Experience building and optimizing ‘big data’ pipelines, architectures and data sets on cloud native environments
Ability to work independently and learn new technologies
2+ years of experience building processes supporting data transformation, data structures, metadata, dependency and workload management
Working knowledge of message queuing, stream processing, and highly scalable ‘big data’ stores.
Proficiency with Python, SQL, Java or Scala.
Experience in data streaming tools and processes (Kafka, AirFlow, ELK, Spark etc.)
Experience with big data solutions like Athena, SnowFlake, Redshift, Redis, Aerospike, Impala, Presto, Bigquery, Hive, etc.
Experience in data modelling, BI solutions development (DWH, ETL) and SW development practices – Agile, CI/CD, TDD.
Experience in real time analytics
Experience with GCP and BigQuery
Experience with building automated validation processes to ensure data integrity