Leave us your email address and we'll send you all the new jobs according to your preferences.

Data Engineer (SC cleared)

Posted 9 days 22 hours ago by Stott and May

Permanent

Not Specified

Other

Surrey, Guildford, United Kingdom, GU1 1

Job Description

Data Engineer (SC cleared)

Start: ASAP
Duration: 12 months
Location: Mostly Remote - must have access to London or Bristol
Pay: negotiable, INSIDE IR35

Responsibilities:
- Design, implement robust ETL/ELT data pipelines using Apache Airflow
- Build ingestion processes from internal systems and APIs, using Kafka, Spark, AWS
- Develop and maintain data lakes and warehouses (AWS S3, Redshift)
- Ensuring governance using automated testing tools
- Collaborate with DevOps to manage CI/CD pipelines for data deployments and ensure version control of DAGs
- Apply best practice in security and compliance

Required Tech Skills:
- Python and SQL for processing
- Apache Airflow, writing Airflow DAGs and configuring airflow jobs
- AWS cloud platform and services like S3, Redshift
- Familiarity with big data processing using Apache Spark
- Knowledge of modelling, schema design and partitioning strategies
- Understanding batch Vs streaming data paradigms
- Docker or Kubernetes (containerization)