Databricks Engineer (Security Clearance Active)

Posted 10 hours 27 minutes ago by IO Associates

£700 Daily
Contract
Not Specified
Factory Jobs
England, United Kingdom
Job Description

Role: Databricks Engineer (Contract)
Clearance: Active SC or DV clearance required
Location: Hybrid (on-site days required)
Rate: 700/Day
Start Date: ASAP
Duration: 6 months (likely extensions)

Overview

We are seeking an experienced Databricks Contractor to support a major UK Government programme delivering secure, scalable, cloud-based data platforms. This role suits a seasoned engineer who can hit the ground running in a fast-paced, security-focused environment.

You'll work across data engineering, optimisation, and platform development using Azure Databricks, Apache Spark, and modern data tooling.

Key Responsibilities

  • Build and optimise Databricks notebooks, pipelines, and Delta Lake workflows.
  • Develop scalable ETL/ELT processes for large, secure datasets.
  • Implement best practices for data governance, quality, and access control.
  • Work with architects and security teams to design secure data solutions.
  • Troubleshoot performance issues and contribute to platform enhancements.
  • Deliver high-quality engineering outputs within agile delivery cycles.

Essential Experience

  • Current SC or DV clearance (mandatory).
  • Strong commercial experience with Azure Databricks.
  • Advanced PySpark or Scala Spark engineering skills.
  • Experience delivering data solutions in secure, regulated, or government environments.
  • Hands-on expertise with Azure Data Lake, Data Factory, Key Vault, DevOps Pipelines.
  • Strong understanding of Delta Lake, distributed processing, and performance optimisation.

Nice to Have

  • Experience with Unity Catalog, MLflow, or CI/CD automation for Databricks.
  • Familiarity with Terraform or IaC for secure environments.
  • Knowledge of data governance frameworks used across UK Government.

Why This Contract?

  • Work on impactful, nationally significant government programmes.
  • Long-term pipeline with strong extension potential.
  • Opportunity to influence architecture and technical direction.
  • Hybrid working with well-structured agile delivery.

How to Apply

If you're an SC/DV-cleared Databricks Contractor ready for your next government engagement, please submit your CV and availability.