Databricks Engineer (Security Clearance Active)

Role: Databricks Engineer (Contract)
Clearance: Active SC or DV clearance required
Location: Hybrid (on-site days required)
Rate: 700/Day
Start Date: ASAP
Duration: 6 months (likely extensions)

Overview

We are seeking an experienced Databricks Contractor to support a major UK Government programme delivering secure, scalable, cloud-based data platforms. This role suits a seasoned engineer who can hit the ground running in a fast-paced, security-focused environment.

You'll work across data engineering, optimisation, and platform development using Azure Databricks, Apache Spark, and modern data tooling.

Key Responsibilities

• Build and optimise Databricks notebooks, pipelines, and Delta Lake workflows.
• Develop scalable ETL/ELT processes for large, secure datasets.
• Implement best practices for data governance, quality, and access control.
• Work with architects and security teams to design secure data solutions.
• Troubleshoot performance issues and contribute to platform enhancements.
• Deliver high-quality engineering outputs within agile delivery cycles.

Essential Experience

• Current SC or DV clearance (mandatory).
• Strong commercial experience with Azure Databricks.
• Advanced PySpark or Scala Spark engineering skills.
• Experience delivering data solutions in secure, regulated, or government environments.
• Hands-on expertise with Azure Data Lake, Data Factory, Key Vault, DevOps Pipelines.
• Strong understanding of Delta Lake, distributed processing, and performance optimisation.

Nice to Have

• Experience with Unity Catalog, MLflow, or CI/CD automation for Databricks.
• Familiarity with Terraform or IaC for secure environments.
• Knowledge of data governance frameworks used across UK Government.

Why This Contract?

• Work on impactful, nationally significant government programmes.
• Long-term pipeline with strong extension potential.
• Opportunity to influence architecture and technical direction.
• Hybrid working with well-structured agile delivery.

How to Apply

If you're an SC/DV-cleared Databricks Contractor ready for your next government engagement, please submit your CV and availability.