Senior Databricks Engineer

Posted 6 hours 20 minutes ago by Careerwise

Contract
Not Specified
Other
Not Specified, United Kingdom
Job Description

Senior Databricks Engineer (Contract) - £525 per day
Location: Remote (UK-based)
Contract: Outside IR35
Rate: £525/day

We are looking for a Senior Databricks Engineer to join our team on a contract basis and take ownership of improving and future-proofing our data platform. This is an exciting opportunity to make a major impact by bringing best practices, standards, and architecture redesign to our Databricks environment.

Key Responsibilities

  • Establish Best Practice: Define and embed standards, principles, and governance for Databricks usage across the organisation.

  • Optimise Performance: Review and tune Databricks clusters, jobs, and notebooks for maximum efficiency and cost-effectiveness.

  • Fix & Stabilise Pipelines: Troubleshoot and resolve broken or underperforming data pipelines.

  • Architectural Review: Conduct a deep-dive assessment of the current Databricks architecture, identifying weaknesses and opportunities for improvement.

  • Redesign & Implement: Deliver a refreshed, scalable Databricks architecture that supports future data engineering and analytics needs.

Essential Skills & Experience

  • Proven experience as a Senior Data Engineer or Databricks Engineer with strong hands-on expertise in Databricks (Spark, Delta Lake).

  • Deep knowledge of data engineering best practices, data pipelines, and ETL/ELT processes.

  • Strong background in performance tuning and cost optimisation within Databricks.

  • Experience leading architecture reviews and delivering redesigned solutions at scale.

  • Solid understanding of modern data platform components: Azure Data Lake, SQL, CI/CD, DevOps tooling.