Lead Azure Databricks Platform Engineer/Architect

Posted 1 hour 58 minutes ago by TXP

£500 Daily
Contract
Not Specified
Other
London, United Kingdom
Job Description

Lead Azure Databricks Platform Engineer/Architect
Hands-on Platform Engineering | Serverless | FinOps | POSIT/RStudio Migration

6 Month contract

Inside IR35 - 500 a day

London/Hybrid

Role Purpose

We are seeking a highly experienced, hands-on Azure Databricks Platform Engineer/Architect to enhance and optimise an enterprise Data Platform. The role combines architecture with direct implementation: the successful candidate must be able to configure, develop, troubleshoot and optimise Azure Databricks rather than operate only at design or governance level.
The role is centred on three outcomes: enabling and optimising Databricks Serverless, strengthening FinOps and platform controls, and enhancing the Databricks Discovery Zone to support workloads currently delivered through POSIT/RStudio.

Key Responsibilities

1. Databricks Serverless Enablement and Optimisation

  • Assess existing workloads and determine suitability for Serverless, classic, job or interactive compute based on duration, utilisation, SLA, concurrency, performance and cost.
  • Enable and configure Serverless for appropriate jobs, SQL workloads, notebooks, analytical processing and data pipelines.
  • Establish workload-placement guidance, including when Serverless is not economical for predictable, heavy or continuously running workloads.
  • Implement compute policies, autoscaling, quotas, budget controls and operational guardrails.
  • Measure cost and performance outcomes, identify idle or oversized compute, and recommend optimisation actions.

2. FinOps and Enterprise Platform Controls

  • Define and embed a practical FinOps operating model covering ownership, accountability, projects, environments, teams, applications and cost centres.
  • Implement mandatory tagging and integrate validation into CI/CD so non-compliant resources are prevented from being provisioned.
  • Provide granular cost attribution by workspace, project, application, workload, job and team/user where technically appropriate.
  • Implement budget policies, thresholds, proactive alerts and usage reporting to prevent uncontrolled spend.
  • Use platform usage and billing data to identify idle compute, inefficient workloads, unnecessary storage/data movement and cost anomalies.

3. Databricks Discovery Zone and POSIT/RStudio Migration

  • Enhance the Databricks Discovery Zone to support migration from POSIT/RStudio
  • Enable application deployment, secure API integrations, external data ingestion, LLM integration, scheduling, BI connectivity, local IDE-based development and operational reporting.
  • Define reusable onboarding and migration patterns that reduce technology sprawl while improving security, supportability and delivery speed.

4. Data Engineering and Integration

  • Design and build reliable ingestion and transformation pipelines using Python, PySpark, SQL and Delta Lake.
  • Implement full and incremental ingestion, CDC where appropriate, schema evolution, reconciliation, error handling and data quality controls.
  • Design reusable integration patterns for REST APIs, SaaS platforms, databases, files, object storage, document repositories, enterprise applications and public/external data providers.
  • Implement secure authentication and credential handling for external and internal integrations.
  • Build end-to-end data flows from source through governed ingestion and curated layers to BI, ML or application consumption.

Required Hands-on Technical Skills

Deep hands-on Azure Databricks implementation and troubleshooting
Databricks Serverless and compute/workload optimisation
Azure identity, networking, security, secrets, monitoring and private connectivity
Databricks SQL, Delta Lake and performance optimisation
Python, PySpark and SQL
Jobs/workflows, incremental processing, CDC and data quality
REST/API and external data integration patterns
FinOps, cost attribution, tagging, budgets, monitoring and operational support

Experience and Candidate Profile

  • Significant experience delivering enterprise Azure Databricks platforms in production environments.
  • Demonstrable ability to move between architecture, implementation, debugging and optimisation without depending entirely on specialist engineering teams.
  • Strong understanding of platform security, data governance, operational support and controlled delivery in regulated or complex enterprises.
  • Experience working collaboratively with data engineers, data scientists, architects, security teams, platform teams and business stakeholders.
  • Clear communication skills and the ability to document standards, patterns, decisions and operational guidance.

Highly Desirable but not Mandatory

  • POSIT/RStudio migration or consolidation experience.
  • Migration of analytical/data science workloads (convert and migrate R development/Libraries to Databricks).
  • AI/ML, LLM integration, model life cycle, RAG/vector retrieval or model-serving experience.
  • Large-scale enterprise platform transformation and regulated-industry experience.
  • Strong cost optimisation and FinOps delivery experience across Azure and Databricks.