Azure Databricks Architect (SC Cleared)

Azure Databricks Architect

6-month contract
Remote, with 2 days on-site in London per month
Active SC clearance required
Inside IR35

Overview

We are seeking an experienced Azure Databricks Architect to lead and assure the design and ongoing development of a cloud-based data and analytics platform within a public sector setting. This position centres on platform-level architecture, data design patterns, governance frameworks, and non-functional requirements. You will collaborate closely with delivery teams and stakeholders to ensure the platform meets organisational and government standards for security, scalability, and compliance.

Key Responsibilities

• Define and oversee the end-to-end architecture of Azure Databricks-driven data platforms.
• Create Lakehouse designs incorporating Azure Databricks, ADLS Gen2, and associated Azure services.
• Produce and maintain High-Level Designs (HLDs) and Low-Level Designs (LLDs).
• Develop and enforce best-practice patterns for data ingestion, transformation, and consumption.
• Shape data security models, including RBAC and data separation strategies.
• Ensure platforms adhere to public sector governance, compliance, and security expectations.
• Address performance, resilience, scalability, and cost-efficiency within architectural decisions.
• Participate in architectural assurance processes and structured design reviews.
• Define integration approaches with Azure Data Factory, event-driven and batch ingestion services, and downstream analytics/reporting platforms.
• Provide architectural guidance for CI/CD pipelines and Infrastructure-as-Code.
• Lead optimisation efforts to support sustainable platform operations.
• Act as a design authority for Databricks decisions across multiple delivery teams.
• Support discovery and early-stage design for emerging data and analytics initiatives.
• Identify technical risks, dependencies, and constraints, contributing to mitigation plans.

Essential Skills & Experience

• Strong track record designing Azure Databricks-based architectures in enterprise environments.
• Deep understanding of Lakehouse principles, Delta Lake, and governance capabilities.
• Architectural expertise with Spark-based platforms.
• Experience designing secure, governed platforms within regulated or public sector contexts.
• Proven ability to produce high-quality HLDs, LLDs, and architectural decision artefacts.
• Excellent knowledge of Azure data services and integration technologies.

Desirable Skills & Experience

• Experience delivering data platforms within UK government or wider public sector organisations.
• Familiarity with UK government cloud and security frameworks.
• Background in architectural assurance or design review bodies.
• Understanding of enterprise analytics and reporting ecosystems.
• Relevant Azure or Databricks certifications.

Engagement Model

This role is primarily advisory and architectural rather than hands-on. You will operate across multiple teams and workstreams, supporting both delivery and assurance activities throughout the life cycle of the data platform.