Platform Engineer (Data bricks)
Role Overview
This is an offshore role responsible for administering and operating the Databricks platform that supports the analytics, data engineering, and ML/AI workloads of an enterprise Agentic AI solution, including the streaming ingestion pipelines that feed it. The role ensures Databricks environments are reliable, secure, governed, and cost-efficient across non-production and production, and operates as part of a broader Azure platform engineering function within a follow-the-sun delivery model alongside onshore engineering and architecture teams.
Key Responsibilities
Administer and operate Databricks workspaces, clusters, instance pools, jobs, and workflows across non-production and production environments.
Design and enforce cluster policies for security, stability, and cost control, and manage autoscaling and job/all-purpose compute.
Operate Spark Structured Streaming ingestion pipelines that consume from Kafka/Confluent and Azure Service Bus or Event Hubs and land data into Delta tables.
Administer Unity Catalog for data governance, access control, and lineage, and manage Delta Lake optimisation, versioning, and retention.
Contribute to data integration patterns between Databricks, upstream messaging, and downstream platform services.
Build and maintain Terraform Infrastructure-as-Code for Databricks and related Azure data services.
Implement CI/CD for notebooks, jobs, and configuration using GitHub Actions and Git-based workflows.
Monitor platform health using Azure Monitor and Dynatrace, and troubleshoot Spark job failures, performance, memory, and scaling issues.
Perform capacity planning, usage analysis, and cost optimisation across Databricks environments.
Apply runtime upgrades and configuration changes following change-management best practices, and provide follow-the-sun / on-call support.
Qualifications & Experience
4+ years experience administering Databricks or Spark-based data platforms in enterprise environments.
Deep understanding of Apache Spark architecture, execution, and performance tuning.
Hands-on experience with Unity Catalog, Delta Lake, and lakehouse architectures.
Experience with Spark Structured Streaming and ingestion from Kafka or Azure messaging services.
Working knowledge of Microsoft Azure, including Azure Data Lake Storage, networking, and core PaaS services.
Experience with Terraform Infrastructure-as-Code and Git-based CI/CD.
Proficiency in Python, SQL, and Bash scripting.
Strong troubleshooting and analytical skills, and strong communication for distributed collaboration.
Bachelor’s degree in Computer Science, Engineering, Data Engineering, or a related discipline.
Preferred Skills
Databricks Administrator or Data Engineer certification.
Hands-on experience with Kafka/Confluent and Azure Service Bus or Event Hubs.
Experience integrating data pipelines with messaging and API-based services.
Familiarity with Azure Monitor, Application Insights, and Dynatrace observability.
Familiarity with ITIL-based enterprise service management processes.
Role Overview
This is an offshore role responsible for administering and operating the Databricks platform that supports the analytics, data engineering, and ML/AI workloads of an enterprise Agentic AI solution, including the streaming ingestion pipelines that feed it. The role ensures Databricks environments are reliable, secure, governed, and cost-efficient across non-production and production, and operates as part of a broader Azure platform engineering function within a follow-the-sun delivery model alongside onshore engineering and architecture teams.
Key Responsibilities
Administer and operate Databricks workspaces, clusters, instance pools, jobs, and workflows across non-production and production environments.
Design and enforce cluster policies for security, stability, and cost control, and manage autoscaling and job/all-purpose compute.
Operate Spark Structured Streaming ingestion pipelines that consume from Kafka/Confluent and Azure Service Bus or Event Hubs and land data into Delta tables.
Administer Unity Catalog for data governance, access control, and lineage, and manage Delta Lake optimisation, versioning, and retention.
Contribute to data integration patterns between Databricks, upstream messaging, and downstream platform services.
Build and maintain Terraform Infrastructure-as-Code for Databricks and related Azure data services.
Implement CI/CD for notebooks, jobs, and configuration using GitHub Actions and Git-based workflows.
Monitor platform health using Azure Monitor and Dynatrace, and troubleshoot Spark job failures, performance, memory, and scaling issues.
Perform capacity planning, usage analysis, and cost optimisation across Databricks environments.
Apply runtime upgrades and configuration changes following change-management best practices, and provide follow-the-sun / on-call support.
Qualifications & Experience
4+ years experience administering Databricks or Spark-based data platforms in enterprise environments.
Deep understanding of Apache Spark architecture, execution, and performance tuning.
Hands-on experience with Unity Catalog, Delta Lake, and lakehouse architectures.
Experience with Spark Structured Streaming and ingestion from Kafka or Azure messaging services.
Working knowledge of Microsoft Azure, including Azure Data Lake Storage, networking, and core PaaS services.
Experience with Terraform Infrastructure-as-Code and Git-based CI/CD.
Proficiency in Python, SQL, and Bash scripting.
Strong troubleshooting and analytical skills, and strong communication for distributed collaboration.
Bachelor’s degree in Computer Science, Engineering, Data Engineering, or a related discipline.
Preferred Skills
Databricks Administrator or Data Engineer certification.
Hands-on experience with Kafka/Confluent and Azure Service Bus or Event Hubs.
Experience integrating data pipelines with messaging and API-based services.
Familiarity with Azure Monitor, Application Insights, and Dynatrace observability.
Familiarity with ITIL-based enterprise service management processes.