Senior Databricks Data Platform Engineer

Job summary

The Lead Databricks Platform Engineer designs and evolves a Lakehouse platform that enables data and AI capabilities at enterprise scale. The role builds reusable Python libraries, APIs, automation, governance and security controls, and integrations with identity, incident management, and observability systems. It also supports platform testing, reliability, and complex production incidents, using technologies including Databricks, REST APIs, ServiceNow, Dynatrace, Prometheus, and Grafana.

Qualifications

• At least 10 years of experience in IT or data.
• At least 3 years of Databricks platform enablement experience.
• At least 3 years designing, writing, and maintaining APIs for platform automation, serverless workflows, deployment pipelines, release management, and repository management.
• Advanced Python skills for production-grade libraries and services.
• Experience designing and integrating REST APIs.
• Advanced SQL skills for troubleshooting and performance optimization, including Oracle or Microsoft SQL Server.
• Experience with Git and CI/CD workflows.
• Knowledge of scheduling and orchestration tools such as Stonebranch, Control-M, or Airflow.
• Familiarity with monitoring and observability tools such as ServiceNow, Prometheus, or Grafana.
• Knowledge of data quality and governance frameworks such as Great Expectations.
• English proficiency at C1 Advanced level.
• Investment data domain knowledge is desirable.

Responsibilities

• Clarify requirements and participate in sprint planning with stakeholders.
• Prototype and productize shared Lakehouse platform capabilities.
• Develop reusable Python libraries, SDKs, and internal frameworks.
• Build automation for provisioning, governance controls, configuration management, and operational standards.
• Integrate the platform with incident management, identity and access management, and observability systems.
• Implement secure API integrations using enterprise authentication and authorization.
• Develop and maintain unit and integration tests, and support acceptance testing.
• Provide third-line support for complex production incidents affecting platform stability.
• Collaborate with cloud and infrastructure teams on secure, scalable, reliable architecture.

Skills

Languages

Industry

Relocation