Senior Data Platform Engineer - Azure Databricks

Job summary

The Senior Data Platform Engineer is tasked with designing, building, and optimizing an enterprise data platform using Azure Databricks in a healthcare setting. This role involves maintaining scalable data processing environments, implementing CI/CD pipelines, and ensuring compliance with data governance standards, including healthcare regulations like HIPAA.

Qualifications

• Strong proficiency in PySpark, SQL, and data lakehouse architecture
• Deep understanding of Azure ecosystem including Azure Data Lake, ARO, AKS, Key Vault, Azure Ansible, and Azure DevOps
• Experience with CI/CD implementation using Azure DevOps
• Working knowledge of PySpark, Collibra, Power BI, and Tableau integration workflows
• Familiarity with data governance, security, and compliance best practices
• Excellent problem-solving and cross-functional collaboration skills
• Experience with Terraform or other IaC tools for Azure
• Familiarity with Delta Live Tables, Unity Catalog, and MLflow

Responsibilities

• Design, deploy, and maintain scalable Databricks clusters and workspaces within Azure
• Automate cluster management, job orchestration, and environment provisioning using Infrastructure-as-Code (IaC) tools
• Optimize Spark jobs, delta pipelines, and data lakehouse performance
• Implement robust CI/CD pipelines for Databricks workflows and notebooks
• Manage environment configurations, libraries, secrets, and access controls
• Implement and manage Azure DevOps pipelines for Databricks code deployment, testing, and release management
• Develop reusable CI/CD templates for data ingestion and transformation workflows
• Support and integrate Databricks and/or dbt workflows within Databricks for modular SQL-based transformations
• Ensure efficient orchestration of ETL/ELT pipelines and integration with data lake and data warehouse environments
• Integrate Databricks and Azure data assets with Collibra for data cataloging, lineage tracking, and governance
• Enable secure and performant data access for Power BI and Tableau through optimized data models and APIs
• Work with stakeholders to assist with platform-related technical issues and support their data needs
• Support implementing data quality, validation, and governance processes to ensure compliance with healthcare regulations

Skills

Certifications

Degrees

Relocation