Databricks Platform Engineer

Job summary

The Databricks Platform Engineer role focuses on owning and evolving the Databricks platform for the Global Finance Data Hub, ensuring scalability, reliability, and security. Key responsibilities include designing data ingestion frameworks, managing Delta Live Tables, and implementing CI/CD pipelines, utilizing technologies such as AWS, PySpark, and Databricks services like Unity Catalog and Delta Lake.

Qualifications

• 5-8 years of experience in Data Engineering, Data Platform Engineering, or Cloud Data Architecture
• Minimum 3 years of hands-on experience owning and managing Databricks platforms
• Strong expertise in Databricks services
• Proven experience designing and supporting data ingestion frameworks
• Advanced proficiency in PySpark, Python, and Databricks SQL
• Solid knowledge of DevOps practices
• Strong AWS experience

Responsibilities

• Own and evolve the Databricks platform
• Design, build, and maintain data ingestion frameworks
• Develop and manage Delta Live Tables (DLT) pipelines
• Administer and optimise Delta Lake environments
• Implement and manage Unity Catalog governance practices
• Build, deploy, and support Databricks Workflows
• Establish and maintain CI/CD pipelines
• Collaborate with cloud and infrastructure teams
• Manage authentication and authorisation mechanisms
• Monitor platform health and performance
• Define and enforce cluster policies
• Ensure platform security
• Partner with data engineering, finance, architecture, and product teams
• Drive platform best practices and continuous improvement initiatives

Skills

Certifications

Relocation