Databricks Administrator / Platform Engineer

Job summary

The platform engineer administers, secures, and optimizes an enterprise Databricks environment to keep it reliable, scalable, and cost-effective. The role supports workspaces, clusters, jobs, SQL warehouses, access controls, monitoring, and troubleshooting, while partnering with data, security, and cloud infrastructure teams. It also supports CI/CD and Infrastructure as Code, using technologies such as Unity Catalog, Apache Spark, and Terraform.

Qualifications

• At least six years of experience in cloud infrastructure, data platform administration, database administration, DevOps, data engineering, or a related technical field.
• Hands-on experience administering Databricks in an enterprise environment.
• Experience with Azure Databricks, AWS Databricks, or Databricks on Google Cloud.
• Understanding of Databricks workspaces, clusters, jobs, notebooks, SQL warehouses, instance pools, and compute policies.
• Experience with Unity Catalog, access controls, identity management, service principals, secrets, and data governance.
• Experience monitoring and troubleshooting Databricks jobs, clusters, integrations, and platform performance.
• Working knowledge of Apache Spark and distributed data processing.
• Experience with cloud storage, networking, security, and identity-management services.
• Knowledge of SQL and at least one scripting language: Python, PowerShell, Bash, or Scala.
• Experience with Git, CI/CD pipelines, and automated deployment.
• Experience with Infrastructure as Code tools such as Terraform, ARM templates, Bicep, or CloudFormation.
• Strong troubleshooting, documentation, communication, and stakeholder-management skills.
• Bachelor’s degree in Computer Science, Information Technology, Engineering, Data Science, or a related field.
• Preferred: Databricks certification or a cloud certification from Microsoft Azure, Amazon Web Services, or Google Cloud.
• Preferred: Experience with Delta Lake, Delta Live Tables, MLflow, Lakehouse architecture, or Databricks SQL.
• Preferred: Experience with Azure DevOps, GitHub Actions, Jenkins, or similar CI/CD platforms.
• Preferred: Experience integrating Databricks with Power BI, Tableau, Snowflake, SQL Server, or other enterprise data platforms.
• Preferred: Experience in regulated or highly controlled environments.
• Preferred: Familiarity with data cataloging, lineage, audit logging, disaster recovery, and business-continuity planning.

Responsibilities

• Administer Databricks workspaces, clusters, jobs, notebooks, pools, and SQL warehouses.
• Manage user accounts, groups, roles, permissions, service principals, and access controls.
• Maintain platform security using Unity Catalog, credential management, and data-governance policies.
• Monitor platform availability, job execution, cluster performance, resource use, and system health.
• Troubleshoot job failures, cluster and connectivity issues, performance bottlenecks, and access incidents.
• Optimize clusters, instance pools, autoscaling, and compute policies for performance and cost.
• Support integrations with cloud storage, databases, business intelligence platforms, APIs, and enterprise applications.
• Manage configurations across development, testing, staging, and production environments.
• Support automated deployments and infrastructure management using CI/CD and Infrastructure as Code.
• Develop monitoring dashboards, alerts, audit reports, and operational documentation.
• Support platform upgrades, configuration changes, testing, and releases.
• Partner with security and governance teams on audits, compliance, logging, and data protection.
• Establish platform standards, administrative procedures, naming conventions, and operational practices.
• Provide technical guidance to data engineers, analysts, data scientists, and application teams.
• Recommend improvements to platform scalability, reliability, performance, and cost.

Skills

Certifications

Degrees

Languages

Industry

Relocation