Lead Data Platform Engineer

Job summary

The Lead Data Platform Engineer is the highest individual contributor engineering role on the EDAP (Enterprise Data Analytics Platform) team. This role involves serving as the primary technical authority for the platform, owning the end-to-end architecture, and driving the largest, most complex engineering initiatives. The Lead Data Platform Engineer will set the architectural vision, define engineering standards, and provide technical leadership and mentorship to the team.

Benefits

Qualifications

• 8–14 years of professional software or data engineering experience.
• At least 3 years in a technical lead or principal capacity on a data platform or distributed systems team.
• Demonstrated track record of leading large-scale, cross-functional data infrastructure projects from conception through production.
• Deep expertise in distributed data systems, including data warehouse/lakehouse architecture, streaming platforms, large-scale batch processing, and cloud-native data infrastructure.
• Expert-level proficiency in Python and SQL.
• Strong working knowledge of at least one JVM language (Scala or Java) for Spark or Flink development.
• Deep knowledge of modern data warehouse and lakehouse platforms (Snowflake, BigQuery, Redshift, Databricks, or equivalent), including storage optimization, compute management, and cost governance.
• Strong command of open table formats (Apache Iceberg, Delta Lake, or Apache Hudi) and their trade-offs in production lakehouse architectures.
• Experience architecting and operating large-scale streaming pipelines with Apache Kafka, Kinesis, or Pub/Sub, including schema management, consumer group design, and exactly-once semantics.
• Exceptional communication skills, including the ability to write clearly, present confidently, and adapt technical depth to various audiences.
• Experience establishing engineering standards, review processes, and quality gates across multi-team engineering organizations.
• Experience designing and implementing federated data platform architectures at scale, such as data mesh (preferred).
• Background in ML infrastructure, feature engineering platforms, or MLOps pipelines (preferred).
• Familiarity with data governance, data cataloging, and lineage tooling (DataHub, OpenMetadata, Collibra, or Alation) (preferred).
• Prior experience working at a company with a mature, scaled data platform serving hundreds of internal consumers or petabyte-scale data assets (preferred).
• Exposure to formal program management methodologies or experience driving large initiatives using structured delivery frameworks (preferred).

Responsibilities

• Own the technical architecture of the data platform end-to-end, including ingestion, storage, transformation, orchestration, serving, and observability layers.
• Author and maintain the platform architectural vision document; lead quarterly architecture reviews.
• Define and evolve the target-state architecture for the platform, establishing a multi-year technology roadmap.
• Evaluate emerging technologies and frameworks, making evidence-based adoption recommendations.
• Serve as the final technical escalation point for complex design questions, cross-team conflicts, and build-vs-buy decisions.
• Lead the planning, execution, and delivery of multi-quarter platform initiatives.
• Break down large programs into scoped workstreams, assign technical leads, define milestones, and manage dependencies and risk.
• Drive initiative kick-offs with clear problem framing, success criteria, architectural constraints, and delivery phasing.
• Maintain stakeholder alignment, communicate status, surface trade-offs, and escalate blockers.
• Lead post-mortems and retrospectives for large initiatives; document and socialize lessons learned.
• Exercise functional oversight across the EDAP team, reviewing technical designs, setting quality gates, and approving architectural decisions.
• Define, document, and socialize platform engineering standards, including coding conventions, testing requirements, CI/CD practices, schema design guidelines, and SLA frameworks.
• Establish and own the platform's technical review process, triaging projects, chairing review sessions, and ensuring decisions are documented.
• Identify and resolve technical debt, redundancy, and architectural drift.
• Partner with security, compliance, and infrastructure teams to ensure platform systems meet governance, data privacy, and regulatory requirements.
• Serve as a primary technical mentor for junior and senior data platform engineers.
• Conduct and lead design reviews, architecture critiques, and technical deep-dives.
• Define technical excellence for engineers and participate in calibration discussions.
• Represent the data platform in engineering-wide forums, all-hands, and external venues.
• Build a culture of documentation, reliability, and platform-as-a-product thinking.

Skills

Languages

Industry

Relocation