Principal Observability Platform Engineer

Job summary

Lead the technical strategy and architecture of an observability platform for GPU clusters, AI workloads, and supporting infrastructure. Set direction across metrics, logs, traces, and alerting, building scalable systems that are simple to operate and quick to diagnose. Partner with SRE, infrastructure, and AI/ML teams, mentor engineers, and use incident postmortems to drive lasting platform improvements.

Qualifications

• At least 8 years of experience in SRE, infrastructure engineering, platform engineering, or observability-focused roles.
• Experience operating observability infrastructure at significant scale.
• Hands-on experience with a subset of Prometheus, Thanos, VictoriaMetrics, Grafana, Loki, Tempo, OpenTelemetry, ClickHouse, and Elastic.
• Proficiency in Python, Go, or a similar programming language.
• Experience with Kubernetes at scale; GPU infrastructure or Slurm familiarity is an advantage.
• Ability to architect systems, write and review code, and explain technical tradeoffs.
• Infrastructure-as-Code experience with Terraform, Ansible, or equivalent.
• Ability to influence technical decisions across teams.

Responsibilities

• Set observability platform strategy and architecture across metrics, logs, traces, and alerting.
• Guide long-term decisions on tooling, data models, ingestion, retention, and cardinality management.
• Identify systemic gaps and design systems that make failures visible and fast to diagnose.
• Partner with SRE, infrastructure, and AI/ML teams to integrate observability into engineering practices.
• Establish reusable engineering standards and patterns.
• Mentor the observability team and lead technical development.
• Lead incident postmortems and drive durable platform improvements.
• Evaluate new tools and retire those that do not improve signal quality, operational efficiency, or scalability.

Skills

Languages

Industry

Relocation