Site Reliability Engineer, AI Observability

Job summary

This role operates a shared observability platform for LLM-based applications used by a global life sciences client, while building the monitoring, alerting, and service levels the platform currently lacks. Responsibilities include operating Langfuse on Kubernetes on AWS, managing ClickHouse, PostgreSQL, and Redis, proving backup and restore procedures, rehearsing safe upgrades, maintaining infrastructure as code through Helm and Argo CD, and creating runbooks and support paths for internal teams.

Qualifications

• Hands-on experience with Kubernetes on AWS (managed EKS), with routine work done through Helm values and Argo CD applications.
• Hands-on experience running ClickHouse in production, including replication and Keeper quorum, shard and replica topology, and backup and restore, ideally run through an operator.
• Experience building monitoring and alerting from scratch, including service levels that reflect what users actually experience rather than what is easy to measure.
• Experience upgrading self-hosted software safely, including schema migrations, rehearsal in a lower environment and a rollback plan for a partly completed migration.
• PostgreSQL and Redis operations deep enough to debug metadata-store and queue problems, including backpressure and worker drain.
• Strong operational writing: runbooks, SOPs and post-incident reviews that a colleague can follow unaided during an incident.
• You have prior experience in consulting

Responsibilities

• Diagnose and resolve failures across ClickHouse, PostgreSQL, Redis, the ingestion workers and the Kubernetes layer beneath them
• Own ClickHouse under the operator model, including Keeper quorum, replication, shard and replica topology, and S3 storage tiering
• Rehearse the restore, time it, document it and test it against its failure modes
• Plan and rehearse upgrades in a lower environment, with a rollback plan that works even if a migration is only partly complete
• Build monitoring, alerting and service levels from scratch
• Maintain Kubernetes manifests, Helm values and Argo CD applications so every change goes through the delivery pipeline
• Write and maintain runbooks and SOPs that let a colleague resolve an incident without the author present
• Run the onboarding and support path for internal teams that depend on the platform, and triage what they bring
• Turn recurring operational work into automation
• Own the migration and its rollback

Skills

Industry

Relocation