Jobs / Microsoft
AI Reliability Engineer, HPC
Microsoft · London, ENG, United Kingdom
London, ENG, United KingdomExp: 4+ yrs93,500-161,800 GBP/yearlyRemote
Remuneration
Benefits and other compensation may be available for certain roles.
Location
London, ENG, United Kingdom
Visa sponsorship
Not specified
Job summary
The AI Reliability Engineer combines software and systems engineering to maintain reliable, efficient, highly available distributed AI infrastructure. The role supports HPC clusters for model training and inference by building observability and automation, managing incidents, and securing operations across GPU, storage, and networking systems. It also partners with ML and platform teams to improve production workflows.
Qualifications
- Bachelor's degree in Computer Science or a related technical discipline and at least four years of relevant technical experience, or equivalent experience.
- Preferred: Master's degree in Computer Science or a related technical discipline and at least two years of relevant technical experience, or equivalent experience.
- Experience with Kubernetes, Docker, container orchestration, and CI/CD pipelines for ML workloads.
- Experience with Azure, AWS, or GCP and infrastructure as code.
- Experience with Grafana, Datadog, OpenTelemetry, or similar observability tools.
- Programming or scripting experience in Python, Go, or Bash.
- Experience with distributed systems, networking, storage, and HPC.
- Experience operating GPU clusters and workload schedulers for ML or AI workloads.
- Experience with ML training or inference pipelines.
- Experience with capacity planning and cost optimization for GPU infrastructure.
Responsibilities
- Maintain uptime, resiliency, and fault tolerance of HPC clusters supporting model training and inference.
- Design and maintain monitoring, alerting, and logging for GPU clusters, storage, and networking.
- Automate deployments, incident response, scaling, and failover in CPU and GPU environments.
- Lead on-call rotations, troubleshoot production issues, conduct blameless postmortems, and drive improvements.
- Ensure data privacy, compliance, and secure operations for model training and serving.
- Partner with ML engineers and platform teams to improve developer experience and research-to-production workflows.
Skills
AWSAzureBashDatadogDockerGCPGoGrafanaKubernetesOpenTelemetryPython
Degrees
Bachelor's degree in Computer Science or a related technical disciplineMaster's degree in Computer Science or a related technical discipline
Work schedule
At least four office days per week for employees within the specified distance o
Industry
Artificial intelligenceHigh-performance computing
Relocation
No