Site Reliability Engineer

Job summary

The Site Reliability Engineer designs and operates reliability practices for a large AI training campus, coordinating across compute, network, storage, power, and cooling systems. The role owns fleet-scale monitoring and alert quality, leads SEV response and blameless postmortems, governs high-risk firmware and facility changes, and drives reliability improvements using technologies such as Prometheus, Grafana, NVIDIA DCGM, and industrial telemetry.

Qualifications

• At least 5 years of experience in site reliability, systems engineering, plant operations, or large-scale production operations.
• Experience in power plants, nuclear power, data centers, high-performance computing, or comparable environments is preferred.
• Proven large-scale incident command experience and calm technical leadership.
• Experience designing fleet- or campus-scale monitoring and observability, including alert hygiene and signal quality.
• Hands-on experience with Prometheus/Grafana, NVIDIA DCGM, or industrial telemetry and controls.
• Experience across at least two of compute, network, storage, power, or cooling systems.
• Experience operating playbooks, runbooks, or SOP/MOP/EOP procedures with a 24/7 operations, control-room, or NOC partner.
• Proficiency in Python or Bash scripting and experience with at least one systems language.
• Strong problem-solving skills and a data-driven approach to reliability engineering.
• Ability to collaborate with NOC, data center operations, hardware failure analysis, and infrastructure engineering teams.

Responsibilities

• Own monitoring architecture and signal quality across GPU, host, and facility telemetry.
• Lead SEV response, coordinate incident bridges with the NOC, manage paging and escalation, and maintain accurate incident timelines and severity.
• Conduct blameless postmortems and drive corrective actions to completion.
• Govern high-risk firmware, BIOS, and BMC rollouts and controlled work on power and cooling systems.
• Lead reliability projects across compute, network, storage, and facility systems.
• Partner with Hardware Failure Analysis to identify systemic failures and turn fleet data into monitoring, stop-rules, and vendor RCA inputs.
• Maintain playbooks and dependency maps, run game days, and improve runbooks with the NOC.
• Define error budgets and availability objectives for campus and service boundaries.
• Participate in on-call rotations and SEV incident response at the Memphis/Southaven data center campus.

Skills

Degrees

Languages

Work schedule

Industry

Relocation