Principal AI Infrastructure Engineer - Highrise AI

Job summary

Brings large-scale GPU compute clusters from physical installation through validation and commercial production readiness, owning cluster bring-up, bare-metal provisioning, systems configuration, performance optimization, and acceptance testing. Provides Tier 3 escalation and reliability support for production environments, using Linux, NVIDIA GPU platforms, NCCL, Kubernetes, and Slurm to diagnose and improve performance across nodes, racks, and clusters.

Qualifications

• Bachelor's degree in computer science, engineering, or a related field, or equivalent practical experience.
• At least 8 years of hands-on Linux systems engineering, HPC, cloud, or large-scale infrastructure experience, including senior-level troubleshooting.
• Experience bringing GPU or HPC clusters into production and qualifying them against performance and reliability targets.
• Expertise in Linux, Python, Bash, Ansible, bare-metal provisioning, configuration management, and systems diagnostics.
• Hands-on experience with NVIDIA GPU platforms, NCCL, Field Diagnostics, and multi-node performance testing.
• Practical experience with Kubernetes and Slurm.
• Familiarity with high-speed Ethernet, InfiniBand/RDMA, and parallel or distributed storage.
• Ability to diagnose issues across hardware, firmware, operating systems, and distributed workloads.
• Strong ownership, systematic testing and documentation practices, and clear communication with technical and customer-facing stakeholders.

Responsibilities

• Bring racked, powered, and cabled AI clusters through production readiness and customer handoff.
• Establish and validate baselines for BIOS/BMC, firmware, GPU drivers, operating systems, and cluster software.
• Deploy and optimize bare-metal provisioning, imaging, node configuration, and lifecycle workflows.
• Configure and troubleshoot Linux and GPU systems, including kernel, driver, accelerator, and host-level performance settings.
• Integrate and validate Kubernetes and Slurm environments with platform, networking, and operations teams.
• Execute NVIDIA Field Diagnostics, L11/L12 validation, burn-in, stability, and hardware health qualification.
• Run and analyze NCCL tests, GPU benchmarks, and distributed workload performance at node, rack, and cluster scale.
• Troubleshoot GPU, NVLink/NVSwitch, PCIe, RDMA, and InfiniBand/Ethernet performance issues with network engineers.
• Define acceptance criteria, document test evidence, track defects, verify remediation, and recommend production readiness.
• Escalate and resolve complex production incidents involving compute nodes, GPU software, provisioning, scheduling, and performance.
• Conduct root cause analysis, coordinate vendor escalations, and lead corrective actions for recurring or systemic issues.
• Support fleet upgrades, configuration changes, cluster expansions, and post-maintenance validation.
• Develop diagnostic tooling, automation, runbooks, and operational handoff materials for SRE, NOC, and site support teams.
• Coordinate cross-functional work and communicate technical status, benchmark findings, risks, and acceptance recommendations.

Skills

Degrees

Work schedule

Travel

Industry

Relocation