Lead Software Engineer - LLM Ops Platform Reliability

Job summary

As a Lead Software Engineer at JPMorgan Chase, you will build and operate large language model serving infrastructure, focusing on reliability, performance, and cost-efficiency. You will work with cloud and Kubernetes-based deployments, ensuring the stability and security of AI systems in production.

Qualifications

• Formal training, certification, or equivalent practical experience in software engineering concepts
• Hands-on experience with system design, application development, testing, and operational stability in production environments
• Advanced proficiency in Python for building production-grade services and tooling
• Proficiency with automation and continuous delivery methods
• Hands-on experience with AWS and Terraform for infrastructure delivery and lifecycle management
• Strong understanding of site reliability engineering practices, including incident management, root-cause analysis, runbooks, and reliability patterns
• Practical knowledge of observability and instrumentation across metrics, logs, and traces
• Comfort with on-call operations and production troubleshooting
• Hands-on production experience operating LLM inference servers such as vLLM and llm-d (or directly equivalent serving stacks)
• Hands-on experience hosting and serving LLMs on Amazon EKS and/or Amazon SageMaker, and on local GPU infrastructure
• Knowledge of LLM reliability and risk considerations, including latency/throughput trade-offs, model and weight versioning, prompt/response logging, and safe rollout patterns

Responsibilities

• Design, develop, troubleshoot, and deliver secure, high-quality production software and services for AI infrastructure
• Build backend services and APIs that enable reliable operation of AI infrastructure in production
• Operate and scale LLM serving infrastructure (such as vLLM and llm-d), including model hosting, request routing, continuous batching, and KV-cache optimization
• Deploy, host, and lifecycle-manage open-source and proprietary LLMs on Amazon EKS and Amazon SageMaker, as well as on-prem and local GPU clusters, using reproducible infrastructure as code and continuous delivery pipelines
• Implement observability (logs, metrics, traces) with dashboards and actionable alerting, including Prometheus metrics and Grafana/Alertmanager integration for LLM and GPU workloads
• Tune GPU and accelerator capacity, autoscaling, and cost efficiency for LLM inference workloads using performance and optimization techniques (e.g., quantization, parallelism, speculative decoding)
• Lead reliability engineering for LLM endpoints through capacity planning, load/soak testing, safe rollouts (blue/green, canary), failover, and incident response for outages and model-quality regressions
• Participate in an on-call rotation, lead incident triage and mitigation, and produce clear post-incident root-cause analyses and follow-ups
• Identify recurring operational issues and automate remediation to improve platform stability and developer experience
• Build and maintain multi-agent systems with strong orchestration (planning, coordination, tool-calling, state/memory, and workflow control) where appropriate
• Contribute to an inclusive team culture grounded in diversity, opportunity, inclusion, and respect, and help drive adoption of leading-edge technologies through communities of practice

Skills

Relocation