Senior AI Platform Engineer – LLM Gateway

Job summary

Implement and evolve an enterprise LLM Gateway that provides secure, governed, and observable access to foundation models for applications, agents, and other AI use cases. The role covers integrating consumers and model providers; implementing identity, secrets, networking, governance, and resilience controls; and operating the platform with metrics, tracing, cost measurement, infrastructure as code, and CI/CD. The environment includes AWS, generative AI inference APIs, and enterprise security requirements.

Qualifications

• At least 6 years of experience in platform engineering, cloud engineering, distributed systems integration, API management, or a related field.
• At least 1 year of demonstrable experience integrating AI Gateways.
• Experience implementing and operating APIs, shared platforms, or cross-functional services in complex enterprise environments.
• Hands-on experience integrating generative AI inference APIs in production, including authentication, streaming, error handling, usage limits, latency, and token usage.
• Experience with API Gateways, API management, or equivalent integration platforms.
• Strong knowledge of API security and corporate identity and access controls, including OAuth2, OIDC, JWT, secrets management, scopes, audiences, and least privilege.
• AWS experience, especially with identity, secrets management, observability, networking, and Infrastructure as Code.
• Knowledge of cloud networking and security, including VPCs, controlled egress, proxies, DNS, TLS, certificates, and external service connectivity.
• Python development and automation experience, including APIs, integrations, middleware, or platform components.
• Experience with Terraform, CloudFormation, or CDK; CI/CD; and version-controlled configuration.
• Ability to independently diagnose incidents and complex issues, including bottlenecks and integration problems.
• Experience with AWS AgentCore Gateway, Amazon Bedrock, or other generative AI capabilities is valued.
• Experience with LLM Gateways, multi-model or multi-provider routing, fallbacks, guardrails, content filtering, prompt-injection protection, DLP, or AI FinOps is valued.
• Experience with Kubernetes/EKS, service mesh, advanced SRE practices, or regulated industries is valued.

Responsibilities

• Implement and configure LLM Gateway components according to the defined architecture and corporate standards.
• Design integrations, consumption patterns, and technical platform improvements.
• Integrate applications, agents, and services with the corporate inference endpoint.
• Integrate identity, authentication, and authorization, including workload identities, OAuth2/OIDC, JWT, scopes, audiences, and least privilege.
• Manage provider secrets and credentials securely, including environment segregation and rotation.
• Configure model-provider connectivity with networking and security teams, including DNS, proxies, TLS, certificates, and controlled egress.
• Implement access policies, rate limits, quotas, routing, consumption limits, guardrails, and traceability.
• Implement timeouts, retries, circuit breaking, error handling, fallbacks, and load testing.
• Instrument the platform with metrics, structured logs, distributed traces, alerts, dashboards, and audit support.
• Measure consumption and costs by application, agent, project, environment, and model.
• Automate deployments, configuration, and tests using Infrastructure as Code and CI/CD pipelines.
• Produce technical documentation, integration guides, operating procedures, and knowledge-transfer materials.
• Collaborate with architecture, cybersecurity, IAM, networking, cloud, SRE, and development teams.

Skills

Industry

Relocation