Senior AI Evaluation Engineer (CICD & Platform)

Job summary

The Senior AI Evaluation Engineer owns continuous evaluation architecture, deployment pipelines, and release gates for an enterprise AI skills platform. The role builds CI/CD and MLOps infrastructure, automated evaluation harnesses, synthetic datasets, regression tests, and production observability workflows to ensure AI agents execute accurately, satisfy schema contracts, and operate safely in customer environments. Relevant technologies include Python, GitHub Actions, GitLab CI, pytest, LangSmith, Braintrust, and DeepEval.

Benefits

Qualifications

• At least 4 years of professional Python software engineering experience building production-grade systems, with strength in CI/CD engineering or AI evaluation and familiarity with the other.
• At least 2 years designing, building, and operating test automation frameworks, CI/CD pipelines, or automated release gates.
• At least 1 year building or evaluating Generative AI applications, including tool-calling, agent workflows, or structured output parsing.
• Experience with AI evaluation and observability frameworks such as LangSmith, Braintrust, DeepEval, Promptfoo, or Phoenix.
• Strong knowledge of schema validation with JSON Schema, Pydantic, or OpenAPI, and API contract testing.
• Familiarity with MCP, tool-calling specifications, or agent execution harnesses.
• Understanding of LLM failure modes, including hallucinations, prompt injection, sensitive-data sanitization, and role-based access boundaries.
• Ability to own technical domains end-to-end, collaborate with platform and product teams, and write technical documentation and architecture designs.

Responsibilities

• Build tiered evaluation pipelines, from static and schema checks to synthetic, multi-turn trajectory, and LLM-as-a-Judge evaluations.
• Architect and maintain CI/CD pipelines, skill registry versioning, and runner environments with automated release gates.
• Develop synthetic data generation pipelines and red-teaming utilities from API schemas and domain specifications.
• Maintain versioned benchmark datasets and run regression tests across model upgrades, prompt changes, and security boundaries.
• Feed production traces and failure logs into benchmark datasets to improve test coverage.

Skills

Degrees

Languages

Work schedule

Industry

Relocation