Senior AI Backend Engineer - Agent Evaluation & Quality

About the role

We build production multi‑agent systems that serve a large user base. As these systems scale, ensuring confidence in agent performance becomes critical. This role owns the evaluation infrastructure that measures, validates, and improves agent quality, enabling faster and safer releases.

Key responsibilities

• Design and implement LLM‑as‑judge systems, calibrating them against human labels to make agent quality measurable per agent and failure mode.
• Develop per‑pull‑request evaluation harnesses and regression detection pipelines integrated into CI/CD.
• Create user simulators that generate test coverage and adversarial scenarios before real users encounter them.
• Feed production failure signals back into evaluation datasets to continuously improve the system.
• Collaborate with product teams to translate qualitative goals into concrete, measurable criteria.
• Gradually contribute to agent development, leveraging deep knowledge of evaluation outcomes.

Required profile

• Strong software engineering fundamentals with production‑grade Python or Typescript, clean API design, testing, and CI/CD experience.
• Hands‑on experience building LLM‑driven agents, RAG pipelines, tool/function calling, and orchestration frameworks such as LangGraph or LangChain.
• Measurement mindset: comfortable defining metrics, calibrating models, and running experiments.
• Production experience running LLM systems at scale, handling reliability, latency, cost, and observability.
• 5+ years of software engineering experience, recent work focused on LLM/agent technologies.

Required skills

• Python
• Typescript
• LangGraph
• LangChain
• CI/CD pipelines
• LLM and agent development
• Retrieval‑augmented generation (RAG)