Forward Deployed Engineer – AI Assurance

About the role

The Forward Deployed Engineer – AI Assurance owns the quality of AI‑native applications, combining functional testing with AI‑specific evaluation such as accuracy, drift, and hallucination. You will ensure that every AI feature meets rigorous standards before reaching production.

Key responsibilities

• Build test plans and automation for functional and AI‑specific features.
• Design evaluation harnesses to measure model/agent outputs (accuracy, consistency, hallucination rate).
• Run regression testing across model, prompt, and configuration changes to detect silent quality drift.
• Red‑team AI features for edge cases and adversarial inputs.
• Develop automated evaluation pipelines integrated into CI/CD.
• Partner with AI Architects to define testability requirements early.
• Own the quality gate and communicate risk to delivery leadership.
• Train delivery teams on AI‑specific testing practices.
• Maintain the evaluation framework (golden datasets, scoring rubrics, LLM‑as‑judge calibration).
• Define acceptance thresholds and gate releases accordingly.
• Build tooling for dataset curation, trace capture, offline/online runs, and dashboards.
• Instrument production evaluations and drift monitoring, feeding failures back into golden datasets.

Required profile

• 5–9 years of QA/test engineering experience, with at least 2 years focused on AI/ML‑powered features.
• Strong test automation expertise using Python‑based frameworks and CI/CD integration.
• Deep understanding of AI failure modes (hallucination, bias, drift, non‑determinism) and ability to design targeted tests.
• Statistical literacy to interpret model evaluation metrics.
• Familiarity with AI red‑team methodologies.
• Clear, assertive communicator willing to block releases over quality concerns.
• Detail‑oriented, methodical under delivery pressure, and able to explain risk in business terms.

Required skills

• Python
• CI/CD integration
• Python‑based test automation frameworks
• OpenAI Evals
• Ragas
• DeepEval
• LangSmith
• Azure AI Foundry evaluations
• Statistical analysis for model metrics