AI QA Engineer – Test LLM and Generative AI Applications

About the role

Eurisko is seeking an AI QA Engineer to ensure the quality and reliability of applications powered by large language models and generative AI. You will work closely with AI engineers, developers, and product teams to design tests, evaluate responses, and prevent regressions across AI‑driven products.

Key responsibilities

• Design and execute test scenarios for prompts, conversational flows, multilingual inputs, and edge cases.
• Evaluate AI‑generated responses for accuracy, relevance, tone, and compliance with business requirements, flagging hallucinations and inappropriate content.
• Validate Retrieval‑Augmented Generation (RAG) pipelines, checking citation accuracy, grounding, and behavior with missing or outdated information.
• Build automated regression suites, create test datasets, define scoring criteria, and integrate checks into CI/CD pipelines.
• Perform safety and adversarial testing, including prompt injection and guardrail verification.
• Test end‑to‑end AI workflows, APIs, tool calls, latency, and error handling, documenting defects and collaborating on fixes.

Required profile

• Professional experience in software QA, including test planning, case design, defect management, API testing, and automation.
• Hands‑on experience testing LLM‑powered or generative AI applications and understanding of AI‑specific failure modes.
• Working knowledge of RAG, embeddings, vector stores, and context management.
• Practical Python skills for automation, dataset preparation, and evaluation scripting.
• Experience in Agile/Scrum environments with strong analytical and communication abilities.

Required skills

• Python
• API testing
• Test automation frameworks
• RAG concepts and vector‑store technologies
• Agile/Scrum methodology