Data Quality & AI Operations Manager

About the role

Own content and data operations for the Extend platform. Build a leaner, more accurate profiling process and use AI fine tuning to progressively replace manual annotation with automation. You own the evaluation and data feedback loop and lead the content operations team. Success is measured by golden‑set accuracy and profile automation rate.

Key responsibilities

• Build and maintain golden sets and evaluation suites; define acceptance criteria for each AI Admin iteration.
• Own the data side of AI Admin iterations: curate training and evaluation data, run benchmarks, analyze failure modes, and partner with Shanghai engineering on fine‑tuning and deployment.
• Design the human‑in‑the‑loop (HITL) mechanism: define the boundary between human review and auto‑approval, and feed human corrections back into prompts, eval sets and training data.
• Author and maintain Arabic/English labeling guidelines; adjudicate edge cases requiring cultural and linguistic judgment.
• Lead the content operations team with end‑to‑end ownership of annotation operations: rater training, workflows, throughput and quality SLAs.
• Establish and track a quality metrics framework: accuracy, automation rate, inter‑annotator agreement, cost and turnaround per profile.

Required profile

• Fluent in Arabic, with strong knowledge of the Arab world's media, brands, publishers and cultural context.
• Hands‑on experience with AI agent / LLM systems: eval design, golden sets and model feedback loops; conversant in finetuning concepts (SFT, RLHF, prompt iteration) without needing to train models yourself.
• 2+ years in AI data operations or applied AI quality, from a data labeling company, an in‑house annotation team at an AI company, or an AI product team.
• Comfortable collaborating across Riyadh and Shanghai time zones.

Required skills

• SFT
• prompt iteration
• taxonomy design
• finetuning concepts
• annotation operations
• LLM systems
• eval design
• model feedback loops
• LLM‑assisted labeling automation
• golden sets
• AI agent
• data labeling
• RLHF