Overview

Senior AI Evaluation Analyst

Responsibilities

• Seeking an experienced and detail-oriented professional to join its Machine Learning Science team, supporting the development and improvement of AI-powered travel products. The team works on conversational AI agents that help travelers discover options, compare products, plan trips, and receive assistance throughout their travel journey.
• You will play a key role in evaluating the quality, accuracy, relevance, and reliability of AI-generated responses against established guidelines and quality metrics. You will analyze evaluation results, identify recurring errors and quality gaps, and translate findings into actionable insights that help improve AI agent performance.
• This role requires a strong foundation in data annotation, quality evaluation, analytical problem-solving, and guideline-based decision-making. The position is suited to professionals who enjoy hands-on evaluation and analysis of AI outputs rather than traditional business reporting or dashboard development.
• Ideal Candidate Profile
• The ideal candidate is an experienced, hands-on professional who combines strong evaluation and quality assurance skills with analytical thinking. They should be comfortable reviewing large volumes of information, applying structured guidelines, identifying recurring quality issues, and converting observations into meaningful insights.
• Candidates with experience in AI evaluation, data annotation, content quality, search relevance, or similar quality-focused roles are encouraged to apply. Direct experience with LLMs or conversational AI is an advantage but is not mandatory for candidates with strong transferable experience.
• Evaluate AI-generated responses from conversational agents against predefined guidelines, evaluation criteria, and quality standards.
• Perform and review data annotation, labelling, and classification to assess the accuracy, relevance, completeness, and appropriateness of AI-generated content.
• Identify and document incorrect, misleading, incomplete, inconsistent, or irrelevant responses, including potential AI hallucinations.
• Conduct quality assurance reviews to ensure consistency, accuracy, and adherence to evaluation guidelines.
• Analyze large and complex datasets to identify recurring errors, quality trends, performance gaps, and opportunities for improvement.
• Interpret quality metrics such as error rates, evaluation consistency, reviewer disagreement, and other relevant indicators.
• Investigate ambiguous or borderline cases and apply sound judgment while adhering to established evaluation criteria.
• Prepare clear documentation of evaluation findings, root causes, observations, and recommended improvements.
• Use Excel or Google Sheets and, where applicable, basic SQL to filter, analyze, and summarize evaluation data.
• Collaborate with Machine Learning Scientists, AI product teams, and other stakeholders to communicate quality insights and support improvements in AI agent performance.
• Contribute to the refinement of evaluation guidelines, quality frameworks, and review processes based on observed issues and emerging patterns.
• Maintain accurate documentation and track identified issues through appropriate ticketing and workflow management systems.

Qualifications

• 5–8 years of relevant experience in data annotation, data labeling, AI evaluation, content quality, quality assurance, search relevance evaluation, or similar guideline-based assessment roles.
• Strong experience evaluating data, content, or system outputs against predefined guidelines, quality standards, and measurable criteria.
• Demonstrated ability to analyze large datasets, identify patterns, investigate quality issues, and derive actionable insights.
• Strong analytical thinking, problem-solving, and decision-making skills, particularly when dealing with ambiguous or inconsistent information.
• Excellent attention to detail and the ability to apply evaluation criteria consistently.
• Proficiency in Microsoft Excel or Google Sheets for data filtering, analysis, quality tracking, and reporting.
• Strong written communication skills and the ability to document findings, explain issues, and communicate insights clearly.
• Ability to work independently on evaluation tasks while collaborating effectively with cross-functional teams.
• Genuine interest in artificial intelligence, conversational AI, and data-driven product improvement.
• Experience with Human-in-the-Loop (HITL) workflows, AI training data validation, model output assessment, or LLM-as-a-judge evaluation.
• Exposure to Large Language Models (LLMs), Generative AI, conversational agents, or AI-generated content evaluation.
• Familiarity with evaluating AI responses for factual accuracy, relevance, completeness, helpfulness, and hallucinations.
• Basic SQL knowledge for extracting, filtering, and analyzing datasets.
• Experience with quality metrics such as error rates, inter-annotator agreement, reviewer disagreement, and evaluation consistency.
• Experience maintaining evaluation guidelines, quality documentation, review workflows, or issue-tracking systems.
• Exposure to travel, hospitality, customer experience, search relevance, policy evaluation, or content quality operations.