Overview

The ideal candidate will have strong hands-on experience with Python, APIs, Generative AI, LLMs, and Agentic Systems, along with a strong understanding of AI evaluation and benchmarking.

Responsibilities

• Test and evaluate LLM-powered Agentic Systems across different business scenarios and user journeys.
• Develop and implement evaluation frameworks, datasets, and benchmarks for AI agents.
• Define and measure key agent performance metrics, including:Tool-calling accuracy and success rateResponse accuracy and relevanceReliability and consistencyTask completion rateMulti-turn conversation performanceOverall system effectiveness Design evaluation datasets, including questions, expected answers, test cases, and multi-turn conversational scenarios. Benchmark agent performance across different models, prompts, tools, workflows, and system configurations.
• Analyze agent failures and identify root causes and areas for improvement. Evaluate the agent's ability to correctly understand user intent, reason through tasks, select appropriate tools, and generate accurate responses.
• Work with large volumes of business documentation, product information, and data to quickly understand complex business domains.
• Develop Python-based scripts and tools for automated evaluation, testing, data processing, and analysis.
• Work with APIs and system integrations to test agent interactions with external tools and services.
• Collaborate with AI/ML engineers, software engineers, product managers, and domain experts to continuously improve the agent. Monitor evaluation results and provide data-driven recommendations to improve agent quality and user experience. Contribute to building scalable processes for continuous evaluation and regression testing of Agentic AI systems.

Qualifications

• Technical Skills Strong programming
• experience in Python. Solid
• understanding of REST APIs, API integrations, and system interactions. Strong
• understanding of Generative AI and Large Language Models (LLMs). Deep
• understanding of Agentic AI systems, including:AI agents and workflowsTool/function callingPrompting and context managementMulti-turn conversationsAgent orchestrationRetrieval and external knowledge integration
• Experience with AI/LLM evaluation and benchmarking. Experience designing datasets and test scenarios for AI systems. Strong analytical skills with the ability to work with evaluation data and identify patterns, failures, and performance gaps. .