Overview

Orion Innovation is a premier, award-winning, global business and technology services firm.

Responsibilities

• Validate ML models, data pipelines, and inference APIs for accuracy, fairness, and robustnessEvaluate model behavior across diverse data sets, edge cases, and unexpected inputsMonitor model drift, performance degradation, and bias over time
• Conduct prompt testing, hallucination detection, and safety validation for LLM-based systemsExperience of testing implementation of AI/ML into applicationManual Testing with automation exposureAPI testing with SOAP UI/ PostmanAgile methodology experienceTest & Defect management with JIRA/ Azure DevOps (ADO)DB Testing Experience - Good to Have Tools - Deepeval/ Langfuse/ PytestQuality Engineering & Automation
• Build and maintain automated test suites for AI features
• Develop test scripts for ML workflows using Python, Playwright, and CI/CD tools
• Design and execute manual and automated tests for functional, performance, and security scenariosIdentify defects, establish test metrics, reporting dashboards, and test traceabilityData Quality & Responsible AIValidate training and inference data quality, labeling consistency, and data governance
• Work with data science and product teams to implement responsible AI testing guidelinesDocument and escalate risks related to bias, privacy, explainability, and complianceCross-Functional CollaborationPartner with Data Scientists, ML Engineers, and Product Managers to define acceptance criteria