Overview

In this AVP-level role, you will be responsible for designing, implementing, and maintaining automated testing frameworks for AI systems.

Responsibilities

• Quality Framework Architecture: Design, build, and maintain scalable automated testing frameworks specifically tailored for AI/ML systems, including LLMs, RAG pipelines, and traditional ML models.
• Model & Output Validation: Design and execute automated evaluation pipelines to measure critical metrics such as hallucination rates, context retention, retrieval precision, semantic similarity, and answer relevance (using tools like RAGAs, DsPy).
• Data Pipeline & ETL Auditing: Validate the quality, integrity, and schema compliance of data pipelines and ETL processes (PySpark, Hive, Kafka, Parquet, Iceberg) to ensure clean data flows into AI models.
• CI/CD Integration: Integrate automated AI quality gates into enterprise CI/CD pipelines (using Tekton, Harness, UDeploy, Jenkins) to enable continuous testing and prevent regressions.
• Risk Stewardship & Compliance: Lead adversarial testing (Red Teaming) to identify vulnerabilities like prompt injection, jailbreaking, and data leakage. Validate real-time safety guardrails to ensure compliance with ethical AI guidelines, corporate policies, and regulatory standards.
• Operational Excellence & Observability: Implement monitoring and observability solutions (using MLFlow, Dagster, Vertex AI, SageMaker) to detect model drift, performance degradation, and latency bottlenecks in production.
• Stakeholder Engagement: Demonstrate exceptional communication and diplomacy skills, effectively engaging with cross-functional stakeholders to articulate quality risks, negotiate solutions, and align on quality benchmarks.

Qualifications

• Programming Languages:
• Primary: Python, Shell Scripting, Jupyter Notebook Scripting
• Secondary: Java
• Machine Learning & AI Evaluation Frameworks:
• Core ML: Scikit-learn, Keras, TensorFlow, PyTorch, spaCy, NLTK
• Advanced NLP & GenAI Evaluation: RAGAs, DsPy, Hugging Face Transformers, RASA, Large Language Models (LLM)
• MLOps & ML Lifecycle Management:
• Platforms: MLFlow, Dagster, Google Cloud Vertex AI, Amazon SageMaker
• Database Management Systems:
• Relational: Oracle, MySQL, PostgreSQL
• NoSQL: MongoDB
• Infrastructure & DevOps:
• CI/CD & DevOps Tooling: UDeploy, Harness, Tekton, Jenkins, Maven
• Version Control: Git, Subversion (SVN)
• Containerization & Orchestration: Docker, OpenShift, Amazon ECS
• Data Engineering & Processing:
• Messaging & Asynchronous Communication: Kafka, RabbitMQ
• ETL & Data Warehousing: Sqoop, PySpark, Hive, Hadoop, HDFS
• File Formats: Avro, Parquet, Iceberg
• Job Scheduling & Orchestration: Autosys, Cron
• Web Technologies:
• Web Frameworks: FastAPI, Flask
• Web Servers & Reverse Proxies: NGINX

Education

• Bachelor’s/University degree or equivalent experience
• ----------------------------------------------------
• ----------------------------------------------------
• ----------------------------------------------------
• ----------------------------------------------------
• ----------------------------------------------------
• ----------------------------------------------------