Overview

Role Details: Leads the engineering of production-grade AI and ML solutions on Azure Databricks, with accountability for model development, applied statistics, evaluation, lifecycle management and operational quality.

Responsibilities

• Leads the engineering of production-grade AI and ML solutions on Azure Databricks, with accountability for model development, applied statistics, evaluation, lifecycle management and operational quality.
• Bangalore/Gurugram/Mumbai
• Recognised Databricks, MLflow, Azure, cloud architecture or machine-learning engineering certification.

Qualifications

• 9-12 years of overall technology experience.
• 5+ years building AI/ML solutions, including strong hands-on Python development, applied modelling and production-system ownership.
• 4+ years in software, data or ML engineering, including conversion of exploratory work into tested and deployable production components.
• 3+ years with Databricks, Apache Spark or a comparable lakehouse and distributed processing environment.
• 2+ years in Generative AI, NLP, RAG, document intelligence or evaluation of LLM and agentic applications.
• 3+ years deploying and operating ML systems, inference pipelines or AI services on Azure or another major cloud platform.
• 2+ years leading engineers, data scientists or complex technical workstreams while remaining actively involved in architecture, coding and reviews.
• Advanced Python, modular and object-oriented design, typing, packaging, dependency management, configuration-driven development and performance optimisation.
• Production model pipelines covering data preparation, feature engineering, training, validation, packaging, deployment and inference.
• API and service integration, batch/streaming interfaces, error handling, retries, idempotency and operational resilience.
• Git, peer review, secure coding, documentation and unit/integration/data-quality/model-regression testing.
• Classification, regression, ranking, recommendation, clustering, forecasting and anomaly detection.
• Feature engineering and selection, model comparison, hyperparameter optimisation, ensemble methods and error analysis.
• NLP, embeddings and transformer fundamentals and practical judgement on pre-trained models, fine-tuning and custom models.
• Explainability, robustness, stability, fairness, model limitations and human-review requirements.
• Probability distributions, sampling, descriptive and inferential statistics, hypothesis testing, confidence intervals and regression.
• Experimental design, A/B testing, statistical power, sample-size considerations and practical versus statistical significance.
• Calibration, threshold optimisation, class imbalance, leakage, confounding, temporal validation, uncertainty and causal pitfalls.
• Ability to translate statistical findings into engineering decisions and measurable business outcomes.
• Python, SQL, PySpark, Delta Lake, lakehouse patterns and distributed data processing.
• Databricks Runtime for ML, notebooks/repos, Jobs, Workflows and deployment bundles or equivalent release practices.
• Feature computation and management, batch and streaming pipelines and workload performance/cost optimisation.
• Experiment and application tracking, parameters, metrics, artefacts, packaging and reproducibility.
• Models and AI artefacts in Unity Catalog, versions and aliases, promotion, rollback, access control, lineage and auditability.
• MLflow tracing, evaluation datasets, scorers, feedback, prompt/application versions and production monitoring.
• End-to-end traceability across source data, features, code, models, prompts, tools, configurations and evaluation results.
• Working knowledge of LLM behaviour, model APIs, structured outputs, context management, RAG and tool-calling patterns.
• Retrieval evaluation using precision/recall at K, ranking relevance, labelled query-document datasets and failure segmentation.
• Generation evaluation covering groundedness, correctness, completeness, consistency, robustness, safety and task completion.
• Deterministic checks, custom scorers, human evaluation, regression datasets and calibrated LLM-as-judge approaches.

Nice to have

• MLflow 3 for GenAI, Databricks Vector Search, AI Gateway, advanced Model Serving and Unity Catalog-governed AI services.
• Azure AI Foundry, Azure OpenAI, Azure AI Search, Azure DevOps and Azure-native security and integration services.
• Experience building reusable ML platforms, feature pipelines, evaluation harnesses, monitoring frameworks or common AI engineering SDKs.
• NLP, document intelligence, multilingual AI, voice, multimodal modelling or evaluation of agentic systems.
• Explainable AI, model risk management and collaboration with independent validation, Information Security and Audit teams.
• Credit, fraud, collections, customer servicing, operations, forecasting or other BFSI use cases.

Education

• Bachelor’s or Master’s degree in Computer Science, Engineering, Statistics, Mathematics, Data Science, Econometrics, Operations Research or a related quantitative discipline.
• Strong production engineering, machine-learning and applied statistical competence is more important than research credentials.