Overview

Azure Databricks & Generative AI Engineer

Responsibilities

• We are looking for an experienced Azure Databricks & Generative AI Engineer to build scalable data and AI solutions using Azure Databricks, Python, and Generative AI technologies. The ideal candidate should have strong expertise in data engineering, Databricks, Azure cloud services, and hands-on experience developing or integrating Gen AI/LLM-based solutions.
• Design, develop, and maintain scalable data solutions using Azure Databricks.
• Develop data processing and transformation pipelines using PySpark and Python.
• Work with Delta Lake, Spark SQL, and Databricks Workflows.
• Build and integrate Generative AI and LLM-based applications.
• Develop AI-powered solutions using Azure OpenAI and Azure AI services.
• Implement RAG (Retrieval-Augmented Generation) solutions using enterprise data.
• Work with embeddings, vector databases, and AI Search for intelligent information retrieval.
• Integrate structured and unstructured data into Gen AI applications.
• Develop data pipelines to support AI/ML and analytics use cases.
• Optimize Databricks jobs, Spark workloads, queries, and data processing performance.
• Implement data quality, security, governance, and access controls.
• Collaborate with data scientists, software engineers, business teams, and cloud architects.
• Monitor and troubleshoot data pipelines and AI workloads.
• Follow best practices for cloud architecture, CI/CD, version control, and deployment.

Qualifications

• Solid hands-on experience with Azure Databricks.
• Strong knowledge of PySpark, Python, and SQL.
• Good understanding of Azure cloud services.
• Hands-on knowledge of Generative AI and Large Language Models (LLMs).
• Understanding of RAG architecture, embeddings, prompt engineering, and vector search.
• Knowledge of data engineering and ETL/ELT concepts.
• Preferred Skills
• Knowledge of MLflow and MLOps.
• Knowledge of Azure Data Lake Storage (ADLS Gen2).
• Familiarity with LangChain, LlamaIndex, or similar Gen AI frameworks.
• Understanding of Responsible AI and enterprise AI security.

Education

• Strong analytical and problem-solving skills.
• Good understanding of data and AI architectures.
• Ability to work with cross-functional teams.
• Strong communication and documentation skills.
• Ability to troubleshoot complex data and AI issues.
• Willingness to learn and adopt emerging Gen AI and Azure technologies.