Overview

About Amplify Health

Responsibilities

• As a Lead AI Data Engineer, you will lead a small team that designs and delivers AI-ready data platforms and GenAI data solutions on Azure and Databricks. You stay hands-on, writing code, reviewing designs and unblocking the team, while owning the technical direction, delivery quality and people development for one or two projects. You will make the architecture trade-offs that decide whether AI solutions are accurate, secure, governed and cost-effective in production.
• Role at a Glance
• Own the end-to-end solution design for data and AI workloads on 1 to 2 projects: ingestion, lakehouse modelling, unstructured-data processing, retrieval, serving and evaluation.
• Make and document architecture decisions (for example, as architecture decision records) on trade-offs such as batch versus streaming, Databricks Vector Search versus Azure AI Search, chunking and embedding strategy, model and endpoint selection, and build versus buy.
• Define project engineering standards: coding conventions, testing strategy, CI/CD, infrastructure-as-code, environment promotion and release management.
• Stay hands-on: build critical or complex components, prototype new patterns and lead deep-dive troubleshooting of production issues.
• Lead delivery of RAG, agentic and ML data pipelines from proof of concept to production, including permission-aware retrieval, freshness service levels and fallbacks.
• Set up evaluation and observability for AI solutions (golden datasets, offline and online evaluation, tracing, guardrails, quality dashboards) and use them as release gates.
• Design how governed data is exposed to AI agents (MCP servers, tools and functions, semantic layers, Genie spaces or Fabric data agents) with the right access controls and human-in-the-loop checkpoints.
• Manage AI cost and capacity: token budgets, model routing, caching, AI gateway policies and GPU / serverless sizing.
• Ensure solutions meet data governance, privacy and security requirements: Unity Catalog / Purview, classification, masking and de-identification, lineage and audit.
• Embed responsible-AI practices (risk assessment, data and model documentation, bias and quality checks, transparency) aligned to frameworks such as the NIST AI RMF, ISO/IEC 42001 and applicable regulations.
• Work with information security and data protection teams on AI threat modelling, including prompt injection, data leakage and excessive agent permissions.
• Translate business and client requirements into technical scope, estimates, plans and milestones; manage risks, dependencies and change.
• Act as the main technical contact for project stakeholders; present designs, demos and progress to technical and non-technical audiences.
• Work with product owners, data scientists, AI engineers, architects and platform teams to align priorities.
• Lead a team of 3 to 6 engineers: allocate work, set clear goals, run code and design reviews, and remove blockers.
• Coach and develop team members through regular feedback and development plans, and contribute to performance reviews.
• Drive team adoption of AI-assisted engineering with clear guardrails (review standards, security, IP and data-handling rules) and measure its impact.
• Take part in hiring through interviews, technical assessments and onboarding.

Qualifications

• Team Leadership: Leads a team of 3 to 6 engineers; may line-manage some. Accountable for technical delivery and team health
• Project scope: Leads 1 to 2 projects at a time
• Stakeholder / client exposure: Primary technical point of contact for project stakeholders and client teams; presents designs, demos and progress
• Hands-on expectation: About 50 to 70% hands-on engineering
• 9 to 12 years in data engineering, with 2+ years leading a team or workstream of engineers.
• Expert Python, PySpark and SQL, with strong system design and distributed data processing fundamentals.
• Proven delivery of production lakehouse platforms on Azure and Databricks (Delta Lake, Unity Catalog, Lakeflow, streaming, CI/CD, infrastructure-as-code).
• Proven delivery of at least two AI or GenAI data solutions to production (for example, RAG, document intelligence, agentic data access, feature / ML pipelines), including evaluation and monitoring.
• Strong retrieval design knowledge: embeddings, chunking, hybrid search, re-ranking, index freshness and access control.
• Experience with MLOps / LLMOps tooling (MLflow, model serving, AI gateway, prompt management) and AI evaluation methods.
• Solid grasp of data governance, privacy, security and responsible-AI practices for sensitive data.
• Experience planning and estimating work, managing delivery risk and communicating with senior stakeholders.
• Demonstrated ability to coach engineers and to set and enforce engineering standards.

Nice to have

• Multi-agent systems and MCP server design; Databricks Agent Bricks or Microsoft Foundry Agent Service.
• Microsoft Fabric and OneLake interoperability with Databricks.
• Ontology, knowledge graph or semantic layer design for AI agents.
• Experience in healthcare, insurance or other regulated industries; familiarity with HL7 FHIR and privacy regulations (for example, PDPA, HIPAA, GDPR).
• Agile delivery certification or formal leadership training.
• Bachelor's or Master's degree in Computer Science, Engineering, Data Science or a related field, or equivalent practical experience.
• Databricks Certified Data Engineer Professional.
• Databricks Certified Generative AI Engineer Associate.
• Microsoft Certified: Azure AI App and Agent Developer Associate (AI-103) or MLOps Engineer Associate (AI-300).
• HashiCorp Certified: Terraform Associate (optional).
• Lead by example: you still ship code and hold yourself to the standards you set.
• Clarity: you turn ambiguous requests into clear plans, decisions and trade-offs.
• Growth mindset: you invest in each team member's development and give honest, timely feedback.
• Responsibility: you make AI risk, quality and cost visible and act on them before release.
• Projects delivered to the agreed scope, quality, timeline and budget, with production service levels met.
• AI solutions go live with defined evaluation thresholds, monitoring and guardrails, and keep meeting them in production.
• Engineering standards adopted across the team: test coverage, CI/CD, infrastructure-as-code and documentation.
• A motivated, growing team: engineers build new skills, earn certifications and stay engaged.
• Positive stakeholder and client feedback on technical leadership and communication.
• Measurable productivity or cost gains from AI-assisted engineering and platform optimisation.