Overview

We are seeking a highly skilled Databricks Platform Engineer to design, build, configure, integrate, and operationalize enterprise Data & AI solutions on the Databricks platform.

Responsibilities

• Design, build, configure, and maintain Databricks workspaces across development, testing, and production environments.
• Implement platform standards, reusable frameworks, templates, and best practices.
• Manage Unity Catalog, clusters, SQL Warehouses, compute policies, workspace configurations, and access controls.
• Automate platform deployment and configuration using Infrastructure as Code (Terraform, CI/CD pipelines).
• Build and integrate scalable data pipelines using Databricks Lakehouse architecture.
• Design ingestion frameworks for batch, streaming, API, database, and file-based integrations.
• Implement Delta Lake, Structured Streaming, and medallion architecture patterns.
• Integrate Databricks with enterprise data platforms such as Snowflake, SAP, Oracle, SQL Server, Azure Data Factory, Kafka, and cloud storage services.
• Enable ML and Generative AI workloads on Databricks.
• Implement MLflow, model lifecycle management, feature stores, and model serving capabilities.
• Support AI engineers and data scientists with scalable development environments.
• Integrate Databricks with LLMs, vector databases, AI gateways, and enterprise AI platforms.
• Establish CI/CD pipelines for data and AI workloads.
• Implement automated testing, deployment, monitoring, and rollback mechanisms.
• Create reusable deployment frameworks and engineering accelerators.
• Support release management and environment promotion processes.
• Implement enterprise security controls, RBAC, data masking, encryption, and audit logging.
• Configure and manage Unity Catalog governance policies.
• Ensure compliance with enterprise security, privacy, and regulatory requirements.
• Partner with cybersecurity teams to implement platform hardening and vulnerability remediation.
• Implement platform observability, monitoring, and operational dashboards.
• Configure logging, alerting, performance monitoring, and incident management processes.
• Optimize platform performance, cost, scalability, and reliability.
• Support production operations and resolve platform issues.
• Collaborate with architects, data engineers, AI engineers, security teams, and business stakeholders.
• Provide technical guidance and platform best practices.
• Participate in architecture reviews and platform roadmap planning.
• Mentor junior engineers and contribute to engineering excellence initiatives.

Qualifications

• Bachelor's or master’s degree in computer science, Engineering, IT, or a related field.
• 5+ years of experience in Data, Platform, or Cloud Engineering.
• 3+ years of hands-on experience with the Databricks Lakehouse Platform.
• Strong expertise in Delta Lake, Unity Catalog, MLflow, Databricks Workflows, Structured Streaming, PySpark, and Spark SQL.
• Experience working with cloud platforms such as Azure, AWS, or GCP.
• Proficiency in Python and SQL.
• Strong knowledge of DevOps, CI/CD, Infrastructure as Code (IaC), and platform automation.
• Hands-on experience with Terraform, GitHub Actions, Azure DevOps, or equivalent CI/CD tools.
• Experience with Generative AI, Agentic AI, and LLM-based applications.
• Experience integrating Databricks with Snowflake and enterprise AI platforms.
• Knowledge of Kubernetes, Docker, APIs, Kafka, and event-driven architectures.
• Databricks Certified Professional or Associate certifications.
• Experience supporting enterprise-scale Data & AI platforms.
• Platform availability and reliability.
• Deployment automation and operational efficiency.
• Security and compliance adherence.
• Data pipeline performance and scalability.
• AI/ML platform adoption and productivity improvements.
• Platform cost optimization and governance effectiveness.