Browse Jobs
Explore open roles across India.Explore active software roles. Sign up to apply and get AI match scores.
Top MatchesTop Job Matches
Loading jobs…
Back to Browse Jobs
True Tech Professionals
Senior Data Engineer Databricks Pipelines (Delhi)
True Tech Professionals
New Delhi•Hybrid•3-5 Yrs•Jul 1
Job description
Hybrid
Overview
Senior Data Engineer Databricks Pipelines
Responsibilities
- Design, develop, and maintain scalable ETL/ELT pipelines using Databricks, PySpark, Delta Lake, and Databricks Workflows .
- Build production-grade data ingestion frameworks supporting both batch and streaming workloads.
- Develop reusable and optimized data processing components using Python and Spark.
- Design and optimize Databricks Lakehouse architecture using Raw, Bronze, Silver, and Gold data layers.
- Implement efficient storage strategies to support analytics, AI workloads, and high-performance retrieval use cases.
- Optimize Spark jobs for performance, scalability, and reliability.
- Build real-time data ingestion pipelines using Delta Live Tables (DLT) and Databricks Auto Loader.
- Integrate multiple enterprise data sources including batch and streaming systems.
- Implement CDC (Change Data Capture) pipelines for incremental data processing.
- Implement data validation frameworks and quality checks within data pipelines.
- Build automated data quality gates, exception handling, and monitoring mechanisms.
- Implement Unity Catalog for data governance, access control, lineage tracking, and security management.
- Troubleshoot and optimize Spark workloads involving:
- Data skew
- Shuffle optimization
- Partition strategies
- Join optimization
- Cluster performance tuning
- Utilize Spark UI, Adaptive Query Execution (AQE), caching, broadcast joins, and optimization techniques.
- Manage CI/CD workflows for Databricks deployments using:
- Git
- Azure DevOps / GitHub Actions
- Databricks Asset Bundles (DABs)
- Promote data solutions across Dev, QA, and Production environments following engineering best practices.
- Collaborate on infrastructure automation using tools like Terraform.
- Design, develop, and maintain scalable ETL/ELT pipelines using Databricks, PySpark, Delta Lake, and Databricks Workflows .
- Build production-grade data inge
Qualifications
- Employment Type: Full-Time
- Job Overview
- We are looking for a highly skilled Senior Data Engineer with strong expertise in Databricks, PySpark, and modern data engineering practices to design, develop, and optimize enterprise-scale data pipelines.
- The ideal candidate will play a key role in building scalable Lakehouse architectures , developing reliable ETL/ELT workflows, and enabling AI-driven analytics solutions by ensuring high-quality, governed, and performance-optimized data platforms.
- You will work closely with data scientists, AI teams, and engineering teams to deliver robust data pipelines that power analytics, chatbot solutions, and intelligent data applications.
- 69 years of experience in enterprise-scale Data Engineering.
- Strong hands-on experience with:
- Databricks
- PySpark
- Python
- Delta Lake
- SQL
- Solid understanding of Lakehouse architecture and modern data platforms.
- Hands-on experience with:
- Delta Live Tables (DLT)
- Databricks Workflows
- Auto Loader
- Unity Catalog
- Delta Change Data Feed (CDF)
- Strong knowledge of Spark optimization techniques.
- Preferred Skills
- Knowledge of vector search or AI-ready data architectures.
- Strong understanding of data governance and security frameworks.
- Build scalable and reliable data pipelines independently.
- Optimize complex Spark workloads.
- Design production-grade Databricks solutions.
- Implement automation, governance, and best practices across the data lifecycle.
- Employment Type: Full-Time
- Job Overview
- We are looking for a highly skilled Senior Data Engineer with strong expertise in Databricks, PySpark, and modern data engineering practices to design, develop, and optimize enterprise-scale data pipelines.