Browse Jobs
Explore open roles across India.Explore active software roles. Sign up to apply and get AI match scores.
Top MatchesTop Job Matches
Loading jobs…
Back to Browse Jobs
Claidroid
Senior Data Engineer / Platform Re-Engineering Lead – Azure Synapse & Databricks Migration
Claidroid
Pune•Hybrid•10-13 Yrs•Jun 23
Job description
Hybrid
Overview
Senior Data Engineer / Platform Re-Engineering Lead – Azure Synapse & Databricks Migration
Responsibilities
- As a Senior Data Engineer / Platform Re-Engineering Lead, you will lead the re-engineering of an existing enterprise data platform built on Azure Synapse Analytics and drive the future migration of validated workloads to Databricks.
- This is not a greenfield implementation role. The role requires deep technical seniority to audit, understand, validate, and improve a complex end-to-end data architecture spanning source ingestion, transformation, governance, and consumption layers.
- You will be responsible for reverse-engineering existing implementations, assessing their correctness, identifying gaps and defects, and owning the technical migration strategy to a modern Databricks Lakehouse platform.
- You will work closely with senior engineers, business stakeholders, data governance teams, and architecture teams to validate embedded business logic, preserve governance and control framework semantics, and ensure platform transformation is delivered with quality, accuracy, and business continuity.
- Lead the technical assessment and re-engineering of an existing enterprise data platform across all layers, from source ingestion through to data consumption.
- Reverse-engineer, document, and validate existing pipeline logic, data models, transformation frameworks, and data governance controls.
- Identify gaps, defects, and technical debt across the platform and remediate incorrect or sub-optimal implementations.
- Ensure correctness of data processing patterns, including change data capture, slowly changing dimensions, deduplication, soft/hard deletes, retroactive change processing, and business reconciliation.
- Design and implement target-state architectures aligned to modern lakehouse principles while ensuring feature parity and business logic fidelity during transition.
- Manage platform evolution initiatives, including parallel-run phases where multiple implementations operate simultaneously.
- Validate output consistency before cutover and contribute to legacy component decommissioning planning.
- Define and execute migration strategies for existing workloads from Azure Synapse to Databricks.
- Re-implement ingestion, transformation, and orchestration pipelines on target platforms while maintaining audit, quality, reconciliation, and governance standards.
- Collaborate with business, data governance, and architecture stakeholders to validate embedded business rules and data quality requirements.
- Provide technical leadership across re-engineering and migration workstreams.
- 10+ years of experience in Data Engineering, with significant platform migration or re-engineering experience.
- Proven experience in auditing and taking ownership of existing, complex enterprise data platforms — not just building from scratch.
- Strong hands-on experience with Azure Synapse Analytics, including Pipelines, Spark Pool, and Dedicated SQL Pool.
- Strong experience with Azure Data Lake Storage Gen2, Delta Lake on Azure, and Synapse Lakehouse patterns.
- Hands-on experience with Azure Databricks, Delta Live Tables, Delta Lake, PySpark, and Spark SQL.
Qualifications
- Employment Type: Full-time
- About Claidroid Technologies
- Claidroid Technologies Pvt. Ltd. is hiring an experienced Senior Data Engineer / Platform Re-Engineering Lead – Azure Synapse & Databricks Migration to join our growing team.
- At Claidroid Technologies, our name is derived from Cloud, AI, and Automation — the three pillars that define our innovation-driven approach to digital transformation.
- Deep understanding of medallion architecture, including Raw, Harmonized, Conformed, and Consumption layers.
- Strong knowledge of SCD Type 0/1/2, CDC patterns, soft/hard deletes, retroactive change processing, deduplication, business reconciliation, and SCD versioning.
- Ability to re-implement governance and control frameworks natively in Databricks, including audit logging, reconciliation, and data quality checks.
- Strong Python, SQL, PySpark, Spark SQL, ETL/ELT, and data modeling skills.
- Familiarity with config-driven and automation-first pipeline patterns using YAML, PySpark, SQL-driven generation, and mapping documents.
- Exposure to containerization tools such as Docker.
- Comfortable operating across both hands-on engineering and technical architecture.
- Strong communication skills with the ability to engage business, governance, architecture, and engineering stakeholders with clarity.
- Exposure to real-time and streaming pipelines such as Event Hub, Kafka, or Kinesis.
- Familiarity with monitoring and observability tools such as Dynatrace.
- Exposure to BI tools such as Power BI and Tableau.
- Immediate joiners or candidates serving a short notice period will be preferred.
- Candidates willing to work from Thiruvananthapuram or Pune in hybrid mode will be preferred.