Overview

We are looking for a hands-on Data Engineer who can independently design, build and operate production-grade data pipelines using Apache Airflow.

Responsibilities

• We are looking for a hands-on Data Engineer who can independently design, build and operate production-grade data pipelines using Apache Airflow. This is a delivery-facing engineering role: you will own complex, multi-dependency DAGs end to end from ingestion and transformation logic through to scheduling, monitoring, failure recovery and performance tuning.
• The ideal candidate writes production-quality Python, is genuinely strong in SQL (including analytical functions, complex joins and query optimization), and has debugged enough real pipeline failures to have opinions about idempotency, backfills and retry strategy. You will work closely with data architects, analysts and business stakeholders, and will be expected to raise the engineering standard of the pipelines you touch.
• Design, develop and maintain complex Airflow DAGs involving multi-stage dependencies, dynamic task generation, branching, sensors, task groups and cross-DAG triggers.
• Build pipelines that are idempotent, restart able and backfill-safe, with correct handling of scheduling intervals, catchup behavior, data intervals and time zones.
• Implement robust error handling and recovery -retries with backoff, SLAs, alerting, callbacks, dead-letter handling and clean partial-failure semantics.
• Develop reusable components: custom operators, hooks, sensors and DAG factory patterns that reduce duplication across the codebase.
• Manage Airflow configuration properly - connections, variables, pools, priority weights, queues and concurrency/parallelism controls.
• Write and optimize complex SQL - multi-table joins, CTEs, window/analytical functions, incremental and merge/upsert logic, slowly changing dimensions.
• Tune query performance: read execution plans, correct inefficient joins and predicates, and apply appropriate partitioning, clustering and indexing strategies.
• Implement data quality and reconciliation checks - row counts, null/duplicate checks, referential integrity and business rule validations as first-class pipeline steps.
• Contribute to dimensional and analytical data model design (star/snowflake schemas, fact and dimension design, grain definition).
• Write clean, modular, testable Python following sound engineering practice - packaging, configuration management, logging, exception handling and type hints.
• Build ingestion and transformation utilities: REST/SOAP API integrations, pagination and rate-limit handling, file parsing (CSV, JSON, Parquet, XML) and data frame processing (pandas / PySpark).
• Write unit and integration tests for pipeline logic, including DAG integrity tests, and participate in code reviews.
• Monitor and support production pipelines - triage failures, perform root cause analysis, drive permanent fixes, and reduce recurring incidents.
• Work within CI/CD practice: Git-based workflows, branching strategy, automated deployment of DAGs, and environment promotion (Dev → QA → Prod).
• Translate requirements from analysts and business stakeholders into technical designs, and document DAG logic, data lineage and operational runbooks.

Qualifications

• B.E. / B.Tech / MCA / M.Sc. in Computer Science, Information Technology or a related discipline.
• Equivalent practical experience will be considered for candidates with a demonstrable engineering track record.