Overview

We are looking for a skilled Mid-Level Data Engineer to design, develop, and maintain scalable data pipelines and cloud-based data processing solutions.

Responsibilities

• Design, develop, and maintain scalable data pipelines for batch and incremental data processing.
• Build ETL/ELT workflows to extract, transform, validate, and load data from APIs, databases, files, and third-party platforms.
• Develop and optimize SQL queries, stored procedures, and database schemas.
• Work with relational databases such as PostgreSQL, SQL Server, or MySQL.
• Develop data integration solutions using Python, REST APIs, and relevant cloud services.
• Build and maintain data processing pipelines using Databricks and PySpark.
• Implement Medallion Architecture with Bronze, Silver, and Gold layers.
• Work with Delta Lake tables, schema evolution, and data transformation processes.
• Implement Change Data Capture (CDC) and incremental loading mechanisms.
• Handle data cleansing, validation, deduplication, reconciliation, and error handling.
• Develop API integration utilities and backend services using TypeScript and Node.js.
• Develop serverless functions using Azure Functions and support workflow orchestration using Durable Functions.
• Schedule, monitor, and troubleshoot data pipeline execution.
• Optimize query performance, Spark jobs, pipeline efficiency, and data storage.
• Collaborate with data analysts, backend developers, and business teams to understand requirements.
• Maintain technical documentation, data mappings, pipeline specifications, and operational procedures.
• Follow data security, access control, coding standards, and version control practices.

Qualifications

• 3–5 years of experience in data engineering, data integration, or a related field.
• Strong proficiency in Python and SQL.
• Hands-on experience with Databricks and PySpark.
• Understanding of Medallion Architecture and Bronze, Silver, and Gold data layers.
• Experience working with Delta Lake and Delta tables.
• Knowledge of ETL/ELT development, CDC, and incremental data loading.
• Experience integrating REST APIs and handling JSON, CSV, and other data formats.
• Working knowledge of TypeScript and Node.js.
• Experience with Azure Functions; familiarity with Durable Functions.
• Experience with relational databases and data modeling.
• Hands-on experience with at least one cloud platform: AWS or Microsoft Azure.
• Familiarity with Git, CI/CD practices, logging, monitoring, and automated testing.
• Understanding of data quality checks, error handling, and pipeline troubleshooting.
• Reliable and maintainable data pipelines.
• Accurate, consistent, and timely data delivery.
• Improved SQL, Spark, and pipeline performance.
• Reduced pipeline failures and effective issue resolution.
• High-quality, reusable, and well-documented code.
• Successful delivery of assigned engineering tasks within agreed timelines.
• Bachelor’s degree in Computer Science, Information Technology, Engineering, or a related field, or equivalent practical experience.

Nice to have

• Experience working with both AWS and Microsoft Azure.
• Familiarity with Amazon S3, AWS Glue, AWS Lambda, and Amazon CloudWatch.
• Familiarity with Azure Data Factory, Azure Data Lake Storage Gen2 (ADLS Gen2), and Azure Monitor.
• Experience with Databricks Workflows and job scheduling.
• Knowledge of Delta Lake optimization and schema evolution.
• Exposure to Docker and cloud-based data platforms.
• Experience with BI and analytics tools such as Power BI or Tableau.
• Familiarity with data governance, data lineage, and automated data reconciliation.