Overview

We are seeking an experienced Senior Data Engineer to lead the design, development, and optimization of scalable, secure, and high-performance data engineering solutions.

Responsibilities

• Design and lead the development of scalable data architectures and end-to-end data pipelines.
• Define data engineering standards, architectural patterns, reusable frameworks, and best practices.
• Build and optimize complex ETL/ELT workflows supporting large-scale data processing.
• Design data ingestion frameworks for databases, REST APIs, files, event streams, and third-party platforms.
• Develop reusable connectors supporting authentication, pagination, rate limits, retries, error handling, and incremental synchronization.
• Design and maintain data warehouses, data lakes, lakehouses, and analytical data models.
• Optimize complex SQL queries, database schemas, partitioning strategies, and processing performance.
• Design and optimize data processing solutions using Databricks, PySpark, and Delta Lake.
• Define data ingestion, cleansing, standardization, enrichment, and aggregation strategies.
• Design Delta Lake tables and implement schema enforcement, schema evolution, and transactional data processing.
• Optimize Spark jobs, joins, partitioning, file sizes, and execution plans.
• Develop curated datasets for business intelligence, analytics, and downstream applications.
• Design and implement scalable Change Data Capture (CDC) and incremental loading frameworks.
• Implement watermarking, checkpointing, and recovery mechanisms.
• Ensure idempotent processing and reliable data consistency during retries and reprocessing.
• Design historical backfill, replay, and recovery strategies.
• Design reusable API integration frameworks using TypeScript, Node.js, and Python where appropriate.
• Develop secure and reliable connectors for third-party platforms and external data sources.
• Implement authentication, pagination, rate-limit management, retries, token refresh, and timeout handling.
• Design scalable serverless processing solutions using Azure Functions.
• Design orchestrated, long-running workflows using Azure Durable Functions.
• Implement workflow orchestration, retry policies, failure recovery, and parallel processing patterns where appropriate.
• Integrate serverless workflows with data pipelines, cloud storage, and downstream processing systems.
• Design, implement, and optimize data engineering solutions across AWS and Microsoft Azure.
• Work with AWS services such as Amazon S3, AWS Glue, AWS Lambda, and Amazon CloudWatch, as applicable.
• Work with Azure services such as Azure Data Lake Storage Gen2, Azure Data Factory, Azure Functions, and Azure Monitor, as applicable.
• Implement secure access controls, identity management, encryption, and secrets management.
• Optimize cloud resource utilization, storage costs, and data processing performance.
• Implement logging, metrics, monitoring, alerting, and pipeline observability.
• Ensure data accuracy, completeness, freshness, and availability.

Qualifications

• The role also involves building API integration services using TypeScript and Node.js, implementing serverless workflows using Azure Functions and Durable Functions, optimizing production workloads, establishing engineering best practices, and mentoring team members.
• 6–10+ years of experience in data engineering, data platforms, or related engineering roles.
• Advanced proficiency in Python, SQL, and PySpark.
• Strong hands-on experience with Databricks and distributed data processing.
• Expertise in Medallion Architecture and lakehouse design.
• Strong experience with Delta Lake, schema evolution, transactional data processing, and table optimization.
• Proven experience designing CDC and incremental loading frameworks.
• Strong understanding of ETL/ELT architecture, data modeling, and source-to-target reconciliation.
• Experience developing API integration services using TypeScript and Node.js.
• Hands-on experience with Azure Functions and Azure Durable Functions.
• Strong expertise in at least one of AWS or Microsoft Azure, with practical experience in the other.
• Experience with cloud security, monitoring, logging, and cost optimization.
• Experience with Git, CI/CD pipelines, automated testing, and production deployment practices.
• Strong troubleshooting, problem-solving, communication, and technical decision-making skills.
• Experience mentoring engineers and leading technical initiatives.
• Familiarity with Databricks Workflows and Unity Catalog.
• Knowledge of Terraform or other infrastructure automation tools.
• Familiarity with data governance, lineage, metadata management, and data observability.
• Bachelor’s or Master’s degree in Computer Science, Information Technology, Engineering, or a related field, or equivalent practical experience.
• Success Criteria
• Scalable, secure, and highly reliable data platform architecture.
• Improved pipeline performance, data quality, and system availability.
• Reduced production incidents and faster issue resolution.
• Optimized cloud infrastructure and data processing costs.
• Reusable data ingestion frameworks and consistent engineering standards.
• High-quality documentation, automated testing, and operational practices.
• Successful delivery of complex data engineering initiatives.
• Effective technical mentorship and cross-functional collaboration.