Overview

We are seeking an accomplished Lead Data Engineer with strong expertise in modern cloud data platforms, with hands-on experience in Snowflake or Databricks.

Responsibilities

• Data Architecture & Development: Lead the design, development, and optimization of scalable, secure, and high-performance data solutions using Snowflake and/or Databricks.
• ETL/ELT Pipeline Engineering: Design, build, and maintain robust data pipelines using appropriate orchestration and data engineering frameworks, ensuring reliability, scalability, monitoring, retries, and error handling.
• Data Modelling & Transformation: Develop scalable data models and transformations following best practices for data warehousing and modern lakehouse architectures.
• Snowflake / Databricks Development: Develop and optimize data solutions on Snowflake or Databricks, including performance tuning, data processing, storage optimization, and workload management.
• Data Quality & Governance: Implement data validation, automated testing, monitoring, governance, and security practices to ensure data integrity, reliability, and compliance.
• Performance Optimization: Identify and resolve performance bottlenecks across data pipelines, queries, jobs, and data processing workloads.
• Cloud Data Engineering: Build and manage data solutions on cloud platforms such as AWS, Azure, or GCP.
• Collaboration: Partner with data analysts, data scientists, architects, and business stakeholders to translate requirements into scalable technical solutions.
• Technical Leadership: Provide technical guidance and mentorship to data engineers, conduct code/design reviews, and establish engineering best practices.
• Innovation & Research: Stay current with advancements in cloud data platforms, data engineering, analytics, and AI/ML technologies and recommend relevant improvements.
• Infrastructure & CI/CD: Work with version control, CI/CD, and infrastructure-as-code practices for reliable deployment and management of data platforms.

Qualifications

• Bachelors or Masters degree in Computer Science, Information Technology, Data Engineering, or a related field.
• 710 years of experience in data engineering, with strong hands-on experience in at least one of Snowflake or Databricks.
• Demonstrated experience in leading and mentoring data engineering teams.
• Strong working knowledge of modern data engineering concepts, including:
• Data modelling and data warehousing
• ETL/ELT processes
• Data Lake / Lakehouse architectures
• Batch and incremental data processing
• Data quality and validation
• Snowflake: Strong hands-on experience in data modelling, performance tuning, access control, and features such as streams, tasks, external tables, or equivalent capabilities.
• Databricks: Strong hands-on experience with data engineering, Spark-based processing, Delta Lake/Lakehouse concepts, and pipeline development.
• Snowflake or Databricks one must be hands-on; knowledge of the other is good to have.
• Proficiency in SQL and Python.
• Spark experience is expected for candidates with Databricks exposure.
• Familiarity with transformation frameworks such as dbt is good to have.
• Familiarity with version control systems such as Git and CI/CD practices.
• Exposure to infrastructure-as-code tools such as Terraform is good to have.
• Strong analytical, problem-solving, collaboration, and communication skills.
• Demonstrated ability to lead technical projects and mentor junior/mid-level engineers.
• Preferred Skills:
• Hands-on experience with both Snowflake and Databricks.
• Familiarity with BI/analytics tools such as Power BI, Tableau, Looker, or similar.
• Knowledge of data governance, security, and compliance frameworks.