Browse Jobs
Explore open roles across India.Explore active software roles. Sign up to apply and get AI match scores.
Top MatchesTop Job Matches
Loading jobs…
Back to Browse Jobs
Zorba AI
Data Engineer with GCP + Pyspark, Scala
Zorba AI
New Delhi•On-site•6-9 Yrs•13 hours ago
Job description
Overview
Job Title: Data Engineer – GCP, PySpark & ScalaExperience
Responsibilities
- Design, develop, deploy, monitor, and optimize batch ETL/data pipelines using Scala, PySpark, and GCP services.
- Build and maintain scalable data processing workflows using Google Cloud Platform technologies such as BigQuery, Dataproc, and Cloud Storage.
- Develop and manage Airflow DAGs, ensuring proper dependency handling, scheduling, retry mechanisms, and idempotent execution.
- Implement Medallion Architecture (Bronze, Silver, Gold) with robust data quality checks and data summarization processes.
- Optimize ETL jobs and Spark applications for performance, scalability, and cost efficiency.
- Monitor production pipelines, create observability dashboards, and develop runbooks for operational support and continuous optimization.
- Troubleshoot production issues and collaborate with cross-functional teams to deliver reliable data solutions.
- Follow coding standards, best practices, and CI/CD processes for data engineering solutions.
- Support data governance initiatives and ensure compliance with enterprise data management standards.
Qualifications
- 6+ years of experience in Data Engineering.
- Strong hands-on experience with Scala and PySpark.
- Expertise in Google Cloud Platform (GCP) services, including BigQuery, Dataproc, Cloud Storage, and related data services.
- Strong knowledge of Apache Airflow, including workflow orchestration, dependency management, retries, and idempotent execution.
- Strong SQL skills and experience with large-scale data processing.
- Good understanding of distributed data processing and Spark optimization techniques.
- Exposure to data governance and metadata management.
- Familiarity with Agile/Scrum development methodologies.
- Knowledge of data security and cloud best practices.