Overview

We are looking for a skilled Data Engineer with expertise in PySpark, AWS, and SQL to support data processing and analytical initiatives.

Responsibilities

• Design, build, and optimize ETL pipelines using PySpark for data ingestion, transformation, and storage on AWS.
• Collaborate with stakeholders to understand data requirements, translating them into scalable data solutions.
• Develop and manage AWS services such as S3, Glue, Lambda, EMR, Redshift, and RDS for data processing and storage.
• Implement data workflows that handle both batch and real-time processing needs, ensuring low-latency and efficient data access.
• Write and optimize complex SQL queries for data extraction and transformation from AWS RDS, Redshift, and other SQL-based databases.
• Leverage indexing, partitioning, and caching techniques to improve query performance on large datasets.
• Ensure data accuracy, completeness, and consistency throughout the data lifecycle.
• Implement best practices for data quality, governance, and security using AWS-native tools and third-party solutions.
• Automate repetitive tasks and optimize workflows, ensuring efficient and resilient data processing.
• Use CI/CD pipelines and version control (e.g., Git) to deploy and manage data workflows.
• Identify and resolve issues within the data pipeline, providing solutions for data access and query performance.
• Support the team in data analytics and data science initiatives by preparing data in an accessible and structured format.

Qualifications

• 5+ years of data engineering experience with a focus on PySpark, AWS, and SQL.
• Strong proficiency in PySpark for ETL and data transformation tasks.
• Hands-on experience with AWS data services (S3, Glue, Redshift, EMR, Lambda).
• Expertise in SQL and relational database management, with experience optimizing complex queries.
• Knowledge of data partitioning, indexing, and caching techniques to enhance performance.
• Familiarity with CI/CD principles and tools, including version control systems like Git.
• Knowledge of streaming data processes and tools like Kinesis or Kafka.
• Background in big data technologies, such as Hadoop or Apache Spark.
• Location : - Anywhere india ,Multiple Locations