Overview

A senior-level Data Engineer specializing in AWS Glue, Snowflake, and related big data technologies to design and manage complex data pipelines and architectures.

Responsibilities

• Design, develop, test, deploy, and maintain large-scale data pipelines using AWS Glue, Python, PySpark, and Big Data technologies.
• Collaborate with cross-functional teams to gather requirements and deliver high-quality solutions on time.
• Develop automation scripts using Jenkins to streamline the build process and ensure efficient deployment of applications.
• Troubleshoot complex issues related to data processing, ETL processes, and database queries.

Qualifications

• 5-10 years of experience in designing and developing scalable data architectures on AWS cloud platform.
• Solid expertise in AWS services such as S3, Glue, Lambda, Redshift, DynamoDB etc.
• Proficiency in programming languages like Python or similar scripting languages (e.g., Shell).
• Experience working with big data technologies like Hadoop ecosystem (HDFS), Spark (PySpark) etc.

Nice to have

• Proficiency in Snowflake and experience working with other cloud-based storage systems.