Overview

ZS is a place where passion changes lives.

Responsibilities

• Design, develop, maintain and support robust data pipelines that ingest, transform and deliver data for analytics, reporting, machine learning and AI use cases.
• Write and review production-grade Python, PySpark and SQL for complex, high-volume data transformations.
• Optimize Spark and SQL workloads for performance, scalability and cost by analyzing execution plans, partitioning, skew, caching, memory pressure and I/O patterns.
• Build reusable ingestion and transformation components, common utilities and automation frameworks that reduce manual intervention and improve delivery consistency.
• Monitor scheduled pipelines and distributed jobs, investigate failures, restore services, and improve observability, alerting, and operational runbooks.
• Perform hands-on troubleshooting across the data-processing layer and supporting cloud infrastructure, including Linux hosts, storage, access, networking and service configuration.
• Conduct root cause analysis for recurring or high-impact incidents and implement long-term remediation to prevent recurrence.
• Apply data-engineering standards for reliability, maintainability, data quality, security and governance throughout the development lifecycle.
• Partner with data producers, consumers, architects, product teams and cloud operations teams to translate requirements into practical technical solutions.
• Contribute to technical design reviews, code reviews, release validation, capacity planning and productionreadiness assessments.
• Support proofs of concept involving LLMs, Amazon Bedrock, Agentic AI and MCP-based integrations where they intersect with enterprise data platforms.
• Work with AWS or other technology vendors and subject matter experts for advanced troubleshooting when required.

Qualifications

• 4+ years of hands-on experience in data engineering, including Apache Spark, Hadoop ecosystems and Data Warehousing.
• Strong expertise in designing, building, testing, and supporting scalable batch or ETL/ELT data pipelines.
• Hands-on expertise in Spark or PySpark development, including distributed processing concepts and job optimization.
• Advanced SQL skills, including complex joins, window functions, aggregations, query tuning and data validation.
• Strong Python skills for data processing, reusable frameworks, automation and operational tooling.
• In-depth hands-on experience with Linux, shell utilities, logs, processes, file systems and troubleshooting.
• Proven ability to implement complex data transformations, including cleansing, deduplication, enrichment, aggregation and slowly changing dimension patterns.
• Sound understanding of data warehousing, data lakes, partitioning, file formats, schema evolution and data quality controls.
• Experience diagnosing and optimizing pipeline performance, including parallelism, partition sizing, data skew, memory usage, shuffle behavior and inefficient SQL patterns.
• Ability to troubleshoot complex production issues, perform root cause analysis and implement sustainable corrective actions.
• Understanding of cloud infrastructure fundamentals relevant to analytics workloads, including compute, storage, identity and access, networking, monitoring and cost management.
• Understanding of Large Language Models, Amazon Bedrock, Agentic AI patterns and Model Context Protocol (MCP) concepts.
• Ability to collaborate with cross-functional and globally distributed teams and communicate technical findings clearly to stakeholders.
• Hands-on experience with AWS analytics services such as Amazon EMR, AWS Glue, Amazon Athena, Amazon SageMaker, etc.
• Experience with Amazon S3-based data lakes and related security, cataloging, partitioning and performance
• Exposure to workflow orchestration or scheduling tools and CI/CD practices for data pipelines.
• Familiarity with infrastructure-as-code or cloud automation tools used to provision and manage analytics
• Knowledge of streaming data, open table formats, data governance, or metadata and catalog solutions.
• Experience supporting secure, regulated or mission-critical analytics platforms.
• Associate-level or higher AWS certification, or relevant data engineering certification
• Fluency in English
• Client-first mentality
• Intense work ethic
• Collaborative spirit and problem-solving approach
• Cross-functional skills development & custom learning pathways
• Milestone training programs aligned to career progression opportunities
• Internal mobility paths that empower growth via s-curves, individual contribution and role expansions