Overview

Synechron is seeking a Data Engineer with strong expertise in PySpark and the Cloudera Data Platform (CDP) to design, develop, and maintain scalable data pipelines.

Responsibilities

• Design and develop scalable data pipelines using PySpark and Cloudera Data Platform (CDP).
• Build, optimize, and maintain data ingestion and transformation processes for large-scale datasets.
• Develop reusable, maintainable, and testable data-processing components.
• Implement data validation and quality checks to support data accuracy, consistency, integrity, and availability.
• Work with Hadoop, Hive, HDFS, Spark, ETL processes, and related big data technologies.
• Apply data warehousing and big data architecture principles when designing data solutions.
• Collaborate with cross-functional teams to understand data requirements and deliver data-driven solutions.
• Monitor data workflows, processing jobs, resource utilization, and pipeline performance.
• Investigate and resolve data-quality issues, job failures, performance bottlenecks, and operational incidents.
• Optimize Spark jobs, PySpark code, queries, data layouts, partitioning, and resource usage where required.
• Document data pipelines, transformation logic, dependencies, operational procedures, and known limitations.
• Support deployment, release, testing, and production-readiness activities for data solutions.
• Contribute to improvements in data engineering standards, development practices, monitoring, and operational support.
• Consider sustainable engineering practices by reducing unnecessary data movement, optimizing compute and storage usage, reusing pipeline components, and supporting maintainable solutions.
• Deliver data pipelines that meet agreed requirements for quality, performance, reliability, security, and availability.
• Escalate material risks, dependencies, data-quality concerns, and delivery constraints through appropriate channels.
• Strong hands-on experience with Python, particularly for PySpark-based data processing.
• Design, develop, test, and optimize PySpark data pipelines and CDP-based ingestion and transformation workflows.
• Collaborate with engineering, analytics, business, and operational teams to clarify data requirements, dependencies, delivery timelines, and implementation options.
• Monitor scheduled and production workflows, investigate failures or performance issues, and deliver fixes, enhancements, documentation, and operational updates.
• Make technical decisions within the agreed solution scope regarding data-processing logic, performance improvements, validation controls, and pipeline reliability; escalate material risks when required.

Qualifications

• Familiarity with workflow orchestration, data- pipeline monitoring, scheduling, and alerting tools.
• Familiarity with data-quality, metadata-management, lineage, and observability tools.
• Exposure to streaming, real-time processing, or event-driven data platforms.
• Ability to use SQL for data querying, validation, transformation, and analysis.
• Familiarity with Scala or another language used in distributed data processing.
• Hands-on experience with Hive, HDFS, Hadoop, and data warehousing concepts.
• Understanding of structured, semi-structured, and large-scale distributed data.
• Knowledge of data quality, data integrity, validation, partitioning, schema management, and data availability.
• Understanding of ETL processes and big data architecture.
• Familiarity with data lakes, lakehouse architectures, metadata management, data lineage, or data cataloguing.
• Exposure to batch and streaming data-processing patterns.
• Understanding of cloud-related considerations such as scalability, storage, compute usage, availability, access controls, and operational monitoring.
• Ability to design or support data pipelines that run reliably in distributed environments.
• Familiarity with cloud monitoring, logging, automation, and cost-optimization practices.
• Strong experience with PySpark and Apache Spark.
• Ability to select appropriate Spark processing patterns for scalability, reliability, and performance.
• Familiarity with structured streaming or other distributed streaming frameworks.
• Exposure to reusable pipeline frameworks and configuration-driven processing.
• Familiarity with source control, code reviews, defect tracking, and controlled release practices.
• Understanding of batch processing, distributed computing, ETL development, and software development lifecycle practices.
• Ability to document technical solutions, operational procedures, dependencies, and data-processing logic.
• Familiarity with continuous integration and continuous delivery practices.
• Exposure to agile development practices and iterative delivery.
• Understanding of secure data handling, access control, authentication, authorization, and protection of sensitive information.
• Ability to apply security and privacy requirements when designing, processing, storing, and transferring data.
• Awareness of secure coding, credential management, auditability, and least-privilege principles.
• Familiarity with encryption, data masking, secure data transfer, and role-based access controls.
• 6–10 years of professional experience in data engineering, big data engineering, ETL development, or a related role.
• Strong production experience with PySpark and Cloudera Data Platform (CDP).
• Hands-on experience with Hadoop, Hive, HDFS, Spark, ETL processes, data warehousing, and big data architecture.