Overview

Senior Data Engineer: Data Pipelines, Orchestration, System Design & A (Kolkata)

Responsibilities

• We are looking for a Senior Data Engineer to design, build, and operate reliable data pipelines and orchestration frameworks, and to enable AI-driven products and workflows on top of them. You will own the data flow from ingestion to consumption, set engineering standards for the team, and work closely with analytics, data science, and ML teams.
• Design, build, and maintain scalable batch and streaming pipelines for structured and unstructured data.
• Develop robust ingestion, transformation, and loading (ETL/ELT) processes from diverse sources (databases, APIs, event streams, files).
• Optimize pipeline performance, cost, and reliability
• Build and manage workflows using orchestration tools
• Implement scheduling, dependency management, retries, alerting, and backfill strategies.
• Build data foundations for machine learning and generative AI use cases, including feature pipelines and data preparation for model training and inference.
• Integrate AI/LLM tools and APIs into data workflows (e.g., document processing, enrichment, classification, summarization).
• Support retrieval-based AI solutions, including embedding pipelines and vector databases
• Collaborate with data scientists and ML engineers to move models from prototype to production.
• Implement data quality checks, validation, lineage, and documentation.
• Lead design reviews and code reviews, and mentor junior and mid-level engineers.

Qualifications

• Senior Data Engineer: Data Pipelines, Orchestration, System Design & AI Tooling
• Location: Kolkata
• Employment Type: Full-time
• 8-10 years of professional experience in data engineering or a closely related field.
• Robust proficiency in Python and advanced SQL.
• Proven experience building and maintaining production data pipelines at scale.
• Hands-on experience with at least one orchestration tool
• Solid understanding of data warehousing and lakehouse concepts, data modeling, and tools.
• Working knowledge of AI/ML workflows and tools, including LLM APIs, embeddings, and vector databases.
• Strong problem-solving, communication, and stakeholder management skills.
• Knowledge of data governance, cataloging, and lineage tools.
• Cloud or data platform certifications.
• Bachelor's or
• Master's degree in Computer Science, Engineering, or a related field. .