Overview

Data Engineer (Immediate Joiners Only) Check salary range prior to applying. Starting date is expected to be October 1st for part-time, and full-time from 2027.

Responsibilities

• Design and build scalable ingestion pipelines for structured, semi-structured, unstructured, and vector data from diverse sources, including Oracle Database, SQL Server, PostgreSQL, Pinecone, MongoDB, REST APIs, JSON, XML, CSV, Excel, logs, documents, SaaS applications, ERP systems, SharePoint, Jira, NetSuite, and Microsoft Dynamics 365.
• Implement appropriate ingestion methods, including full loads, incremental loads, Change Data Capture (CDC), batch, micro-batch, and streaming, with watermarking, checkpointing, upserts, and deletion handling.
• Design and develop reliable ETL/ELT pipelines using SQL, Python, PySpark, Databricks, Spark SQL, Delta Lake, and medallion data layers.
• Transform raw data into clean, standardized, enriched, and analytics-ready datasets while managing validation, deduplication, schema evolution, reconciliation, retries, error handling, recovery, and pipeline monitoring.
• Develop and optimize complex SQL queries, stored procedures, views, indexes, partitions, and other database objects.
• Administer databases, including user access, security, backup and recovery, monitoring, patching, migration, high availability, and performance tuning.
• Prepare data for analytics, machine learning, and Generative AI applications, including embeddings, vectorization, semantic search, vector databases, and Retrieval-Augmented Generation (RAG).
• Implement data-quality checks, metadata management, security controls, and data-governance standards.
• Troubleshoot production database and data-pipeline issues and perform root-cause analysis.
• Collaborate with data architects, application teams, analysts, data scientists, and AI engineers.
• Mentor junior team members and contribute to data-engineering and database best practices.

Qualifications

• Strong hands-on experience with Oracle Database, SQL, and PL/SQL.
• Working knowledge of MongoDB, including document modelling, aggregation, indexing, and administration.
• Knowledge of database administration, including backup and recovery, security, monitoring, migration, and high availability.
• Proficiency in PySpark, Python, or another data-engineering scripting language.
• Knowledge of vector databases, embeddings, semantic search, and RAG-based data preparation.
• Strong analytical, troubleshooting, communication, and documentation skills.