Overview

Role Purpose Build and operate the data pipelines that feed the Entity Hub.

Responsibilities

• Ingestion development — build and maintain pipelines to land the six in-scope sources (Secretary of State, D&B, ARROW, E1, hCue, DocCentral) into the Fabric Bronze/raw layer.
• Mirroring & CDC — implement Fabric Mirroring for supported structured sources and establish change-data-capture patterns; implement watermark/incremental load logic where mirroring is unavailable.
• Raw layer management — maintain one Delta table per source on an append-only basis, retaining evidence records and full source provenance.
• Standardization & transformation — implement name normalization, address parsing and attribute standardization logic in Spark notebooks; support identifier-spine construction.
• Data quality — implement data quality checks, validation rules, threshold alerts and exception handling; support reconciliation against source.
• Pipeline operations — schedule, monitor and troubleshoot pipeline runs; investigate failures and performance issues; maintain run documentation.
• Performance tuning — optimise Spark jobs, Delta file sizes, partitioning and pipeline efficiency to manage Fabric capacity consumption.
• Documentation — produce and maintain source-to-target mappings, transformation logic documentation and lineage records.

Qualifications

• 4+ years hands-on data engineering with strong PySpark and SQL
• Production experience building ingestion pipelines from multiple heterogeneous sources
• Working knowledge of Delta Lake and medallion/lakehouse architecture
• Experience implementing incremental loads and CDC-style processing
• Experience implementing data quality checks and troubleshooting pipeline failures
• Key Deliverables Owned
• Operational ingestion pipelines for all agreed sources
• Bronze/raw layer with one Delta table per source and CDC retained
• Standardization and parsing transformation logic
• Data quality checks, monitoring and exception handling
• Source-to-target mapping and run documentation
• Dual Role / Complementary Skills
• Complementary with the Entity Resolution engineering workstream — both are PySpark-on-Fabric disciplines, so this role can cross-train on Splink tuning and candidate-pair generation to provide cover. Also supports the Sr. Data Engineer (Lead) on identifier-spine construction, and can assist the VectorDB Engineer with document/attribute preparation in Phase 2.

Nice to have

• Microsoft Fabric hands-on experience (Mirroring, Copy Jobs, Environments)
• Exposure to entity/master data standardization (name and address parsing)
• Familiarity with libraries such as Great Expectations for data quality