Overview

Grade Level (for internal use): 13Key Responsibilities Data Pipeline Transition and Platform Delivery • Partner with the core Databricks team to plan and execute the transition of existing data pipelines to the target enterprise data platform.

Responsibilities

• Partner with the core Databricks team to plan and execute the transition of existing data pipelines to the target enterprise data platform.
• Drive implementation of repeatable engineering patterns for ingestion, transformation, testing, deployment, and monitoring across onboarded datasets.
• Guide the design and operation of cloud-native data pipelines leveraging relevant AWS services such as:
• Amazon S3 for durable storage
• AWS Glue for integration and catalog-driven processing
• AWS Lambda for event-driven processing
• Amazon Kinesis for streaming use cases
• AWS Lake Formation for governed data lake controls
• Promote the use of AWS IAM, encryption, and environment-level controls to enforce secure access to platform resources and data products in line with enterprise governance expectations.
• Influence architectural decisions related to Databricks, Delta Lake, Apache Iceberg, Unity Catalog, metadata-driven processing, and governed data lake design.
• Champion the adoption of AI-powered development tools, including Claude, GitHub CoPilot, LLMs, and other AI-assisted engineering solutions, to increase engineering velocity, code quality, and operational effectiveness.
• Apply LLM-based tools to support common data engineering activities such as:
• Code generation
• Code refactoring
• SQL optimization
• PySpark optimization, etc.
• Develop and implement context engineering practices that provide AI tools with the necessary technical background, architectural constraints, data schemas, coding standards, platform patterns, security requirements, and governance expectations.
• Create reusable context packs, prompt libraries, and AI-enabled engineering playbooks for common data platform tasks, including Databricks pipeline migration, ingestion framework development, test generation, code review support, and operational troubleshooting.
• Use LLMs to accelerate understanding and modernization of legacy pipelines, including dependency analysis, code explanation, transformation logic interpretation, metadata extraction, and refactoring recommendations.
• Provide technical direction for offshore teams supporting data onboarding to the enterprise platform in an asset-agnostic manner.
• Define and operationalize onboarding patterns that can support a broad range of data assets, domains, and source systems without requiring bespoke platform redesign for each use case.
• Work with partner teams to simplify and standardize how data is ingested, transformed, governed, and published to the platform.
• Establish reusable ingestion and processing patterns across batch and streaming use cases using technologies such as AWS Glue, AWS Lambda, Amazon Kinesis, or event-driven integrations where appropriate.
• Support integration of platform pipelines and datasets with the enterprise data mastering platform.
• Collaborate with upstream and downstream stakeholders to ensure mastered data can be consumed reliably through standardized interfaces and governed data flows.
• Help establish data quality controls, reconciliation processes, metadata alignment, and stewardship workflows required to support trusted mastered data in the platform.
• Contribute to issue resolution and continuous improvement related to mastering-related ingestion and distribution workflows.
• Support data mastering capabilities aligned with platforms such as NeoXam DataHub, including: Data acquisition, Cleansing, Enrichment, Mastering, Reconciliation, Golden copy generation & Downstream distribution of trusted data products.
• Support semantic modeling practices that help create consistent business definitions, reusable data concepts, and governed consumption patterns across the enterprise data platform.
• Partner with business, architecture, data governance, and platform teams to align technical data structures with business-friendly semantic definitions.

Qualifications

• 10+ years of experience in data engineering, data platforms, cloud data architecture, software engineering, or related engineering domains.
• Demonstrated experience operating as a senior Individual Contributor, technical lead, principal engineer, staff engineer, distinguished engineer, architect, or equivalent role in complex data platform environments.
• Proven ability to provide technical guidance, influence architectural decisions, and drive engineering best practices across distributed teams.
• Ability to translate broader architectural direction into actionable technical designs, implementation patterns, and delivery plans.
• Comfortable operating in a role that blends hands-on engineering, technical strategy, architecture influence, delivery enablement, and cross-team technical mentorship.
• Strong hands-on experience with modern data engineering and pipeline development, including batch and/or streaming data workflows.
• Strong familiarity with AWS cloud-native data engineering, including services such as Amazon S3, AWS Glue, AWS Lambda, AWS Lake Formation, Amazon Kinesis, and related orchestration, security, and monitoring capabilities used to build scalable and governed data platforms.
• Working knowledge of technologies and concepts such as Delta Lake, Apache Iceberg, Databricks Unity Catalog, metadata-driven pipelines, data catalogs, and governed lakehouse architectures.
• Solid understanding of data quality, schema management, lineage, metadata, access control, encryption, observability, and operational support for production data pipelines.
• Familiarity with engineering best practices such as version control, automated testing, CI/CD, infrastructure automation, release management, and monitoring.
• Practical experience using AI-assisted engineering tools such as Claude, GitHub CoPilot, LLMs, or similar AI-enabled development tools in software engineering or data engineering workflows.
• Demonstrated understanding of context engineering, including how to structure technical requirements, architectural constraints, data schemas, data contracts, examples, coding standards, logs, and acceptance criteria to improve AI-generated outputs.
• Ability to create reusable context packs, prompt templates, engineering playbooks, and documentation patterns that improve consistency and quality of AI-assisted engineering work.
• Ability to critically evaluate AI-generated code, recommendations, and documentation for correctness, security, performance, maintainability, and alignment with enterprise standards.
• Strong communication and stakeholder engagement skills, with the ability to work effectively across offshore and global teams.
• Ability to influence technical direction without relying on formal people-management authority.
• Knowledge of semantic modeling is a strong plus, particularly for designing consistent business definitions, improving data discoverability, supporting governed data consumption, and aligning business terminology with technical data products.
• Databricks certifications, AWS certifications such as AWS Certified Data Engineer or AWS Certified Solutions Architect, or equivalent practical platform experience are a plus.

Nice to have

• Familiarity with enterprise data cataloging, metadata management, business glossary, lineage, operational metadata platforms, or semantic data layers.
• Knowledge of NeoXam DataHub is a strong plus, particularly for firms using it to centralize financial data, manage the data lifecycle from acquisition to distribution, enable automated workflows, and deliver a single source of truth for mastered datasets.

Benefits

• Health & Wellness: Health care coverage designed for the mind and body.
• Flexible Downtime: Generous time off helps keep you energized for your time on.
• Continuous Learning: Access a wealth of resources to grow your career and learn valuable new skills.
• Invest in Your Future: Secure your financial future through competitive pay, retirement planning, a continuing education program with a company-matched student loan contribution, and financial wellness programs.
• Family Friendly Perks: It’s not just about you. S&P Global has perks for your partners and little ones, too, with some best-in class benefits for families.
• Beyond the Basics: From retail discounts to referral incentive awards—small perks can make a big difference.
• ---------------------------------------------------------
• ---------------------------------------------------------