Overview

Data Engineer Ratings Engine & Billing Transformation

Responsibilities

• We are seeking an experienced Data Engineer to join a large-scale Billing Transformation programme. The engineer will be responsible for sourcing telemetry data, building scalable data pipelines, supporting rating logic, and preparing billing-ready outputs for SAP invoicing.
• This role requires strong hands-on experience in AWS Glue, Snowflake, Python, PySpark, SQL, and CI/CD practices. The candidate will also be responsible for implementing data quality checks, reconciliation controls, audit trails, exception handling, and operational monitoring to ensure billing accuracy.
• This is a critical engineering role supporting the transformation of raw telemetry usage data into rated, validated, and invoice-ready billing data.
• Build and maintain data ingestion pipelines from telemetry and usage data sources.
• Develop scalable ETL/ELT pipelines using AWS Glue, PySpark, Python, and SQL.
• Support both batch and event-driven data ingestion patterns.
• Ensure data pipelines are reliable, performant, scalable, and cost-efficient.
• Create and maintain curated usage datasets in Snowflake.
• Design efficient data models, tables, views, and processing layers for downstream billing use cases.
• Optimize SQL queries, warehouse usage, partitioning strategies, and performance.
• Support analytics, validation, and reconciliation use cases on Snowflake.
• Support implementation of rating logic for usage-based billing.
• Prepare billing-ready datasets for invoice processing.
• Generate data outputs and feeds required for SAP invoicing.
• Work on pricing, usage calculation, billing rules, and financial data transformation logic.
• Ensure raw telemetry data is transformed into accurate, validated, and invoice-ready records.
• Design and implement data quality checks across ingestion, transformation, rating, and billing layers.
• Build reconciliation controls to validate data completeness, accuracy, and consistency.
• Implement audit trails for billing data movement and transformation.
• Create exception-handling and error-management processes for failed or inconsistent records.
• Support root cause analysis for billing discrepancies and data issues.
• Build and support CI/CD pipelines for data engineering deployments.
• Work with tools such as GitHub Actions, GitLab CI, Jenkins, AWS CodePipeline, or similar.
• Implement operational monitoring, alerts, dashboards, and pipeline health checks.
• Maintain technical documentation, runbooks, and support procedures.
• Collaborate with engineering, finance, billing, and operations teams to ensure smooth delivery.

Qualifications

• Budget: 22 LPA
• Contract Duration: 12 months
• Strong hands-on experience in data engineering, ETL/ELT development, and large-scale data pipeline design.
• AWS Glue
• PySpark
• Python
• SQL
• Cloud-based data processing
• Strong experience working with Snowflake as a data warehouse or enterprise data platform.
• Advanced SQL skills for:
• Data analysis
• Validation
• Troubleshooting
• Query optimization
• Performance tuning
• Understanding of:
• Data modelling
• Partitioning
• Performance tuning
• Cost optimization
• Scalable pipeline design
• Valuable understanding of one or more of the following:
• Usage-based billing
• Rating engines
• Invoice feeds
• SAP invoice integration
• Financial data processing
• Billing reconciliation
• Data Quality & Governance
• Data quality checks