Overview

• Role: Data Engineer

Responsibilities

• Data Architecture and Design:
• Design scalable, reliable, and secure data solutions that meet real business needs
• Develop data models, data flows, and integration architectures from the ground up
• Own the full lifecycle from requirements gathering to production deployment
• Ensure solutions meet business requirements and align with best practices.
• Data Ingestion and Integration
• Build and manage data pipelines to ingest data from various sources, including third-party and RESTful APIs.
• Build ETL/ELT processes using ADF, Fabric Data Flows, and Notebooks.
• Embed MDM solutions into pipelines, ensuring seamless data sync across systems.
• Data Storage and Management
• Design and implement data storage solutions using Microsoft Fabric, Azure SQL Database, and Azure Delta Lake.
• Define match, merge, and survivorship rules for master data management and ensure platform integration with other systems.
• Data Transformation and Processing
• Use Python, PySpark, and Fabric Notebooks to process and transform data at scale.
• Implement transformation logic to prepare data for analysis and reporting.
• Optimize data processing workflows for efficiency and scalability.
• Analytics and Reporting
• Design and develop SSAS OLAP cubes, tabular & semantic models to support complex analytical queries and reporting.
• Define dimensions, hierarchies, and measures; write MDX and DAX queries to retrieve and manipulate data.
• Integrate models with reporting tools such as Power BI, Excel, etc. to deliver dashboards and insights.
• Data Quality
• Implement data quality checks, validation processes, and monitoring to ensure accuracy and consistency.
• Maintain compliance with data governance policies across the full data lifecycle.
• Monitoring and Optimization
• Monitor data solutions for performance, reliability, and scalability; troubleshoot and resolve issues promptly.
• Optimize query and price performance through indexing and partitioning strategies.
• Security
• Implement role-based access control, auditing, and encryption to protect data assets.
• Collaboration and Communication
• Work closely with analysts and business stakeholders to understand data requirements.

Qualifications

• Python + SQL: Strong Python and SQL proficiency, including advanced queries and query optimization.
• Big Data: Apache Spark / PySpark for large-scale distributed data processing.
• Data Pipelines: Building ETL/ELT pipelines using Azure Data Factory, Fabric Data Flows, and Fabric Notebooks.
• Cloud + Warehouse: Hands-on experience with Microsoft Azure services and Microsoft Fabric for cloud data warehousing and lakehouse architecture.
• Demonstrated experience analyzing complex business and technical needs to develop data architecture and solutions.
• Excellent communication skills to articulate complex data concepts to diverse stakeholders.
• Strategic thinker with strong problem-solving and analytical skills.
• Data security and privacy compliance: role-based access control, identity management, auditing, encryption, and API security best practices.
• Data quality management, monitoring, and full data lifecycle management.
• Familiarity with SSAS for OLAP cube and tabular model development.
• Minimum 5 years of experience in database development, data architecture, or data engineering.
• Bachelor's or master's degree in computer science, Data Science, Information Technology, or related field (or equivalent experience).
• Certifications in Azure, Microsoft Fabric, or related areas are desirable.
• Project management experience is a plus.
• Power BI for dashboard development and self-service reporting.
• Data modeling best practices and advanced performance tuning techniques.
• API management tools such as Azure API Management, Postman, or Swagger.
• Master data management using Profisee software.
• Data estate governance using Microsoft Purview for data quality monitoring.
• Containerization and orchestration tools for deployment.
• Familiarity with accounting principles and the financial services industry.
• Basic AI/ML concepts, including experience with training, validation, and testing datasets.