Kagool
Create a free account
One signup unlocks everything for this job:
- Apply to this job and track your application
- See your personalized profile match score
- Generate an AI cover letter for this role
Enterprise Software & Network Solutions · 201 to 500 Employees · Birmingham, United Kingdom
Hiring: Engineering
What employees say
+ Literally none. Awful leaders. No hold on their finance and takes them months to pay
− I honestly would run away. It’s just filled with people who went to college together and refuse to pay you
More reviews on the company profile.
Interview experience · Data Engineer
First HR will call you after screening Next round with tech lead/architect for 30-45 mins The interviewers was soft spoken, respectful and trying to understand your hands on live experience with tech stack and projects I
India · 6 reports
More interview insightsJob description
Overview
We are a fast-growing IT consultancy specializing in the transformation of complex Global enterprises that use SAP.
Responsibilities
- Design, develop, and maintain scalable batch and real-time data pipelines using Apache Spark and PySpark.
- Develop and support Kafka-based real-time streaming pipelines on Microsoft Azure.
- Implement real-time data processing using Spark Structured Streaming.
- Work with Kafka topics, partitions, consumer groups, offsets, retention, and message delivery mechanisms.
- Implement Kafka producers and consumers for high-volume data ingestion and processing.
- Integrate Apache Kafka with Azure data services and downstream data platforms.
- Develop data transformation, cleansing, aggregation, and enrichment processes.
- Implement checkpointing, fault tolerance, error handling, retry mechanisms, and recovery strategies for streaming applications.
- Design and optimize Spark jobs for performance, scalability, and efficient resource utilization.
- Work with Azure data services such as Azure Databricks, ADLS Gen2, Azure Event Hubs, Azure Data Factory, and Azure Synapse Analytics.
- Develop SQL queries and work with relational and analytical databases.
- Implement data quality and validation frameworks.
- Troubleshoot production issues and perform root-cause analysis.
- Collaborate with Data Architects, Application Teams, DevOps, and Business stakeholders.
- Participate in technical design discussions, code reviews, and CI/CD activities.
- 7+ years of experience in Data Engineering / Big Data.
- Strong hands-on experience with Apache Spark and PySpark.
- Strong hands-on experience with Apache Kafka.
- Mandatory hands-on experience with Kafka on Azure.
- Mandatory Microsoft Azure experience.
- Strong experience with Spark Structured Streaming.
- Strong programming skills in Python.
- Strong SQL skills.
Qualifications
- Good understanding of distributed computing and Big Data architecture.
- Topics and partitions
- Producers and consumers
- Consumer groups
- Offsets
- Retention
- Partitioning
- Error handling and recovery
- Partitioning
- Shuffling
- Joins
- Caching
- Serialization
- Performance tuning
- Hands-on experience with Kafka:
- Strong understanding of Spark concepts including:
- Hands-on experience with Azure Databricks and/or Azure data engineering services.
- Delta Lake
- Delta Live Tables
- Kafka Connect
- Confluent Kafka / Confluent Cloud
- Schema Registry
- Avro
- Azure Event Hubs
- Kubernetes
- Terraform
- Azure DevOps
- Prometheus / Grafana
- CI/CD automation
Verify Python for this role
Free PRACTICE for Python adds a verified skill signal that can boost how roles like this rank for you — optional, not required to browse or apply.
Sign up — verify Python free →Want match scores and deeper insight? Create a free candidate profile
Job application FAQ
Last updated 5 October 2026
- How do I apply for Senior Data Engineer – Spark Streaming & Kafka at Kagool?
- Open this job on Feedback AI and sign up free (or sign in). You can apply from the job detail page; match scores and skill gaps appear when your profile is complete. Kagool may attach structured assessments to interview rounds after you apply.
- Does Feedback AI show a match score for this Senior Data Engineer – Spark Streaming & Kafka role?
- Yes after you add a resume or skills. Match score ranks this role against your profile — skills met, gaps, level, and location — so you prioritise applications where fit is strongest.
- Is Senior Data Engineer – Spark Streaming & Kafka remote or on-site in New Delhi, Delhi?
- This listing is marked on_site with location New Delhi, Delhi. Confirm details on the job description before applying.
- What skills matter for this Senior Data Engineer – Spark Streaming & Kafka role at Kagool?
- This is a senior Senior Data Engineer – Spark Streaming & Kafka role. Key skills include skills listed in the job description. Feedback AI highlights matched vs missing skills on your profile after signup.