Browse Jobs
Explore open roles across India.Explore active software roles. Sign up to apply and get AI match scores.
Top MatchesTop Job Matches
Loading jobs…
Data Engineer: Big Data-AWS
IBM
Pune•Hybrid•1-3 Yrs•2 hours ago
Job description
Hybrid
Overview
Introduction In this role, you'll work in our IBM Client Innovation Center (CIC), where we deliver deep technical and industry expertise to a wide range of public and private sector clients around the world.
Responsibilities
- Design, build, optimize and support new and existing data models and ETL processes based on our clients business requirements.
- Build, deploy and manage data infrastructure that can adequately handle the needs of a rapidly growing data driven organization.
- Coordinate data access and security to enable data scientists and analysts to easily access to data whenever they need too.
- Developed the Pysprk code for AWS Glue jobs and for EMR.. Worked on scalable distributed data system using Hadoop ecosystem in AWS EMR, MapR distribution..
- Developed Python and pyspark programs for data analysis.. Good working experience with python to develop Custom Framework for generating of rules (just like rules engine).
- Developed Hadoop streaming Jobs using python for integrating python API supported applications..
- Developed Python code to gather the data from HBase and designs the solution to implement using Pyspark. Apache Spark DataFrames/RDD's were used to apply business transformations and utilized Hive Context objects to perform read/write operations..
- Re - write some Hive queries to Spark SQL to reduce the overall batch time
- Understanding of Devops.