AIMLEAP
Create a free account
One signup unlocks everything for this job:
- Apply to this job and track your application
- See your personalized profile match score
- Generate an AI cover letter for this role
Information Technology Support Services · 51 to 200 Employees · New York, US
AIMLEAP is an ISO 9001:2015 and ISO/IEC 27001:2013 certified global technology consulting and service provider offering Digital IT, AI-augmented Data Solutions, Automation, and Research & Analytics Services. AIMLEAP has been recognized as ‘The Great Place to Work®’. With focus on AI and automation-first approach, our services include end-to-end IT application management, Mobile App Development, Data Management, Data Mining Services, Web Data Scraping, Self-serving BI reporting solutions, Digital Marketing, and Analytics solutions. We started in 2012 and successfully delivered projects in IT & digital transformation, automation driven data solutions, and digital marketing for more than 750 fast-growing companies in the USA, Europe, New Zealand, Australia, Canada; and more. - An ISO 9001:2015 and ISO/IEC 27001:2013 certified - Served 750+ customers - 11+ Years of industry experience - 98% Client Retention - Great Place to Work® Certified - Global Delivery Centers in the USA, Canada, India & Australia Email: sales@fullstacktechies.com USA: 1-30235 14656 Canada: +1 4378 370 063 India: +91 810 527 1615 Australia: +61 402 576 615
Hiring: Engineering
What employees say
+ Great learning opportunities, supportive leadership, positive work culture, and excellent work-life balance. Employees are encouraged to explore new ideas, take ownership, and cont
− No major concerns to highlight.
More reviews on the company profile.
Job description
Overview
Python & JavaScript Developer – AI & Web Scraping
Responsibilities
- Design, develop, and maintain scalable web crawling and scraping systems for dynamic and JavaScript-heavy websites.
- Develop browser automation workflows using Playwright, Selenium, Puppeteer, or similar frameworks.
- Investigate and resolve crawling issues such as 403/429 responses, redirects, timeouts, rendering failures, session issues, and anti-bot challenges.
- Develop robust retry, fallback, and failure-handling mechanisms to improve crawler reliability.
- Work with cookies, sessions, browser contexts, headers, proxies, and related crawling mechanisms to maintain state and improve crawling reliability.
- Build and optimize concurrent and distributed scraping workflows using asynchronous processing, queues, and worker-based architectures.
- Design data pipelines covering URL processing, crawling, extraction, validation, transformation, and storage.
- Optimize crawler performance, including concurrency, browser lifecycle, resource utilization, timeouts, and request handling.
- Implement monitoring and observability for crawl success rates, failure types, latency, retries, and worker performance.
- Debug complex crawling problems and identify root causes rather than relying only on one-off fixes.
- Develop reusable crawling components and frameworks that can support multiple websites and use cases.
- Collaborate with data engineering, AI, backend, and product teams to deliver reliable and structured web data.
- Evaluate and adopt new web crawling, browser automation, and data extraction technologies where appropriate.
Qualifications
- Location: Remote
- Mode of Engagement: Full-time
- No of Positions: 3
- 3–5 years of hands-on experience in web scraping, web crawling, browser automation, or web data engineering.
- Strong programming experience in Python. JavaScript/Node.js is a plus.
- Strong hands-on experience with Scrapy, Playwright, Selenium, Puppeteer, or similar browser automation frameworks.
- Strong understanding of HTTP, HTML, DOM, JavaScript rendering, redirects, cookies, sessions, headers, and browser contexts.
- Good understanding of asynchronous programming, concurrency, and parallel processing.
- Understanding of retry mechanisms, error handling, idempotency, rate limiting, and backpressure in distributed systems.
- Hands-on experience with proxy management, session handling, and anti-bot challenges.
- Strong debugging and analytical skills, with the ability to investigate and resolve complex crawling failures.
- Familiarity with Docker and cloud platforms such as AWS, GCP, or Azure is a plus.
- Strong hands-on engineering mindset rather than purely project or delivery management experience.
- Ability to independently debug, investigate, and solve difficult crawling problems.
- Strong understanding of how browsers, HTTP requests, sessions, and websites interact.
- Ability to design systems that are reliable, scalable, and fault tolerant.
- Good understanding of how to distribute and coordinate large-scale scraping workloads.
- Ability to identify the root cause of crawling failures and develop generalized, reusable solutions.
- Willingness to work across scraping, backend services, distributed processing, data pipelines, and infrastructure when required.
- Strong problem-solving skills and curiosity to understand how websites behave rather than relying solely on existing scraping tools.
Education
- Industry: IT / Software Development
- Notice Period: Immediate Joiners Preferred
- About the Role
- We are looking for a hands-on Web Scraping / Crawling Engineer with 3–5 years of experience in web scraping, browser automation, and scalable data extraction.
- The ideal candidate should have strong experience working with dynamic and JavaScript-heavy websites, building reliable crawling workflows, handling crawling failures, and working with distributed processing systems.
- This is a highly technical, hands-on role. You will be expected to design, develop, debug, optimize, and maintain web crawling and data extraction systems.
- Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field is preferred.
- Equivalent practical experience in software engineering or web scraping will also be considered.
Verify JavaScript for this role
Free PRACTICE for JavaScript adds a verified skill signal that can boost how roles like this rank for you — optional, not required to browse or apply.
Sign up — verify JavaScript free →Want match scores and deeper insight? Create a free candidate profile
Job application FAQ
Last updated 21 September 2026
- How do I apply for Python Web Scraping Engineer-4-6 Years at AIMLEAP?
- Open this job on Feedback AI and sign up free (or sign in). You can apply from the job detail page; match scores and skill gaps appear when your profile is complete. AIMLEAP may attach structured assessments to interview rounds after you apply.
- Does Feedback AI show a match score for this Python Web Scraping Engineer-4-6 Years role?
- Yes after you add a resume or skills. Match score ranks this role against your profile — skills met, gaps, level, and location — so you prioritise applications where fit is strongest.
- Is Python Web Scraping Engineer-4-6 Years remote or on-site in Anywhere?
- This listing is marked remote with location Anywhere. Confirm details on the job description before applying.
- What skills matter for this Python Web Scraping Engineer-4-6 Years role at AIMLEAP?
- This is a mid-level Python Web Scraping Engineer-4-6 Years role. Key skills include Remote. Feedback AI highlights matched vs missing skills on your profile after signup.