Python Developer - Web Scraping Specialist

About NIQ for Digital Commerce

In today's fast-evolving e-commerce landscape, companies are inundated with vast volumes of data which pose challenges in extracting actionable insights and measuring their impact. NIQ for Digital Commerce addresses this by gathering over 60 billion data points daily, transforming them into cutting-edge analytics and monitoring tools to empower industry professionals with real-time market visibility. As a rapidly growing start-up in retail analytics with a dynamic and experienced team, NIQ offers substantial career growth opportunities within a supportive and autonomous environment.

Role Overview

We seek a Python Developer to join our scraping team focused on enhancing and maintaining our data collection mechanisms. This role involves designing and implementing resilient Scrapy spiders capable of adapting to website changes, conducting rigorous code reviews and data validations, and utilizing deep expertise in HTTP protocols and modern browsers to circumvent sophisticated anti-bot defenses.

Key Responsibilities

• Develop, maintain, and document durable Scrapy spiders to ensure long-term operational stability despite evolving website structures.
• Execute thorough code reviews and validate datasets to uphold stringent quality standards.
• Apply in-depth understanding of HTTP, web browser technologies, and proprietary tools to engineer advanced web crawling strategies that bypass anti-bot protections.
• Design and contribute to major enhancements across diverse codebases, provide detailed technical documentation, and deliver training sessions to relevant teams when required.

Qualifications and Experience

• Master's degree in Computer Science, Information Technology, or an allied discipline.
• Minimum of three years' professional experience in software engineering roles.
• Proficiency with web scraping frameworks such as BeautifulSoup and Scrapy.
• Solid knowledge of HTML, JavaScript, and single-page application architectures.
• Experience working with RESTful and GraphQL APIs.
• Hands-on familiarity with prominent Python web frameworks like Django and FastAPI.
• Advantageous to have expertise in Scrapy scraping framework and time series databases such as InfluxDB.
• Strong command of Docker, Git, pandas, regular expressions, Linux environments, and shell scripting (bash).
• Proven experience with major cloud platforms including AWS, GCP, or Azure.
• Highly motivated, eager to take initiative, autonomous, and passionate about technology and continuous learning.
• Comfortable tackling intellectually challenging software engineering problems.

Additional Information and Benefits

• Flexible work environment supporting work-life balance.
• Volunteering leave provisions.
• Access to online learning resources such as LinkedIn Learning.
• Employee Assistance Program (EAP) for well-being support.

NIQ integrates artificial intelligence tools in various recruitment stages (resume screening, candidate evaluation, scheduling, job matching, communications, and administrative tasks) to streamline processes and ensure fairness and uniformity based on job-relevant criteria. These AI tools operate under strict principles focusing on fairness, transparency, human oversight, and inclusion, with final hiring decisions made by human evaluators. Regular reviews are conducted to mitigate bias and comply with laws. Candidates needing accommodations or human review requests may contact local HR representatives.

About NIQ

NIQ is a global leader in consumer intelligence, providing unparalleled insights into consumer purchasing behavior to unlock growth opportunities. In 2023, NIQ merged with GfK, combining global reach and expertise in retail analytics. Operating across over 100 markets and covering above 90% of the global population, NIQ delivers comprehensive consumer insights via advanced analytic platforms, embodying a Full View™ perspective. NIQ is owned by Advent International.

Diversity, Equity, and Inclusion Commitment

NIQ is dedicated to cultivating an inclusive workplace reflecting the diverse populations served. We believe a multitude of perspectives fuels innovation and excellence. Employment decisions are made irrespective of race, color, religion, gender, sexual orientation, national origin, age, disability, genetic data, marital status, veteran status, or any other legally protected attribute. We encourage candidates who value inclusivity and equity to join and contribute meaningfully. For details, visit NIQ’s ongoing diversity initiatives page.