Web Scraping Engineer

Senior Python Scraping Engineer - Permanent

Location: Remote

A technology company is seeking a Senior Python Scraping Engineer to design, build, and operate large-scale, resilient web-scraping systems. The role focuses on solving complex scraping challenges across dynamic, JavaScript-heavy environments and platforms with strong anti-automation defences. This position sits at the intersection of data engineering, reverse engineering, and reliability engineering.

What You'll Do

• Design and maintain large-scale Python scraping systems that operate reliably against highly defended websites.
• Build scrapers for dynamic, JavaScript-heavy pages using browser automation and hybrid techniques.
• Continuously adapt to changes in structure, request flows, and anti-automation mechanisms.
• Engineer robust extraction pipelines focused on accuracy, consistency, and observability.
• Implement Proxy, fingerprinting, and request-routing strategies to increase stability and throughput.
• Monitor scrape health, detect failures early, and quickly resolve issues in production.
• Optimise performance, cost, and latency across scraping infrastructure.
• Collaborate with data engineering, data science, and product teams to ensure scraped data is reliable and usable.
• Maintain clear documentation and operational runbooks for long-term maintainability.

Required Skills and Experience

• Strong professional experience with Python, including production-grade scraping systems.
• Deep hands-on experience scraping Google or similarly protected platforms.
• Solid understanding of HTTP, TLS, cookies, headers, redirects, and browser-network behaviour.
• Experience with tools such as Playwright, Selenium, Puppeteer, or similar browser-automation frameworks.
• Strong knowledge of HTML parsing, DOM traversal, and data-extraction techniques.
• Experience handling rate limiting, CAPTCHAs, IP rotation, and bot-detection systems.
• Comfortable working with asynchronous and concurrent scraping architectures.
• Experience running scrapers at scale in cloud environments.
• Strong debugging skills and the ability to reason about complex failure modes.
• Experience with headless browser fingerprinting and anti-bot evasion techniques.

Nice to Have

• Familiarity with containerisation and orchestration technologies such as Docker and Kubernetes.
• Experience with distributed task systems and job queues.
• Background in data-quality monitoring and anomaly detection.
• Experience working with search, advertising, or competitive-intelligence data.

What We Value

• Pragmatic problem-solving in hostile and fast-changing technical environments.
• High ownership of systems that run continuously and at scale.
• A strong bias toward reliability, correctness, and observability.
• Ability to explain complex technical behaviour to non-specialists.
• Curiosity about how large platforms evolve their defences and infrastructure.

Please send me a copy of your CV if you're interested and meet the above