Case study: replacing a legacy web-scraping system at scale — without downtime

In 2023, Coflnet took on a project for an advertising company from northern Germany with around 10 employees. The company was already running a web scraping system it had purchased years earlier, and it had collected millions of articles from various websites in that time. Over time, though, the old system reached its limits: it became unreliable, inefficient, and was no longer able to reliably scrape the pages that mattered most to the business.

The company needed a web scraping service that could specifically target the relevant websites, keep up with layout changes, and hold up at the volume the business had grown into. This is where Coflnet came in — we had already built a different system for this client with good results, so taking on the scraping replacement was a natural next step.

The problem

The central problem wasn’t building a scraper from scratch — it was how to replace a legacy web scraper that the company depended on daily, without losing what it had already built up. The purchased system was overwhelmed: important information was too often captured incorrectly, which led to inaccurate data and, downstream, wrong decisions based on that data. At the same time, several million articles had already been collected with the old system, so it couldn’t simply be switched off and rebuilt from zero.

The challenge, in short, was to migrate the scraping system without downtime while developing a new, optimized engine focused on the websites that actually mattered to the advertising company.

The approach

Coflnet went into the project with two clear objectives: keep the existing data reliable and accessible, and build a new scraping system that was faster, more robust, and better tailored to the client’s needs.

1. A new scraping engine. We built a completely new scraping engine designed for the dynamic, constantly changing structure of real websites, and specifically tuned to the pages that mattered most to the advertising company.
   • The scrapers were built to be more adaptable, handling layout changes and popups without breaking.
   • The system was optimized for speed, so new content could be scraped and saved quickly — the most relevant articles for an advertising business are usually the most recently published ones, so freshness mattered as much as coverage.
2. Compatibility with the old system. The data generated by the new engine had to work with everything already built around the old one. So the new system feeds the same downstream interface that the old scraping system used, meaning the target system and everything consuming that data kept working unchanged.

What “scraping at scale” actually requires

This project is a useful example of a pattern that comes up whenever scraping needs to move from “a script that works” to a data extraction service for businesses that has to run continuously and be trusted. A few things tend to matter regardless of the specific target sites:

• Rate limiting. Requests need to be paced per domain rather than fired as fast as possible — both to avoid overloading the source and to avoid triggering blocks that take the whole pipeline down.
• Respecting robots.txt and terms of service. Scraping at scale means treating each source’s published rules as a constraint on the crawler’s design, not an afterthought (this is general practice, not legal advice — a specific legal review is the right call before scraping any particular site at volume).
• Deduplication. The same article or listing often gets seen more than once, from re-crawls or from multiple entry points. Without a dedup step, storage and downstream processing costs grow for no benefit.
• Monitoring. A scraper that silently starts returning wrong or empty data is worse than one that fails loudly — the old system in this case had exactly that problem, which is part of why it had become unreliable. Alerting on drops in yield or changes in page structure catches breakage before it reaches the data.
• Storage that scales. Millions of articles need a storage layer that keeps writing and querying fast as the dataset grows, not one that degrades as volume increases.

None of this is specific to advertising or media monitoring — it’s the same shape of problem for anyone extracting data from sources they don’t control, at a volume where manual checking isn’t an option.

Results

The migration to the new scraping system was a complete success. The advertising company now has a fast, reliable, and highly targeted web scraping service focused on the sites that matter most to them.

• Improved reliability. The new system is significantly more accurate in the information it collects, meaning fewer losses and better data quality overall.
• Faster scraping. The optimized scrapers collect and process information faster, and the most important pages are now scraped at a shorter interval than before.
• Data continuity. Thanks to the careful migration process, downtime was avoided entirely — the advertising company switched to the new system without any interruption to data availability.

Outlook

The new system continues to support the advertising company’s growth. With a scalable database and a flexible scraping engine, the company is well-equipped to process more data, expand into new markets, and adapt as the sites it depends on keep changing.

This project is a good illustration of why replacing a legacy scraper isn’t just a technical rebuild — it’s a migration problem first, and a scraping problem second. Getting scraping at scale right means the new system has to earn its place gradually, proving itself against the old one before anything gets switched off.

FAQ

How do you migrate a scraping system without downtime? By keeping the old system running while the new one is built and validated, and by making sure the new system writes to the same downstream interface the old one used. That way everything consuming the data — dashboards, databases, other services — keeps working unchanged during and after the cutover, and the switch can happen once the new system has proven itself rather than on a fixed deadline.

What makes a legacy web scraper become unreliable over time? Websites change their layout, add anti-bot measures like popups and interstitials, and shift how they structure content — a scraper built for how a site looked years ago gradually falls behind. Without ongoing maintenance and monitoring, accuracy degrades quietly until the data it produces can’t be trusted.

When does it make sense to replace a scraping system instead of patching it? When the failure pattern is structural rather than occasional — when a system is regularly missing or misreading data rather than failing on rare edge cases. At that point, incremental fixes tend to cost more over time than building a new engine designed for the sites you actually need to cover, migrated carefully against the old one.

If your business depends on data extraction from sources you don’t control and the system behind it is starting to show its age, a one-hour consultation is a practical way to scope out what a replacement or a new pipeline would look like — and our data platforms page has more on how we build and operate the infrastructure behind it.