Data platforms & pipelines
The same architecture that processes tens of millions of marketplace records a day for our own products — built and operated for you.
What we build
Five building blocks we combine into your pipeline.
Web scraping & data extraction at scale
Structured data pulled reliably out of websites and marketplaces, even when the source keeps changing.
Price & market monitoring with alerts
Track competitor prices and market listings continuously, and get notified the moment something moves.
Real-time pipelines
Kafka → Cassandra/Scylla → API/dashboards, built to keep up with high-volume, high-frequency data.
Dashboards & reporting
Turn raw pipeline output into dashboards and reports the rest of the business can actually use.
How we work
We start with one data source rather than trying to boil the ocean, and keep your existing interfaces working while we build around them. Once it's live, we operate it: monitoring, alerting and ongoing updates, on EU hosting, following data-minimisation practice under GDPR. We're not lawyers — for anything legally contentious around a specific data source, we'll point you to one rather than guess.
To scope the first version, we clarify source access, update frequency, retention and the system that will consume the data. These choices affect both development effort and ongoing operating costs; our pricing and engagement models explain how we agree them. If audio is one of your sources, dialect speech recognition can turn recordings into text for the pipeline.
Proof it works at scale
Real numbers from our own production systems.
- • SkyCofl, our own product, has tracked over 800 million auctions.
- • ANE collects listings from supported marketplace sources for product search, price references and alerts. Coverage depends on source access.
- • Read the case study on replacing a legacy scraping system at scale, without downtime.
Related reading
- Custom Price Monitoring at Marketplace Scale: What We Learned Building ANE →
Off-the-shelf price monitors assume a fixed catalog. Here's how we built custom crawling, classification, and alerting for open marketplace data.
- From machine to dashboard: turning CAN bus and machine data into something your business can act on →
How machine data nobody looks at becomes a dashboard with alerts — architecture, pitfalls, and where our integration stops: at the library on the device, not at the bus.
- Processing Tens of Millions of Records a Day With a Three-Person Team: the Pipeline Behind SkyCofl →
How a three-person team built and operates the Kafka and Cassandra pipeline that ingests and prices tens of millions of marketplace records a day.
- Case study: replacing a legacy web-scraping system at scale — without downtime →
How Coflnet replaced a client's unreliable legacy web scraper with a faster, more robust scraping system at scale, migrating without downtime.
One-hour consultation for €150
60 minutes with one of our managing directors about your project. Within two working days you receive a written summary with a recommended approach and a rough cost and time range.
€150+ VAT
For businesses and founders.
- Request a slot
- Short written confirmation by email
- Invoice after the call
Frequently asked questions
Is scraping legal?
It depends on the source, what you extract, and how you use it — there are general considerations around terms of service, robots.txt, rate limits and personal data. We assess this per case together with you; this isn't legal advice, and for anything contentious we'd point you to a lawyer.
How much data can you handle?
Our own products already run pipelines processing tens of millions of records a day, and the same architecture scales down cleanly for smaller data sources. Exact throughput depends on your source and infrastructure, which is part of what we scope together.
Do we own the data and code?
Your data stays your data. For what is built specifically for you, you normally receive broad usage rights — the exact terms, such as exclusivity and reused building blocks, are set out in the project contract.
Can it feed our BI/Excel?
Yes. Pipeline output can land wherever you need it — a database your BI tool reads from, a scheduled export, or an API your spreadsheets pull from.