Processing Tens of Millions of Records a Day With a Three-Person Team: the Pipeline Behind SkyCofl
“We don’t have a data team” is usually the point where a founder decides a real-time data platform is out of reach. It sounds like something only a company with a dedicated engineering org can operate: constant ingestion, a database that doesn’t fall over under write load, a pricing or matching layer that has to run in seconds, not overnight batch jobs.
We build and operate exactly that kind of platform at Coflnet, and we do it with a team of three. The clearest example is SkyCofl, our data platform for Hypixel SkyBlock, a Minecraft marketplace. It’s a gaming context, but the engineering problem is not gaming-specific: continuously ingest a high-volume marketplace feed, price and match it in real time, and keep serving it to a live API and a desktop client without downtime. That’s the same shape of problem an SME faces with price feeds, IoT telemetry, or any other high-volume data source it needs turned into something usable.
This post walks through the architecture, what actually broke as the volume grew, and the cost decisions that keep a platform like this operable by a small team instead of a large one.
The problem
A marketplace feed doesn’t arrive as a tidy batch file. It arrives continuously, as thousands of small events, and every one of them can change what a user should be shown next. SkyCofl tracks over 800 million historical auctions and more than 550 million bazaar price points, and both numbers keep growing in real time as the game’s auction house and bazaar update. Ingesting, deduplicating, pricing, and indexing that volume of events every day adds up to tens of millions of records moving through the pipeline daily, from raw event to a value a player can act on.
The naive approach — poll the source, write to a database, query the database when someone asks — falls apart at this scale in two places. First, writes: a relational database happily handling thousands of rows a day starts to choke once you’re inserting and updating at real-time marketplace speed. Second, reads: any query that has to scan the full history to answer “is this item worth buying” has to run in milliseconds, not seconds, or the result is useless by the time it comes back.
The approach
The pipeline is built around a straightforward shape that generalizes well beyond gaming:
- Ingestion pulls the raw feed continuously and pushes it onto Kafka topics — the buffer that decouples “data is arriving” from “data is being processed,” so a slow downstream step never blocks ingestion.
- Processors consume those topics: one service finds newly profitable items in the auction data, another tracks bazaar buy/sell pressure, another deduplicates records that arrived twice from overlapping polls (a dedicated Kafka deduplication service consumes a source topic and forwards clean, deduplicated records to a target topic, with backlog recovery for anything up to 24 hours old).
- Storage is tiered by how the data is actually used. Recent, frequently-queried data lives in ScyllaDB/Cassandra for fast reads and writes; Redis holds hot, short-lived state; older data moves to cheaper object storage (S3-compatible, Cloudflare R2) once it’s mostly queried for historical lookups rather than live decisions, with a bloom filter so the system can answer “does this record exist in the archive” in constant time without a full storage read.
- The API layer (SkyApi) serves all of that back out with rate limiting tuned separately for anonymous, logged-in, and premium clients, so one heavy caller can’t degrade the experience for everyone else.
- The frontend and in-game mod are the last mile — the same pipeline feeds both a web UI and a Minecraft client mod, so the underlying data model has to serve two very different consumption patterns from one source of truth.
None of these pieces is exotic on its own. What makes it work at volume is that each stage is sized and optimized for exactly the job it does, instead of one general-purpose database trying to be good at everything.
What broke, and what we changed
Two changes stand out because they were forced by scale, not planned upfront.
The first was in the filter engine that matches marketplace listings against user-defined criteria. The original implementation expressed every filter as a .Where() query against an abstract data source (IQueryable), which is elegant and lets the same filter logic run against a database or an in-memory list. It worked fine until the rest of the pipeline got faster and the filter step became the bottleneck — the IQueryable abstraction adds real overhead, and some users had built filter lists with 300+ conditions instead of the expected handful. The fix was to compile filters into .NET expression trees instead: the same filter definition can now be translated into a database query when scanning history and into fast compiled code when matching a single new record in real time, without duplicating the logic. It’s a good example of a decision that looks like premature optimization until the volume proves it isn’t.
The second was storage economics: keeping every historical record on the fast, expensive tier doesn’t scale financially. The current design keeps roughly the most recent three months of data on ScyllaDB, where reads and writes need to be fast, and archives everything older to object storage, where it’s cheaper to hold indefinitely and still fully queryable when someone needs it.
Cost discipline
A three-person team can’t operate a platform this size by throwing infrastructure at every scaling problem — the cost has to stay proportionate to the company. That shapes every decision above: cheap archival storage instead of one expensive database tier for everything, a bloom filter instead of a full lookup for existence checks, and Kubernetes clusters on European providers (Hetzner and OVHcloud) that autoscale instead of running for peak capacity around the clock. The same discipline shows up in ANE, a marketplace deal-finding platform we built that grew directly out of the pattern-matching logic SkyCofl uses to spot underpriced auctions.
Outlook
Nothing about this architecture depends on the data being about a video game. A price feed from a set of suppliers, sensor readings from a fleet of devices, listings from a marketplace you don’t control — the pipeline shape is the same: ingest onto a message queue, process in dedicated stages, store on tiers that match how the data is actually read, and serve it through an API sized for real usage patterns. The part that’s hard to justify hiring for internally, on an SME budget, is exactly the part that’s cheapest to have built and operated for you.
FAQ
Do I need a full data engineering team to process millions of records a day? No. The architecture that makes this manageable — Kafka for ingestion, tiered storage, a purpose-built API layer — is designed to be operated by a small team once it’s built correctly. The initial setup is the hard part, not the ongoing operation.
What’s the difference between batch processing and real-time data processing? Batch processing collects data and processes it on a schedule (hourly, nightly). Real-time processing acts on each event as it arrives, which is necessary whenever a stale result — a wrong price, a missed alert — has real cost. SkyCofl and ANE are both real-time by necessity.
Is Kafka overkill for a smaller data volume? Kafka’s main value isn’t raw throughput, it’s decoupling ingestion from processing so a slow or failing downstream step doesn’t stop data collection. That’s useful well before you hit “millions of records a day” — it’s a question of reliability requirements, not just volume.
How do you keep infrastructure costs down at this scale? Mainly through tiered storage (fast, expensive storage only for data that’s actually queried live) and autoscaling Kubernetes rather than fixed capacity sized for peak load. Both are architectural decisions made early, not cost-cutting retrofitted later.
If your business generates more data than your team has time to turn into something usable, a one-hour consultation is a practical way to find out what a pipeline built for your actual volume and budget would look like.
Next step: a one-hour consultation for €150
60 minutes with one of our managing directors about your project – our regular hourly rate, credited against a follow-up project. For businesses and founders.