Context
A new trading-pair listing is a short-lived edge, and the information behind it exists across nine exchanges the moment it's published — each with a different publishing mechanism, a different rate limit, and, on three of them, active defenses against automated access.
The problem
Treating nine heterogeneous data sources as one uniform polling target doesn't scale cleanly in either direction. Poll everything fast and the three exchanges with active bot defenses fail constantly or get the service blocked outright. Poll everything at the safe, slow interval those three require and the six exchanges with clean, fast APIs get under-served for no reason.
What we built
A tiered ingestion architecture that matches infrastructure cost to what each source actually supports. Six exchanges — Binance, OKX, Bybit, Bitget, Coinbase, Deribit — expose usable REST APIs and get polled directly every 15 seconds. The three that don't — MEXC, BingX, Bitfinex — sit behind Cloudflare protections that block direct access regardless of retry strategy, so they run through a dedicated Puppeteer microservice on its own 5-minute cadence, with a Redis cache in front of it to bound how often the expensive path actually executes.
Alert delivery runs on the same architectural principle of shared infrastructure over duplicated logic: a Telegram bot and a Discord bot route through one rate limiter and one subscription-tier check, so a free-tier user gets identical gating no matter which channel they're on, and a REST API with SHA-256-hashed API keys serves programmatic access on the same backend.
Scaling the aggregation layer as the dataset grew
The architecture's harder engineering problem showed up on the database side as the tracked pair count crossed 13,000: the aggregation cycle's per-pair query pattern meant total database load scaled linearly with pairs tracked, not with pairs that actually changed. The redesign batch-loads existing pairs and base assets per exchange in bulk, then filters to only what changed before any per-pair processing begins — so cycle cost now scales with the size of the delta each poll produces, not the size of the dataset being polled. The same principle applied to notification delivery, moving subscriber fan-out from a full in-memory load to a cursor-based stream.
Outcome
The platform now runs each of nine exchanges at the fastest cadence its own API actually supports, with an aggregation layer whose cost tracks the rate of real change rather than the size of the dataset — see CEX Tracker.
Lessons learned
Design the polling tier to the source, not to the average. A single uniform interval either overloads the restrictive sources or under-serves the permissive ones; a tiered architecture lets each source run at its own sustainable rate.
Scale cost with the size of the change, not the size of the dataset. Filtering to what actually changed before doing expensive work is the same architectural move whether it's applied to a database write path or a notification fan-out — treat "what changed" as the unit of work, not "what exists."
Shared infrastructure across delivery channels keeps behavior consistent. Routing Telegram and Discord through one rate limiter and one tier check means a policy change is one code path to update, not two that can drift apart.