When the exchange API doesn't exist: reaching for a headless browser on purpose

Puppeteer is the wrong default for almost everything. It's slow relative to a direct HTTP request, expensive in memory and CPU, and fragile in ways a typed API response never is — a page redesign breaks a selector in a way a JSON schema change never breaks a field name. So the interesting engineering decision in a multi-exchange tracker isn't "use Puppeteer," it's "use it for exactly three exchanges, and nothing else."

Six exchanges, one polling strategy; three exchanges, another

Of the nine centralized exchanges this service tracks, six expose trading pairs and announcements through APIs that behave the way APIs are supposed to: authenticate, request, get structured JSON back. Those six get polled directly every 15 seconds, because there's no reason not to — the cost of a fast poll against a real API is low.

The other three — MEXC, BingX, Bitfinex — sit behind Cloudflare protections that block direct HTTP requests regardless of what headers or retry strategy you throw at them. For those three, and only those three, a separate microservice runs puppeteer-extra with the stealth plugin, polled on a much slower 5-minute cadence, with a Redis cache in front of it so repeated requests within that window don't each spin up a fresh browser context.

The two-tier cadence is the actual decision

It would be simpler to run everything through the browser-based path and accept one polling interval everywhere. It would also mean the six exchanges with perfectly good APIs get treated as if they didn't have one — polled as slowly and as expensively as the three that genuinely require it. The two-tier design is a direct trade-off made visible in the code instead of hidden in an average: fast and cheap where an API allows it, slow and deliberately resource-bounded only where nothing else is available.

What the fallback path still has to get right

Running a stealth-plugin browser doesn't mean the problem is solved once and done. Two smaller decisions mattered as much as the initial choice to use Puppeteer at all:

Cache before you scrape again. A 5-minute TTL on scraped results means a burst of internal requests for the same exchange's data doesn't translate into a burst of browser launches. The cache exists specifically to keep the expensive path expensive only as often as it needs to be.

Log failure as failure, not as an empty success. An earlier version logged "announcements scraped for X with count: 0" whenever the scraper actually failed to load the page — which reads, in a log stream, exactly like a legitimate empty result. It's a small distinction, but it's the difference between an alert firing when scraping breaks and a silent gap in coverage that nobody notices until a listing was missed.

The general principle: reach for a headless browser when an API genuinely doesn't exist, size the cost you're willing to pay for that path deliberately, and never let its failure mode look identical to a legitimate empty response.