Your Ethereum RPC load balancer is serving stale data

Ethereum RPC is JSON over HTTP, so it is tempting to treat it like any other HTTP service: put nodes behind a load balancer, health check the port, spread the load.

That setup has three failure modes, and the dangerous thing about all of them is that they return 200 OK.

Failure 1: the node cannot answer, but tries anyway

Not every node can serve every method.

Tier matters: minimal, full and archive nodes hold different amounts of history. Ask an archive-only question of a pruned node and you get an error at best.

Client matters more than people expect. erigon_* and ots_* methods exist only on Erigon. txpool_status and txpool_inspect work on Geth and Reth, while Erigon supports only txpool_content. A round-robin balancer will send each of these somewhere arbitrary, and your caller sees an intermittent failure that looks like a node fault rather than a routing fault — which is a genuinely miserable thing to debug, because retrying sometimes works.

The fix is to make the router protocol-aware: classify every method by the capability it needs, probe each node with web3_clientVersion to learn what it is, and route accordingly. Capability mismatches become structurally impossible rather than statistically unlikely.

Failure 2: the answer is stale and looks fine

This is the one that actually hurts.

A node three blocks behind the chain head will answer eth_blockNumber or eth_getBalance immediately, with a well-formed response, containing data that is simply out of date. There is nothing at the HTTP layer that distinguishes it from a correct response. Your health check passes, because the node is healthy — it is just behind.

Routing live queries to the node with the highest block height solves the common case. But that conflicts with cost: archive nodes are the most expensive hardware in the fleet, and you do not want cheap queries landing on them just because they happen to be a block ahead.

The resolution is tier-preferred routing with a canonical fallback: light methods prefer minimal nodes, and escalate to the highest-block node only when the preferred tier drifts more than a couple of blocks behind. Cost optimisation never silently costs correctness, and the threshold is a number you can tune rather than an argument you keep having.

Failure 3: re-orgs

Chain re-organisations mean a block your node reported can stop being canonical. Serve it afterwards and you are returning data from a fork that no longer exists.

Detecting this requires tracking block hashes across nodes rather than trusting heights alone — two nodes at the same height can disagree about which block that is, and that disagreement is the signal.

Stateful methods break silently too

eth_newFilter returns an ID that only means something on the node that created it. Poll that filter through a load balancer that has moved you to a different node, and you get an empty result rather than an error. Subscriptions have the same problem.

Filter and subscription sessions need affinity to their originating node. And if your clients speak WebSocket, forward JSON-RPC to nodes over WebSocket rather than translating it into HTTP requests — the translation adds a round trip per call for exactly the workloads that are most latency-sensitive.

Make the routing observable, or you will not trust it

Every routing rule above is a decision you will eventually need to defend. Export:

• routing decisions, labelled by method
• tier fallbacks, labelled by source tier, target tier and reason
• re-org detections, as a counter

One warning from experience: keep the metric method allowlist current. An early version of ours bucketed unrecognised methods into unknown_*, and after a traffic-mix change a meaningful share of requests vanished into that bucket. A dashboard that has quietly stopped describing your traffic is worse than no dashboard, because you are still trusting it.

The general lesson

Every failure here comes from the same root cause: the balancer can see TCP connections, and the correctness of an Ethereum RPC response is a property of the chain state, the node's client implementation, and its sync position. None of that is visible at layer 4.

If your load balancer cannot see the protocol, it cannot protect you from the protocol.