Context
A mixed fleet of Ethereum nodes — minimal, full and archive, running a mix of Erigon, Geth and Reth — sat behind a conventional load balancer. It worked, in the sense that requests got answers. It was wrong in ways that only surface under load or during a re-org.
The problem
Three distinct failure modes, all invisible to a balancer that only counts connections:
Capability mismatches. Methods like erigon_* and ots_* exist only on
Erigon. txpool_status and txpool_inspect work on Geth and Reth, while Erigon
only supports txpool_content. A round-robin balancer sends them anywhere, and
the caller sees an error that looks like a node fault rather than a routing fault.
Cost inversion. Archive nodes are the most expensive hardware in the fleet, and were absorbing cheap queries that a minimal node could have answered — while genuinely archive-only requests queued behind them.
Silent incorrectness. A node a few blocks behind returns a well-formed, entirely stale answer. During a re-org, a node can return a block from a fork that has since been abandoned. Neither is distinguishable from a correct response at the HTTP layer.
What we built
A Node Weighter middleware behind HAProxy that makes routing a function of the request:
- Method classification by cost and required capability, routed to the cheapest tier that can answer correctly.
- Automatic client detection via
web3_clientVersion, so capability differences are handled by the router rather than by every caller. - Canonical fallback — when the preferred tier drifts more than two blocks behind, the request escalates to the highest-block node. Cost optimisation never silently costs correctness.
- Re-org detection by tracking block hashes across nodes.
- Session affinity for filters and subscriptions, so stateful RPC survives.
- Native WebSocket forwarding, removing HTTP overhead for subscription-heavy clients.
Challenges
The hardest part was not the routing logic — it was observability of the routing
logic. An early version exported metrics with an incomplete method allowlist, so
a meaningful share of traffic aggregated into unknown_* buckets. A dashboard
that has quietly stopped describing your traffic is worse than no dashboard,
because it is trusted. Expanding the allowlist and adding a method label to the
routing-decision counter made tier behaviour legible.
The second was tuning fallback aggressiveness. Escalate too eagerly and archive nodes are back to serving everything; too reluctantly and you serve stale data. A two-block threshold, with fallbacks counted by source tier, target tier and reason, made the trade-off a tunable number rather than an argument.
Outcome
Routing is now explainable per request. Cheap methods stay on cheap nodes, capability mismatches are structurally impossible rather than merely unlikely, and fork-derived responses are caught before they reach a caller. Tier fallbacks are a metric, so drift in the traffic mix is visible well before it becomes an incident — see Ethereum RPC Router.
Lessons learned
Protocol awareness beats connection counting. Every problem here came from a balancer that could see TCP but not JSON-RPC.
Correctness needs a metric, not just a code path. The re-org and fallback logic only became trustworthy once we could count how often each fired.
Instrument the allowlist, not just the happy path. Metrics that silently bucket unknown values will hide exactly the change you needed to see.