Context
A multi-GPU host serving several different models, fronted by conventional load balancing. Average latency looked acceptable. The p99 did not, and nobody could explain any individual slow request.
The problem
GPUs serving LLMs violate the assumptions load balancers are built on. Concurrency does not degrade them gracefully — two generations on one card split KV-cache and compute so that both become unpredictably slow. Request cost varies by two orders of magnitude, so "least connections" treats a 10-token completion and a 2,000-token generation as identical work. And capacity is bound to a loaded model, not to a host, so you cannot scale reactively when a model takes minutes to load.
The result was a system whose tail latency was effectively random, and an operations team with no way to attribute a slow request to a cause.
What we built
Admission control instead of distribution. The router decides whether a request runs now, and where, by scoring every eligible healthy worker on active requests, queue depth, latency, job cost, throughput, KV pressure and GPU utilisation. Lowest score wins.
Averages that survive restarts. Latency and throughput are exponential moving averages persisted in Redis, so a worker that has historically been slow is still known to be slow after a deploy.
Decisions you can read. Every routing decision is logged with its complete scoring breakdown.
Provisioning as code. An Ansible playbook maps each GPU to a model, port and parameters, with a systemd unit per instance so one crashed model server does not take its neighbours with it.
Challenges
The instructive bug was a race condition in a deterministic scheduler. With four idle GPUs and four requests arriving simultaneously, you would expect one per GPU. Instead all four landed on GPU 0. The scoring function is deterministic: given identical worker state it always ranks the same worker best. All four requests read state before any of them had committed its own pick, so all four independently concluded GPU 0 was optimal.
That is not a flaw in the scoring maths, and treating it as one would have led to adding randomness — which would have degraded every well-behaved case to fix a burst case. It is a race in when state is read relative to when it is written, and it belongs to be fixed there.
Outcome
The explicit goal was never peak throughput. A consistent 30 tok/s is more valuable than a spiky 45 tok/s, because everything downstream — timeouts, UI affordances, capacity planning — can be built around the former and cannot be built around the latter. Scheduling is now explainable per request, and rebuilding a node is a playbook run — see LLM Router and Multi-GPU LLM Provisioning.
Lessons learned
Predictability is a feature, and usually the one being asked for. "Make it faster" often means "stop surprising me".
Deterministic does not mean correct under concurrency. A pure function of shared mutable state still races.
Separate provisioning from scheduling. Keeping them apart is what allows a node rebuild without touching routing policy, and vice versa.