Why your load balancer is wrong about GPUs

Put a llama.cpp server behind nginx, HAProxy, or a cloud load balancer, and it will work. Requests will get answers. Average latency will look fine on the dashboard.

Then someone will complain that the assistant "sometimes takes forever", you will look at p99, and you will find that it is roughly four times p50 with no pattern you can attribute to anything. This is not a tuning problem. The load balancer is built on three assumptions, and a GPU serving an LLM violates all of them.

Assumption 1: concurrency degrades gracefully

A web backend under 2× load gets roughly 2× slower, and mostly stays predictable. This is what connection-based balancing is designed around.

A GPU does not behave this way. Two generations running on one card do not each take twice as long — they contend for KV-cache and compute in ways that make both unpredictably slow. You do not get graceful degradation, you get a cliff whose location depends on the specific pair of requests that happen to collide.

The practical consequence: admitting a second request to a busy worker can be worse than queueing it. A load balancer has no concept of "I would rather wait".

Assumption 2: requests are roughly interchangeable

"Least connections" is a good heuristic when requests cost about the same.

Inference requests differ by two orders of magnitude. One completion emits 10 tokens in 300 ms. The next emits 2,000 tokens over 90 seconds. To a connection counter these are both "1". A worker holding one long generation looks exactly as busy as a worker holding one trivial one, and traffic keeps arriving at the wrong place.

Worse, you cannot know the cost in advance — output length is not known until the model stops. Any scheduler that needs to know request cost up front is unbuildable here. What you can do is measure what each worker has actually been achieving and let that inform the next decision.

Assumption 3: capacity is a property of the host

Autoscaling assumes you can add capacity roughly on demand. Loading a model takes minutes and occupies the card while it happens. Capacity is bound to a loaded model on a specific card, not to a host, and it cannot be conjured reactively during a traffic spike.

This inverts the usual approach. You cannot scale into a burst, so you must schedule through it.

What to do instead

Stop distributing and start controlling admission. For each request, score every eligible worker on things that actually predict how it will perform:

score = (100 * active_requests)     # concurrency hurts most
      + (10  * queue_length)
      + (5   * normalized_latency)
      + (2   * normalized_job_cost)
      - (3   * normalized_throughput) # reward workers that are genuinely fast
      + (15  * kv_pressure)
      + (15  * gpu_util)

The exact weights matter less than the shape: active concurrency dominates, throughput is a reward rather than a penalty, and memory pressure is visible to the scheduler.

Two details make the difference between this working and merely looking clever:

Persist the averages. Latency and throughput should be exponential moving averages stored outside the process, so a worker that has historically been slow is still known to be slow after a restart. Otherwise every deploy resets the scheduler's memory and it spends the next few minutes rediscovering reality.

Log the full breakdown of every decision. When someone asks why a request was slow, "it scored lowest on a formula" is not an answer. The individual terms are.

The trap: deterministic scoring races

Here is the bug you will write, because we did.

Four idle GPUs. Four requests arrive within the same few milliseconds. You expect one per GPU. Instead all four land on GPU 0, while the other three sit idle.

The scoring function is deterministic — given identical worker state it always ranks the same worker best. All four requests read state before any of them committed its own pick, so all four independently computed the same answer.

The tempting fix is to add randomness. Resist it: that degrades every well-behaved case to paper over a burst case. This is a race between reading and writing shared state, and it should be fixed there — by making the pick and the state update atomic, so the second request sees the first one's decision.

The goal is predictability, not peak

The counterintuitive part: a consistent 30 tok/s is more valuable than a spiky 45 tok/s.

Everything downstream — timeouts, streaming UI, capacity planning, the promise you make to a customer — can be built on a number you can rely on. None of it can be built on an average that hides a factor-of-four tail. When someone asks you to make inference faster, check whether they actually mean stop surprising me.