LLM Router

The problem

A llama.cpp instance on a GPU breaks the assumptions every normal load balancer is built on.

Concurrency doesn't scale. Two generations on one GPU don't run twice as slowly — they split KV-cache and compute in ways that make both unpredictably slow. A web backend degrades gracefully under concurrency; a GPU degrades chaotically.

Requests are wildly non-uniform. One request generates 10 tokens in 300 ms, the next generates 2,000 tokens over 90 seconds. "Least connections" treats them as equal.

Capacity is per-model, not per-host. Loading a model takes minutes and owns the card while it does. You cannot scale reactively the way you would a stateless service.

The approach

The router controls concurrency rather than merely distributing it: it decides whether a request should run now, and where. The design goal is predictability over peak — a consistent 30 tok/s is preferred over a spiky 45 tok/s, because everything downstream can plan around the former.

Routing algorithm

For each incoming request the router filters to workers with the matching model and healthy status, then scores each eligible worker:

score = (100 * active_requests)
      + (10  * queue_length)
      + (5   * normalized_latency)
      + (2   * normalized_job_cost)
      - (3   * normalized_throughput)
      + (1   * node_distance_penalty)
      + (15  * kv_pressure)
      + (15  * gpu_util)

The worker with the lowest score wins. Latency and throughput are exponential moving averages persisted in Redis, so scores reflect real historical performance across worker restarts rather than resetting on every deploy.

Every routing decision is logged with its full scoring breakdown. When a request is slow, you can say why.

The interesting failure mode

Deterministic scoring has a race condition that surprises people, and it is worth spelling out because it is a general lesson about stateless schedulers.

Picture four idle GPUs and four requests arriving at nearly the same instant. Naively you would expect one request per GPU. But the scoring formula is deterministic: given the same worker state, it always ranks the same worker best. If all four requests read worker state before any of them has committed its own pick, all four independently compute "GPU 0 is best" and all four land on GPU 0 — while GPUs 1, 2 and 3 sit idle.

That is not a bug in the scoring maths. It is a race in when each request reads state relative to when the others write it. Recognising the difference is what determines whether you fix it correctly.

Operational design

• Redis as the only state backend. Queues, worker registration, health and the moving averages all live in one place, which keeps the router itself stateless and horizontally scalable.
• Priority handling. critical jobs go to the front of a worker's queue and get a wider allowance against per-worker queue-length caps, so an urgent request isn't excluded from a worker merely because its queue looks full by normal standards.
• OpenAI-compatible API alongside a native API, so existing clients work without modification.
• Observability first. Prometheus metrics and Grafana dashboards for queue depth, latency distribution, throughput and per-worker health.

Case study

See this router in production: Making multi-GPU inference predictable instead of fast.

Availability

This project is not currently public. The engineering patterns above are transferable, and we're happy to walk through the design in detail on a call.