The problem
A 4× GPU inference host is not difficult to configure — it is difficult to configure again, identically, six months later. Each GPU needs its own server process, its own model, its own port and often its own chat template. Done by hand, the real configuration ends up distributed across shell history, a half-updated wiki page and one engineer's memory.
The approach
One playbook owns the whole host. Each GPU is mapped to a model and a port, a launcher script is templated per instance, and each instance is supervised by its own systemd unit — so a crashed model server restarts without taking its neighbours with it.
A representative mapping:
| GPU | Model | Port | Chat template | | --- | ---------------- | ---- | ------------- | | 0 | deepseek-r1-8b | 8008 | yes | | 1 | minicpm-v-latest | 8009 | yes | | 2 | qwen3.5-9b | 8010 | no | | 3 | qwen3-8b | 8011 | yes |
Validated before it runs
The pipeline has three stages, and all of them are manual — nothing deploys automatically on push, because a playbook that reconfigures GPU hosts is not something you want triggered by a typo in a README:
| Job | Stage | What it does |
| --------- | -------- | ---------------------------------- |
| lint | validate | --syntax-check only |
| dry-run | validate | --check mode, no changes applied |
| deploy | deploy | full playbook run |
Locally the same gates are one command each:
ansible-playbook --syntax-check -i inventory.ini deploy_llama_multi_gpu.yaml
ansible-playbook --check -i inventory.ini deploy_llama_multi_gpu.yamlWhy it pairs with the router
This playbook produces the workers; LLM Router decides what runs on them. Provisioning and scheduling are separate concerns, and keeping them separate is what lets you rebuild a node without touching routing policy, or change routing policy without redeploying a node.
Case study
See this playbook in production: Making multi-GPU inference predictable instead of fast.
Availability
This project is not currently public.