Building an LLM inference layer that survives any single provider's outage

Most integrations with an LLM provider are written against one provider. That's a reasonable default until the pipeline depending on it needs to keep running when that provider has a bad day — and once uptime matters, an LLM API is infrastructure like any other external dependency: something to route around, not something to trust unconditionally.

Treat the provider like any other single point of failure

The pattern is the same one you'd apply to a database read replica or an upstream API: a circuit breaker in front of the call, and a configurable fallback order behind it. Analysis in our dependency-upgrade pipeline can run against OpenAI, OpenRouter, a self-hosted Ollama instance, or a local LM Studio endpoint. The circuit breaker trips on repeated failures against one provider and routes to the next in the fallback order, so an outage on one provider degrades the pipeline's latency, not its availability.

Two Redis instances back this, deliberately kept separate: one dedicated to queue coordination for the workers doing the analysis, one dedicated to caching LLM output and changelog data. Keeping them apart means cache pressure under heavy analysis load never competes with the queue throughput that keeps the whole system responsive.

The hard part is the response layer, not the request layer

Every major hosted provider documents its request format thoroughly, and "OpenAI-compatible" has become a reasonable shorthand for how to talk to a huge range of providers, including self-hosted ones. What it doesn't guarantee is response format. Local reasoning models frequently return their answer in a separate field from the one hosted-provider SDKs check by default, and a naive integration reads that as an empty response rather than a differently shaped one.

Building a resilience layer that includes self-hosted models in its fallback chain means the response parser has to be defensive by design: normalize multiple response shapes, strip reasoning markup before it reaches anything downstream expecting clean output, and retry with a larger token budget when a reasoning model runs out of room mid-answer rather than treating that as a hard failure. None of this is optional once a local provider is a real link in the fallback chain rather than a nice-to-have.

Test the resilience layer, not just the happy path

A stub provider exists specifically so the full integration suite exercises this response-parsing logic in CI, deterministically, without live API keys or a running local model. The value isn't testing that the LLM gives a good answer — that's not something CI can verify anyway. It's testing that the plumbing around the LLM — the parser, the fallback order, the circuit breaker — behaves correctly regardless of which provider actually answers, which is exactly the part of the system that has to be right for the resilience guarantee to mean anything.

The general principle

If a pipeline depends on an LLM provider the way it depends on a database, it should be architected the way any other external dependency is: fail over automatically, cache aggressively at the boundary, and build the response layer for the least standardized member of the fallback chain, not the most polished one.