Circuit Breakers, Hedging, and Provider Fallbacks for Production LLM APIs in September 2026: Keep Latency SLOs When an Upstream Model Stalls

When a production LLM API slows down or returns 5xx errors, your product feels it immediately. Chat UIs hang. Agents stall mid-tool loop. Batch jobs burn their wall-clock budget waiting on a single upstream. Cascades and cost-aware routing help you pick a cheaper model when quality allows. Canary and shadow traffic help you swap models safely. This post is about a different failure mode: the primary provider is healthy most of the time, then suddenly is not. You need timeouts, circuit breakers, hedging, and explicit fallbacks so one slow dependency does not take your SLO with it.

Image: mikemacmarketing / photo on flickr via Wikimedia Commons (CC BY 2.0)

Image: Midjourney; prompt suggested by Grok via Wikimedia Commons (Public domain)

This is a practical playbook for API and platform teams running generative AI in production in September 2026. No fluff: what to measure, what to open and close, and how to wire fallbacks without inventing duplicate answers or double-billing your users.

What breaks first when an LLM provider stalls

LLM calls are long-lived compared with typical microservice RPCs. Time-to-first-token (TTFT) for a cold or contended model can be hundreds of milliseconds to several seconds. Full completion for a long answer can run tens of seconds. If your gateway treats every call like a 200ms CRUD request, you will either time out healthy generations or hold worker threads forever.

Typical failure patterns:

• Tail latency spikes — p50 looks fine, p99 TTFT jumps, and interactive users abandon the session.
• Partial outages — some regions or model IDs fail while others succeed; naive retries hammer the bad shard.
• Queue buildup — your own concurrency limit fills with stuck requests, so even healthy capacity cannot accept new work.
• Agent fan-out — one tool-calling step waits on a stalled model, so the whole agent run exceeds its deadline.

Design for these before you optimize for another percentage point of cache hit rate.

Define deadline budgets, not a single timeout

Give every user-facing request a total deadline. Split that budget across stages: auth and policy checks, retrieval (if any), primary LLM call, fallback LLM call, and post-processing. A concrete pattern:

1. Total client deadline: 30 seconds for interactive chat.
2. Primary model attempt: up to 12 seconds to first token, then stream until 25 seconds total.
3. If the primary has not produced a first token by 8 seconds, start a hedge (see below) or trip toward fallback.
4. Reserve 4–5 seconds for a smaller fallback model so the user still gets something before the hard deadline.

Put the remaining budget in headers your gateway understands (for example a deadline timestamp). Propagate it to retries so a late retry does not start a 20-second call with 2 seconds left.

Separate connect timeouts from read timeouts. A TCP connect should fail fast (1–2 seconds). A read timeout during streaming should be idle-based: reset the idle clock on every token chunk, and only abort if no bytes arrive for N seconds. That avoids killing a slow but progressing generation.

Circuit breakers that match LLM traffic

A classic circuit breaker opens after a threshold of failures, then probes with half-open requests. For LLM APIs, tune it to the signals that matter:

• Open on elevated 5xx rate, connection errors, or sustained TTFT above a hard threshold for a given model ID and region.
• Do not open on ordinary 4xx validation errors, content-policy refusals, or short rate-limit responses you can back off from.
• Scope the breaker per provider + model + region, not one global switch for “all AI.” A outage on a large reasoning model should not black-hole a small classifier model on the same vendor.
• Half-open carefully — use a small probe concurrency (one or a few requests) and only close after consecutive successes within a short window.

Emit breaker state as a metric and as a structured log field. On-call should see “primary-gpt-open” in the same dashboard as error rate, not discover it from user reports.

When the breaker is open, fail fast into your fallback path instead of queueing. Queuing behind an open breaker recreates the pile-up you were trying to avoid.

Hedging: a second request when the first is late

Request hedging sends a duplicate call to a second replica or provider if the first has not answered by a hedge delay. It is useful when latency variance is high and capacity exists. It is expensive if you hedge too early or too often.

Practical rules:

• Hedge only for latency-critical, user-visible paths (chat, autocomplete). Skip hedging for offline batch jobs where cost dominates.
• Set the hedge delay around a high percentile of healthy TTFT (for example p95), not the mean. Hedging at p50 doubles spend for little gain.
• Cancel the loser as soon as the winner produces a first token (or a complete response, if you cannot cancel mid-stream). Track wasted tokens from hedges you could not cancel.
• Prefer hedging across independent failure domains (another region or another provider) rather than two identical replicas behind the same congested cluster.

Cap hedge rate. If more than a small fraction of traffic is hedging, you are masking a capacity or model problem. Fix the primary instead of paying forever for duplicates.

Provider fallbacks without double answers

A fallback is a different model or vendor used when the primary is open-circuited, timed out, or explicitly unavailable. Cascades pick a smaller model for cost when the task is easy. Fallbacks pick an alternate when the primary cannot serve at all. Keep those policies separate in config so product and SRE can change them independently.

Implementation checklist:

1. Map capabilities — if the primary supports tools and structured JSON, the fallback must support the same contract or you must degrade the feature (for example return plain text and skip tools).
2. Normalize errors — translate provider-specific error codes into a small internal set: retryable, non-retryable, content-policy, capacity.
3. Idempotency keys — for non-streaming calls that may create side effects via tools, pass an idempotency key so a retry or fallback does not fire the same purchase or ticket twice.
4. Label the response — record which model actually answered in logs and (when appropriate) in the API response metadata for debugging.
5. Quality gate — if the fallback is much weaker, shorten max tokens or tighten system instructions so you do not promise primary-tier quality while serving a tiny model.

Example flow for an interactive completion:

1. Call primary with deadline budget and streaming enabled.
2. If breaker is open or connect fails, go straight to fallback.
3. If TTFT exceeds hedge delay, start hedge or fallback in parallel; cancel the slower path on first token.
4. If primary returns retryable 5xx and budget remains, one bounded retry with jitter, then fallback.
5. If nothing succeeds before the client deadline, return a typed error the UI can render (“AI temporarily unavailable”) instead of a generic 500.

What to measure every week

Instrument at least:

• Success rate and TTFT / end-to-end latency by provider, model, and region.
• Breaker open time and probe success rate.
• Hedge fire rate, hedge win rate, and wasted token cost.
• Fallback share of traffic and user-visible error rate when both paths fail.
• Queue depth and rejected-at-gateway count (capacity exhaustion vs upstream failure).

Alert on sustained breaker-open and on rising fallback share, not only on raw 5xx. A silent shift to fallback can hide a primary outage until quality tickets arrive.

A minimal rollout plan

Week 1: add deadline propagation, idle-based stream timeouts, and per-model breakers in shadow mode (log would-open, do not trip).

Week 2: enable fail-fast on open breakers to a single fallback model for one low-risk endpoint. Track quality and cost.

Week 3: add hedging for the hottest interactive path only, with a high percentile delay and loser cancellation.

Week 4: document runbooks — how to force-open a breaker, how to disable hedging during a cost incident, and how to pin traffic to one provider during a migration.

Resilience for LLM APIs is not a single library toggle. It is a budget, a breaker scope, a hedge policy, and a fallback contract that match how generative calls actually behave. Get those four right and provider blips stop looking like product outages.