Published on
November 16, 2026
Updated on
November 16, 2026

Four LLM Failure Modes Your Fallback Doesn't Catch

Four LLM Failure Modes Your Fallback Doesn't Catch

Most teams set up an LLM gateway to switch providers when one goes down. That setup handles one of four failure modes: the endpoint that returns an error. It misses the model that keeps returning 200 OK with worse answers, it cannot help when access to your model is withdrawn overnight by regulatory order, and it does nothing when the gateway itself is the breach.

In short

  • LLM failover breaks in four distinct ways: endpoint errors, silent answer degradation, access withdrawal, and a breached gateway. Error-based failover only catches the first.
  • On October 21, 2026, a malicious LiteLLM package on PyPI stole credentials and dumped provider key tables. The layer doing the failover was the breach.
  • Silent degradation is the hardest to catch. Anthropic's own eval team missed real quality drops twice in seven months, and users noticed first both times.
  • Cross-model voting has research support but no published production deployment with a false-positive rate. Different model names do not guarantee independent votes.
  • The defense is layered: failover classified by error source, a second model family that passes your evals, a canary with a fixed reference, and the gateway treated as a vault.

What Are the Four Ways LLM Failover Breaks?

LLM failover breaks in four ways, and each needs a different defense. The endpoint returns an error, the answer degrades without any error code, access to the model is withdrawn entirely, or the gateway layer itself becomes the breach.

Failure typeWhat happensDoes failover catch it?
Endpoint errorProvider returns 5xx, 429, or timeoutYes, if the fallback leaves the failing path
Silent degradationModel returns 200 OK with worse contentNo. The gateway logs a success.
Access withdrawalModel is pulled by regulatory or vendor orderNo. No routing setting restores a withdrawn model.
Gateway breachThe failover layer itself is compromisedNo. The layer meant to protect you becomes the attack surface.

A team building on a single LLM provider tends to plan for the first failure and stop there. The gateway handles it, the dashboard goes green, and the architecture diagram shows a clean fallback chain. The other three failures do not show up in request logs or status pages. They show up in your users' experience, or in a security incident report.

How Does Error-Based Failover Handle Provider Outages?

Error-based failover works when the failure is visible in your request outcomes. The provider returns an error, the gateway sees it, and traffic moves to a backup path. This is the failure mode gateways were built for, and it is close to solved, with two caveats.

What the Anthropic Incident Showed

On October 25, 2026, Anthropic's status page opened an incident covering multiple models, with elevated errors lasting about thirty minutes. It was the fourth elevated-error incident in under two days. A gateway handles this well if the fallback leaves the serving path that is failing. A chain that drops from Opus to Sonnet to Haiku on the same API walks from one error into the next.

When Vendor Outages Overlap

Crossing vendors helps, with less certainty than the architecture diagram suggests. On September 3, ChatGPT, Claude, and Grok had overlapping outages. Three different vendor names did not guarantee a healthy route. The error code can also lie. On June 2, OpenAI's Codex requests got HTTP 429s for about fifty minutes because a failing rate-limiting dependency flagged them by mistake. A policy that only backs off on 429 would have kept waiting on the broken path.

A Second Failure Domain

Brightwave, whose agents process billions of tokens a day, does this properly. Their engineering writeup puts the reason in one line: "If every agent owns its own SDK calls, every provider incident becomes a product incident." Their router serves the same Sonnet through Anthropic, Bedrock, or Vertex, and they have seen external provider status pages turn red while their customer-facing dashboards stay flat.

A cheaper step exists if you cannot build that router: stop calling the vendor directly. Enterprise teams go through Bedrock or Azure. Bedrock serves the same model weights on AWS hardware, with reserved capacity if you pay for it. Of the three bugs in Anthropic's September 2025 postmortem, two never reached Bedrock at all. A cloud endpoint gives you a different failure domain.

It has limits. Azure OpenAI had its own multi-region outage on May 29, about three hours of elevated errors. The region is still a single point. When DynamoDB's DNS broke us-east-1 in October 2025, a primary and a backup in that region went down together.

What Happens When the Model Returns 200 but the Answer Is Wrong?

The second failure type does not throw an error. The provider answers with a valid response, the gateway logs a success, and the content is worse than it was last week. This is the failure nobody pages you about.

Anthropic's September 2025 postmortem is the clearest case. A routing bug sent Sonnet 4 requests to the wrong servers. On Anthropic's own platform, it affected 16% of requests at the worst hour. On Bedrock, its peak was 0.18%. Users got worse answers, and no errors. The postmortem admits that "the evaluations we ran simply didn't capture the degradation users were reporting."

The April 2026 postmortem, about the Claude Code product layer, says it again: "neither our internal usage nor evals initially reproduced the issues." Twice in seven months, a team with the weights, the infrastructure, and a dedicated eval function heard about a drop from users first. If it can happen there, it can happen in any stack that relies on vendor evals alone.

One public instrument for tracking quality over time is LiveNerf, an open-source tracker that asks Opus 5.5 the same 78 hard questions every day through Claude Code, with a control model alongside. Through October 23 it shows no change. Its author flags eight questions that look wrong and thirty ambiguous ones, and reruns the result without them. What it offers is a panel, settings, error bars, and a decision rule anyone can inspect.

The vendor's evals missed real drops twice. The tracker measures a narrower slice of work than any one team's production traffic. A fixed reference does not settle every complaint. It tells you which claim you have actually tested.

Building your own LLM evaluation framework is the anchor that catches this failure. The gateway cannot do it. A compiler, a schema validator, a fixed question panel with known answers, or a domain reviewer can. For a broader look at how AI systems fail in production, see our breakdown of the biggest AI fails of 2025.

Can a Second Model Catch Degradation the First One Hides?

The obvious fix for a missing reference is a second model. Run two vendors side by side and treat disagreement as the signal. The research is more developed than expected, but the production evidence is still thin.

A study covering more than 350 models found that, on one leaderboard dataset, models agreed 60% of the time when both were wrong. More accurate models had more correlated errors. A nine-judge panel drawn from seven model families measured out at about two independent votes. Different model names do not buy independent votes.

ApproachWhat it foundProduction-ready?
PoLL (Panel of LLM evaluators)Juries of smaller models from different families beat a single large judge across six datasetsBenchmark only. No published production alarm with false-positive rate.
Cross-model consistencyAWS researchers used disagreement between families to catch hallucinated passagesSignal exists. No published live deployment as an alarm.
Correlated error analysisMore accurate models have more correlated errors, reducing panel independenceResearch finding. Changes how you interpret panel votes.

What nobody has published is running it as a live production alarm with a false-positive rate. That would change the assessment, and it could appear any month. Until it does, the distance between a benchmark result and a page at 3 AM is where the cost sits. A voting panel decides. A detector only raises its hand, and something outside the models has to decide whether the hand means anything.

What Happens When Access to Your Model Is Withdrawn Overnight?

Both of the first two failures assume the model is still there. The third one took it away while it was healthy. On June 12, the Commerce Department sent Anthropic an export directive, and Anthropic switched Fable 5 and Mythos 5 off for everyone.

No routing setting can restore a withdrawn model. The model was withdrawn on purpose by people outside the vendor. Fable came back nineteen days later. Moving to a cloud host does nothing here either. AWS also revoked access to Fable 5 and Mythos 5 on Bedrock "for all users in all regions," at Anthropic's request.

The only defense is a second model family that already passes your evals, so the gateway has somewhere to switch. A model name sitting in a fallback array, untested against your workload, is a hope. One that passes your evals is a plan. The plan costs something every month you do not need it: prompts tuned for two families, evals run against both, a second contract. Most teams will not pay for it until they have lost a model once.

This is the same pattern we see in vendor lock-in. The cost of switching is paid before the switch, or it is paid under pressure after the model is gone. An eval run is mandatory. A swap without evals is a 200 OK with garbage inside.

For guidance on deploying LLMs with data control built in, download our resource on building a privacy-first LLM.

How Can the Gateway Itself Become the Breach?

On October 21, 2026, a malicious version of LiteLLM went up on PyPI. LiteLLM is the gateway a lot of teams put between their code and the model vendors, so traffic can move when one provider goes down. It was the second time this year. In March, attackers stole its publishing token and shipped two bad versions. Seven months later, it still gets more than 90 million downloads a month.

The October payload stole credentials. On a proxy with a database, it also dumped the tables holding every provider key that proxy stores, according to the first issue report. LiteLLM's own root-cause writeup says the attacker was after "upstream inference API tokens."

The way in was ordinary. A master key committed to a public repo in May and never revoked. An owner's GitHub token that was never scoped down. A release gate loosened that same week "to accommodate the temporary push for more frequent releases." No AI was needed for any of it. Release pressure loosening a security gate is a failure mode most governance frameworks miss.

The case where the layer doing the switching is the breach has a different detection signature. It throws no error code and shows up on no status page. The first signal is an issue report, after the keys are gone. Error-based failover is not looking for it. The gateway's own credential hygiene needs its own check, separate from failover. For a framework on governing AI agents in regulated environments, see our AI agent governance framework.

Brightwave cites the March LiteLLM compromise as one reason it owns that layer itself. That works if you have Brightwave's engineers. If your gateway's database holds every provider key you own, October 21 was a dry run of your incident. For strategies on building reliable guardrails around generative AI, download our guide on effective guardrails for your GenAI apps.

What Should You Put in Place for LLM Resilience?

A layered defense covers all four failure modes. Each layer addresses a specific gap that the others cannot.

LayerWhat it catchesHow to build it
Classified failoverProvider-wide 5xx, timeouts, suspicious 429sBack off on 429 tied to your own quota. Investigate unexplained 429s across keys. Timeouts in seconds, not minutes.
Second host, different regionSame model on a different failure domainAnthropic's 2025 numbers show hosts fail differently. Same model on Bedrock before a second vendor.
Second model familyAccess withdrawalMust already pass your evals before you need it.
Gateway as vaultSupply chain attacks on the failover layerPin reviewed releases and digests. Scope credentials per workload. Revoke exposed keys immediately.
Canary with fixed referenceSilent quality degradationFixed panel, pinned client, explicit settings, control arm, decision rule written before data comes in.

A canary on your own task is the most underbuilt layer. LiveNerf's design is public and small: a fixed panel, a pinned client, explicit settings, a control arm, and a decision rule written down before the data comes in. Its rule needs two ten-day windows to call a change, which is fine for a study and slow for an alarm. Set your own effect size, check frequency, and false-positive budget. Run one on your own task. Put a second model on a sample of traffic as a detector, and treat a disagreement as a reason to look. Page only on a threshold you set in advance, backed by a test, a task-level canary, or a domain reviewer.

The vault is the item that failed at LiteLLM in October. Nobody there needed a new technique. The key had been spotted in May and was never revoked.

NineTwoThree has shipped 27 AI systems to production with a 97% project success rate. 24 of those projects were ROI-positive. We build layered failover, eval-driven model selection, and supply-chain controls into every LLM system we ship.

If your primary model got a third worse tomorrow and kept returning 200, what in your stack would notice before your users did? And if it disappeared on a Friday by order of someone who has never heard of your product, how long until you were running on something else?

If your team is building AI systems that need to stay up when providers go down, contact NineTwoThree to talk about resilience architecture, or see how our AI development services take LLM systems to production. For a deeper dive on the infrastructure behind AI agents, read about what an AI harness is and why it matters.

written by
Share on

Read more from

LLM