LLM provider failover: how to keep an AI app answering when a model API fails

Every LLM API fails some of the time. Requests get rate-limited, responses slow down, servers return 5xx errors and streams drop halfway through an answer. If your app depends on one provider, that provider’s bad hour is your outage. Failover is how you keep answering.

What actually fails

• Rate limits. The API returns a 429 because you, or everyone on a shared tier, sent too much.
• Slow responses. The request succeeds eventually, but the first token arrives too late for a user to wait.
• Server errors. A 500, 502 or 503 from the provider.
• Dropped connections. A stream starts and then stops before the answer is complete.

Each needs a decision: retry the same provider, move to another one, or give up with a clear message.

Pattern 1: an ordered cascade

List providers in the order you prefer them, usually cheapest and fastest first. Try the first. If it fails, try the next. Honey uses Cloudflare Workers AI (Llama 3), then Groq, then Gemini 2.5 Flash. A sketch in TypeScript:

type Provider = {
  name: string;
  generate: (prompt: string) => Promise<string>;
};

// Preferred provider first.
const cascade: Provider[] = [workersAi, groq, gemini];

export async function answer(prompt: string) {
  for (const provider of cascade) {
    // Skip a provider that keeps failing.
    if (breaker.isOpen(provider.name)) continue;
    try {
      const text = await withTimeout(
        provider.generate(prompt),
        8_000,
      );
      breaker.recordSuccess(provider.name);
      return { text, provider: provider.name };
    } catch (error) {
      breaker.recordFailure(provider.name);
      console.warn("provider failed", provider.name, error);
    }
  }
  throw new Error("All providers failed");
}

withTimeout and breaker are small helpers, covered next.

Pattern 2: a timeout for every attempt

A provider that answers in forty seconds has failed for a chat user. Give each attempt its own time budget, ideally for the time to first token when you stream, and keep the total inside what a user will tolerate. Otherwise a slow first provider uses up the whole budget before the fallback gets a turn.

Pattern 3: retries with a budget

Transient errors deserve a retry, but retries add load to a provider that is already struggling. Three rules keep them safe:

• Use exponential backoff with jitter, so many clients do not retry at the same instant.
• Cap the retries per request, and ideally across all requests, as a retry budget.
• Do not retry errors that will not change, such as a malformed request. For a 429, respect the Retry-After header when the provider sends one.

Pattern 4: a circuit breaker

If a provider has failed several times in a row, stop sending it traffic for a short cooldown, then let a single request through to test whether it has recovered. This saves every user from waiting on a timeout you already expect, and it gives the provider room to recover.

Make the fallback honest

• Models answer differently. Test your prompts against the last model in the cascade, not only the first.
• Log which provider answered. You cannot improve a cascade you cannot see.
• Decide what happens when everything fails. A clear message beats a spinner.

Test the failures on purpose

A cascade you have never seen fail is a guess. Chaos testing means simulating the failures: slow responses, random 500s, dropped sockets and jitter. Hiccup is an endpoint that misbehaves on purpose for exactly this, so you can test retry budgets and backoff without building flaky mock servers. On the Honey page, the outage demo shows a cascade reacting to a failure.

Need this on your AI app?

Failover, retry budgets, circuit breakers and a chaos test suite are the core of my LLM reliability engineering service, typically a one to two week engagement. Send a brief and you get a direct reply within 12 hours.