Skip to content
Free$1 of credit for every new organization
All articles

Engineering

Handling provider outages in your app

Usefully fails over between providers serving the same model. Here's what that covers, what it can't, and the few lines of code that cover the rest.

· 3 min read · Usefully team


Every model provider has outages: elevated error rates, overloaded capacity, slow responses during a launch week. If your product depends on a model, a provider's bad hour becomes your bad hour unless something sits in between. Usefully handles a large part of this for you. This post explains which part, and how to handle the rest in your own code.

What Usefully handles

Many models are served by more than one provider. Usefully keeps a health score for each provider and a circuit breaker that temporarily demotes one that keeps failing. When a request fails in a way that's the provider's fault (a 5xx, an overload, a rate limit, a timeout, a connection error), it moves to the next healthy provider for the same model.

  • Up to 3 providers are tried, within a failover budget of about 30 seconds.
  • You're charged once, for the attempt that succeeded. Failed attempts aren't charged.
  • Errors about the request itself (such as a prompt that's too long) are returned straight away, because another provider would reject them too.
  • Streams fail over only before the first chunk.

For most outages, that's the whole story: your users get an answer, and the provider column in Logs shows who served it.

What it can't handle

  • A model with only one provider. If that provider is down, there's nowhere to fail over to.
  • Every provider for a model failing at once. Rare, but possible during large incidents.
  • A stream that fails mid-answer. The text already sent can't be taken back.
  • Your own constraints. A data_policy or provider restriction can leave fewer providers to choose from, by design.

In these cases you'll get a 502 provider_error, a 503 service_unavailable or a 504 timeout, which means every eligible provider already failed.

Fall back to another model

The most effective app-level defence is a second model, ideally from a different provider family. With one API for all models, that's a list and a loop:

import openai

MODELS = ["anthropic/claude-sonnet-5-5", "openai/gpt-5", "google/gemini-2.5-flash"]

def complete(messages):
    last_err = None
    for model in MODELS:
        try:
            return client.chat.completions.create(model=model, messages=messages, timeout=60)
        except (openai.InternalServerError, openai.APITimeoutError, openai.APIConnectionError) as err:
            last_err = err          # 5xx, timeout, network: try the next model
        except openai.NotFoundError as err:
            last_err = err          # model unavailable: try the next one
    raise last_err

Don't fall back on 400 (fix the request), 401 (fix the key) or 402 (top up): another model won't help. Test your prompts on each fallback model in advance, so the backup answer is one you'd be happy to ship.

Degrade gracefully

Sometimes the right fallback isn't a model at all:

  • Show a cached or previous answer, clearly labelled.
  • Queue the request and notify the user when it's done, for non-interactive work.
  • Hide an optional AI feature and keep the rest of the page working.
  • Show a clear message with a retry button, rather than a spinner that never ends.

Watch it happen

During an incident, Logs show which provider served each request and which requests failed. After it, they show whether your fallback path ran and what it cost. That's useful evidence when deciding whether to make a different model your default.

Test the fallback path deliberately, too: temporarily put an invalid model ID first in your list on staging and confirm the next model answers.

The routing details are in how automatic fallback works. Each model's page on models lists the providers serving it.

More articles

Get started

Your first request is five minutes away.

  • $1 of free credit to start
  • No card, no subscription
  • Keep the SDK you already use
  • Real people at support@usefully.info
Handling provider outages in your app · Usefully