Engineering
Handling provider outages in your app
Usefully fails over between providers serving the same model. Here's what that covers, what it can't, and the few lines of code that cover the rest.
· 3 min read · Usefully team
Every model provider has outages: elevated error rates, overloaded capacity, slow responses during a launch week. If your product depends on a model, a provider's bad hour becomes your bad hour unless something sits in between. Usefully handles a large part of this for you. This post explains which part, and how to handle the rest in your own code.
What Usefully handles
Many models are served by more than one provider. Usefully keeps a health score for each provider and a circuit breaker that temporarily demotes one that keeps failing. When a request fails in a way that's the provider's fault (a 5xx, an overload, a rate limit, a timeout, a connection error), it moves to the next healthy provider for the same model.
- Up to 3 providers are tried, within a failover budget of about 30 seconds.
- You're charged once, for the attempt that succeeded. Failed attempts aren't charged.
- Errors about the request itself (such as a prompt that's too long) are returned straight away, because another provider would reject them too.
- Streams fail over only before the first chunk.
For most outages, that's the whole story: your users get an answer, and the provider column in Logs shows who served it.
What it can't handle
- A model with only one provider. If that provider is down, there's nowhere to fail over to.
- Every provider for a model failing at once. Rare, but possible during large incidents.
- A stream that fails mid-answer. The text already sent can't be taken back.
- Your own constraints. A
data_policyor provider restriction can leave fewer providers to choose from, by design.
In these cases you'll get a 502 provider_error, a 503 service_unavailable or a 504 timeout, which means every eligible provider already failed.
Fall back to another model
The most effective app-level defence is a second model, ideally from a different provider family. With one API for all models, that's a list and a loop:
import openai
MODELS = ["anthropic/claude-sonnet-5-5", "openai/gpt-5", "google/gemini-2.5-flash"]
def complete(messages):
last_err = None
for model in MODELS:
try:
return client.chat.completions.create(model=model, messages=messages, timeout=60)
except (openai.InternalServerError, openai.APITimeoutError, openai.APIConnectionError) as err:
last_err = err # 5xx, timeout, network: try the next model
except openai.NotFoundError as err:
last_err = err # model unavailable: try the next one
raise last_errDon't fall back on 400 (fix the request), 401 (fix the key) or 402 (top up): another model won't help. Test your prompts on each fallback model in advance, so the backup answer is one you'd be happy to ship.
Degrade gracefully
Sometimes the right fallback isn't a model at all:
- Show a cached or previous answer, clearly labelled.
- Queue the request and notify the user when it's done, for non-interactive work.
- Hide an optional AI feature and keep the rest of the page working.
- Show a clear message with a retry button, rather than a spinner that never ends.
Watch it happen
During an incident, Logs show which provider served each request and which requests failed. After it, they show whether your fallback path ran and what it cost. That's useful evidence when deciding whether to make a different model your default.
Test the fallback path deliberately, too: temporarily put an invalid model ID first in your list on staging and confirm the next model answers.
The routing details are in how automatic fallback works. Each model's page on models lists the providers serving it.