Skip to content
Free$1 of credit for every new organization
All articles

Engineering

How automatic fallback works

When a provider errors, overloads or times out, Usefully moves the request to another provider serving the same model. Here's what triggers it, and what doesn't.

· 2 min read · Usefully team


Most models are served by more than one provider. Usefully tracks each provider's health and, when one fails, sends the request to the next one, so a provider outage becomes a retry your users never see.

How a request is routed

  1. 1Usefully lists the providers serving the model and drops any excluded by your preferences or currently unhealthy.
  2. 2It ranks the rest (a curated priority per model, unless you ask to sort by price, latency or throughput) and sends the request to the first.
  3. 3If that provider fails in a way that says something about the provider, the request moves to the next one.

What triggers a fallback

Provider-side failures: a 5xx error, an overload, a rate limit, a timeout or a connection error. At most 3 providers are tried, and no new attempt starts after 30 seconds, so a request never hangs on a long chain of retries.

What doesn't

A failure about the request itself, such as a 400 because the prompt is too long for the model's context window, is returned straight away. Another provider would reject it too, so retrying would only waste time.

Streaming

A stream can fall back only before its first byte is sent. Once text has started reaching your app, switching providers would mean starting the answer again, so a later failure ends the stream with an error event instead.

Circuit breakers

A provider that keeps failing is temporarily demoted, so one outage doesn't slow every request down with a failed first attempt. It's tried again once it recovers. An open circuit breaker even overrides your own order preference, unless no other provider is left.

Your control

Add a provider object to any request: order to set your own priority, only or ignore to include or exclude providers, sort by price, latency or throughput, max_price to skip expensive providers, and allow_fallbacks: false to use one provider only.

{
  "model": "anthropic/claude-sonnet-5-5",
  "messages": [{ "role": "user", "content": "Hi" }],
  "provider": { "sort": "latency", "allow_fallbacks": true }
}

And you're charged once: a request that fails over to a second provider costs the same as one that succeeded first time. More in models and routing.

More articles

Get started

Your first request is five minutes away.

  • $1 of free credit to start
  • No card, no subscription
  • Keep the SDK you already use
  • Real people at support@usefully.info
How automatic fallback works · Usefully