Engineering
How automatic fallback works
When a provider errors, overloads or times out, Usefully moves the request to another provider serving the same model. Here's what triggers it, and what doesn't.
· 2 min read · Usefully team
Most models are served by more than one provider. Usefully tracks each provider's health and, when one fails, sends the request to the next one, so a provider outage becomes a retry your users never see.
How a request is routed
- 1Usefully lists the providers serving the model and drops any excluded by your preferences or currently unhealthy.
- 2It ranks the rest (a curated priority per model, unless you ask to sort by price, latency or throughput) and sends the request to the first.
- 3If that provider fails in a way that says something about the provider, the request moves to the next one.
What triggers a fallback
Provider-side failures: a 5xx error, an overload, a rate limit, a timeout or a connection error. At most 3 providers are tried, and no new attempt starts after 30 seconds, so a request never hangs on a long chain of retries.
What doesn't
A failure about the request itself, such as a 400 because the prompt is too long for the model's context window, is returned straight away. Another provider would reject it too, so retrying would only waste time.
Streaming
A stream can fall back only before its first byte is sent. Once text has started reaching your app, switching providers would mean starting the answer again, so a later failure ends the stream with an error event instead.
Circuit breakers
A provider that keeps failing is temporarily demoted, so one outage doesn't slow every request down with a failed first attempt. It's tried again once it recovers. An open circuit breaker even overrides your own order preference, unless no other provider is left.
Your control
Add a provider object to any request: order to set your own priority, only or ignore to include or exclude providers, sort by price, latency or throughput, max_price to skip expensive providers, and allow_fallbacks: false to use one provider only.
{
"model": "anthropic/claude-sonnet-5-5",
"messages": [{ "role": "user", "content": "Hi" }],
"provider": { "sort": "latency", "allow_fallbacks": true }
}And you're charged once: a request that fails over to a second provider costs the same as one that succeeded first time. More in models and routing.