Engineering
Timeouts and the failover budget
Usefully tries up to three providers within about 30 seconds. How to set your own client timeouts around that, for chat, streaming and background jobs.
· 3 min read · Usefully team
Timeouts are one of those settings nobody thinks about until a request hangs for two minutes in production. With an API gateway in the middle, there are two layers to think about: what Usefully does when a provider is slow, and what your client does while it waits. Getting them to agree avoids both hung requests and needlessly abandoned ones.
What Usefully does
When a provider errors, is overloaded, rate-limits or times out, Usefully moves the request to the next healthy provider serving the same model. Two limits keep that bounded:
- At most 3 providers are tried for one request.
- About 30 seconds of failover budget: no new attempt starts after that point.
Health scores and circuit breakers mean a provider that's already failing is skipped up front, so most requests that need failover spend little of that budget. And however many providers were tried, you're charged once, only for the attempt that succeeded.
Setting your client timeout
Your client timeout should cover the failover budget plus the time the model needs to generate the answer. Generation time depends mostly on output length and the model. A few rules of thumb:
- Short, non-streaming answers (classification, extraction): 45 to 60 seconds.
- Long, non-streaming answers (reports, large code edits): 120 seconds or more, or switch to streaming.
- Streaming: use a short timeout for the first chunk and a separate idle timeout between chunks, rather than one long total timeout.
from openai import OpenAI
import httpx
client = OpenAI(
base_url="https://api.usefully.info/v1",
api_key=USEFULLY_API_KEY,
timeout=httpx.Timeout(90.0, connect=10.0), # total per attempt, and connect
max_retries=2,
)Why too short is expensive
If your client gives up at 15 seconds while the model is still writing, the request may well finish on our side and be billed, and your retry starts a second, billed request. Failed requests aren't charged, but a request you abandoned isn't a failure from the provider's point of view. A timeout that's too short turns slow into double-priced.
Why too long hurts users
In an interactive app, nobody waits two minutes for a spinner. For user-facing features, stream the response so the user sees progress, and keep max_tokens proportionate to the answer you need. A long answer that streams feels faster than a short one that doesn't.
Streaming specifics
Failover for a stream happens only before the first chunk. Once text is flowing, a provider failure ends the stream with an error event. So for streams:
- 1Allow up to about 30 seconds for the first chunk; that covers failover.
- 2After that, use an idle timeout (say 20 to 30 seconds without a chunk) rather than a total limit.
- 3If the stream ends with an error, show what arrived and offer a retry. You pay only for what was delivered.
const controller = new AbortController();
let idle = setTimeout(() => controller.abort(), 30_000); // first chunk
const stream = await client.chat.completions.create(
{ model: "anthropic/claude-sonnet-5-5", messages, stream: true },
{ signal: controller.signal },
);
for await (const chunk of stream) {
clearTimeout(idle);
idle = setTimeout(() => controller.abort(), 20_000); // between chunks
send(chunk.choices[0]?.delta?.content ?? "");
}
clearTimeout(idle);Background jobs
Batch work can afford long timeouts and more retries. Give each request a generous timeout, retry 5xx and 429 with backoff, and size your worker pool below your rate limit. See rate limits and retries.
Whatever values you choose, write them down next to the code that sets them, with the reason. Timeouts are easy to change in a hurry during an incident and hard to reason about later.
The routing and failover rules are described in how automatic fallback works. Latency for every request is in the dashboard Logs.