Engineering
Rate limits and retries with exponential backoff
Which errors to retry, how long to wait, and how the rate-limit headers tell you to slow down before you hit a 429.
· 3 min read · Usefully team
Every production AI app eventually meets a 429 or a 503. The difference between a blip and an incident is usually the retry logic: retry the right errors, wait the right amount, and stop at the right time. Usefully already retries provider failures for you on other providers, so your own retries are a second, smaller layer. Here's how to write it.
Know which errors are retryable
Every error from Usefully has the same shape, whatever the provider: an error object with a type, a code, a message and a request_id. Branch on type and code; messages may change.
- Retry:
429 rate_limited(afterRetry-After),502 provider_error,503 service_unavailable,504 timeout, and connection errors. - Don't retry:
400 invalid_request(fix the request),401 authentication,403 permission_denied,404 not_found. - Retry after topping up:
402 insufficient_credits. Retrying immediately won't help.
A 502 or 503 from Usefully means every eligible provider for that model already failed, because failover happens before you see an error. So it's worth a retry or two, but also worth a fallback plan, such as a different model.
Use the SDK's retries first
The official OpenAI and Anthropic SDKs already retry connection errors, 408, 409, 429 and 5xx responses with exponential backoff and jitter, and they honour Retry-After. Often the only change you need is the retry count:
from openai import OpenAI
client = OpenAI(
base_url="https://api.usefully.info/v1",
api_key=USEFULLY_API_KEY,
max_retries=4, # default is 2
timeout=60, # seconds per attempt
)Writing your own
If you call the API with plain HTTP, implement the same policy: exponential backoff with full jitter, a cap on the wait, a cap on attempts, and Retry-After taking precedence when present.
const RETRYABLE = new Set([408, 409, 429, 500, 502, 503, 504]);
export async function callWithRetry(body: unknown, attempts = 4): Promise<Response> {
for (let i = 0; ; i++) {
const res = await fetch("https://api.usefully.info/v1/chat/completions", {
method: "POST",
headers: {
Authorization: `Bearer ${process.env.USEFULLY_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify(body),
});
if (res.ok || !RETRYABLE.has(res.status) || i === attempts - 1) return res;
const retryAfter = Number(res.headers.get("retry-after"));
const backoff = Math.min(30_000, 500 * 2 ** i) * Math.random(); // full jitter
await new Promise((r) => setTimeout(r, retryAfter > 0 ? retryAfter * 1000 : backoff));
}
}Jitter matters more than it looks. Without it, every worker that hit the limit at the same moment retries at the same moment, and hits it again.
Slow down before the 429
Every response carries the current rate-limit state, so a batch job can pace itself instead of bouncing off the limit:
x-ratelimit-limit: requests per minute allowed for this key.x-ratelimit-remaining: requests left in the current window.x-ratelimit-reset: seconds until the window is full again.
When remaining gets low, sleep until reset. For large backfills, a fixed-size worker pool sized below your limit is simpler and more predictable than retrying your way through.
Limits you set yourself
Per-key limits (requests per minute, tokens per minute and a daily spend limit) return the same 429 rate_limited, with codes rpm_exceeded, tpm_exceeded or daily_spend_limit_reached. The last one doesn't clear in seconds: it resets at midnight UTC, so don't retry it in a loop. Check the code and surface it.
Retries and cost
Failed requests aren't charged, and credits reserved for a failed request are released straight away. So a retry costs you time, not money. The exception is a retry of a request that actually succeeded but whose response you lost (a client-side timeout, say): that's two billed requests. Keep client timeouts generous enough that you rarely abandon a request that's still working; see timeouts and the failover budget.
One last habit: log the request_id from any error you give up on. It's the fastest way for support to see what happened to that request.
The full error table is in the API reference. Include the request_id when you write to support@usefully.info.