Appearance
Models & routing
Choosing a model
Models are named <author>/<model>, e.g. anthropic/claude-sonnet-5-5, anthropic/claude-haiku-5-5. A bare name such as claude-sonnet-5-5 also works when it identifies exactly one model.
GET /v1/models (no key needed) lists every model with its type (chat, search), context length, capabilities, the providers serving it, and prices as exact decimal strings:
json
{
"id": "meta-llama/llama-3.3-70b-instruct",
"type": "chat",
"providers": ["groq", "together"],
"pricing": { "currency": "USD", "input_per_million_tokens": "0.59", "output_per_million_tokens": "0.79" }
}Any chat model works through both /v1/chat/completions (OpenAI format) and, for Claude models, /v1/messages (Anthropic format). Usefully translates between formats, including tool calls and streaming.
How a request is routed
- Usefully lists the providers serving the model and drops any that are excluded by your preferences (below) or currently unhealthy.
- It ranks the rest (by default, a curated priority per model) and sends the request to the first.
- If that provider fails in a way that says something about the provider (5xx, overload, rate limit, timeout, connection error), the request goes to the next one. At most 3 providers are tried, and no new attempt starts after 30 seconds.
- If the failure is about the request (a 400 such as a context overflow), it's returned immediately: another provider would reject it too.
Streams fall back only before the first byte is sent. After that, a failure ends the stream with an error event (see Streaming).
Providers that keep failing are temporarily demoted (a circuit breaker) and retried once they recover, so one outage doesn't slow down every request.
Routing preferences
Add a provider object to any chat or search request:
json
{
"model": "meta-llama/llama-3.3-70b-instruct",
"messages": [{ "role": "user", "content": "Hi" }],
"provider": {
"order": ["groq", "together"],
"allow_fallbacks": true,
"sort": "price",
"max_price": { "input": 1.0, "output": 2.0 }
}
}| Field | Effect |
|---|---|
order | Provider IDs to try first, in this order |
only | Use only these providers |
ignore | Never use these providers |
allow_fallbacks | false sends the request to one provider only (default true) |
sort | price, latency or throughput instead of the default ranking |
max_price | Skip providers priced above these USD-per-million-token limits |
data_policy | zdr (zero data retention), no_training, or any |
The provider object is never forwarded to providers. If no provider satisfies your preferences, the request fails with 503 no_provider_available.
An open circuit breaker overrides order: a provider that's down is skipped even if you listed it first, unless no other provider is left.
Pricing
Token prices are the providers' list prices, passed through unchanged. Each response's cost appears in Logs, and a request is only charged for tokens actually used (see Billing).
Response caching
Add Usefully-Cache: <seconds> (1 to 604,800) to a non-streaming chat request to cache its response. An identical request from your organization within that time returns the stored response instantly, without calling a provider, for 10% of the original cost.
bash
curl https://api.usefully.info/v1/chat/completions \
-H "Authorization: Bearer $USEFULLY_API_KEY" -H "Content-Type: application/json" \
-H "Usefully-Cache: 3600" \
-d '{"model": "anthropic/claude-sonnet-5-5", "messages": [{"role": "user", "content": "Summarize our refund policy"}]}'x-usefully-cache: hitormisssays what happened. Hits show providerusefully-cachein Logs.- Requests match when everything that affects the answer is identical (model, messages, temperature, tools…). Routing preferences and
userare ignored. - Caches are per organization and never shared. Streaming requests aren't cached.
- Use it for deterministic prompts (classification, extraction, fixed FAQs); with
temperature > 0a cached answer is one sample, reused.