Engineering
Reducing latency: model choice, max_tokens and streaming
Most of an AI request's latency is generation time. Five practical levers for making responses arrive sooner, in rough order of impact.
· 4 min read · Usefully team
When an AI feature feels slow, the instinct is to blame the network or the API. Usually the time goes somewhere more predictable: the model generating tokens one after another. That's good news, because it means latency is mostly under your control. Here are the levers, roughly in order of impact.
Where the time goes
A request's latency has three parts: getting to the model (network, routing, queueing), reading the input (longer prompts take longer), and generating the output (each token after the previous one). For most chat and writing tasks, the last part dominates. A 600-token answer takes several times longer than a 100-token one on the same model.
The dashboard Logs show latency next to output tokens for every request. If the two rise and fall together, you're looking at generation time, and the levers below will help.
1. Stream
Streaming doesn't make the full answer arrive sooner, but it changes what the user waits for: the first words instead of the last. For anything a person reads, it's the biggest perceived improvement for the least work. Set "stream": true; see streaming responses without surprises.
2. Ask for less output
Fewer output tokens is the most direct speed-up there is, and it saves money too, since output is usually priced higher than input.
- Set
max_tokensclose to the longest answer you actually want. - Say how long the answer should be: "in two sentences", "under 100 words", "return only the JSON".
- Don't ask the model to restate the question or summarize what it's about to do.
- For classification, ask for a label, not a label plus an explanation, unless you use the explanation.
reply = client.chat.completions.create(
model="anthropic/claude-haiku-5-5",
max_tokens=60,
messages=[
{"role": "system", "content": "Classify the ticket as billing, bug, or other. Reply with the label only."},
{"role": "user", "content": ticket},
],
)3. Use a smaller model where it's good enough
Smaller models generate faster. For routing, classification, extraction and short rewrites, they're often as accurate as larger ones on your actual inputs. Test on twenty real prompts and compare; if the small model passes, you've made the feature faster and cheaper at once. The models page lists every model's price and context window side by side.
4. Send less input
Long prompts take longer to process before the first token appears. Common sources of bloat:
- Retrieval that stuffs in twenty documents when five would do.
- Full conversation history in long chats. Summarize older turns or keep the last few.
- Large tool results passed back verbatim. Return only the fields the model needs.
- System prompts that grew a paragraph per incident. Prune them occasionally.
5. Split the work
If one request does several jobs (classify, then look something up, then write), consider splitting it. A fast model can handle the quick steps while the user sees progress, and independent steps can run in parallel. Two parallel 2-second requests beat one 4-second request.
What about provider failures?
When a provider fails, Usefully fails over to another one serving the same model. Health scoring and circuit breakers skip providers that are already failing, so most requests don't pay for a failed first attempt. When a failover does happen, it adds the time of the failed attempt, bounded by the failover budget of about 30 seconds. For a stream, failover only happens before the first chunk, so once text is flowing there's no added delay.
A quick checklist
- 1Stream anything a person reads.
- 2Set
max_tokensand ask for concise output. - 3Try the smallest model that passes your quality check.
- 4Trim the input: fewer documents, shorter history, smaller tool results.
- 5Check latency against output tokens in Logs after each change.
Compare models' speed and cost on your own prompts in the dashboard playground, then confirm with real traffic in Logs. Model details: models.