Engineering
Streaming responses without surprises
Streaming makes an app feel fast, but it changes how errors, usage and billing work. A practical guide to streaming through Usefully in Python and TypeScript.
· 3 min read · Usefully team
A model that takes eight seconds to write an answer feels slow if the user stares at a spinner, and fast if the first words appear in half a second. That's what streaming buys you: the same total time, but text arrives as it's generated. On Usefully you turn it on with one field, and it works the same for every model on the models page, whichever provider ends up serving the request.
Streaming also changes a few things you might take for granted with a single JSON response: where usage numbers come from, what an error looks like, and what you pay if the stream stops early. This post walks through all three.
Turning it on
Set "stream": true on POST /v1/chat/completions. The response is a stream of server-sent events in OpenAI's chunk format, ending with data: [DONE]. Any OpenAI SDK pointed at Usefully handles the parsing for you.
from openai import OpenAI
client = OpenAI(base_url="https://api.usefully.info/v1", api_key=USEFULLY_API_KEY)
stream = client.chat.completions.create(
model="anthropic/claude-sonnet-5-5",
messages=[{"role": "user", "content": "Explain circuit breakers in three short paragraphs."}],
stream=True,
stream_options={"include_usage": True},
)
for chunk in stream:
if chunk.choices and chunk.choices[0].delta.content:
print(chunk.choices[0].delta.content, end="", flush=True)
if chunk.usage:
print("\n", chunk.usage)The same thing in TypeScript, for a Node server or a Next.js route handler:
import OpenAI from "openai";
const client = new OpenAI({ baseURL: "https://api.usefully.info/v1", apiKey: process.env.USEFULLY_API_KEY });
const stream = await client.chat.completions.create({
model: "anthropic/claude-sonnet-5-5",
messages: [{ role: "user", content: "Explain circuit breakers in three short paragraphs." }],
stream: true,
stream_options: { include_usage: true },
});
for await (const chunk of stream) {
process.stdout.write(chunk.choices[0]?.delta?.content ?? "");
}Getting usage numbers
A normal response carries a usage object with input and output tokens. A stream doesn't, unless you ask: add stream_options: {"include_usage": true} and a final chunk arrives with the totals and an empty choices list. That's why the Python loop above checks chunk.choices before reading from it. Either way, every request appears in the dashboard Logs with its tokens and cost, so you don't need to record usage yourself just to know what you spent.
What happens when something fails
Usefully can fail over to another provider serving the same model, but a stream can only fail over before its first chunk is sent. After that, switching providers would mean restarting the answer halfway through. So there are two cases:
- Before the first chunk: the request quietly moves to the next healthy provider, like any other request. Your code never sees the failure.
- After text has started: the stream ends with an error event instead of
[DONE]. The SDKs raise it as an exception inside your loop.
In practice that means your streaming loop needs a try around it, and your UI needs a state for "the answer stopped part-way". A short message such as "The response was interrupted. Try again?" with a retry button is usually enough. Don't silently retry and append: the second attempt will start from the beginning, and the user would see the opening twice.
try:
for chunk in stream:
...
except Exception as err: # openai.APIError and friends
show_interrupted_banner(partial_text, err)What you pay for an interrupted stream
Billing follows what was actually generated. If the user closes the tab or your server cancels the request, you pay for the input and the output delivered so far, not the full max_tokens that was reserved. The rest of the reservation goes back to your balance. These requests show as partial in Logs so you can tell them apart. A request that fails before any output isn't charged at all.
Practical tips
- Cancel on disconnect. If your server proxies the stream to a browser, abort the upstream request when the client goes away. You stop paying for tokens nobody will read.
- Flush early. Buffering layers (some reverse proxies, compression middleware) can hold chunks back and undo the benefit. Disable buffering on the streaming route.
- Keep keys on the server. Stream from your backend to the browser; never put a Usefully key in client code.
- Set a sensible `max_tokens`. It caps both the worst-case cost and how long a runaway answer can go on.
The quickstart has a complete streaming example, and the API reference lists every chunk field. Questions: support@usefully.info.