Skip to content
Free$1 of credit for every new organization
All articles

Guide

Structured JSON output you can actually parse

Getting a model to return clean JSON every time takes more than asking nicely. JSON mode, schemas in the prompt, validation and a retry loop, on any model.

· 3 min read · Usefully team


Extraction, classification and agent steps all want the same thing from a model: a JSON object your code can read without guessing. Models are good at this, but "good" isn't "always", and the failure cases (a stray sentence before the brace, a trailing comma, a missing field) are exactly the ones that break a pipeline at 3am. Here's the approach we recommend, from cheapest to most robust.

1. Pick a model with JSON mode

Many models support a JSON mode that constrains output to valid JSON. The models page has a filter for it. Turn it on with response_format, exactly as you would with OpenAI's API:

from openai import OpenAI
import json

client = OpenAI(base_url="https://api.usefully.info/v1", api_key=USEFULLY_API_KEY)

reply = client.chat.completions.create(
    model="openai/gpt-5-mini",
    response_format={"type": "json_object"},
    messages=[
        {"role": "system", "content": "Return JSON with keys: company (string), amount_usd (number), due_date (YYYY-MM-DD)."},
        {"role": "user", "content": invoice_text},
    ],
)
data = json.loads(reply.choices[0].message.content)

JSON mode guarantees syntax, not shape. You still need to describe the fields you want, and still need to check them.

2. Put the schema in the prompt

Be explicit: list every key, its type, and what to do when a value isn't in the input. "Use null if the due date isn't stated" prevents the model from inventing one. A short example of a correct object helps more than a paragraph of rules.

  • Name keys the way your code names them, so you don't need a mapping layer.
  • Give enums as a closed list: "status": one of "paid", "unpaid", "overdue".
  • Ask for numbers as numbers, not strings with currency symbols.
  • Keep the object flat where you can. Deep nesting raises the error rate.

3. Validate, then retry once

Parse with a schema library and, if validation fails, send the error back to the model and ask for a corrected object. One retry fixes almost every miss; if the second attempt fails too, log it and move on rather than looping.

from pydantic import BaseModel, ValidationError
from typing import Literal

class Invoice(BaseModel):
    company: str
    amount_usd: float
    due_date: str | None
    status: Literal["paid", "unpaid", "overdue"]

def extract(text: str) -> Invoice | None:
    messages = [{"role": "system", "content": SCHEMA_PROMPT}, {"role": "user", "content": text}]
    for _ in range(2):
        reply = client.chat.completions.create(
            model="openai/gpt-5-mini",
            response_format={"type": "json_object"},
            messages=messages,
        )
        raw = reply.choices[0].message.content
        try:
            return Invoice.model_validate_json(raw)
        except ValidationError as err:
            messages += [
                {"role": "assistant", "content": raw},
                {"role": "user", "content": f"That didn't match the schema: {err}. Return the corrected JSON only."},
            ]
    return None

4. Or use tool calling

For models that support tools, defining a single function whose parameters are your schema is another reliable way to get structured arguments back. The model fills in the arguments as JSON, and you never execute anything: you just read them. See tool calling across providers.

Cost notes

  • Extraction is a good fit for smaller, cheaper models. Test a small model first and only move up if accuracy demands it.
  • Set max_tokens close to the size of the object you expect. A runaway answer then can't cost much.
  • A retry is a second request, billed normally. Logs show both, so you can see how often your retry path runs.
  • Failed requests (provider errors) aren't charged, so a provider hiccup doesn't double your cost.

Common mistakes

  • Parsing without stripping. Some models wrap JSON in a code fence when JSON mode isn't on. If you can't use JSON mode, strip a leading and trailing fence before parsing.
  • Trusting types. A field described as a number can still arrive as "12.50". Let your schema library coerce or reject it explicitly.
  • Huge schemas. Twenty optional fields invite the model to fill some with guesses. Ask only for what you'll use.

Try your schema prompt in the dashboard playground on two or three models before writing code, then compare cost per request in Logs. Model capabilities are listed on the models page.

More articles

Get started

Your first request is five minutes away.

  • $1 of free credit to start
  • No card, no subscription
  • Keep the SDK you already use
  • Real people at support@usefully.info
Structured JSON output you can actually parse · Usefully