Guide
Evaluating models on your own prompts
Leaderboards don't know your task. A small, repeatable evaluation with twenty real prompts tells you which model to use, and what it will cost.
· 3 min read · Usefully team
Public benchmarks are useful for a shortlist and nearly useless for a final decision. They measure someone else's tasks, in someone else's format. What you want to know is narrower: on your prompts, which model gives acceptable answers at the lowest cost and latency? With every model behind one API key, answering that takes an afternoon.
Step 1: collect real prompts
Take 20 to 50 inputs from real usage, or realistic drafts if you haven't launched. Include the awkward ones: very long inputs, ambiguous questions, inputs in other languages, cases where the right answer is "I don't know". A small set of hard cases tells you more than a large set of easy ones.
Step 2: decide what "good" means
Write it down before you look at outputs, or you'll grade generously. Depending on the task:
- Extraction and classification: exact match against a known answer. Easy to automate.
- Structured output: does it parse, and does it pass validation?
- Summaries and writing: a short rubric (accurate, complete, right length, right tone) scored by a person, or by a strong model as a judge with spot checks.
- Code: does it compile and pass the tests?
Step 3: run every candidate
Because the API is the same for every model, the evaluation is a loop over model IDs. Record the answer, tokens and time for each.
import csv, time
from openai import OpenAI
client = OpenAI(base_url="https://api.usefully.info/v1", api_key=USEFULLY_API_KEY)
MODELS = ["anthropic/claude-haiku-5-5", "anthropic/claude-sonnet-5-5", "openai/gpt-5-mini", "google/gemini-2.5-flash"]
with open("eval.csv", "w", newline="") as f:
out = csv.writer(f)
out.writerow(["model", "case", "ok", "input_tokens", "output_tokens", "seconds"])
for model in MODELS:
for case in CASES: # [{"id":..., "messages":..., "expected":...}]
t = time.time()
r = client.chat.completions.create(model=model, messages=case["messages"], temperature=0)
ok = grade(r.choices[0].message.content, case["expected"])
out.writerow([model, case["id"], ok, r.usage.prompt_tokens,
r.usage.completion_tokens, round(time.time() - t, 2)])Use a dedicated key with a daily spend limit for evaluation runs. It keeps eval spend separate in the dashboard and caps the cost of a bug in your loop.
Step 4: compare on three axes
- 1Pass rate. Discard models below your bar.
- 2Cost per request. Multiply average tokens by each model's prices from the models page, or read the costs straight from Logs.
- 3Latency. Median and slowest, from your timings or Logs.
The winner is usually the cheapest model that clears the bar, not the one with the highest score. If two models pass, the cheaper, faster one wins unless the quality gap matters to users.
Pitfalls
- Prompts tuned for one model. A prompt polished against one model can disadvantage others. Give each candidate a fair try at a neutral prompt.
- Too few hard cases. Easy cases make every model look the same.
- Randomness. Use
temperature=0for comparisons where possible, and run important cases more than once. - Judge bias. If a model grades outputs, check a sample by hand, and avoid judging a model with itself.
Keep it running
Save the cases and the script. When a new model appears on Usefully, or you change a prompt, rerun it. An evaluation you can repeat in minutes turns "should we switch models?" from a debate into a table.
Finally, share the results table with your team. A clear record of which models were tried, on which cases, and why one was chosen saves the same discussion from happening again in three months.
For guidance on the shortlist, read choosing a model. The dashboard playground is handy for trying a single prompt on several models before you script it.