01
Why "which model is best" has no answer, and what does
You should be able to: Reframe model choice as a per-task measurement rather than a global ranking.
There is no best model, only best-for-this-task-at-this-cost — the point the Choosing Inference course made. That reframing has a direct consequence for evaluation: a public leaderboard cannot answer your question, because it measures a general capability and you have a specific task. A model mid-table on a benchmark may be perfect for your classification job and overkill for your autocomplete.
So the unit of evaluation is your task, not the model. The question you are actually answering is narrow and answerable: for this specific job, at this quality bar, which of these candidate models pass, and which is cheapest among the ones that do? That is a measurement you can run in an afternoon, and it is worth more than any leaderboard because it is about the work you actually do.
Everything else in this course is machinery for answering that one question honestly. The machinery matters because the easy way to answer it — try a model, eyeball a few outputs, decide it "seems good" — is exactly how teams end up paying for a frontier model on a task a small one would pass, or shipping a cheap model that fails in a way nobody sampled.
Source: Why Choosing Inference Matters
02
Define the bar before you look at any output
You should be able to: Write a task-specific rubric that says what a passing answer is, in advance.
Write down what a correct answer looks like before you run a single model — the same discipline the estate applies to directing agents: define done before the work, so you check against a standard rather than an impression. For evaluation this rubric is the whole game, because a bar you set after seeing the outputs is a bar the outputs set for you.
A good rubric is specific to the task and checkable. For extraction: are the required fields present and correct? For a summary: does it keep the load-bearing facts and add none? For classification: does the label match? Where you can state the pass condition as something a script or a second model could check, you have a rubric you can run at scale rather than one you re-argue every time.
The rubric is also where you decide how good is good enough, which is a product decision, not a technical one. A moderation classifier needs a higher bar than a tag suggester. Setting that bar explicitly is what lets a cheaper model win honestly: it passes the bar the task actually needs, not the highest bar you could imagine.
Source: Directing Agents Well
03
The golden set — small, real, and labelled
You should be able to: Assemble an honest evaluation set that represents the work rather than flattering a model.
You evaluate against a golden set: a collection of real task inputs with known-good answers. It does not need to be large — a few dozen well-chosen cases usually separate the models that pass from the ones that do not — but it does need to be real and representative, drawn from the actual work rather than invented to be easy.
The cases that earn their place are the hard and the typical, not the convenient. Include the edge cases that break things: the ambiguous input, the adversarial one, the empty one, the one in the wrong format. A golden set of only easy cases will pass every model and tell you nothing, which is the evaluation equivalent of a test suite that asserts nothing.
Label the expected answers once, carefully, ideally by more than one person where judgement is involved — because the golden set is now your standard, and a wrong label is a wrong verdict applied to every model equally. The set is an asset: it outlives any single model, and re-running it against next month's new model is how you keep the routing decision current instead of frozen at whatever was true the day you first chose.
04
Run the comparison fairly, and score it the same way every time
You should be able to: Compare candidate models on identical inputs with a consistent scorer, and read cost alongside quality.
A fair comparison is the same inputs, the same prompt, through the same gateway, to each candidate model, scored the same way. A gateway makes this cheap: because switching model is a one-word change to the model id, you can run your golden set through a dozen models without touching anything else, which is exactly the leverage the Building with Gatewayz course describes turned toward measurement.
Scoring is where honesty is won or lost. Where the rubric is mechanical — an exact field, a label, a number — score it with code, which never gets tired or generous. Where it needs judgement, a capable model can act as a scorer against your rubric, but hold it to the same standard as any other measurement: a scorer you cannot spot-check is a scorer you cannot trust, so audit a sample of its verdicts by hand. Never let a model both produce and grade its own answer in the same pass.
Read cost next to quality, never alone. For each model, record the pass rate on the golden set and the cost per request from the token counts — the per-request attribution the build course teaches. Now the comparison is a table with two columns that matter, and the decision falls out of it: among the models that clear the bar, the cheapest wins.
Source: Building with Gatewayz
05
From verdict to routing — cost as a managed number
You should be able to: Turn an evaluation result into a routing rule, and keep it honest as models change.
The evaluation produces a decision, and the decision becomes a routing rule: this class of request goes to the cheapest model that passed its bar, escalating only where the eval showed the cheap model fails. Now the claim "we route to save money without losing quality" is not a hope — it is backed by a measurement you can show, on the tasks you actually run.
This is what makes cost a managed number rather than a mystery. When the bill moves, you can ask whether a routing rule changed or a model regressed, and re-run the golden set to find out. Without the eval you are guessing; with it you are reading. The measurement is the thing that turns the whole Inference Economy from a set of good ideas into an operation you can actually run.
Re-run it on a schedule and whenever a promising model appears, because the frontier moves: a model too weak or too expensive last quarter may clear the bar cheaply now, and only a re-run tells you. Hold every eval result to the estate's standard for any figure — a number you can raise at will by loosening the rubric is a gate, not a measurement — and the routing decision it feeds stays one you can defend to anyone who asks what a request cost and why.
Source: Inference as Yield
