Somewhere in this week's queue is a task that ran on a frontier API and could have run on a machine in your office. Which one? Most teams answer that by reading benchmark tables, and the tables cannot say. They rank models against public test sets. Your queue is private, narrow, and comes with its own files attached, so the question it poses is different from the question any benchmark answers.
The gap between the two questions used to be papered over by a large capability gap: local models trailed by enough that the details did not matter. That cushion is thinner now. In an interview published on November 19, Pete Warden, one of the longest-serving engineers in deep learning, put local open models within months of the big commercial ones on benchmark capability, and argued the real barrier has moved to usability. Two more open-weight models landed the same week, Radixark's DaisyRose-27B on November 18 and Tencent's Hy4 Flash on November 17. When the benchmark gap is months and shrinking, benchmarks stop settling the argument. Your own work has to.
A golden set is the cheapest way to make your own work do that. It is ten real tasks pulled from your recent queue, each with an answer you can check mechanically, run against a candidate model to decide where that class of work should live. The name borrows from gold-standard test data in machine learning. The habit is closer to an acceptance test: small, specific, and binding, with the verdict written down before anyone argues from vibes.
We run agents on our own hardware all day, so we have an interest in where this lands, and we will get to it near the end. The instrument itself needs nothing from us.
A benchmark measures general ability across a public distribution of tasks. Three things go missing on the way to your queue.
The distribution is wrong. Benchmark items are chosen to separate models from each other. Your items are chosen by your customers and your calendar, and they cluster tightly: the same invoice layouts, the same handbook, the same four report formats. A model can lag on olympiad math and sail through every task your ops queue produced this quarter. Nothing in the leaderboard tells you which situation you are in.
Context is the second mismatch. Benchmarks mostly test what a model can supply from its own weights. Office work mostly tests whether a model can follow what you hand it: the price sheet is attached, the policy document is in the folder. When the material rides along with the task, the model's stored knowledge matters less and its ability to read carefully matters more, and that ability saturates earlier in model size.
And the scoring hides the cost that actually hurts. A benchmark scores itself in milliseconds. Your real cost sits in the human minutes spent checking output, and that number is invisible to every leaderboard. A Harvard working paper that circulated in October 2026 looked at 718 companies using AI coding agents and found code volume up about 30 percent without matching output gains, while review time rose 49 percent. The bottleneck moved to the checking. An instrument for deciding where work runs has to price the checking in, which is exactly what the expected-answer column of a golden set does.
The set only works if every item can be scored without a meeting. Four check modes cover most queues, in descending order of how much you should lean on them.
Exact match. The answer is a value: a vendor ID, a due date, a reorder quantity. Extraction and calculation tasks live here, and they make the best golden items because a miss is unarguable.
Contains. The answer must include certain strings, in any phrasing: a retrieval question over your own handbook ("what is the refund window for annual plans?") checks as pass if the number of days and the one exception both appear.
Schema-valid. The output has to parse: the meeting transcript becomes action-item JSON that validates against a schema, the product table comes back as CSV with the right columns. Formatting work checks itself.
Rubric. A person scores the output against two or three written criteria. Drafting tasks end up here. Use this mode sparingly, because it reintroduces the human minutes the other three modes were saving, and write the rubric down anyway so two reviewers fail the same answer.
A workable rubric is shorter than people expect. For a customer-reply draft, ours is three lines: states the resolution in the first two sentences; quotes the actual policy clause rather than gesturing at it; contains nothing the customer did not ask about. Pass requires all three. The discipline is the same as the other modes, written before the run, binary per criterion, no points for style. If a criterion needs a debate to apply, it is two criteria, or it is taste, and taste does not belong in the set.
Picking items teaches you something before any model runs. A task you cannot write a check for is a task you have not finished defining, and that is useful to learn while it is still cheap. It usually means the task is judgment work wearing a routine task's clothes, and judgment work was never a local-first candidate to begin with.
Four steps, one afternoon.
Pull ten real items of one task type from last month's queue. Real, because synthetic prompts flatter models: they arrive clean, and your queue does not. Last month's, because the mess is representative: the scanned PDF, the vendor who renamed a column, the email with the forwarded thread stapled to it. One task type per set, because a mixed set produces a score that describes nothing in particular. Sample on purpose rather than at random: seven typical items, plus the two worst inputs the quarter produced, plus one known trap (the vendor whose name collides with another, the invoice with a credit line in it). A set of ten clean items certifies that the model handles clean items, which nobody doubted.
Freeze the materials. The local candidate gets exactly what the current flow gets: the same attachments, the same instructions, the same reference files. If you quietly improve the prompt while switching models, the comparison leaks, and you will not know which change did the work.
Write the expected answer next to each item before the first run. Afterwards, you will grade on a curve. The expectations go in a file, and the file goes in version control next to your prompts.
Run, score, record. Model name, quantization level, date, score. A score without its configuration becomes a rumor within a month.
The file format matters less than having one. This shape has worked for us:
task_type: invoice-extraction
model: "local-9b-q4" # record quantization; scores do not transfer across it
date: 2026-11-20
items:
- id: t01
input_ref: "queue/2026-10/inv-0441.pdf"
materials: ["vendors.csv"]
check: exact
expect: {"vendor_id": "V-209", "total": "4180.00", "due": "2026-12-01"}
- id: t02
input_ref: "queue/2026-10/inv-0457.pdf"
check: schema
expect_schema: "invoice.schema.json"
- id: t03
input_ref: "queue/2026-10/policy-q-112.txt"
materials: ["handbook-2026.md"]
check: contains
expect_contains: ["14 days", "annual plans excluded"]
python
import yaml
spec = yaml.safe_load(open("task_fit.yaml"))
passed = 0
for item in spec["items"]:
answer = ask_local(item) # your client here
passed += run_check(item, answer) # exact / contains / schema / rubric
print(f'{spec["task_type"]}: {passed}/{len(spec["items"])} on {spec["model"]}')
The quantization comment earns its place. Quantization stores each model weight in fewer bits so the model fits smaller memory; Warden's November 19 interview walks the arithmetic for a 9-billion-parameter model, roughly 9 GB at 8-bit and small enough for a 4 GB graphics card at Q4. A model that scores 9 out of 10 at one quantization level has told you nothing about a more aggressive one, so the score file records the level, and you test at the level you will actually run.
The lines encode one judgment: a task you must re-check more than a third of the time costs more in attention than it saves in fees. The fee side of that trade is small and public. Across current list prices, one agent task spans roughly $0.00086 to $0.0603 depending on the model (IntuitionLabs, 2026-09-05; InventiveHQ, 2026-10-19; assuming 4,000 tokens in, 800 out, 10% retries, 70% cache hits), which is why what one AI task costs rarely decides this question on its own. The attention side is the expensive one, and the Harvard review-time number above is what it looks like when nobody prices it.
So the top band tolerates one miss in ten only because the check that caught it in testing keeps running in production as a spot-check. The middle band is honest about its cost: every item gets a human signature, which pays off when review takes seconds (confirm three extracted fields) and burns money when review re-does the task (re-read the whole contract). If review costs what the task costs, 6 to 8 is a polite way of saying no. The bottom band is a verdict about today, with an expiry date attached.
One more rule that has saved us arguments: a task that scores 10 on the golden set but fails twice in its first production week goes back to the middle band without a meeting. The set is a sample. Production is the population.
When an item fails, sort the failure before touching anything, because the three causes point at three different fixes. A material problem means the input itself defeated the model: the scan too blurry to read, the attachment missing. That is a preprocessing fix or a routing rule (scans go elsewhere), and no model change is implied. An instruction problem means the model did the wrong thing confidently: read the European date as American, summarized when you wanted extraction. One added sentence in the task instruction usually clears it, and the fix is free, which is why you re-run before concluding anything. A capability problem is what remains when the input was clean and the instruction was explicit and the answer is still wrong. Only the third bucket argues against the model, and in our runs it is the smallest of the three, which is worth knowing before you let one failed run end the experiment.
Models now improve on a monthly cadence; two open-weight releases landed within 48 hours of each other this week alone. The set is how you notice without re-litigating. Same ten items, same checks, new line in the score log each quarter. A task moving from 7 to 9 across two quarters is a diff you can discuss in review, and the score log becomes the history of what your office learned to trust.
Two maintenance habits keep the instrument honest. When an item ages out, the vendor redesigned their invoice, the handbook chapter was rewritten, replace it and note the swap in the log; a set that silently drifts measures nothing. And resist upgrading the set to twenty items. Ten is the budget that keeps the quarterly re-run an hour instead of an afternoon, and an instrument that is cheap to run is an instrument that actually gets run.
Ownership settles the last argument. The set belongs to whoever owns the queue, the ops lead for invoices, the support lead for replies, because they are the ones who can tell a representative item from a convenient one, and they feel the standing cost of each band in their own week. Scores go wherever that team already posts its numbers. The day the set has no owner, it stops being re-run, and six months later someone quotes a stale verdict in a planning meeting as if it were current. An unmaintained golden set fails in the direction of overconfidence, which is the expensive direction.
The same ten-question habit, pointed at memory instead of tasks, is how you verify an agent's memory survived a model switch. Build it once, use it at both doors.
Consider an ops team that handles supplier invoices, forty a week, currently extracted by a cloud model and pasted into the ledger by hand. They pull ten October invoices, including the two scanned ones everyone hates, write expected values for vendor, total and due date (an hour and a half, mostly spent finding the ugly cases), and run them against a 9B model at Q4 on an office machine.
First run: 7 out of 10. The two scans fail on the total field, and one digital invoice fails because the model reads a European date format as American. That is the middle band, local drafts with a signature on each, and already workable since checking three fields takes seconds.
They make one change, an instruction spelling out the date convention, and re-run: 9 out of 10, with the remaining miss on the worst scan. Top band, with the spot-check watching for layout drift, and the scanned invoices routed separately. The whole exercise fit in an afternoon, and the verdict came out of their own paper, which is what makes it hold up in the next planning meeting.
The two scans get a decision of their own rather than a shrug. Either they go through OCR cleanup before the model sees them, and the preprocessed versions join next quarter's set as items, or the routing rule sends anything tagged as a scan to the stronger cloud model and the log records why. Both are fine answers. The bad answer is averaging them into the score and letting one input class quietly drag a whole task type back to the expensive path.
Numbers in this sketch are for the shape of the exercise; your pass rates will differ, which is the point of running it on your queue instead of ours.
Our own product, KarmaBox, is built for exactly this split: the routine work runs on a machine you own, and the hard cases go to built-in cloud models only when you approve the spend. Here is how the pieces above map onto it.
A golden answer, in production form, looks like this. In an internal test of our cross-border commerce avatar team (2026-11-15, test account), an avatar that had been taught a cost structure once took a single instruction and returned the landed-profit calculation in 28 seconds: $12.55 net per unit, break-even ACoS 41.84%, matching a manual recalculation. When any cost input is missing, it asks instead of estimating a range. Specific, numeric, checkable against a human redo: exactly the kind of item that belongs in a set, and exactly the kind of work the top band is for.
A verdict only matters if it sticks, so the boundary is enforced rather than suggested. Since the local-loop release (2026-10-22), a task pinned to run locally fails loudly if it cannot complete on the machine; it does not quietly fall back to a cloud call. Your golden-set verdict stays true after you stop watching, and a failed local run arrives as a report, which is itself a data point for the next retest. The behaviour is documented under what a sovereign AI avatar is, alongside our unplugged test (2026-10-28 to 30): 197 local inferences in one day with zero business data leaving the machine.
The bottom band has a path that does not involve provisioning anything. Cloud models are built into the platform, no API keys to set up; when a task would spend money on a cloud model, the avatar asks first and waits for approval. Moving a task between bands is a configuration change plus a yes or no from the person who owns the budget.
Volume, once a task passes, is a dial rather than a re-verdict. The same machine runs several tasks in parallel, and instances scale on demand; in the published small-business example, a team ran 2 to 3 parallel instances in normal weeks and turned the dial to 15 for the holiday rush, then back down after. The golden set certifies the task type once; the instance count is just how many copies of a certified worker you want this month.
And if you are deciding whether to own the machine at all, the honest operations ledger (hardware, electricity, the hour a month of upkeep) is in our self-hosted agent guide; the device itself runs the mid-size open models this whole exercise is about. If none of that fits your stack, the YAML, the bands and the quarterly re-run work the same without us.
A golden set addresses allocation risk: the cost of putting work in the wrong place. It does not raise the capability ceiling: a task every local candidate fails stays a cloud task, whatever your preferences, and no re-test schedule changes that until the models do. Ten items are a sample, so a rare input class that never made the set will fail in production first; the spot-check exists to catch it, and the set absorbs it as a new item afterwards. A set also scores one configuration at a time, so comparing two local candidates means two full runs of the same ten items, never one blended score. Scores age with model releases, which is why the verdict carries an expiry date. And the set says nothing about where data may sit or which actions need a human signature; capability was never the whole decision, and the three decisions no model makes for you stay yours in either band.
Which ten items from last month's queue would make your first set?
The Data Exchange, Pete Warden on Local AI, Open Models, Voice Interfaces, and the Future of Hardware, 2026-11-19
Radixark, DaisyRose-27B, 2026-11-18
Tencent, Hy4 Flash, 2026-11-17
ComplexDiscovery, Harvard Researchers Find AI Coding Agents Shifted the Bottleneck to Code Review, 2026-10
IntuitionLabs, LLM API Pricing: Cost per Task Comparison, 2026-09-05
InventiveHQ, LLM API Cost Comparison, modified 2026-10-19