{"slug": "the-measurement-was-harder-than-the-model", "title": "The Measurement Was Harder Than the Model", "summary": "A developer building a specialist AI agent to query Bureau of Labor Statistics data found that initial accuracy claims of 93% were invalid due to data leaks, stale evaluation files, and incorrect labels, with the true exact-match rate measured at 13.3%. The project, which distilled Qwen3-1.7B into a small model using MLX LoRA and 229 training examples, required redesigning the validation split to hold out phrasings rather than concepts and adding build-time assertions to prevent errors. The developer discovered that validation loss and actual tool-call accuracy disagreed violently, with the best exact-match score of 37.2% occurring at iteration 200 despite rising loss.", "body_md": "#\n[The Measurement Was Harder Than the Model\n](https://kovashikawa.github.io/ai/projects/distilling-bls-agent/)\n\n## The idea\n\nTake Qwen3-1.7B, give it a set of BLS economic data tools, and distill it into a\nspecialist small enough to run on an M4 Mini. Ask it “what happened to food\nprices since 2021?” and it should emit ```\nget_series(series_id=\"CUUR0000SAF1\",\nstart=\"2021\")\n```\n\n.\n\nThe pipeline: 205 hand-written seed questions over 82 economic concepts, 229 training examples, MLX LoRA, a 67MB adapter, six tools.\n\nThat part worked. Everything I initially believed about *how well* it worked was\nwrong, in several separate ways, and finding each one required disproving the\nprevious one. The measurement was harder than the model. The fix, when it\nfinally came, was not a hyperparameter but a change to what the model was asked\nto say.\n\n## Act 1: the number that wasn’t real\n\nRound one looked clean. Train loss fell from 3.4 to 0.027. Fifteen held-out examples, 87% accuracy. I fixed some data issues, retrained, got 93%. Base model scored roughly zero. Ship it.\n\nThen I ran a separate adversarial review of the pipeline, and none of it survived.\n\n**The held-out set was not held out.** I had split the seeds before expanding\nthem, which sounds right. But the expander drew from shared pools (a list of\nseries IDs, a list of search terms) regardless of which split a seed belonged\nto. So a training seed and a test seed could independently emit byte-identical\nrows. 34% of my validation set appeared verbatim in training. Worse, the eval\nfile itself was stale: it was the *first* round’s holdout, and the second round’s\nreshuffle had moved all 15 of those questions into training. The 93% was\nmeasured on training data.\n\n**The metric was the easy half.** “Accuracy” meant *did it pick the right tool*:\na 6-way classification. Whether the emitted call was actually correct was never\nscored.\n\n**A ninth of the labels were wrong.** Eleven series IDs named the wrong concept.\n`CUUR0000SETG01`\n\nwas mapped to “energy”; the BLS catalog calls it **airline\nfares**. `CUUR0000SAH1`\n\nwas “housing” but means *shelter*; `CUUR0000SAM1`\n\nwas\n“medical care” but means *medical care commodities*. Four IDs (`SAR1`\n\n, `SAC1`\n\n,\n`SAS1`\n\n, `SEHA01`\n\n) do not exist in any BLS catalog at all. I had invented them by\npattern-matching the real ones.\n\nHonest number, measured properly: **13.3% exact match.**\n\nThe lesson I’d draw is not “be careful.” It’s that *every one of these bugs\npushed the number up*. Nobody investigates a pleasant surprise as hard as a\ndisappointing one, and that asymmetry is the whole problem.\n\n## Act 2: testing the impossible\n\nFixing the leak exposed a deeper design error.\n\nThe split held out whole *concepts*. If “medical care” appeared only in the test\nset, the model was being asked to produce `CUUR0000SAM`\n\nhaving never once seen\nthat mapping. Four of eleven scored items were unanswerable by construction.\n\nThis is a lookup task. `\"medical care\" → CUUR0000SAM`\n\ncannot be derived from\nfirst principles; it can only be recalled. So the split should hold out\n**phrasings**, not concepts:\n\n- Every concept contributes at least one phrasing to training\n- Remaining phrasings go to val/test\n- The build\n**fails** if any concept is held out entirely\n\nI rewrote the seed data as concept tables (82 concepts, each with 2+ distinct phrasings) and added build-time assertions: every series ID must exist in the bundled 8,103-row catalog, no example may appear in two splits, no concept may be missing from train. Assertions, not intentions. Two of them have fired on me since.\n\n## Act 3: the discovery\n\nWith clean data I swept checkpoints from 100 to 1400 iterations, planning to pick the lowest validation loss like every tutorial says.\n\nVal loss bottomed at iteration 250 and rose steadily afterward. Textbook overfitting. But I was also scoring actual tool-call accuracy, and the two disagreed violently:\n\n| iter | val loss | exact match |\n|---|---|---|\n| 200 | 0.135 (min) |\n37.2% |\n| 400 | 0.135 | 65.1% |\n| 600 | 0.147 | 90.7% |\n| 800 | 0.156 | 81.4% |\n| 1000 | 0.169 | 88.4% |\n| 1400 | 0.169 | 74.4% |\n\nStopping at the val-loss minimum would have shipped a 37% model instead of a 91%\none. I wrote this up as a finding: *val loss is a trap for structured-output\ntasks.* Cross-entropy punishes a confidently-wrong series ID exactly as hard as\ngibberish, while a task evaluator sees a near-miss. I had citations lined up.\n\nIt was a config bug.\n\n## Act 4: disproving my own finding\n\nBefore publishing I checked one number I had never looked at: the ratio of completion length to prompt length.\n\n```\nmean prompt tokens     : 134\nmean completion tokens :  19\ngeneration ratio       : 0.141\n```\n\nEvery training row carried a ~120-token system prompt, byte-identical across the\ndataset, and a ~19-token tool call. And I was training with prompt loss\n**unmasked**.\n\n**87.7% of every gradient was the model re-predicting a fixed preamble.**\n\nThat explains all of it. Train loss of 0.027 was never impressive: most of it\nwas copying a constant. And validation loss was ~88% a measurement of *preamble\nreproduction*, which is uncorrelated with whether the tool call is right. The two\ncurves weren’t in tension for any deep reason. One of them was mostly noise.\n\nThis is documented. Huerta-Enochian and Ko (2024) found a statistically significant effect of prompt-loss weight specifically for short-completion data, and Vaughn’s walkthrough of the same phenomenon on a multiple-choice dataset (generation ratio 0.01) reports that stopping at the full-sequence val-loss minimum yields 53% accuracy while completion loss is still falling. I reproduced a known failure mode and mistook it for a discovery.\n\nSo I masked the prompt and reran. Prediction: val loss should re-couple with accuracy.\n\n| iter | val loss (masked) | exact match |\n|---|---|---|\n| 200 | 0.046 | 62.8% |\n| 400 | 0.028 | 86.0% |\n| 600 | 0.027 (min) |\n90.7% |\n| 800 | 0.028 | 90.7% |\n\n| unmasked | masked | |\n|---|---|---|\n| pick by val-loss minimum | 37.2% | 86.8% |\n| best checkpoint available | 90.7% | 88.4% |\npenalty for trusting val loss |\n~50 pts |\n1.6 pts |\n\nThe trap was self-inflicted. Mask the prompt and standard practice works fine.\n\nSwitching to chat-format data to enable masking fixed two other things I had been\ncarrying without noticing: Qwen3’s chat template emits an empty\n`<think></think>`\n\nblock, which is the *actual* mechanism for non-thinking mode;\nI had been asking for it in English in the system prompt, which is not the same\nthing. It also removed a stray leading space before every tool call that made the\ntraining target tokenize differently from anything an inference path would\nproduce.\n\n## Act 5: the thing that actually mattered\n\nHere is the result I should have led with, and it is the least exciting one.\n\nI had been quoting 90.7%. To check reproducibility I retrained with three different random seeds, changing nothing else:\n\n| seed | exact match |\n|---|---|\n| 0 | 76.7% |\n| 1 | 81.4% |\n| 2 | 90.7% |\n| 3 | 88.4% |\n\n**Mean 84.3%, standard deviation 6.4 points, range 14 points.**\n\n90.7% was not the result. It was the best of four draws. At this dataset size a single run tells you almost nothing, and every comparison I had made (including one where I concluded a data fix had caused a 14-point regression) was inside the noise. That “regression” was seed 0 landing at the bottom of the distribution. I nearly reverted a correct fix because of it.\n\nAfter the config fixes (prompt masking, all 28 layers instead of 16, rank 16, cosine schedule with warmup), across three seeds:\n\n**Mean 88.4%, standard deviation 4.0.** Better mean, tighter spread, and a\nvalidation signal that now points the right way.\n\n## An interlude on measurement\n\nSmall-data fine-tuning has terrible measurement ergonomics. Nearly every mistake\nin this project was a *measurement* mistake, not a modelling one:\n\n- A test set that wasn’t held out\n- A metric that scored the easy half of the task\n- Labels that were confidently wrong\n- A test that asked for things never taught\n- A loss dominated by a constant\n- Single-run numbers with 14 points of spread\n- Gold labels in a stale format, scoring correct answers as wrong\n\nThe model was never the hard part. None of these were visible from the loss\ncurve. The first six all made the number look *better* than reality, which is why\nthey all survived. The seventh made it look catastrophically worse, and I nearly\nabandoned a good idea because of it.\n\n## Act 6: the part hyperparameters couldn’t fix\n\nThe config work moved 84% to 88% and stalled. Every remaining failure was sibling\nconfusion between near-identical codes: `CUUR0000SAF11`\n\n(food at home) vs\n`CUUR0000SAF1`\n\n(food), `CUUR0000SAM`\n\n(medical care) vs `CUUR0000SEMD`\n\n(hospital\nservices). Those aren’t bugs. That’s what memorizing a codebook into weights looks\nlike at the margin, and no learning rate fixes it.\n\nI was training a 1.7B model to recall 13-character opaque strings where a one-character slip silently fetches a different economic series. Lookup tables don’t belong in weights. So before writing that as an opinion, I measured it.\n\n### Measuring the alternative\n\nFirst a correction to my own framing. I had been saying the catalog has 8,103\nseries, so memorization covers 0.4% of it. That number is misleading. The catalog\nis really **400 distinct items repeated across ~20 area and seasonal-adjustment\ncombinations**. “Housing” matches 120 titles because the same concept appears for\nUS city average, Northeast, New England, Chicago, seasonally adjusted and not.\n\nConstrained to the namespace every one of my seeds actually uses (US city\naverage, not seasonally adjusted): the corpus is **400 rows, and the item name is\na unique key**. That reframes the problem: not 8,103 codes to memorize, but 400\nwell-named items to *look up*. Still 11x more than the 35 I could afford to\nteach, but a completely different kind of problem.\n\nSo I built a ~30-line BM25 index over those 400 item names and measured recall of the correct series ID on the same 43 held-out questions:\n\n| query | recall@1 | recall@5 |\n|---|---|---|\n| oracle (the gold item’s own name) | 100% |\n100% |\nthe raw user question, no model at all |\n84.4% |\n93.8% |\n\nThe first row says the retriever has no ceiling problem: given a decent query it is perfect. The second row is the uncomfortable one. Throwing the user’s raw question at BM25 (no model, no training, no GPU, no adapter) retrieves the correct series 84.4% of the time, against 88.4% for the fine-tuned model. Those are within noise of each other.\n\nA week of distillation is currently tied with a text search over 400 strings.\n\nAnd the two questions raw BM25 misses are precisely the two my fine-tuned model gets wrong:\n\n```\nwant  Medical care                          from \"Show me healthcare CPI...\"\nwant  Owners' equivalent rent of primary…   from \"What did OER do between…\"\n```\n\nBoth are vocabulary gaps: “healthcare” and “OER” don’t appear in the official item names. So the obvious move is to have the model rewrite the user’s words into catalog vocabulary and let retrieval do the lookup. Much easier than emitting a 13-character code from memory, and it generalizes to all 400 items.\n\nI tested that too, and it does not work yet:\n\n| query source | recall@1 | recall@5 |\n|---|---|---|\n| oracle (gold item name) | 100% | 100% |\nraw question, no model at all |\n84.4% |\n93.8% |\n| base Qwen3-1.7B rewrites the question | 75.0% | 84.4% |\nmy fine-tuned model rewrites it |\n62.5% |\n71.9% |\n\nEvery model in the chain makes retrieval *worse*. The base model degrades good\nqueries. Asked about “tuition, other school fees, and childcare”, which is\nverbatim the official item name, it helpfully rewrites it to “Education\nexpenses”. Asked about OER it expands the acronym to “Educational Resources\n(OER)”, which is a real term from a different field entirely.\n\nMy fine-tuned model is worse still, and the reason is the interesting part. Asked for a search phrase, it emits series IDs:\n\n```\n\"What did education and communication prices do…\"  →  CUUR0000SAEDECU01\n\"Show me tuition and childcare CPI…\"               →  CUUR0000SEEB\n\"Show me healthcare CPI…\"                          →  search_phrase:CUUR0000SEMD Healthcare\n```\n\nIt has specialized so hard on emitting codes that it can no longer paraphrase.\nNote the second line: `CUUR0000SEEB`\n\nis the *correct answer*. The model knows the\nmapping. It just can’t express it in a form the retriever can use, because the\nindex is over item names and it only speaks in IDs.\n\nThat is a useful negative result. It means retrieval cannot be bolted onto this adapter: the memorization fine-tune destroyed the exact capability retrieval depends on. The two-step system has to be trained from base, on traces that include the search step, and Anthropic’s caution about small models and query formulation is now something I’ve measured on my own data rather than quoted.\n\n### Act 7: the fix was the output format\n\nThe agent ecosystem has converged on retrieval for a structurally identical\nproblem one level up. Hermes Agent, mcp-sieve, and various tool-router plugins\nall replace “load every tool schema” with a `search`\n\n→ `describe`\n\n→ `call`\n\nbridge. Anthropic’s MCP evaluations show accuracy *improving* when tools are\ndeferred rather than preloaded (49% → 74% on Opus 4) because large catalogs\ncause decision paralysis.\n\nTheir bottleneck is too many tools. Mine was 400 values in one argument slot. So\nI was about to build the two-step system, and then noticed I could get most of\nthe benefit by changing one thing: **what the model is asked to emit.**\n\n```\nget_series(series_id=\"CUUR0000SAF11\")   →   get_series(item=\"Food at home\")\n```\n\nThe model names the item; forty lines of code resolve the name to an ID. This\nworks only because of the 400-item finding above: item name is a *unique key*\nin that namespace, so the resolution is deterministic, not a guess.\n\nWhy it helps is the same reason the failures were what they were. `SAF11`\n\nvs\n`SAF1`\n\nis a one-character discrimination. *“Food at home”* vs *“Food”* is a\nsemantic one, which is the kind of distinction a language model is actually built\nto make. And a name fails *loudly*: `resolve_item`\n\nreturns `None`\n\nfor something\nit can’t place, where one wrong character in a code silently fetches a different\nseries and returns plausible numbers.\n\nFive seeds, same 43 held-out phrasings:\n\n| target format | exact | sd |\n|---|---|---|\n`get_series(series_id=\"CUUR0000SAF11\")` |\n88.8% | 3.0 |\n`get_series(item=\"Food at home\")` |\n92.1% | 1.3 |\n+ hierarchy-aware resolver |\n94.4% |\n1.3 |\n\n**+5.6 points, t = 3.8, p ≈ 0.005**, and seed variance more than halved.\n\nExact match on 43 held-out phrasings\n\nSame held-out set throughout. The originally reported 93% is not shown — it was measured on training data. The final row (+alias) is the same model with the resolver alias table; see Act 8 on why it is not the headline number.\n\n**base Qwen3-1.7B** no fine-tuning\n\n**V2, honestly measured** the real starting point\n\n**BM25 over 400 items** no model at all\n\n**+ prompt masking, r16** config fixes\n\n**+ item-name targets** output format\n\n**+ hierarchy resolver** current\n\n**+ alias table**†see Act 8\n\n## Table view\n\n| Stage | Exact | sd | Kind |\n|---|---|---|---|\n| base Qwen3-1.7B | 9.3% | — | baseline |\n| V2, honestly measured | 13.3% | — | fine-tuned |\n| BM25 over 400 items, no model | 84.4% | — | baseline |\n| + prompt masking, all layers, rank 16 | 88.8% | ±3.0 | fine-tuned |\n| + item-name targets | 92.1% | ±1.3 | fine-tuned |\n| + hierarchy-aware resolver | 94.4% | ±1.3 | fine-tuned |\n+ alias table † | 96.7% | ±1.3 | fine-tuned |\n\nThe hierarchy rule is small: when a bare term is a whole-word prefix of several\ncatalog entries, prefer the general one: “tobacco prices” means *Tobacco and\nsmoking products*, not *Tobacco products other than cigarettes*.\n\nThe only remaining failures at this point are “healthcare” → *Medical care* and\n“OER” → *Owners’ equivalent rent*. Both are vocabulary gaps the resolver cannot\nbridge without knowing which official term the user meant. The alias table was\nthe obvious fix, and the obvious problem: both failures are visible from\nreading the test set.\n\n**And the first run of this experiment reported 23.3%.** A catastrophic\nregression that would have killed the idea. It was a scoring bug: the held-out\nfile still stored raw `series_id`\n\nwhile the model now emitted item names, so\nevery *correct* answer scored wrong. I caught it because 10 of 43 passed, and\nexactly 10 test items have no series ID. Gold is now rendered by the same\nfunction that builds training targets, so the two cannot drift again.\n\nThat’s the second phantom regression in this project. Both times the number said “your change broke it” and both times the number was the broken thing.\n\nA note on the “100% accuracy” claims circulating in the tool-router ecosystem: I read them. One is 10 out of 10 test cases. Another is 100% on synthetic validation data generated the same way as its training data. Both are the same species of number as my original 93%. Copy the architecture; don’t quote the benchmarks.\n\n### Act 8: the alias table\n\nThe two failures were “healthcare” → *Medical care* and “OER” → *Owners’\nequivalent rent*. I said I was leaving them in. Then I shipped the alias table.\n\nThe number, same five adapters, model untouched:\n\n| exact | sd | |\n|---|---|---|\n| fine-tuned, item names | 94.4% | 1.3 |\n| + resolver alias table | 96.7% | 1.3 |\n\nThe accounting matters. There are roughly 50 aliases in the table: groceries,\ngas, core CPI, airfare, public transport. Against this eval they move nothing;\nits 43 questions don’t use that vocabulary. The entire +2.3pp is `healthcare`\n\n.\nThat alias was added knowing it was one of my two held-out failures.\n\nWhich makes it contaminated by the definition this post uses. I reported 94.4% as the model’s number and put the alias row in a footnote.\n\nThe tension is that `healthcare`\n\nis simultaneously the most obvious synonym a\nreal user types and one of my two known test failures. Omitting it from a\nuser-facing resolver to protect a benchmark would be absurd. So no option was\nboth maximally useful and maximally clean.\n\nWhat that means more broadly: once you’ve read your test failures, you can’t un-read them. The honest options narrow permanently. The stricter path was to build the alias table from general domain vocabulary, validate it on the val split, and leave test untouched. I didn’t do that. I reported both numbers and disclosed which one was contaminated. Honest, but not maximally rigorous, and the distinction is real.\n\nOER is still failing. The model emits `item=\"Education and communication\"`\n\n, a\nhallucination the resolver cannot repair. That is a model error, not a vocabulary\ngap, and an alias does nothing for it.\n\nOne item remains genuinely open: enumerate all 400 names in context and let the model select from a visible list rather than recall from memory. At 2,274 tokens it fits, and it removes the lookup problem entirely for anything the model can phrase correctly.\n\n## What I would do differently\n\n**Check your generation ratio before anything else.** If completions are short\nrelative to prompts, mask the prompt. Otherwise your loss is mostly measuring\nwhether the model can copy a constant, and any conclusion you draw from that\ncurve is suspect.\n\n**Report a distribution, not a run.** Three seeds minimum, five if the difference\nmatters. At n=43 with 6 points of seed variance, a single number is close to\nmeaningless. If you only rerun the results you dislike, you will systematically\npublish your luckiest draws.\n\n**Write assertions, not intentions.** “The splits are disjoint” is a claim. A\nbuild that fails when they aren’t is a guarantee. Mine now refuses to run if any\nseries ID is absent from the catalog, if any example appears twice, if any\nconcept is missing from training, or if any label mentions dates its question\ndoesn’t.\n\n**Split by what the model must actually generalize over.** For a lookup task,\nhold out phrasings. Holding out concepts measures the impossible.\n\n**Ask what you’re making the model say, not just how you’re training it.** The\nsingle largest improvement in this project (+5.6 points, variance halved) came\nfrom changing the output format from an opaque ID to a name, not from any\nhyperparameter. Errors that are one character apart are hard for a language\nmodel; errors that are semantically apart are easy. And a name can fail loudly\nwhere a code fails silently.\n\n**Measure the dumb baseline first.** Thirty lines of BM25 over 400 strings scores\n84.4% here. I was five days in before I knew that, and for all five of those days\nI was comparing my results against nothing. If the baseline had come out ahead,\nthe right answer would have been to delete the model.\n\n**Be most suspicious of results you like.** Six of the seven bugs above inflated\nmy numbers. That is not coincidence: unpleasant surprises get debugged, pleasant\nones get published.\n\n**Once you’ve read your test failures, you can’t un-read them.** The honest\noptions narrow. If you’re building something that touches items you know failed,\nvalidate it on val, not test. I didn’t; I reported both numbers and disclosed the\ncontamination. Honest, not maximally rigorous. The difference matters.\n\n## Results\n\n43 held-out phrasings, none seen in training, every referenced concept present. Mean over five training seeds, +/- one standard deviation across seeds:\n\n| tool | entity | exact | |\n|---|---|---|---|\n| base Qwen3-1.7B | 72.1% | 9.3% | 9.3% |\n| fine-tuned, emitting series IDs | 99.1% +/- 1.3 | 89.8% +/- 2.1 | 88.8% +/- 3.0 |\nfine-tuned, emitting item names |\n99.1% +/- 1.3 |\n94.4% +/- 1.3 |\n94.4% +/- 1.3 |\n| + resolver alias table | 99.1% +/- 1.3 | 96.7% +/- 1.3 | 96.7% +/- 1.3 † |\n| raw BM25 over 400 items, no model | – | – | 84.4% |\n\n**tool**— correct function chosen, out of six** entity**— plus the load-bearing argument (series ID / query / survey)** exact**— every argument identical\n\n† Same five adapters, model untouched. The entire +2.3pp is the\n`healthcare`\n\nalias, added knowing it was a held-out failure. See Act 8.\n\nA 67MB adapter on an M4 Mini, for the 35 concepts it was taught. I want to resist rounding that 99.1% to “100%”; four of five seeds hit 43/43 and the temptation to quote the clean number is exactly the reflex this whole exercise was about.\n\nWorth stating the cost honestly: the config fix took the adapter from 19MB to 67MB, because it raised LoRA rank from 8 to 16 and adapted all 28 transformer blocks instead of the last 16. So +4.5 points of exact match came with 3.5x the adapter size. I did not sweep that tradeoff; rank 8 across all layers might get most of the gain at half the size, and I have not checked.\n\nThe third row is there because a serious writeup should include the baseline that embarrasses it.\n\nOne last operational note, learned the annoying way: I ran an evaluation concurrently with a training job on the same GPU and got different numbers for identical weights. Not non-determinism (two isolated runs are byte-identical), but under memory pressure MLX returned different results rather than simply running slower. I nearly wrote up a phantom regression from it. Score on a quiet machine.\n\nCode: [github.com/kovashikawa/bls_data](https://github.com/kovashikawa/bls_data).\n\n## References\n\n- Huerta-Enochian, M. & Ko, S. Y. (2024). “Instruction Fine-Tuning: Does Prompt\nLoss Matter?” EMNLP 2024.\n[arXiv:2401.13586](https://arxiv.org/abs/2401.13586) - Guo, H., Dennis, S., Patil, R., & Shabahang, K. (2026). “When Mean CE Fails:\nMedian CE Can Better Track Language Model Quality.”\n[arXiv:2605.24667](https://arxiv.org/abs/2605.24667). Finds mean cross-entropy rising while held-out accuracy stays near peak, in Qwen2.5-1.5B SFT. - Apicella, A., Isgrò, F., Pollastro, A., & Prevete, R. (2026). “Don’t stop me\nnow: Rethinking Validation Criteria for Model Parameter Selection.”\n[arXiv:2602.22107](https://arxiv.org/abs/2602.22107). Finds early stopping on validation*accuracy*performs worst for neural classifiers, favouring loss-based criteria. Worth reading against this post: their objection is to early stopping specifically, and they find post-hoc selection across all epochs comparable to loss-based selection. - Chatterjee, A., Renduchintala, H. S. V. N. S. K., Bhatia, S., & Chakraborty, T.\n(2025). “On the Effect of Instruction Tuning Loss on Generalization.”\n[arXiv:2507.07817](https://arxiv.org/abs/2507.07817). Finds fully masking prompt tokens is rarely optimal either; a low-to-moderate prompt weight usually beats both extremes. - Vaughn, D. (2024). “To Mask or Not to Mask: The Effect of Prompt Tokens on Instruction Tuning.” Towards Data Science.\n\n#### Share on\n\n[X](https://x.com/intent/tweet?text=The+Measurement+Was+Harder+Than+the+Model%20https%3A%2F%2Fkovashikawa.github.io%2Fai%2Fprojects%2Fdistilling-bls-agent%2F)\n\n[Bluesky](https://bsky.app/intent/compose?text=The+Measurement+Was+Harder+Than+the+Model%20https%3A%2F%2Fkovashikawa.github.io%2Fai%2Fprojects%2Fdistilling-bls-agent%2F)", "url": "https://wpnews.pro/news/the-measurement-was-harder-than-the-model", "canonical_source": "https://kovashikawa.github.io/ai/projects/distilling-bls-agent/", "published_at": "2026-07-27 00:00:00+00:00", "updated_at": "2026-07-29 21:22:00.066106+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-agents", "ai-tools", "ai-research"], "entities": ["Qwen3-1.7B", "Bureau of Labor Statistics", "MLX LoRA", "M4 Mini"], "alternates": {"html": "https://wpnews.pro/news/the-measurement-was-harder-than-the-model", "markdown": "https://wpnews.pro/news/the-measurement-was-harder-than-the-model.md", "text": "https://wpnews.pro/news/the-measurement-was-harder-than-the-model.txt", "jsonld": "https://wpnews.pro/news/the-measurement-was-harder-than-the-model.jsonld"}}