{"slug": "my-fine-tuned-model-scored-100-the-benchmark-was-lying", "title": "My fine-tuned model scored 100%... The benchmark was lying", "summary": "A developer fine-tuned Mistral 7B on a laptop to detect personal data in log lines and support messages, achieving a perfect 100% score on an initial test set. However, after rebuilding the test set from real public data, the fine-tuned model dropped to 95% while few-shot prompting collapsed to 66%, revealing that the original benchmark was misleading. The developer details the LoRA fine-tuning process, dataset construction, and reproducible commands on Apple Silicon Macs.", "body_md": "I fine-tuned Mistral 7B on my laptop to detect personal data in log lines and support messages. On my first test set it scored 100%. Perfect. Every single line classified correctly.\n\nI did not publish that number, because the same test set gave few-shot prompting 94%, and a six-point gap over a prompt you can write in five minutes is not a reason to fine-tune anything. The honest conclusion looked like \"this was a waste of an afternoon.\"\n\nThen I threw my test set away and rebuilt it from real public data. The fine-tune dropped to 95%. Prompting collapsed to 66%.\n\nSame model, same code, same training recipe. A 6 point gap became a 29 point gap, and the conclusion flipped completely. My benchmark had been choosing my answer for me, and it had chosen wrong.\n\nThis article is the whole run: what LoRA actually does, how to build a dataset that does not lie to you, the exact commands, and the measured results. Everything is reproducible from [the repository](https://github.com/jguillaumesio/lora-pii-detection-mlx) on any Apple Silicon Mac.\n\nWorth settling first, because \"LoRA is not really fine-tuning\" comes up constantly.\n\nIt is. LoRA trains the model with gradient descent on your data exactly like full fine-tuning. The difference is which weights move. Instead of updating all 7 billion parameters, it freezes them and learns two small low-rank matrices per targeted layer. Their product approximates the weight update that full fine-tuning would have made, and it can be merged back into the base weights afterwards, giving you a genuinely different model.\n\nIn my run, that meant **0.145% of the parameters were trainable**: 10.5 million out of 7.25 billion. That is the entire reason this fits on a laptop, and why the output is a 42 MB adapter file instead of a new 4 GB model.\n\nWhat it is not is prompting or retrieval. The weights actually change. The honest caveat is that full fine-tuning can push a bit further on hard tasks, at ten to a hundred times the memory cost. For teaching a model a format, a taxonomy, or a behaviour, LoRA is what practitioners actually ship, and it is what the fine-tuning APIs from the major providers run under the hood.\n\nThe model gets one line of text, in English or French, and must answer with strict JSON:\n\n```\n{\"pii\": true, \"types\": [\"email\", \"name\"]}\n```\n\nSix types: `email`\n\n, `phone`\n\n, `name`\n\n, `iban`\n\n, `address`\n\n, `dob`\n\n. Empty list when there is nothing.\n\nThis is a real problem, not a toy. Personal data leaks into logs, staging dumps and exports far beyond your users table, which I wrote about in [PII and data masking](https://jguillaumesio.com/blog/pii-data-masking/), and knowing where it is is the precondition for [retention and deletion](https://jguillaumesio.com/blog/gdpr-data-retention-and-deletion/) that actually works.\n\nHardware: a MacBook with an Apple M5 and 16 GB of unified memory. Base model `mlx-community/Mistral-7B-Instruct-v0.3-4bit`\n\n, running on [MLX](https://github.com/ml-explore/mlx-lm), Apple's array framework. No cloud, no API keys, total cost 0 EUR.\n\nEvery tutorial shows you `mlx_lm.lora --train`\n\n. That command is four minutes of work. The dataset is the other three hours, and it is where the result is actually decided.\n\nI generated 800 examples from my own templates: log lines with fake emails, support messages with fake IBANs, plus hard negatives full of UUIDs and invoice numbers so the model could not just flag anything that looks like an identifier.\n\nIt gave the fine-tune a perfect score. The reason is obvious in hindsight: **the test set came from the same templates as the training set.** The model had seen every sentence shape before. I was measuring memorisation of my own imagination.\n\nIf your fine-tune scores 100%, your test set is too easy. That is not a nice problem to have, it is a broken measurement.\n\nSo I went looking for public corpora, and immediately walked into two problems that are worth more than the rest of this article.\n\nThe obvious choice is `ai4privacy/pii-masking-200k`\n\n, which everybody cites. Its licence is dual: free for individuals, non-profits, and companies with three staff or fewer, paid otherwise. A blog that markets a consulting practice is commercial use. Depending on your situation you may be fine, but \"everybody uses it\" is not a licence.\n\nFor negatives I grabbed [LogHub](https://github.com/logpai/loghub), 32,000 lines of real production logs from 16 systems. Two problems. Its licence covers \"research or academic work\" only. And, far worse for an article about detecting personal data, several of those systems contain real personal data:\n\n`/home/`\n\npaths from national laboratoriesMy screening regex caught four email addresses and missed all of it, because a username inside a file path does not look like contact data. I had built a training set that teaches a model that real people's names are **not** personal data. That is worse than no model at all.\n\nBoth sources were thrown out. The final pipeline uses two Apache-2.0 corpora: [kiji](https://huggingface.co/datasets/DataikuNLP/kiji-pii-training-data) for positives, which covers all six types including IBAN and date of birth in English and French and ships its own train/test split, and [witfoo syslog](https://huggingface.co/datasets/witfoo/syslog-to-artifact) for negatives, 155,000 lines of real firewall and system logs.\n\nHere is the trap that would have quietly ruined everything. Positives are business prose. Negatives are raw syslog. A model can separate those two by **writing style** and score brilliantly without ever learning what personal data is.\n\nSo every quadrant has to exist:\n\n| Positive (has PII) | Negative (no PII) | |\n|---|---|---|\nProse |\nkiji sentences | same sentences, PII replaced by role words |\nLog |\nreal syslog with real PII injected | real syslog, untouched |\n\nThe prose negatives are built by rewriting each annotated span into a generic role word, so \"contact Alice Dupont at [alice@example.fr](mailto:alice@example.fr)\" becomes \"contact the customer at the support address\". Same sentence structure, same vocabulary, same language, no personal data.\n\nThen I found the leak inside my own fix. Filler phrases like \"the customer\" appeared **only** in negatives, so they became a perfect giveaway. The model could learn my filler vocabulary instead of the task. So 10% of the rows are hybrids: role words everywhere, except one real value left in. Now the filler carries both labels and is useless as a signal.\n\nPublic datasets are not clean. Screening mine turned up:\n\n**Blanket filtering costs three quarters of the corpus.** Most kiji rows mention a passport or an SSN somewhere, types I do not model. Dropping those rows left 4,180 usable. Replacing just those spans with role words instead left 15,810.\n\n**The annotations are incomplete.** Kiji has a `coreferences`\n\nfield, and it is empty on every single row. So a later mention survives: \"Alice Dubois\" is annotated, but \"Dubois a signé\" three sentences down is not, and that real surname lands inside a \"no personal data\" example. Fix: screen every negative against a 1,519-token name vocabulary harvested from the corpus itself.\n\n**4.7% of the French rows are encoding-corrupted.** Accented characters arrive as NUL bytes, so `étude`\n\nis stored as `\\x00tude`\n\n. Train on those and you are teaching the model mojibake French. 406 rows dropped.\n\n**4% of the syslog carries account names in prose**, like `Accepted password for johndoe`\n\n. Too few to be worth parsing, so they are dropped wholesale rather than mislabelled as clean.\n\nFinal dataset: 8,000 train, 448 validation, 800 test. 4,572 positives against 4,676 negatives. 2,544 French rows. Zero overlap between splits, verified.\n\nMy first smoke test crashed:\n\n```\njinja2.exceptions.TemplateError: Conversation roles must alternate user/assistant/user/assistant/...\n```\n\nMistral's chat template **rejects a standalone system role**. Unlike Llama or Qwen, it wants strictly alternating user and assistant turns. So the instructions have to ride on the first user turn:\n\n```\n{\n  \"messages\": [\n    {\"role\": \"user\", \"content\": f\"{SYSTEM_PROMPT}\\n\\nLine: {text}\"},\n    {\"role\": \"assistant\", \"content\": '{\"pii\":true,\"types\":[\"email\"]}'}\n  ]\n}\n```\n\nThe dangerous part is not the crash. It is that if you fix this in your evaluation script and forget your dataset builder, training and inference use different prompt shapes, and your adapter looks broken for no visible reason. Build both from the same function.\n\n```\npython -m mlx_lm lora \\\n  --model mlx-community/Mistral-7B-Instruct-v0.3-4bit \\\n  --train --data ./data \\\n  --fine-tune-type lora \\\n  --batch-size 4 --num-layers 16 --iters 500 \\\n  --learning-rate 1e-5 --max-seq-length 512 \\\n  --mask-prompt --grad-checkpoint \\\n  --steps-per-report 50 --steps-per-eval 250 --save-every 500 \\\n  --val-batches 25 --seed 42 \\\n  --adapter-path ./adapters 2>&1 | tee training.log\n```\n\nWhy these values:\n\n** --mask-prompt is the one you must not skip.** It computes the loss on the answer only, not on the input line. Without it, most of the tokens the model is learning to predict are the log line itself, which is not the task.\n\n** --batch-size 4 --num-layers 16** came from measurement, not guesswork. A first run at batch 1 with 8 layers peaked at 4.8 GB on a 16 GB machine, so there was room to double both.\n\n** --learning-rate 1e-5** is the consensus for small datasets. 1e-4 oscillates, 1e-6 barely moves.\n\n** --grad-checkpoint** trades compute for memory, and\n\n`--save-every`\n\nThe loss curve is the interesting part:\n\n| Iteration | Validation loss |\n|---|---|\n| 1 | 3.537 |\n| 250 | 0.508 |\n| 500 | 0.491 |\n\n**Almost everything happens in the first 250 iterations.** The next 250 bought a 3% improvement, so I stopped there rather than running the 2,000 I had planned. If you take one operational lesson: watch validation loss and stop when it flattens, because \"train longer\" is mostly a way to spend electricity.\n\nFinal cost: **45 minutes, 6.0 GB peak memory, a 42 MB adapter, 0 EUR.**\n\nEvery mode uses the same prompts, the same test set and temperature 0. \"Few-shot\" means six worked examples in the prompt, which is what a sensible engineer tries before reaching for training.\n\n| Metric | Zero-shot | Few-shot (6) | LoRA |\n|---|---|---|---|\n| Accuracy | 66% | 66% | 95% |\n| Precision | 0.639 | 0.610 | 0.926 |\n| Recall | 0.686 | 0.840 | 0.974 |\n| F1 | 0.662 | 0.707 | 0.950 |\n| False positives | 75 | 104 | 15 |\n| Missed PII | 61 | 31 | 5 |\n| Valid JSON | 100% | 100% | 100% |\n| Seconds per line | 0.83 | 1.67 | 0.91 |\n\nLook at few-shot's precision: adding six examples made it **worse** than zero-shot, 104 false positives against 75. It found more personal data and cried wolf far more often.\n\nThe per-type breakdown shows why:\n\n| Type | Zero-shot | Few-shot | LoRA |\n|---|---|---|---|\n| 0.745 | 0.782 | 1.000 |\n|\n| phone | 0.628 | 0.575 | 0.983 |\n| name | 0.531 | 0.663 | 0.855 |\n| iban | 0.358 | 0.194 |\n0.950 |\n| address | 0.500 | 0.597 | 0.914 |\n| dob | 0.383 | 0.366 | 0.875 |\n\nIBAN is the story. Six examples cannot teach a model the boundary between an IBAN, an invoice reference and a `whsec_`\n\nwebhook secret across two languages. Few-shot scores 0.194 there, worse than saying nothing. The fine-tune reaches 0.950, because 4,000 examples of that boundary is what it takes.\n\nI also wrote 30 lines by hand, in phrasings the corpora never produced, as a final honesty check:\n\n| Metric | Zero-shot | Few-shot | LoRA |\n|---|---|---|---|\n| Accuracy | 80% | 90% | 100% |\n| False positives | 2 | 1 | 0 |\n| Missed PII | 4 | 2 | 0 |\n\nThe fine-tune got all 30. Small sample, so I would not put \"100%\" on a slide, but it did not fall apart off-distribution, which was the real question.\n\nThe fine-tune is **1.8x faster per line than few-shot**, 0.91 seconds against 1.67. The six examples are gone from the prompt, so every single inference is shorter, forever. Better and cheaper is a rare combination.\n\nFine-tuning teaches behaviour, format and taxonomy. It does not teach facts.\n\nIf your problem is \"the model does not know our internal documentation\", fine-tuning is the wrong tool and retrieval is the right one. Facts change; weights do not. You will retrain forever and still get confident wrong answers.\n\nAnd prompting deserves a fair trial first. On my synthetic dataset it genuinely tied the fine-tune. It only collapsed when the task got hard enough that six examples could not express the rules. That threshold is the actual decision point, and you cannot find it by reading blog posts, including this one. You find it by measuring both, which costs an afternoon.\n\nThe technical work here was easy. MLX is excellent, the command is one line, and it ran on a laptop while I did something else.\n\nThe hard part, and the part that decided the outcome, was the data: the licence that quietly excludes commercial use, the \"clean\" log corpus full of real usernames, the empty coreference field, the NUL-corrupted French, and the style shortcut that would have handed me a beautiful meaningless number.\n\nI got two completely different answers to the same question, on the same model, on the same day. The only thing that changed was the quality of what I measured against. Before you trust any fine-tuning result, including the ones in this article, ask what the test set is made of.\n\nEverything is public, including the training log and the raw predictions:\n\n```\ngit clone https://github.com/jguillaumesio/lora-pii-detection-mlx\ncd lora-pii-detection-mlx\npython3 -m venv .venv && .venv/bin/pip install mlx-lm datasets\n.venv/bin/python build_dataset.py\n```\n\nThe dataset is not committed. It rebuilds from the two Apache-2.0 corpora with that one command, so nothing is redistributed that should not be. The 30 hand-written test lines, the results and the training log are all in the repo.\n\n*Originally published on jguillaumesio.com. I write about payments, AI agents in production, and running SaaS infrastructure without a platform team.*", "url": "https://wpnews.pro/news/my-fine-tuned-model-scored-100-the-benchmark-was-lying", "canonical_source": "https://dev.to/jguillaumesio/my-fine-tuned-model-scored-100-the-benchmark-was-lying-48fo", "published_at": "2026-08-11 20:09:08+00:00", "updated_at": "2026-08-11 20:47:08.402858+00:00", "lang": "en", "topics": ["machine-learning", "large-language-models", "developer-tools"], "entities": ["Mistral 7B", "MLX", "Apple", "LoRA", "ai4privacy"], "alternates": {"html": "https://wpnews.pro/news/my-fine-tuned-model-scored-100-the-benchmark-was-lying", "markdown": "https://wpnews.pro/news/my-fine-tuned-model-scored-100-the-benchmark-was-lying.md", "text": "https://wpnews.pro/news/my-fine-tuned-model-scored-100-the-benchmark-was-lying.txt", "jsonld": "https://wpnews.pro/news/my-fine-tuned-model-scored-100-the-benchmark-was-lying.jsonld"}}