{"slug": "a-small-not-so-smart-model-went-from-10-to-99-6-at-reading-property-deeds-for-1", "title": "A small, not-so-smart model went from 10% to 99.6% at reading property deeds for $1.15, and it says 'check this line' instead of bluffing", "summary": "A developer built DeedReader, an open-source tool that parses metes-and-bounds property deed descriptions using a fine-tuned Qwen3-8B model (LoRA, trained on Tinker for about $2.50) and verifies every computed number in deterministic code. In the developer's tests on synthetic deeds, the untuned base model read 10% of deeds exactly right and produced 5 of 60 wrong with no warning, while the tuned system flags mismatches as NEEDS HUMAN REVIEW, points to the offending line, and searches one-digit corrections to close the parcel. The developer cautions that all figures come from synthetic deeds and real-deed accuracy is unmeasured.", "body_md": "*This is a submission for the [Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady](https://dev.to/challenges/hacktoberfest-week3-2026-10-19)*\n\n**N 86°29' E, 103.2 feet.**\n\nThat is one line of a property deed. Walk the rest of the lot and you should end up where you started.\n\nYou end up **27 feet away.**\n\nOne digit is wrong. Maybe a clerk typed `103.2` for `130.2`. Maybe the scan smudged. Maybe the AI that read it got creative. The drawing you'd make still looks fine, and nobody finds out until the survey crew does.\n\n**DeedReader** reads that deed, closes the shape in plain code, plots it on a real satellite map, and when something doesn't add up it refuses to draw it. It stamps the result **NEEDS HUMAN REVIEW**, points at the line, and tries every one-digit fix to find the one that closes the parcel. In the example above: `103.2 → 130.2`.\n\nEvery parcel of land has a legal description, and it reads like this:\n\n*Commencing at a cedar post at the corner of Tract 7, thence S 46°30′11″ W 1,011.25 feet to a corner post, thence along a curve to the right, R = 757.19 feet, to a point in the center of the road, thence N 23°16′52″ W 1,206.14 feet to a 4 inch concrete monument…*\n\nSomeone has to turn that into a drawing. Convert every bearing. Plot every side, curved ones included. Check the shape closes. Work out the acreage. That someone is a surveyor, a title abstractor or a paralegal, and the job takes **fifteen to thirty minutes a deed**.\n\nThe scary part isn't the time. It's that **one wrong digit quietly poisons everything downstream**, and the drawing still looks plausible. The failure you have to fear isn't \"the tool is wrong.\" It's \"the tool is wrong and didn't tell me.\"\n\n| Option | What it gives you | What it can't tell you | \n|---|---|---|\n| By hand | A drawing you trust | It takes 15–30 minutes a deed | \n| A regex parser | Instant and free | Only reads clean `N 41°00' E` calls, not 1800s prose or numbers spelled out in words | \n| A base open model, no training | Reads messy prose | **10% exactly right** in my test, and 5 of 60 wrong with no warning at all | \n| **DeedReader** | A closed parcel on a real map in seconds | Says **NEEDS HUMAN REVIEW** when anything doesn't add up, and points at the line | \n\nDeedReader is the lazy version of that job: **paste, glance, approve.**\n\n**Why it fits Slow and Steady:** the tortoise didn't win by being fast. It won by never doing something stupid. A small model, a dollar of training, plain code for every number, and a system that would rather say \"I don't know\" than be quick and wrong. It automates the slowest job in land work, and earns trust the slow way: by showing its arithmetic.\n\nThe video above is the whole tour. The studio replays **real recorded outputs of my fine-tuned model** on deeds it never saw in training, while every safety check runs **live in your browser**.\n\n**Paste, glance, approve.** A clean deed: every check passes, the parcel closes to 1:2.9 million, and the studio reports 43.44 acres. Click any sentence and its side lights up in the drawing. Export to SVG, DXF or GeoJSON in one click.\n\n**Break it on purpose.** \"Simulate a misread\" corrupts a single number. Three independent checks fail at once, the banner turns red, and it names the line:\n\nAnd the lazy payoff, side by side:\n\nPaste a metes-and-bounds legal description. Get structured calls, a closure check done in plain code, a plotted parcel, and an honest **NEEDS HUMAN REVIEW** whenever anything is off. A fine-tuned open-weight model (Qwen3-8B + LoRA, trained on [Tinker](https://thinkingmachines.ai/tinker/) for about $2.5) does the reading; deterministic code does everything that must be right.\n\nBuilt for the DEV Hacktoberfest Open-Source AI Challenge, Week 3. **Read [EVAL_REPORT.md](https://github.com/advikdivekar/deed/EVAL_REPORT.md) before trusting any number here: everything is measured on synthetic deeds, and real-deed accuracy is unmeasured.**\n\n| base Qwen3-8B, 3-shot | tuned (v2) |  | \n|---|---|---|\n| deeds read exactly right (in-distribution) | 10% (n=60) | 99.6% (n=3,000) | \n| flagged for a human | 82% | 0.4% | \n| silent errors (wrong AND unflagged) | 5 of 60 | 2 of 3,000 (0.07%, 95% upper bound 0.24%); 0 if `87 link` =`0.87 chain` | \n| training cost | n/a | about $1.15 (est.) | \n\nUnseen style/unit, zero-shot: spelled-out numbers 85.6% exact (14.4% flagged, 0 silent)…\n\n**The rule behind all of it: the model reads, code decides.**\n\nLanguage models read messy text brilliantly and are not accountable for arithmetic. So Qwen3-8B plus a LoRA does exactly one job, turning prose into structured calls. Everything after that is plain code with tests: closure, area, unit conversion (feet, chains, poles, varas, links) and plotting.\n\nThen five independent checks have to agree before an answer may look confident:\n\n**The data: truth first, prose second.** There is no free pile of labelled real deeds, so I turned the problem around. A generator draws a **closed polygon with known bearings and distances**, then writes it up in many styles: symbols, dashes, words, archaic chains and links, and damaged OCR, with easements, deed-book numbers and stray acreage as distractors. The truth exists before the text does, so every label is exact. I deliberately held back one **writing style** (every number spelled out in words) and one **unit** (Texas varas) to see what the model does with things it has never met.\n\n**Qwen3-8B is a small model. Out of the box it doesn't know what a deed is, and it has nowhere near the knowledge of a giant LLM. It scored 10%. Same model, same prompt, same checks: the only thing that changed was one dollar and fifteen cents of Tinker training.**\n\nThat's the point. I didn't need a model that knows everything. I needed one that does one narrow job perfectly, and fine-tuning is how a small model gets there.\n\n|  | base Qwen3-8B, untrained | **Tinker LoRA** | \n|---|---|---|\n| Exactly right | 6 / 60 (10%) | **2,988 / 3,000 (99.6%)** | \n| Sent to a human | 82% | **0.4%** | \n| Wrong and unflagged | 5 / 60 | **2 / 3,000 (0 by physical length)** | \n| Cost | n/a | **$1.15** | \n\nMy goal wasn't a good number. It was **a good number at the lowest price I could defend.** Here is exactly how.\n\nTinker bills per token, so I started with the output. My first format was JSON at **547 tokens** per deed. I replaced it with one line per call:\n\n```\nPOB iron pin\nL N 41 0 0 E 121.9 ft | iron pin\nC N 3 45 22 E 366.15 ft R 944.33 ft L | stone\n```\n\nCode converts it back to JSON. **145 tokens, 3.8× smaller**, which cuts training, evaluation and local inference all at once, and costs nothing to run.\n\nTraining per million tokens: **Qwen3-8B $0.44** vs Qwen3.5-4B $0.737. A smaller model is not automatically a cheaper one. Qwen3-8B was cheaper to train and strong enough for the job.\n\n```\ntc = await svc.create_lora_training_client_async(base_model=\"Qwen/Qwen3-8B\", rank=32)\n\n# loss only on the answer tokens, never on the deed prose\nDatum(model_input=full[:-1],\n      loss_fn_inputs=dict(weights=[0.0]*(len(prompt)-1) + [1.0]*len(completion),\n                          target_tokens=full[1:]))\n\nfb = await tc.forward_backward_async(batch, loss_fn=\"cross_entropy\")\nop = await tc.optim_step_async(AdamParams(learning_rate=lr_at(s), beta1=0.9, beta2=0.95, eps=1e-8))\n```\n\n**2,300 training deeds**, batch 16, peak learning rate 2e-4 with warmup and linear decay, **429 steps, about 2.6M tokens.** Loss is masked to the answer, so the model learns to write calls instead of memorizing deed prose. Steps are pipelined (step *k+1* is submitted before step *k* returns), a checkpoint is saved every epoch, and a budget guard prices every job from token counts and refuses to launch one that would burn half the cap. Loss falls from 1,806 on step 1 to under 0.1 by step 41.\n\nI didn't train on everything and hope. Four runs, one epoch each, scored on the same 500 held-out deeds:\n\n| training deeds | est. cost | read exactly right | wrong and unflagged | \n|---|---|---|---|\n| 250 | $0.04 | 38.6% | 19 | \n| 500 | $0.08 | 92.6% | **0** | \n| **1,000** | **$0.17** | **99.4%** | **0** | \n| 2,000 | $0.34 | 99.6% | 0 | \n\n**Seventeen cents buys 99.4%.** And look at the zeros: between 250 and 500 deeds the model stops making *silent* mistakes. That jump, not the accuracy, is where it becomes safe to trust.\n\nI paired the base model against the fine-tune on **the same 60 deeds**, deed by deed:\n\n|  | base, untrained | fine-tuned | \n|---|---|---|\n| exactly right | 6 / 60 (10%, CI 4.7–20.1%) | **60 / 60** | \n| wrong and unflagged | 5 | **0** | \n| deeds only this model got right | 0 | **54** | \n\nEvery deed the base model got right, the fine-tune also got right, and it won the other 54. Paired sign test: **p < 0.0001.**\n\nThen the real exam: **3,000 fresh deeds**, generated before I froze the validator and scored exactly once. **2,988 exactly right. 99.6%.**\n\nOf the 12 imperfect outputs, **10 were caught by parse and closure alone.** The other 2 are one ambiguity in my own generator: it writes `87 links` as \"87 link\" in one style and \"0.87 chain\" in another, which is the same physical length. By physical length, the silent count is **0**. I report both.\n\nHeld out on purpose, zero-shot, from the first model:\n\n| unseen test | exactly right | wrong and unflagged | \n|---|---|---|\n| numbers spelled out in words | 85.6% | **0** (14.4% flagged for review) | \n| Texas varas | every deed flagged | **0** | \n\nVaras is the fun one. The model read every bearing and distance correctly but wrote the unit as `var`, a token it had never seen, and the strict decoder rejected it. So **everything was flagged and nothing wrong got through.** That is the safety net doing its job on a surprise.\n\nI didn't patch the decoder to accept each spelling the model improvised, because that would be fitting the test set. The real fix was data: **150 examples each**, retrain, score on **fresh** deeds:\n\n| fresh test set | n | exactly right | wrong and unflagged | \n|---|---|---|---|\n| in distribution | 3,000 | **99.6%** | 2 (0 by physical length) | \n| spelled-out numbers | 250 | **99.6%** | 0 | \n| varas | 250 | **100%** | 0 | \n\nOnboarding a new style or unit is a 150-example, sub-dollar job. That's the workflow Tinker makes cheap enough to just do.\n\nEvaluation was the bigger bill (**$1.35** vs $2.46 for all training), so every sampled output is cached and the checks score from the cache. Improving the validator and re-scoring 3,000 deeds cost **$0.** One deed through Tinker costs about **$0.00015.**\n\nI downloaded the **369 MB adapter** and ran it with **MLX on the Apple GPU** over a 4-bit Qwen3-8B base:\n\n**150 of 150 deeds read exactly right, 0 flagged, median 7.5 s per deed.** M3 Pro, 18 GB, no network, no API key.\n\n| item | est. cost | \n|---|---|\n| baseline, no training | $0.11 | \n| learning-curve runs | $0.29 | \n| first full run | $1.01 | \n| **final model (2,300 deeds, 3 epochs)** | **$1.15** | \n| all evaluation (4,000+ deeds) | $1.35 | \n| **everything on Tinker** | **about $3.80** | \n\n*Estimated from token counts at Tinker's list prices.*\n\nI'd rather you hear these from me, with numbers.\n\n`;` or `thence`\n**What went wrong on the way (and made it better):** my validator flagged *perfect* answers until I ran it on the generator's own ground truth and fixed three reading bugs. Two silent errors slipped past closure at 500 deeds (a dropped arc-minute and a flipped curve direction), which is exactly why the independent second reader exists.\n\nA closed API gives you a prompt box. Open weights gave me a **steering wheel.**\n\nThe honest caveat: training ran on Tinker's servers on generated deeds only. No real person's land record was ever sent anywhere, and at run time nothing leaves the laptop.\n\nI built DeedReader with Claude Code as a pair, from a written plan with a cost estimate up front and a budget guard on every training job.\n\n**Best Use of Tinker.** The LoRA fine-tune (pipelined forward/backward passes, loss-masked completions, per-epoch checkpoints), the baseline, the data-size learning curve, 4,000+ evaluated deeds and the adapter download behind the offline story all ran on Tinker. **$1.15 for the final model, about $3.80 for everything.** Same base model, same prompt, same checks: **10% (6/60) → 99.6% (2,988/3,000)**, and every deed the base model got right, the fine-tune got right too.\n\n*Built solo, with Claude Code.*\n\n**Three epochs. $1.15. 10% became 99.6%.** If that sounds like a good trade, a ❤️ or 🦄 helps more people find it.\n\nWhat's the strangest phrase you've ever seen in a property deed? Mine so far is \"to a white oak, thence along the run.\" Leave yours in the comments.", "url": "https://wpnews.pro/news/a-small-not-so-smart-model-went-from-10-to-99-6-at-reading-property-deeds-for-1", "canonical_source": "https://dev.to/advikdivekar/a-small-not-so-smart-model-went-from-10-to-996-at-reading-property-deeds-for-115-and-it-says-490k", "published_at": "2026-10-25 16:39:52+00:00", "updated_at": "2026-10-25 16:44:49.094012+00:00", "lang": "en", "topics": ["large-language-models", "ai-tools", "ai-agents", "developer-tools"], "entities": ["DeedReader", "Qwen3-8B", "Tinker", "Thinking Machines", "Hacktoberfest", "DEV"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/a-small-not-so-smart-model-went-from-10-to-99-6-at-reading-property-deeds-for-1", "markdown": "https://wpnews.pro/news/a-small-not-so-smart-model-went-from-10-to-99-6-at-reading-property-deeds-for-1.md", "text": "https://wpnews.pro/news/a-small-not-so-smart-model-went-from-10-to-99-6-at-reading-property-deeds-for-1.txt", "jsonld": "https://wpnews.pro/news/a-small-not-so-smart-model-went-from-10-to-99-6-at-reading-property-deeds-for-1.jsonld"}}