Can a Local Auditor Ensemble Match a Hosted Decision Model?

A local ensemble of three QLoRA fine-tuned LLM auditors (Qwen2.5, SmolLM3, and phi4) was benchmarked against Jev, TypeSafe's hosted typed decision model, on 20,000 industrial record-linkage pairs at a 41.3% match rate. Jev's recall on the gold set rose from 14.9% with the base rubric to 89.1% with the MSF rule alone at 98.4% precision, and reached 92.2% with all five rules, but 6,193 of the pool's 8,253 true matches (75%) are MSF pairs, so the gain is one rule meeting the dominant product family. On the non-MSF subset, the full rubric ladder added 15 points of recall while precision held near 90%.

Here are two records from a client’s corrugated-sheet catalog: 275B 32.2500 X 36.0000 122.65 MSF 275B 32.2500 X 36.0000 122.18 MSF Same board grade, same sheet dimensions. The last number differs, and it’s followed by MSF: thousand square feet. That’s how much was bought, not what was bought. These two records are the same product. Now these two, from the same review: 4X6 1 MIL CLEAR poly bag 4X6 4 MIL CLEAR poly bag Also nearly identical, differing by one number. This time, the number is the bag’s thickness, and the records are different products. In industrial record linkage, knowing which numbers define the product and which merely describe a transaction is often the difference between a correct match and a corrupted catalog. My pipeline embeddings, blocking, a re-ranker, calibrated thresholds, clustering proposes candidate matches. Three locally hosted LLM auditors adjudicate them the way an analyst would: three QLoRA fine-tuned models Qwen2.5, SmolLM3, and phi4 . Production has to stay local for data privacy reasons. I wanted to know whether a different kind of auditor would do better. Jev, TypeSafe’s hosted typed decision model https://www.towardsdeeplearning.com/typesafe-spent-two-years-building-jev-in-secret-open-source-cloned-it-in-4-days-6020238ae08f , answers one multiple-choice question per pair MATCH, NONMATCH, or INSUFFICIENT EVIDENCE and returns a probability distribution. Its instructions, the rubric, are a text file I can version and test. I can’t deploy Jev, so for me it’s a benchmark: if a hosted model with written-down domain rules is clearly better, that’s the bar a local model has to reach. I also ran Laya, an open-source, self-hostable implementation of the same request format https://github.com/NandhaKishorM/laya , from its base checkpoint, on my RTX 5090. The question I care about in production isn’t abstract accuracy. It’s how many wrong automatic calls stand for a given amount of analyst review. I wrote five rules, one at a time, and recorded each hypothesis before running inference. Three were confirmed by domain review: MSF is quantity, a trailing service-run number is a per-run ID, and packs-per-pallet is packaging. Two I proposed from leftover misses: naming style abbreviations, prefixes is incidental unless the difference is substantive, and per-unit packaging counts are incidental when the product matches. My reference-labeled evaluation sample “gold” is 20,000 pairs drawn from the pipeline’s labeled pair pool, which spans several clients’ catalogs, at the pool’s natural 41.3% match rate. The negatives are blocked candidates, pairs that the pipeline actually considered. 19,530 labels are derived from identity groupings, each human-reviewed when it was built, rather than from pair-by-pair review; the other 470 are direct pair reviews. All seven rubric versions were frozen before gold was built, so nothing on it fed back into the rules. Jev’s recall on gold went from 14.9% with the base rubric to 89.1% with the MSF rule alone, at 98.4% precision. With all five rules, it reached 92.2%. Then I checked what the pool was made of. 6,193 of its 8,253 true matches 75% are MSF pairs. Precision on everything else held at about 90% throughout. So the headline jump is one rule meeting the product family that dominates the pool. On the non-MSF pairs, the full ladder bought 15 points of recall. The service-run and pallet rules barely moved the totals, and that’s measurable: gold has 175 service-run pairs and 9 pallet-count pairs. The full rubric-by-rubric results are in the appendix. I ran all seven rubric versions on gold, and only the base and full rubrics on the second client. I ran the three production auditors on the same 20,000 pairs. On the non-MSF subset, which isolates performance away from the dominant MSF convention: Two conventions matter here. Jev can abstain with INSUFFICIENT EVIDENCE, and in this table, an abstention counts as not-MATCH. Jev abstained on 916 of these 13,787 pairs 6.6% . Counting only the 12,871 pairs it decided, which changes the denominator, its recall is 74.4% at the same 90.6% precision. The fine-tuned auditors return a binary decision on every pair phi4 failed on 29 here . In this table, “unanimous” means all three say MATCH; anything else counts as NONMATCH. In a shadow audit, an abstention goes to a person, so it’s review work rather than an error. Because every model scored the same pairs, I compared them directly: a paired bootstrap on the differences, resampling by query item so repeated items don’t inflate confidence. Jev is about 2 points more precise than each fine-tuned auditor, and the majority vote reflects that difference for example, +2.4 points vs Qwen2.5, 95% interval +0.9 to +3.7 . Recall against either fine-tuned model is indistinguishable. The majority vote has 3.4 points more recall than Jev, and that holds up too. The unanimous vote is the most precise of all: 1.8 points more precise than Jev interval +0.5 to +3.2 , at 8.5 points less recall. The totals tell a different story because the auditors only partially learned the MSF convention from their training data: 88.8% and 79.4% recall on MSF pairs, compared with Jev’s 100%. The fine-tuned phi4 which was trained on a more general set than the other two auditors rejects most of them 35.4% , the way Jev’s base rubric did. So on gold, a written-down rule gave Jev a large lead on one product family. For the non-MSF pairs, Jev was more precise than either the fine-tuned auditor or the majority vote, with similar recall, and the unanimous vote was more precise still, at a recall cost. The fine-tuned auditors were trained on a single amalgamated set from multiple clients, with human-reviewed pairs, and the same reviewer labeled part of that training data and gold’s 470 direct-review rows. Excluding those rows changes the auditors’ numbers by less than 1 point in precision and less than 3 points in recall. About 0.1% of gold’s pairs also appear in their training data. At that point, the story looked settled : a hosted model with one written rule beat my local auditors on the product family that dominates my labeled data, and edged them on everything else. That’s the trap. Gold is a pool I built from my own labeled history, and its composition was doing much of the work. The test that mattered was a second client’s catalog, with no rule written for its products, scored on what the pipeline actually hands the auditors. Gold is a labeled pool, not what the auditors see in production. In production, they adjudicate what the pipeline couldn’t settle on its own. So I took two March link runs from a second client, a catalog of facilities, janitorial, and packaging supplies, and scored every pair the pipeline had labeled accept or sent to review: 37,392 pairs. In that workflow, an analyst still checks the pipeline’s accepts before anything merges, so “accept” is the pipeline’s own call, not a production merge. Everything below is about this audit stage: pairs the pipeline retained. It says nothing about matches lost earlier, in blocking or below the review line. I hand-labeled 1,000 of the 37,392 pairs, sampling more heavily from some confidence bands than others; each labeled pair then counts in proportion to the number of pairs its band contains, so the estimates describe all 37,392 pairs. About 38% are true matches. One person labeled each run without a second review pass. None of these pairs contains MSF. The rule that drove gold has nothing to act on here. Single-model 95% intervals are about ±4-5 points, so the paired comparisons are what matter. Jev, with all five rules, left 1,148 of the 37,392 pairs 3.1% undecided. The fine-tuned auditors return a binary decision on every pair, and phi4 failed on 559. Here too, “unanimous” means all three say MATCH and anything else counts as NONMATCH; in the routing table below, a split goes to an analyst instead. Paired against Jev with all five rules: These intervals are clustered by query item within each confidence band: pairs that share a query item are resampled together. Precision and recall don’t decide anything on their own. The production question is how many incorrect automated calls correspond to a given amount of analyst review. My auditors already work this way: when all three agree, the call stands; any split goes to an analyst, including the 559 pairs where phi4 failed to answer. Jev’s comparable policy is my routing threshold: its call stands only when the probability is at least 0.90. This is an ideal-review comparison: I treat every pair sent to an analyst as correctly resolved, so the table compares the automatic calls that stand, not measured end-to-end accuracy. Review shares are exact counts over all 37,392 pairs. The error rates are estimated from the 1,000 labeled pairs, weighted back to the full set, and they cover only these retained candidates. Weighted equally, the two policies result in a statistically indistinguishable number of incorrect automatic calls 8.6% vs. 10.5%; auditors minus Jev: −1.8 points, interval: −4.7 to +1.0 . But my pipeline doesn’t run on equal error costs. False merges have a downstream detection-and-correction mechanism, an anomaly detector. A valid pair dismissed at this stage has no comparable path: nothing downstream in my current workflow independently rediscovers it. That changes how I read the table. Among pairs that reached this stage, the local auditors’ automatic dismissals miss true matches, amounting to 1.4% of the evaluated population, compared with 10.3% for Jev at 0.90, while sending fewer pairs for review. They let more false merges stand 7.2% against 0.2% , and those have a path to correction. I checked whether a different Jev threshold changes that: Raising Jev’s threshold does rescue some missed matches, at a steep review cost. At 1.00 its probabilities come rounded to two decimals, so that’s the top of the scale , it sends 66.5% of pairs to analysts and still misses 3.8%, above the auditors’ 1.4% at 32.5% review. None of the thresholds I tested matched the local policy on both review load and missed matches. These rates come from the same 1,000 labeled pairs, so they carry the same sampling uncertainty as the table above. I didn’t test other routing designs, such as separate thresholds for MATCH and NONMATCH, beyond a few simple variants. Jev does return a numerical distribution I can route on; my auditors give a three-vote agreement signal. Neither is a calibrated confidence until it’s tested as one. The rule ladder still helped Jev by 7.7 points of recall 47.2% to 54.9% with precision flat. I only ran the base and full rubrics here, so I can’t say which rules produced that gain. The models fail differently. In the holdout, 18 of Jev’s 57 false matches 32% are pairs in which one side omits an attribute, such as thickness or size; it treats missing information as compatible. That suggests a rule to test, not one to adopt yet: when an identity-critical attribute is missing, and no stronger identifier a matching part number, say establishes the match, answer INSUFFICIENT EVIDENCE. Why this pipeline is recall-first. The link run compared 57,000 query items against 660,000 reference items and, across its four output sets, accepted or sent to review 72,177 pairs; I scored the first and last sets. The pipeline’s accept call is only about 52% precise within the accept stratum, 43 of 82 labeled accepts are true matches; interval 41% to 63% , and it finds only 4% of the matches. That’s deliberate. Most matches are caught in the review band, and every accept still gets human review before merging. 28 of the 39 wrong accepts were dimension differences, like 6" X 3" X 12" 32 ECT FOL CORRUGATED BOX against 30 x 12 x 6" Flat Corrugated Boxes . In production, I set the review line to a calibrated score of 0.15 rather than the canonical 0.5, a choice informed by earlier below-the-line testing. At that line, none of the 9,000 sampled scored-and-rejected candidates 1,000 in each of nine runs was labeled a match. Under a representative, independent-sampling approximation, that gives an approximate one-sided 95% upper bound of about 0.03% on the match rate in that rejected band. It’s a ceiling, not an observed rate. It doesn’t measure matches blocking never retrieved, and a low rate over a large band can still hide some misses. On gold, the base Laya checkpoint’s precision ranged from 22% to 38% across all seven rubrics, below the 41.3% you’d get by calling every pair MATCH. The rules changed its behavior from base to full ladder; precision rose from 30.1% to 38.5%, while recall fell from 25.3% to 12.1% , but didn’t make it more useful. On the second client, it achieved about 54% precision and 53-55% recall under both rubrics. That claim is limited to what I tested: the base checkpoint, not fine-tuned on my data, run at instruction lengths beyond its reviewed range, under my thresholds. Laya’s README says the shipped checkpoints work zero-shot and that fine-tuning on your own decisions “is where accuracy jumps.” On its own benchmark accuracy on 2,000 typed decisions across four workflows , it scores the fine-tuned checkpoint at 0.766 and the base at 0.362. I tested the base. A skewed base rate. An earlier test set 36,695 pairs that my auditors had already accepted was 81.7% true matches. There, calling everything MATCH would have beaten Jev’s 78.0% decisive accuracy. Gold makes the effect measurable. Its 470 direct-review rows are 87.4% matches, and on that basis Laya looks competent 90.1% precision, 73.2% recall . On the full gold set, the same configuration scores 30.1% and 25.3%. A silently clipped prompt. Laya’s checkpoint has a reviewed budget of 192 tokens for the question’s instructions. My rubrics ran from 149 to 980 tokens. At the defaults, the server would have truncated six of seven rubrics without an error, and the runs would have looked normal while answering a shorter question than the one I registered. My tokenizer counter refused instead. I ran Laya at a 2,048-token budget and recorded that in every run’s identity. A response contract mistaken for model behavior. Across the development runs, about 1,100 of Jev’s response attempts 0.35% of attempts, including retries failed to meet my response contract. 92% of those were probability distributions that summed to 0.99 because the API serializes probabilities to 2 decimal places. The rest named a choice that wasn’t the highest probability. Neither rate climbed with prompt length. If I’d read those as model flakiness, I’d have tuned the wrong thing. One client’s rule was mistaken for a general one. That’s the gold result. The second client is what caught it. Jev’s API charges came to $16.16 across 394,869 scored pairs: $2.64 on the development sample, $5.31 on the audit-accepted set, $5.07 on gold, and $3.13 on the second client rounded components; they sum to $16.15 . That’s API charges only, not hardware, local compute, or the labeling time. Jev is priced at $0.042 per million input tokens. A 10,000-pair run took about 6 minutes. Laya and the auditors ran locally with no API charges; the three auditors took just under three hours on gold. Jev was never deployable for me. It was the bar. On my development data, with a rule written for one client, it looked clearly ahead of my local auditors. On a second client’s real pipeline output, three local models voting unanimously, with splits sent to an analyst, make about as many wrong automatic calls as Jev at a similar review load, and far fewer missed matches at the stage where my workflow can’t recover them. My next move is to test the rules on the auditors I already run, starting with the missing-attribute cases behind Jev’s false matches. Then Part 2: fine-tune Laya on the same reviewed pairs and compare it with the auditors on missed matches and wrong merges per unit of review, with its probabilities checked for calibration before I route on them. The caveats are the difference between a field report and a benchmark: development data and one-second client, 1,000 labeled pairs from one labeler per run, single runs per model, and Laya outside its reviewed envelope. 20,000 pairs, 41.3% matches. DA = decisive accuracy, the share of MATCH and NONMATCH calls that were correct. INSUFFICIENT EVIDENCE counts as not-MATCH for recall. On the 19,530 identity-grouping rows alone, Jev v007 scores 98.5% precision and 92.8% recall. Auditors on all 20,000 P / R : Qwen2.5 97.4% / 83.7%, SmolLM3 97.1% / 76.9%, phi4 89.9% / 46.8%, majority 97.2% / 79.1%, unanimous 96.4% / 33.3%. 10,000 candidate pairs from a scored pipeline export, 202 of them human-reviewed 48 matches . After the first run, I reviewed the 36 pairs where Jev disagreed with the labels: 34 matches it had dismissed and 2 non-matches it had accepted. I changed two labels the poly bag to non-match, a caster kit to match , and Jev’s call agreed with both changes. Seventeen of the 33 dismissed matches left after review were MSF pairs. That review is error-selected and post-inference: a development reference, not validation. On the 202 pairs, the base rubric called only 2 of 154 non-matches MATCH but missed 36 of 48 matches 34 dismissed, 2 undecided . Each confirmed rule fixed its own family with zero new false matches. Combined, they reached 97.6% precision and 83.3% recall; all five, 95.5% and 87.5%. The 295 identity-grouping pairs in this sample were excluded from these numbers, a conservative choice made before their provenance was confirmed. 36,695 pairs from one client, accepted by majority vote of the production auditors at that time Qwen and SmolLM models accepted all of them; phi4, rejected or abstained on 156 . 81.7% are matches. v005 to v007 moved Jev’s recall from 77.9% to 81.4% with precision flat 87.1% to 86.8% . The two proposed rules were mined from this set’s own misses, so they're in-sample for this set. The packaging rule’s “ when the underlying product matches ” clause generalized well past its target: 47 of its 649 rescues were packaging pairs, 191 naming-style. Both proposed rules remain flagged unconfirmed. Two March link runs 18,637 and 18,755 pairs: every auto-accept plus every pair sent to review . 500 pairs labeled per run, drawn by seed across five strata that depend only on the pipeline the accept set and four calibrated-probability bands in review , each weighted by stratum size over sample size. Intervals: stratified bootstrap, 1,000 draws. Paired differences use the same stratified resampling. One labeler per sheet, no adjudication pass; every non-match carries a reason. phi4 failed on 559 pairs in one run an Ollama repeat-limit error on one machine , scored as undecided. 23 of those fall in the labeled holdout; recounting all 559 as ACCEPT moves the majority vote from 70.4% / 82.7% to 70.3% / 83.0% and leaves unanimous recall unchanged. The operating-policy table uses actual review counts over all 37,392 pairs and holdout-weighted error rates. The unit of analysis is a query item, reference item pair; 44 pairs appear in both scored runs and are counted once per run. Historical re-ranker figures behind the 0.15 review line: 97.2% precision and 94.1% recall at the canonical 0.5 threshold on 279,429 pairs from a random pair-level split of an earlier training set. That split isn’t entity-disjoint, so the numbers are likely optimistic and aren’t an estimate of performance on this client. The 0/9,000 below-the-line result pools equal 1,000-pair samples from nine runs; if those runs’ rejected bands differ much in size, the pooled bound isn’t weighted to their aggregate, and repeated products reduce the effective sample size. Per run, most models were 3 to 8 points less precise and more recall-heavy on the second set than the first; per-run intervals are about ±6 to 8 points. Can a Local Auditor Ensemble Match a Hosted Decision Model? https://pub.towardsai.net/can-a-local-auditor-ensemble-match-a-hosted-decision-model-48faac0f5111 was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.