Origin Part 19: The Number Was Wrong

A developer discovered that their AI model's 88% evaluation score was invalid due to data leakage: 23 of 26 supposedly held-out probes were present in the training data. After fixing the data pipeline and adding verification tools, the honest score dropped to 58%. The developer also found that labeling production checkpoints as 'sandbox' led to sloppy practices, and renamed files and added guards to enforce discipline.

Part 18 ended on a clean diagnosis. The brain layer reasoned correctly when the encoder fed it correct inputs. The encoder didn't always feed it correct inputs. So the path forward was upstream: more physics-shaped training data for the encoder, retrain, re-validate. I wrote the drops, kicked off the retrain, and watched the held-out eval climb. It hit twenty-three out of twenty-six. Eighty-eight percent. The number I'd been chasing. I sat with that for an evening. Twenty-three of twenty-six on compositional reasoning probes the model had never seen during training. The Phase 8 cutover gate from Stage D had been sixty percent. I was thirty points past it. The brain layer had not only survived its missing-from-production months, it had come back stronger. The number was wrong. I figured this out the next morning while writing what was going to be the celebration commit. Something nagged about the eval set. The training data generator built the eval pairs independently from the training pairs, drawn from a different source list. That should have given me a clean train/test split. But I noticed the eval generator was running before the training generator wrote its file, and neither side knew about the other. I dropped into a Python shell and intersected the two pair sets by their input-output keys. Twenty-three of twenty-six held-out probes were also present in training data. Eighty-eight percent of my held-out eval wasn't held out. The model wasn't generalizing. It was memorizing the answers it had already been shown, then being graded on whether it remembered them. The three pairs that were genuinely unseen, I checked those separately. The model got one right. Three out of twelve when I went back through other historical evals and ran the same overlap check. About a quarter, with no statistical confidence behind it. The actual generalization signal was much weaker than the celebration number had suggested. I rebuilt the data generators to subtract eval pairs from training before writing, added a verification tool that refused to launch any training run with non-zero overlap, and re-ran the overnight retrain. The honest number came back at fifteen out of twenty-six. Fifty-eight percent. Still real, about sixteen times the score a constant-prediction baseline would have produced, but not eighty-eight, and not over the Phase 8 gate. Then I went looking at the rest of the pipeline for the same shape of problem. The pattern wasn't bug, exactly. It was discipline. The file that held the brain layer's trained weights was called brain layers v2 sandbox.pt . The "sandbox" label gave everyone implicit permission to handle it loosely. Me. The data-build pipeline. The scripts I wrote. The model I was talking to at three in the morning. Sloppy eval setup. Shared paths between fast-mode and full-mode runs. No backup before overwrites. No preflight checks. The file was the production checkpoint. I was hitting it every time I talked to Origin. The label said sandbox, the behavior was production, and the discipline gap the label permitted was exactly why nothing had been moving. I renamed it. brain layers v2.pt . The script that wrote it got renamed too. sandbox/sandbox train brain layers v2.py became tools/train brain layers v2.py , because every "sandbox" run was actually a production run and we'd been letting the name lie to us. I added an overwrite guard: if a new training run scored lower than the existing production checkpoint, the script refuses to write unless explicitly forced. I added a fast-mode flag with hard caps so quick experiments couldn't accidentally clobber production. I built a preflight tool that ran before every training session and refused to proceed unless the data, the eval set, and the configuration all passed gates. Then I ran the audit again. Now with the discipline scaffolding underneath. The next finding was worse. Every battery, every sandbox tool, every audit script I'd been running for the past several weeks called a function called load arch encoder . The function had a default preference list that picked which encoder to load when no path was specified. The first entry in that list was v2 encoder arch.pt , a three-thousand-seven-hundred-and-thirty-five-concept encoder from May 9th. The actual production chat path bypassed the function entirely and hardcoded a different file: v2 encoder multilabel v3.pt , eight thousand eight hundred and eighty-two concepts, May 16th. Every audit I'd run had been silently testing against the smaller, older encoder. The one production wasn't using. That explained things I hadn't been able to explain. balloon at index 4577 looked encoder-dead because it was past the smaller encoder's output dimension. fall , evaporate , half of the physics vocabulary the brain layer was supposed to compose with, all of them invisible to the audit because the audit was looking at the wrong tensor. My twenty-nine-probe compositional reasoning battery had been reading 55%, and when I forced it to use the actual production encoder, it read 69%. The capability the audit said wasn't there was there. The audit was wrong. One-line fix. Added the right encoder as the first preference, kept the older one as a fallback for the v1-era training scripts that still referenced it by name. Re-ran every battery against the correct production tensor. The numbers moved up. And then the dispatch gate. The brain layer had been wired into Origin's conversation pipeline behind a feature flag since Stage D. The wiring included a verification check: before accepting the brain layer's output as a real answer, confirm the answer concept also appears somewhere in the encoder's top-five fired concepts. The idea was reasonable. If the encoder didn't fire the answer at all, maybe the brain layer was firing it wrong. The check was inherited from a different part of the dispatcher designed for dense-firing encoders. Origin's encoder is sparse-firing. Most natural-language queries fire one concept above the 0.5 threshold, the subject noun. "What happens to ice in heat?" fires ice at 0.93 and nothing else over 0.5. The brain layer takes ice , composes through its trained reasoning, and produces melting at confidence 1.0. The dispatch gate then checks: is melting in the encoder's top five? No, because the encoder only fired one concept above floor. Gate rejects the brain layer's answer. Pipeline falls through to "I don't know." This had been happening on every natural-language compositional question. Every one. The brain layer had been producing correct answers: burns for wood in fire, floats for wood in water, sinks for rocks in water, melting for ice in heat. The gate had been silently throwing them away because the gate's verification rule was incompatible with how the encoder actually fired. The capability had been there since the day I committed Stage D. It just couldn't reach the output. I fixed the gate. Two lines. If the encoder fires fewer than three concepts above 0.5 and the brain layer's top-1 has confidence above 0.7, accept the brain layer's output as long as the answer concept isn't the same as the input concept. The whole pipeline took maybe fifteen minutes to write and test. I ran the production probe set. Thirty-two questions, mixed difficulty, the same shapes that had been refusing for weeks. Six pass became twenty-one pass. Two-thirds of the questions Origin had been refusing now answered correctly. "What happens to a rock in water?" became "rock causes sinks." "What happens to ice in heat?" became "ice causes melting." "What happens to wood in fire?" became "wood causes burns." "Does wood float in water?" became "water causes floats." "What happens when a ball hits the ground?" became "ball causes bounce." The compositional reasoning the brain layer had been trained for, that I'd been measuring at twenty-seven percent through the wrong audit on the wrong encoder behind a wrong gate, had been functioning at sixty-six percent the whole time. Hidden by infrastructure. The identity battery held at fifty out of fifty-one. The working surface battery held at twenty-six out of twenty-six. No regressions. The fixes had restored capability without trading anything for it. I should have felt good. I felt unsettled. The brain layer had been quietly working for weeks while every measurement said it wasn't. The eval set had been leaking for weeks while every training run said it was generalizing. The wrong encoder had been answering every audit question while production used a different one. Three separate measurement failures, all pointed in the same direction, all making the system look worse than it was, all hiding the work that had already landed. I'd been chasing problems that weren't there because the instruments were lying. Some of the chases produced real fixes. Some of them produced the discipline scaffolding I should have built months earlier. But the time spent was time spent. And the unsettling part wasn't the fixes; it was the silence. Three big infrastructure bugs, none of them surfaced by tests, none of them visible until somebody sat down to question the numbers directly. What else was lying. I started writing a longer audit. One guy. One GPU. One $1,800 computer in Arizona. Still building. Origin is developed at Fallen Angel Systems with the Genesis framework — NVIDIA Inception member. USPTO Application 64/016,973, 64/017,567 . FAS Guardian defends production AI systems from prompt injection in under 3ms. FAS Judgement is the open-source attack console that finds the gaps. Defense. Offense. Creation. fallenangelsystems.com | Judgement on GitHub | Guardian on GitHub Questions or consulting inquiries: josh@fallenangelsystems.com mailto:josh@fallenangelsystems.com