It printed "verifying sha256 digest." Then "success."
The file it had just verified was 561 bytes of nulls, and its own filename said so.
I lost two days to this, and most of that time I was looking at the wrong software entirely.
I run AI agent frameworks in a hardened VM, because those frameworks execute code a language model writes, and their own documentation tells you not to run them anywhere near your real filesystem. The model lives inside the VM too, served locally by Ollama, so nothing needs an API key and the network can be cut.
Then the VM started hanging. Not crashing, hanging. The console froze, SSH would accept my password and then sit there forever, and the only way out was a hard power off.
I did that four or five times over two days.
After one of those resets, a model stopped working. Every attempt to use it returned the same thing:
invalid character '\x00' looking for beginning of value
That error surfaced through my agent framework, wrapped in three layers of Python traceback, five frames deep in an HTTP client library. So that is where I looked. I read the framework's retry logic. I read the HTTP handler. I checked whether the request payload was malformed.
Meanwhile I kept trying to reinstall the model. Every time, the download bar filled to 100%, the tool printed "verifying sha256 digest," then "writing manifest," then "success."
And every time, the model was still missing from the list of installed models.
A tool told me four times that it had succeeded at something it had not done. I believed it four times, because it said "verifying," and because the failure was appearing somewhere else.
I want these in the record, because the clean version of this story is a lie.
I thought the disk was full. It was at 42%.
I thought the model file itself was corrupt, so I deleted it and reinstalled. Same result.
I thought the tool's index had drifted from what was on disk, so I restarted the service. Same result.
Then I compared what the server reported it had against what was actually in the folder, and they disagreed. Something was on disk that the running process refused to acknowledge.
Ollama stores model data in a folder where every file is named after the SHA256 hash of its own contents. That is the entire design of a content-addressed store. The name is the checksum. You do not need a manifest, a database, or a network call to know whether a file is intact. You hash it and compare it to what it is already called.
So I did that:
$ sudo sha256sum .../blobs/sha256-34bb5ab01051a11372a91f95f3fbbc51173eed8e7f13ec395b9ae9b8bd0e242b
d69d411b56d3a2b64ee95edb489d83fd913e8dfa12a620656ba2ace261723f6e .../sha256-34bb5ab0...
The file was named 34bb5ab0
. Its contents hashed to d69d411b
.
One command. It had been available since the first failure.
Then I checked every file in the store the same way, thirteen of them, and found a second one just as dead.
The word was in the output. The check was not.
Once a file exists at the expected path, the tool trusts it. It does not re-hash it. So a corrupt file gets skipped over as "already have that one," the progress bar fills instantly, and "success" prints.
I later confirmed this deliberately. I zeroed a file in place, keeping its exact size, which is the state an interrupted write leaves behind. Then I restarted the service and reinstalled the model.
The startup routine logged this:
msg="total blobs: 12"
msg="total unused blobs removed: 0"
It counted the file and left it. The model list loaded with failures=0
. The reinstall printed "verifying sha256 digest" and "success." The file still hashed to the wrong value.
Every layer reported that everything was fine. The only thing that ever disagreed was the filename, and nothing was reading it.
There is a second lesson buried in the same two days, and it is a different failure than the first.
The VM hangs turned out to be my fault. I had given the VM eight virtual CPUs on a machine with eight physical cores, so when the model server grabbed all of them, the guest kernel itself could not get scheduled. The console eventually told me exactly that, in a message I had not been around to see:
watchdog: BUG: soft lockup - CPU#3 stuck for 371s! [llama-server:2626]
Five CPUs locked. SSH was authenticating and then hanging because there was no CPU left to hand me a shell.
So I capped the service's CPU quota, which stopped the lockups immediately. It also destroyed the thing it was protecting. With the cap in place, a prompt still ran at normal speed, but generating a response slowed to six words in twenty five minutes.
The cap works in bursts. The process gets its budget, spends it, then gets frozen until the next window. That is survivable for work that runs in parallel and catastrophic for work where every step waits on the previous one.
The fix was to give the process fewer CPUs continuously rather than all of them intermittently. Same intent, opposite outcome.
I had built a control that succeeded at its stated goal and broke the system anyway. That is not the same failure as a control that does nothing, but you find both the same way: by measuring the thing you were protecting, not the thing you were protecting it from.
Four days ago I wrote about a script I built to verify my VM was isolated, which printed a green OK it was structurally incapable of not printing. I thought that was a story about my own carelessness.
Then I found a production tool doing a stronger version of the same thing, in a store whose entire premise is that verification is free and local.
A system that reports success without checking is not lying to you. It is telling you the truth about something else: that it ran.
The checks that fail this way are never the ones you are watching. They are the ones that print a word that sounds like a check.
Mine said [OK]
. This one said verifying
.
Neither one looked.
I filed it as ollama/ollama#17520, with a reproducer anyone can run in about thirty seconds.
Worth noting what was already in that tracker. There is an open issue from a maintainer about a manifest pointing at a file that is missing, which is adjacent but not the same: my file existed, at the right path, at the exact size the manifest declared. Existence and size both check out. Only hashing catches it.
And there is a closed feature request asking for exactly this kind of integrity checking, rejected in June with the advice to restart the server, because the server sweeps up broken downloads at startup.
I have the startup log where it swept, counted the corrupt file, and removed nothing.
Why the symptom moved. The store holds several files per model. Corrupt the small config file and the model vanishes from the installed list entirely. Corrupt one of the layers and the model lists as healthy, then dies at load with the same cryptic error. Same root cause, two completely different symptoms, and the error message names neither the model nor the file in either case.
Where the nulls came from. Almost certainly the hard resets. A file gets allocated, the machine dies before the data flushes, and the filesystem leaves you the right number of zeroed blocks. Which means my own bad fix for problem two is what created problem one, and I spent two days treating them as unrelated.
The sandbox and its verification scripts are at ai-security-lab.