This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.
Project Overview
The Vesuvius Challenge uses machine learning to read carbonized Herculaneum scrolls, which is 2,000-year-old papyrus that got buried by the eruption of Vesuvius and can never be physically unrolled. Its open-source monorepo, ScrollPrize/villa, contains the vesuvius Python package that researchers use to stream multi-terabyte CT scan volumes and train ink-detection models.
I was setting up that package on my Windows 11 machine (the project's CI only tests Ubuntu, and the workflow file literally says "Extend this list once the build scripts for macOS and Windows are confirmed"), working with an AI coding assistant to run the test suite on a platform it had never been tested on. One test failed. Then it passed. Then it failed again.
Bug Fix or Performance Improvement
The test, test_shared_cache_multiprocess_reads_are_not_torn, spawns four processes that read one scroll volume through a shared on-disk chunk cache. Run it once and you might not see anything wrong. So I ran it twelve times: 7 failures out of 12, all PermissionError: [WinError 5] Access is denied.
A 58% flake isn't a flake. It's a bug with a coin flip attached.
The cache is on the hot path for real usage. It's the component behind the package's documented volume_cache_dir config and the --cache-dir flag of its inference CLI. Any PyTorch Data with num_workers > 0 puts multiple processes into exactly this concurrent pattern, so on Windows, training runs would randomly die mid-epoch.
Once I dug in (a standalone reproducer that propagated full worker tracebacks instead of repr(exc)), the failure turned out to have three separate surfaces, each one hiding behind the previous one:
Cache-entry commit. The zarr library commits each cache entry with a write-temp-then-os.replace pattern. On POSIX, rename(2) over a file another process has open is legal. On Windows, MoveFileEx(MOVEFILE_REPLACE_EXISTING) returns ERROR_ACCESS_DENIED. Four workers populating the same content-addressed keys collide constantly. (This is upstream zarr-developers/zarr-python#3522, open since October, three confirmations, no fix.)
Cache read. Fix the write path and a second surface shows up: a concurrent commit can deny the reader's open too.
Eviction accounting. The package's own LRU sweep only caught FileNotFoundError when deleting old entries. Windows raises PermissionError for in-use files, and the sweep then subtracted the file's bytes from the size budget anyway, under-evicting a cache whose entire job is staying under a byte budget. This one is a genuine bug on every operating system, not just Windows.
The fix follows one principle: a cache is an optimization, never a source of truth, so a refused cache operation must never abort the read that triggered it. I wrapped only the cache-side store. Refused writes degrade to no-ops (logged at debug), and unreadable entries report as a miss, which extends the store's own existing "missing file = miss" semantics, so the outer cache just refetches from the source. The eviction sweep now skips undeletable entries without crediting their bytes and evicts the next-oldest instead. No platform-specific branches anywhere, and the trade-off (a full disk becomes slow refetches rather than a crash) is deliberate and documented.
Measured result: 58% failure to 0 failures in 12 consecutive runs. The package's Windows test suite went from 47 passed / 2 failed to 52 passed / 0 failed.
Because a 58% race is a terrible CI signal, I also added three deterministic regression tests that force each PermissionError surface via monkeypatching instead of racing for it, each one verified to fail against the unfixed code on any OS. (Plus a bonus find while I was in there: the LRU test stamped files with 1-3 nanosecond timestamps, which NTFS at 100 ns resolution collapses to st_mtime_ns == 0, silently destroying the ordering the test depends on. ext4 has 1 ns resolution, which is why Linux CI never noticed.)
Code
The full fix, tests, and methodology:
https://github.com/ScrollPrize/villa/pull/1545 Three files: the cache wrapper and eviction fix in vesuvius/src/vesuvius/ink_detection/volume_io.py, the regression tests in vesuvius/tests/ink_detection/test_volume_io.py, and a platform marker in pyproject.toml (the CUDA-only cucim-cu13 dependency ships manylinux wheels only, which made uv sync --extra all unresolvable on Windows and macOS).
My Improvements
A component that crashed 58% of the time under multiprocess access on Windows now runs clean, on the hot path used by anyone training ink-detection models with Data workers and a disk cache.
An OS-independent eviction accounting bug is gone.
CI gets a deterministic signal for a whole class of failure it previously couldn't see, since the racing test only catches the bug half the time even on the affected platform.
The diagnosis is documented down to the Win32 semantics, including what the upstream zarr issue was missing (mechanism, failure rate, minimal repro), so it's actionable beyond just this one repo.
Two lessons I'm keeping: run flaky tests twelve times, not twice, because a 58% failure rate read as "flaky" for months since nobody actually measured it. And fix one surface at a time, because two of the three bugs here were invisible until the one in front of them was gone.
Workflow transparency: I did this with an AI coding assistant (Claude) driving the investigation under my direction, measuring the failure rate, bisecting the three surfaces, and drafting the fix, with every measurement re-run and verified on my machine. The villa project explicitly welcomes LLM-assisted contributions with human commentary, and this writeup plus the PR discussion is exactly that.