Our Rust file watcher ate 23.6 GB of RAM, and our ignore rules never had a chance

A developer building a local-first file indexer for macOS discovered that the notify-debouncer-full crate's RecommendedCache (FileIdMap) walked the entire watch root, ignoring custom ignore rules, and cached file IDs for all files under $HOME, consuming 23.6 GB of RAM. Switching to NoCache reduced memory from 24.8 GB to 1.45 GB, though it sacrifices rename-event stitching. The team also fixed nested root rescans that caused duplicate embeddings.

We build a local-first indexer for macOS. It watches the user's home folder so that when a file changes, we reindex just that file. Watching one project directory is a solved problem. Watching a real human's $HOME is a different sport, and it broke us in a way that took a heap profiler to see. The app sat at roughly 25 GB of resident memory. Not a slow leak either. It climbed there shortly after launch and stayed. One core also pinned itself for about ten minutes at every boot. The obvious suspects were wrong. It was not the embedding model, and it was not our index. We had a careful ignore list caches, node modules , Library , build output , so surely we were not touching millions of files. We were not. But something else was. We use the excellent notify crate, specifically notify-debouncer-full , which is what you reach for when you want raw filesystem events collapsed into something sane. The setup you copy from the README uses RecommendedCache . RecommendedCache resolves to FileIdMap . From the docs, the debouncer "can optionally keep track of the file system IDs all files and stitches rename events together". That is a genuinely useful feature: on macOS FSEvents and on Windows, a rename shows up as two unrelated events, and pairing them requires knowing that the file at the old path and the file at the new path are the same inode. To do that, it has to know the file ID of every file under the watch root. So it walks the entire watch root and caches a PathBuf, FileId for every entry it finds. Here is the part that mattered: that walk does not know about your ignore rules. Our ignore list filters events . The cache is built underneath that, from the watch root down. We told the debouncer to watch $HOME , so it faithfully walked all of $HOME , including everything we thought we had excluded, following symlinks as it went. Under malloc history , the damage was specific: 54 million allocations to remember the inode of files we had explicitly said we did not care about. We switched the cache to NoCache . That is not free, and it is worth being precise about the tradeoff rather than pretending we outsmarted the library. FileIdMap exists for a reason: with NoCache , the debouncer can no longer stitch rename events together. A rename stops arriving as "this moved from A to B" and degrades into an unrelated delete at A and a create at B. We could absorb that, for two reasons that are specific to our design: So a rename becomes delete-plus-create, and the system settles into the right state on its own. If your indexer is not idempotent, or you have no reconcile pass, this trade is not available to you and you should fix the scope instead. Result: 24.8 GB down to 1.45 GB. While measuring, we caught something else. When FSEvents drops events it does this under load, by design, and tells you so , the debouncer emits a rescan. Our code fanned that out as one rescan per configured root. Our roots were nested. We watched $HOME , and we also watched Documents , Downloads , and Desktop because they were configured separately. So a single dropped event triggered four concurrent, roughly $HOME -scale re-walks, and every file under the nested roots got embedded twice. You could see it plainly in the logs: two EmbeddingsGenerated lines per document. The fix was to de-nest the roots into independent subtrees before emitting any rescan, so overlapping configuration collapses into one walk. Library defaults are calibrated for the common case, and the common case is a project folder with a few thousand files. Nothing in the API warns you, because nothing is wrong with the API. FileIdMap is correct. It is correct at 5,000 files and it is a memory bomb at 6.4 million, and the difference is entirely in what you point it at. So, two things: Know what your watcher caches before you widen the scope. Filtering events is not the same as filtering what the library indexes internally. Ours sat below our filter, where we could not see it. Measure the heap, do not reason about it. We would never have found 54.2 million path allocations by reading code. malloc history found it in one pass. Every hour we spent theorizing about the embedding model was an hour we did not spend attaching a profiler. If you are watching a user's whole home directory, assume every default in your stack was written by someone who was not.