{"slug": "our-rust-file-watcher-ate-23-6-gb-of-ram-and-our-ignore-rules-never-had-a-chance", "title": "Our Rust file watcher ate 23.6 GB of RAM, and our ignore rules never had a chance", "summary": "A developer building a local-first file indexer for macOS discovered that the notify-debouncer-full crate's RecommendedCache (FileIdMap) walked the entire watch root, ignoring custom ignore rules, and cached file IDs for all files under $HOME, consuming 23.6 GB of RAM. Switching to NoCache reduced memory from 24.8 GB to 1.45 GB, though it sacrifices rename-event stitching. The team also fixed nested root rescans that caused duplicate embeddings.", "body_md": "We build a local-first indexer for macOS. It watches the user's home folder so that when a file changes, we reindex just that file. Watching one project directory is a solved problem. Watching a real human's `$HOME`\n\nis a different sport, and it broke us in a way that took a heap profiler to see.\n\nThe app sat at roughly 25 GB of resident memory. Not a slow leak either. It climbed there shortly after launch and stayed. One core also pinned itself for about ten minutes at every boot.\n\nThe obvious suspects were wrong. It was not the embedding model, and it was not our index. We had a careful ignore list (caches, `node_modules`\n\n, `Library`\n\n, build output), so surely we were not touching millions of files.\n\nWe were not. But something else was.\n\nWe use the excellent `notify`\n\ncrate, specifically `notify-debouncer-full`\n\n, which is what you reach for when you want raw filesystem events collapsed into something sane. The setup you copy from the README uses `RecommendedCache`\n\n.\n\n`RecommendedCache`\n\nresolves to `FileIdMap`\n\n. From the docs, the debouncer \"can optionally keep track of the file system IDs all files and stitches rename events together\". That is a genuinely useful feature: on macOS FSEvents and on Windows, a rename shows up as two unrelated events, and pairing them requires knowing that the file at the old path and the file at the new path are the same inode.\n\nTo do that, it has to know the file ID of every file under the watch root. So it walks the entire watch root and caches a `(PathBuf, FileId)`\n\nfor every entry it finds.\n\nHere is the part that mattered: **that walk does not know about your ignore rules.** Our ignore list filters *events*. The cache is built underneath that, from the watch root down. We told the debouncer to watch `$HOME`\n\n, so it faithfully walked all of `$HOME`\n\n, including everything we thought we had excluded, following symlinks as it went.\n\nUnder `malloc_history`\n\n, the damage was specific:\n\n54 million allocations to remember the inode of files we had explicitly said we did not care about.\n\nWe switched the cache to `NoCache`\n\n.\n\nThat is not free, and it is worth being precise about the tradeoff rather than pretending we outsmarted the library. `FileIdMap`\n\nexists for a reason: with `NoCache`\n\n, the debouncer can no longer stitch rename events together. A rename stops arriving as \"this moved from A to B\" and degrades into an unrelated delete at A and a create at B.\n\nWe could absorb that, for two reasons that are specific to our design:\n\nSo a rename becomes delete-plus-create, and the system settles into the right state on its own. If your indexer is not idempotent, or you have no reconcile pass, this trade is not available to you and you should fix the scope instead.\n\nResult: 24.8 GB down to 1.45 GB.\n\nWhile measuring, we caught something else. When FSEvents drops events (it does this under load, by design, and tells you so), the debouncer emits a rescan. Our code fanned that out as one rescan per *configured* root.\n\nOur roots were nested. We watched `$HOME`\n\n, and we also watched `Documents`\n\n, `Downloads`\n\n, and `Desktop`\n\nbecause they were configured separately. So a single dropped event triggered four concurrent, roughly `$HOME`\n\n-scale re-walks, and every file under the nested roots got embedded twice. You could see it plainly in the logs: two `EmbeddingsGenerated`\n\nlines per document.\n\nThe fix was to de-nest the roots into independent subtrees before emitting any rescan, so overlapping configuration collapses into one walk.\n\nLibrary defaults are calibrated for the common case, and the common case is a project folder with a few thousand files. Nothing in the API warns you, because nothing is wrong with the API. `FileIdMap`\n\nis correct. It is correct at 5,000 files and it is a memory bomb at 6.4 million, and the difference is entirely in what you point it at.\n\nSo, two things:\n\n**Know what your watcher caches before you widen the scope.** Filtering events is not the same as filtering what the library indexes internally. Ours sat below our filter, where we could not see it.\n\n**Measure the heap, do not reason about it.** We would never have found 54.2 million path allocations by reading code. `malloc_history`\n\nfound it in one pass. Every hour we spent theorizing about the embedding model was an hour we did not spend attaching a profiler.\n\nIf you are watching a user's whole home directory, assume every default in your stack was written by someone who was not.", "url": "https://wpnews.pro/news/our-rust-file-watcher-ate-23-6-gb-of-ram-and-our-ignore-rules-never-had-a-chance", "canonical_source": "https://dev.to/jacksonxly/our-rust-file-watcher-ate-236-gb-of-ram-and-our-ignore-rules-never-had-a-chance-49pn", "published_at": "2026-07-13 13:40:17+00:00", "updated_at": "2026-07-13 13:46:27.103560+00:00", "lang": "en", "topics": ["developer-tools"], "entities": ["notify-debouncer-full", "FileIdMap", "NoCache", "macOS", "FSEvents"], "alternates": {"html": "https://wpnews.pro/news/our-rust-file-watcher-ate-23-6-gb-of-ram-and-our-ignore-rules-never-had-a-chance", "markdown": "https://wpnews.pro/news/our-rust-file-watcher-ate-23-6-gb-of-ram-and-our-ignore-rules-never-had-a-chance.md", "text": "https://wpnews.pro/news/our-rust-file-watcher-ate-23-6-gb-of-ram-and-our-ignore-rules-never-had-a-chance.txt", "jsonld": "https://wpnews.pro/news/our-rust-file-watcher-ate-23-6-gb-of-ram-and-our-ignore-rules-never-had-a-chance.jsonld"}}