We build a local-first indexer for macOS. It watches the user's home folder so that when a file changes, we reindex just that file. Watching one project directory is a solved problem. Watching a real human's $HOME
is a different sport, and it broke us in a way that took a heap profiler to see.
The app sat at roughly 25 GB of resident memory. Not a slow leak either. It climbed there shortly after launch and stayed. One core also pinned itself for about ten minutes at every boot.
The obvious suspects were wrong. It was not the embedding model, and it was not our index. We had a careful ignore list (caches, node_modules
, Library
, build output), so surely we were not touching millions of files.
We were not. But something else was.
We use the excellent notify
crate, specifically notify-debouncer-full
, which is what you reach for when you want raw filesystem events collapsed into something sane. The setup you copy from the README uses RecommendedCache
.
RecommendedCache
resolves to FileIdMap
. From the docs, the debouncer "can optionally keep track of the file system IDs all files and stitches rename events together". That is a genuinely useful feature: on macOS FSEvents and on Windows, a rename shows up as two unrelated events, and pairing them requires knowing that the file at the old path and the file at the new path are the same inode.
To do that, it has to know the file ID of every file under the watch root. So it walks the entire watch root and caches a (PathBuf, FileId)
for every entry it finds.
Here is the part that mattered: that walk does not know about your ignore rules. Our ignore list filters events. The cache is built underneath that, from the watch root down. We told the debouncer to watch $HOME
, so it faithfully walked all of $HOME
, including everything we thought we had excluded, following symlinks as it went.
Under malloc_history
, the damage was specific:
54 million allocations to remember the inode of files we had explicitly said we did not care about.
We switched the cache to NoCache
.
That is not free, and it is worth being precise about the tradeoff rather than pretending we outsmarted the library. FileIdMap
exists for a reason: with NoCache
, the debouncer can no longer stitch rename events together. A rename stops arriving as "this moved from A to B" and degrades into an unrelated delete at A and a create at B.
We could absorb that, for two reasons that are specific to our design:
So a rename becomes delete-plus-create, and the system settles into the right state on its own. If your indexer is not idempotent, or you have no reconcile pass, this trade is not available to you and you should fix the scope instead.
Result: 24.8 GB down to 1.45 GB.
While measuring, we caught something else. When FSEvents drops events (it does this under load, by design, and tells you so), the debouncer emits a rescan. Our code fanned that out as one rescan per *configured* root.
Our roots were nested. We watched $HOME
, and we also watched Documents
, Downloads
, and Desktop
because they were configured separately. So a single dropped event triggered four concurrent, roughly $HOME
-scale re-walks, and every file under the nested roots got embedded twice. You could see it plainly in the logs: two EmbeddingsGenerated
lines per document.
The fix was to de-nest the roots into independent subtrees before emitting any rescan, so overlapping configuration collapses into one walk.
Library defaults are calibrated for the common case, and the common case is a project folder with a few thousand files. Nothing in the API warns you, because nothing is wrong with the API. FileIdMap
is correct. It is correct at 5,000 files and it is a memory bomb at 6.4 million, and the difference is entirely in what you point it at.
So, two things:
Know what your watcher caches before you widen the scope. Filtering events is not the same as filtering what the library indexes internally. Ours sat below our filter, where we could not see it.
Measure the heap, do not reason about it. We would never have found 54.2 million path allocations by reading code. malloc_history
found it in one pass. Every hour we spent theorizing about the embedding model was an hour we did not spend attaching a profiler.
If you are watching a user's whole home directory, assume every default in your stack was written by someone who was not.