When one embedding per chunk stops working

A team running semantic search over documents found that 186 of 3,000 chunks on its reference test store were never returned by any query, because heading-only chunks with empty bodies embedded as empty vectors; embedding the title for heading-only chunks brought all 186 back. The team then stopped generating one embedding per chunk, deriving separate search units for heading paths, prose blocks, code blocks and transclusions while a resolution index maps each unit back to its full chunk. Adding a predicate that excludes documents from the shared per-scope vector plane cut memory recall from 928 ms to 30 ms on a 2,942-row scope, roughly 31 times faster; the system runs on Postgres because SQLite lacks a working vector tier in practice, and a full retrieval-quality report is due with the next release.

Any system that does semantic search over documents has to decide what to embed. The obvious choice is one embedding per chunk. That is how ours worked for months. Then we noticed some chunks were invisible. 186 out of 3,000 chunks on our reference test store never came back in any query. Not ranked poorly. Never returned. Every invisible chunk was a heading-only chunk whose body was empty because the meaningful content was in the sub-chunks below it. When you embed an empty body, you get an empty vector, and nothing matches. Fixing that part was one line: a heading-only chunk should embed its title. The 186 came back. The bigger issue was the chunks that were NOT invisible. A chunk with a heading, a code block, prose, and a transclusion, all averaged into one vector, loses queries to whichever piece dominates the average. So we stopped generating one embedding per chunk. Each chunk now derives multiple search units: a heading path, a prose block, a code block, a transclusion each get their own embedding. The resolution index knows which chunk each unit belongs to; a query matching the code unit returns the whole chunk with the code block highlighted, not the code block alone. Also folded in: the side-effect bug we caught along the way. Memory recall had been accidentally reaching into document embeddings because we had put both into the same per-scope vector plane, and adding a predicate that excludes documents took memory recall from 928 ms to 30 ms on a 2,942-row scope about 31 times faster . Two limits worth naming: not every chunk has clear semantic pieces a chunk that is one long paragraph does not benefit from derived search units, because there is nothing to derive; it still embeds as one vector, and that is the correct behaviour ; and this all runs on Postgres, because SQLite does not have a working vector tier in practice. The full retrieval-quality report lands with the next release, running the same paired benchmark used for memory. Follows the tool-help arc; the memory-retrieval post that comes after this one closes the September through October arc.