I described 1,245 tables with an LLM and retrieval got worse

A developer catalogued a 1,245-object database schema with an LLM, generating natural-language descriptions for each table, and found that retrieval quality dropped sharply rather than improving. The token "contact" went from appearing in 17 documents to roughly 1,072 of 1,245 after enrichment, collapsing its IDF score from 4.27 to 0.15, while BM25 length normalisation penalised the long, central `contacts` table and favoured short peripheral tables like `contact_import_log`. The developer attributes the regression to correlated noise — generated descriptions share a common vocabulary that inflates document frequency for exactly the terms users query.

The cataloguing step is supposed to be the easy win. You have a schema whose tables are called ecm template link and v pmpm , your users ask questions in English, and the gap between those two vocabularies is why retrieval misses. So you point a model at every table, get back a sentence describing each one, index the sentences alongside the names, and now the corpus speaks English too. I did that to a real 1,245-object schema. Recall went down . Not by a little. The table literally called contacts sat at rank 3 for the question "show the contacts of xmagnet" before cataloguing. After cataloguing it was below rank 40 — off the end of anything I would put in a prompt. The descriptions were fine. I read them. They were accurate, specific, and they made the system worse. This post is what was actually happening, why the obvious fix doesn't work, and the one that does. None of it is specific to text-to-SQL. If you are enriching documents before indexing them — summaries, generated titles, hypothetical questions, keyword expansion, anything — the same mechanism is available to bite you, and it will not announce itself. Every description you generate is written in the same vocabulary as every other description, so enrichment raises the document frequency of exactly the words your users type. BM25 scores a term by inverse document frequency: idf t = log 1 + N - n t + 0.5 / n t + 0.5 N is the corpus size, n t the number of documents containing the term. A term in few documents is informative and scores high; a term in most documents is worthless and scores near zero. That is the entire point of IDF, and it is normally a good instinct. Now think about what a model writes when you ask it to describe a table in a CRM schema. It writes about contacts. It writes about contacts when describing contacts , and also when describing contact lists , campaign recipients , email events , tenants , users , and the audit table that logs changes to any of them — because in a CRM, almost everything is about contacts in some defensible sense. The descriptions are not wrong. They are correlated. After cataloguing, the token contact appeared in roughly 1,072 of the 1,245 documents — a figure I can reconstruct from the IDF it produced, For comparison, here is what the same schema gives you for a genuinely rare term: | term | documents containing it | idf | |---|---|---| | contact , after cataloguing | ~1,072 of 1,245 | 0.15 | | contact , in the name field only | 17 of 1,245 | 4.27 | | tenant , in the name field only | 30 of 1,245 | 3.71 | | the | 0 of 1,245 | — | The tokenizer does not strip stopwords, so the and and are in that index too, sitting near zero because they are in everything. At 0.15, contact had joined them. The word the user typed was, for scoring purposes, a function word. IDF collapse alone would flatten the ranking. What actively inverted it was length normalisation. The b parameter in BM25 penalises long documents, on the sound theory that a long document containing your term is less about your term than a short one that contains it: score += idf t f k1 + 1 / f + k1 1 - b + b len / avg len Ask which object in a CRM schema has the longest document, and the answer is the central one. contacts in this schema has 55 columns. Add a generated description and a row of alias words and its document is several times the corpus average. Meanwhile contact import log has six columns and a one-line description, so it is short, tidy, and — as far as the length prior is concerned — much more about contacts. So the two effects compound in the same direction: contacts from the forty other tables mentioning contacts. Cataloguing didn't add noise. It added correlated noise, and correlated noise attacks the exact query it was meant to help. The questions that degraded most were the ones cataloguing exists to serve — plain English, no schema words. Questions that named a table outright were mostly fine, because they had a rare token to hang on. That is a nasty failure profile: the feature looks fine on your smoke tests and fails on your users. The obvious response is to trust descriptions less. One index, but weight the generated text below the real text. I did this first. It helps a bit and it is the wrong lever, for a reason that took me a while to see: weight and dilution act at different stages. Down-weighting scales the contribution of a term after IDF has already been computed over a corpus the descriptions polluted. contact is still worth 0.15 in the name's own score, because name and description live in one bag of words and IDF is a property of the bag. You have made a bad channel quieter without making the good channel accurate again. And the cost is real. Starving the prose weight cost me "per member per month cost" → v pmpm , which is the single best example in the whole schema of a question only a description can answer. There is no lexical path from that phrase to that name. The description was the only bridge and I had just defunded it. So: down-weighting trades away the wins to partially mitigate the losses. You end up tuning a scalar that makes both worse than they need to be. Score the fields separately and fuse the rankings, rather than concatenating the fields and scoring once. Three BM25 indexes over the same objects: self. bm25 = BM25 doc.embed text for doc in docs everything self. bm25 name = BM25 name text doc for doc in docs identifiers self. bm25 prose = BM25 prose text doc for doc in docs written text where python def name text doc : """Just the identifiers: schema, name, and the name split on underscores.""" return " ".join x for x in doc.schema, doc.name, doc.name.replace " ", " " if x def prose text doc : """Everything written about the object: hint, description, comments.""" parts = doc.hint or "", doc.description or "" parts.extend c.comment or "" for c in doc.columns return " ".join x for x in parts if x Then fuse by reciprocal rank rather than by score: for q in candidates: s = 0.0 if q in vec rank: s += vector weight / RRF K + vec rank q + 1 if q in lex rank: s += lexical weight / RRF K + lex rank q + 1 if q in name rank: s += NAME WEIGHT / RRF K + name rank q + 1 if q in prose rank: s += PROSE WEIGHT / RRF K + prose rank q + 1 Both halves of the bug die at once, and it is worth being precise about why, because "just use fielded search" is advice people give without the mechanism: IDF is recomputed per field. In the name index, the only text is identifiers. Nothing a model writes can ever enter it. contact appears in 17 names out of 1,245, so its IDF is 4.27 instead of 0.15 — 28× the discriminating power, restored by construction rather than by tuning. Length is per field too. The name index's document length is the length of the name. contacts is two tokens whatever else you attach to the object. The 55 columns cannot inflate it, so the length prior stops punishing centrality. Fusion is over ranks, not scores. This is the part that contains a bad catalogue, and it's why I could raise the prose weight back to parity. A channel can only ever contribute its own ranking. If a weak model writes "Stores data about users and their settings" about all 1,245 objects, the prose channel becomes uniformly useless — every object ranks the same, the channel contributes nothing that discriminates, and the name and body channels decide the result unchanged. The floor becomes "no better than before cataloguing" instead of That last property is the one I actually care about. It means pointing a small local model at your schema is safe. Not good, necessarily — a 1.5B model writes considerably worse descriptions than a frontier model, and I'd rather you use the good one. But safe: bad prose can no longer bury the object it describes, so the downside of trying is bounded. The pattern is not about databases. It is: Generated text about a corpus is written in the corpus's own vocabulary, so enrichment inflates document frequency for the domain's central terms — the ones users search with — and inflates document length most for the items that matter most. Anywhere you generate text and index it next to original text, in the same field, you have signed up for both effects: None of these are bad ideas. I still catalogue schemas; recall on business-phrased questions is far better with descriptions than without. The claim is narrower: enrichment belongs in its own field, always. The cost of separating fields is one more index and a fusion step. The cost of not separating them is a regression that shows up only on your most important queries and looks like "retrieval is just hard". If you want to check whether this is happening to you, it is one query and no instrumentation: take the ten nouns your users actually type, and print their document frequency before and after your enrichment step. If any of them are now in more than half your documents, that term is doing nothing, and it was probably doing something before. Honesty about the edges, since the above reads tidier than the week did: Fielded scoring does not make a bad catalogue good. It makes it harmless. If your descriptions are generic, you get the pre-cataloguing ranking back, not a better one — which is the right outcome, but don't read it as a licence to skip evaluating the model that writes them. It also introduces a knob per field, and I do not have a principled method for setting them. Mine are all at parity because that tested best across six schemas, not because parity is theoretically correct. And separating fields cannot fix a term that is genuinely common in the names too. A schema with 300 tables actually called contact something has a real ambiguity problem, and no amount of field isolation invents the information to resolve it. The measurements here come from schemagate, an open-source library Apache-2.0 that does the retrieval step for text-to-SQL. The relevant code is in catalog.py — the comments around the three BM25 constructions are where I wrote this down while it was still fresh. There's a browser demo at ashishsinha1602.github.io/schemagate https://ashishsinha1602.github.io/schemagate/ that runs the real selector client-side on six sample schemas, if you'd rather poke at the ranking than read about it. If you run the document-frequency check on your own corpus, I'd like to know what it says — particularly if it says nothing is wrong, because I'd like to know what makes a corpus immune.