{"slug": "i-described-1245-tables-with-an-llm-and-retrieval-got-worse", "title": "I described 1,245 tables with an LLM and retrieval got worse", "summary": "A developer catalogued a 1,245-object database schema with an LLM, generating natural-language descriptions for each table, and found that retrieval quality dropped sharply rather than improving. The token \"contact\" went from appearing in 17 documents to roughly 1,072 of 1,245 after enrichment, collapsing its IDF score from 4.27 to 0.15, while BM25 length normalisation penalised the long, central `contacts` table and favoured short peripheral tables like `contact_import_log`. The developer attributes the regression to correlated noise — generated descriptions share a common vocabulary that inflates document frequency for exactly the terms users query.", "body_md": "The cataloguing step is supposed to be the easy win. You have a schema whose\n\ntables are called `ecm_template_link` and `v_pmpm`, your users ask questions\n\nin English, and the gap between those two vocabularies is why retrieval\n\nmisses. So you point a model at every table, get back a sentence describing\n\neach one, index the sentences alongside the names, and now the corpus speaks\n\nEnglish too.\n\nI did that to a real 1,245-object schema. Recall went **down**.\n\nNot by a little. The table literally called `contacts` sat at rank 3 for the\n\nquestion \"show the contacts of xmagnet\" before cataloguing. After\n\ncataloguing it was below rank 40 — off the end of anything I would put in a\n\nprompt. The descriptions were fine. I read them. They were accurate,\n\nspecific, and they made the system worse.\n\nThis post is what was actually happening, why the obvious fix doesn't work,\n\nand the one that does. None of it is specific to text-to-SQL. If you are\n\nenriching documents before indexing them — summaries, generated titles,\n\nhypothetical questions, keyword expansion, anything — the same mechanism is\n\navailable to bite you, and it will not announce itself.\n\nEvery description you generate is written in the same vocabulary as every\n\nother description, so enrichment raises the document frequency of exactly\n\nthe words your users type.\n\nBM25 scores a term by inverse document frequency:\n\n```\nidf(t) = log(1 + (N - n_t + 0.5) / (n_t + 0.5))\n```\n\n`N` is the corpus size, `n_t` the number of documents containing the term.\n\nA term in few documents is informative and scores high; a term in most\n\ndocuments is worthless and scores near zero. That is the entire point of\n\nIDF, and it is normally a good instinct.\n\nNow think about what a model writes when you ask it to describe a table in a\n\nCRM schema. It writes about contacts. It writes about contacts when\n\ndescribing `contacts`, and also when describing `contact_lists`,\n\n`campaign_recipients`, `email_events`, `tenants`, `users`, and the audit\n\ntable that logs changes to any of them — because in a CRM, almost everything\n\nis *about* contacts in some defensible sense. The descriptions are not\n\nwrong. They are correlated.\n\nAfter cataloguing, the token `contact` appeared in roughly **1,072 of the 1,245** documents — a figure I can reconstruct from the IDF it produced,\n\nFor comparison, here is what the same schema gives you for a genuinely rare\n\nterm:\n\n| term | documents containing it | idf | \n|---|---|---|\n| `contact` , after cataloguing | ~1,072 of 1,245 | **0.15** | \n| `contact` , in the name field only | 17 of 1,245 | **4.27** | \n| `tenant` , in the name field only | 30 of 1,245 | 3.71 | \n| `the` | 0 of 1,245 | — | \n\nThe tokenizer does not strip stopwords, so `the` and `and` are in that\n\nindex too, sitting near zero because they are in everything. At 0.15,\n\n`contact` had joined them. The word the user typed was, for scoring purposes,\n\na function word.\n\nIDF collapse alone would flatten the ranking. What actively inverted it was\n\nlength normalisation.\n\nThe `b` parameter in BM25 penalises long documents, on the sound theory that\n\na long document containing your term is less *about* your term than a short\n\none that contains it:\n\n```\nscore += idf(t) * f * (k1 + 1) / (f + k1 * (1 - b + b * len / avg_len))\n```\n\nAsk which object in a CRM schema has the longest document, and the answer is\n\nthe central one. `contacts` in this schema has 55 columns. Add a generated\n\ndescription and a row of alias words and its document is several times the\n\ncorpus average. Meanwhile `contact_import_log` has six columns and a\n\none-line description, so it is short, tidy, and — as far as the length\n\nprior is concerned — much more *about* contacts.\n\nSo the two effects compound in the same direction:\n\n`contacts` from\nthe forty other tables mentioning contacts.\nCataloguing didn't add noise. It added *correlated* noise, and correlated\n\nnoise attacks the exact query it was meant to help. The questions that\n\ndegraded most were the ones cataloguing exists to serve — plain English, no\n\nschema words. Questions that named a table outright were mostly fine, because\n\nthey had a rare token to hang on. That is a nasty failure profile: the\n\nfeature looks fine on your smoke tests and fails on your users.\n\nThe obvious response is to trust descriptions less. One index, but weight the\n\ngenerated text below the real text.\n\nI did this first. It helps a bit and it is the wrong lever, for a reason\n\nthat took me a while to see: **weight and dilution act at different stages.**\n\nDown-weighting scales the contribution of a term *after* IDF has already been\n\ncomputed over a corpus the descriptions polluted. `contact` is still worth\n\n0.15 in the name's own score, because name and description live in one bag of\n\nwords and IDF is a property of the bag. You have made a bad channel quieter\n\nwithout making the good channel accurate again.\n\nAnd the cost is real. Starving the prose weight cost me\n\n\"per member per month cost\" → `v_pmpm`, which is the single best example in\n\nthe whole schema of a question only a description can answer. There is no\n\nlexical path from that phrase to that name. The description was the only\n\nbridge and I had just defunded it.\n\nSo: down-weighting trades away the wins to partially mitigate the losses. You\n\nend up tuning a scalar that makes both worse than they need to be.\n\nScore the fields separately and fuse the rankings, rather than concatenating\n\nthe fields and scoring once.\n\nThree BM25 indexes over the same objects:\n\n```\nself._bm25       = _BM25([doc.embed_text() for doc in docs])   # everything\nself._bm25_name  = _BM25([_name_text(doc)  for doc in docs])   # identifiers\nself._bm25_prose = _BM25([_prose_text(doc) for doc in docs])   # written text\n```\n\nwhere\n\n``` python\ndef _name_text(doc):\n    \"\"\"Just the identifiers: schema, name, and the name split on underscores.\"\"\"\n    return \" \".join(x for x in (doc.schema, doc.name,\n                                doc.name.replace(\"_\", \" \")) if x)\n\ndef _prose_text(doc):\n    \"\"\"Everything written *about* the object: hint, description, comments.\"\"\"\n    parts = [doc.hint or \"\", doc.description or \"\"]\n    parts.extend(c.comment or \"\" for c in doc.columns)\n    return \" \".join(x for x in parts if x)\n```\n\nThen fuse by reciprocal rank rather than by score:\n\n```\nfor q in candidates:\n    s = 0.0\n    if q in vec_rank:   s += vector_weight  / (RRF_K + vec_rank[q]   + 1)\n    if q in lex_rank:   s += lexical_weight / (RRF_K + lex_rank[q]   + 1)\n    if q in name_rank:  s += NAME_WEIGHT    / (RRF_K + name_rank[q]  + 1)\n    if q in prose_rank: s += PROSE_WEIGHT   / (RRF_K + prose_rank[q] + 1)\n```\n\nBoth halves of the bug die at once, and it is worth being precise about why,\n\nbecause \"just use fielded search\" is advice people give without the\n\nmechanism:\n\n**IDF is recomputed per field.** In the name index, the only text is\n\nidentifiers. Nothing a model writes can ever enter it. `contact` appears in\n\n17 names out of 1,245, so its IDF is 4.27 instead of 0.15 — 28× the\n\ndiscriminating power, restored by construction rather than by tuning.\n\n**Length is per field too.** The name index's document length is the length\n\nof the name. `contacts` is two tokens whatever else you attach to the object.\n\nThe 55 columns cannot inflate it, so the length prior stops punishing\n\ncentrality.\n\n**Fusion is over ranks, not scores.** This is the part that contains a bad\n\ncatalogue, and it's why I could raise the prose weight back to parity. A\n\nchannel can only ever contribute its own ranking. If a weak model writes\n\n\"Stores data about users and their settings\" about all 1,245 objects, the\n\nprose channel becomes uniformly useless — every object ranks the same, the\n\nchannel contributes nothing that discriminates, and the name and body\n\nchannels decide the result unchanged. The floor becomes *\"no better than before cataloguing\"* instead of \n\nThat last property is the one I actually care about. It means pointing a\n\nsmall local model at your schema is safe. Not good, necessarily — a 1.5B\n\nmodel writes considerably worse descriptions than a frontier model, and I'd\n\nrather you use the good one. But safe: bad prose can no longer bury the\n\nobject it describes, so the downside of trying is bounded.\n\nThe pattern is not about databases. It is:\n\nGenerated text about a corpus is written in the corpus's own vocabulary,\n\nso enrichment inflates document frequency for the domain's central terms —\n\nthe ones users search with — and inflates document length most for the\n\nitems that matter most.\n\nAnywhere you generate text and index it next to original text, in the same\n\nfield, you have signed up for both effects:\n\nNone of these are bad ideas. I still catalogue schemas; recall on\n\nbusiness-phrased questions is far better with descriptions than without. The\n\nclaim is narrower: **enrichment belongs in its own field, always.** The cost\n\nof separating fields is one more index and a fusion step. The cost of not\n\nseparating them is a regression that shows up only on your most important\n\nqueries and looks like \"retrieval is just hard\".\n\nIf you want to check whether this is happening to you, it is one query and\n\nno instrumentation: take the ten nouns your users actually type, and print\n\ntheir document frequency before and after your enrichment step. If any of\n\nthem are now in more than half your documents, that term is doing nothing,\n\nand it was probably doing something before.\n\nHonesty about the edges, since the above reads tidier than the week did:\n\nFielded scoring does not make a bad catalogue good. It makes it harmless. If\n\nyour descriptions are generic, you get the pre-cataloguing ranking back, not\n\na better one — which is the right outcome, but don't read it as a licence to\n\nskip evaluating the model that writes them.\n\nIt also introduces a knob per field, and I do not have a principled method\n\nfor setting them. Mine are all at parity because that tested best across six\n\nschemas, not because parity is theoretically correct.\n\nAnd separating fields cannot fix a term that is genuinely common in the\n\n*names* too. A schema with 300 tables actually called `contact_something` has\n\na real ambiguity problem, and no amount of field isolation invents the\n\ninformation to resolve it.\n\nThe measurements here come from schemagate, an open-source library\n\n(Apache-2.0) that does the retrieval step for text-to-SQL. The relevant code\n\nis in `catalog.py`\n\n— the comments around the three `_BM25` constructions are where I wrote this\n\ndown while it was still fresh. There's a browser demo at\n\n[ashishsinha1602.github.io/schemagate](https://ashishsinha1602.github.io/schemagate/)\n\nthat runs the real selector client-side on six sample schemas, if you'd\n\nrather poke at the ranking than read about it.\n\nIf you run the document-frequency check on your own corpus, I'd like to know\n\nwhat it says — particularly if it says nothing is wrong, because I'd like to\n\nknow what makes a corpus immune.", "url": "https://wpnews.pro/news/i-described-1245-tables-with-an-llm-and-retrieval-got-worse", "canonical_source": "https://dev.to/ashish_sinha_5241c7673d93/i-described-1245-tables-with-an-llm-and-retrieval-got-worse-58a", "published_at": "2026-09-12 23:35:28+00:00", "updated_at": "2026-09-12 23:54:56.937906+00:00", "lang": "en", "topics": ["large-language-models", "natural-language-processing", "ai-tools", "developer-tools"], "entities": ["BM25", "LLM"], "alternates": {"html": "https://wpnews.pro/news/i-described-1245-tables-with-an-llm-and-retrieval-got-worse", "markdown": "https://wpnews.pro/news/i-described-1245-tables-with-an-llm-and-retrieval-got-worse.md", "text": "https://wpnews.pro/news/i-described-1245-tables-with-an-llm-and-retrieval-got-worse.txt", "jsonld": "https://wpnews.pro/news/i-described-1245-tables-with-an-llm-and-retrieval-got-worse.jsonld"}}