{"slug": "spider-2-0-deleted-a-claim-i-made-this-morning", "title": "Spider 2.0 deleted a claim I made this morning", "summary": "A developer found that swapping a hashed vectoriser for a sentence-transformer in a schema retriever produced a strict recall-at-10 gain from 82.6% to 88.1% on a pooled Spider 1.0 catalog (n=1,034), but the effect disappeared on Spider 2.0-lite, where hashed and sentence-model recall were statistically indistinguishable at 82.9% and 82.3% (n=158, a one-question difference). The developer attributes the discrepancy to Spider 2.0's real table descriptions, which let lexical scoring close the vocabulary gap that embeddings were compensating for on Spider 1.0, and labels the explanation a hypothesis fitted to two data points.", "body_md": "This morning I wrote that swapping the hashed vectoriser in my schema\n\nretriever for a sentence-transformer was a consistent win. I had numbers:\n\non a pooled Spider 1.0 catalog, strict recall at k=10 went from 82.6% to\n\n88.1%. Consistent, I said. Larger than my own benchmarks suggested.\n\nThis evening I ran the same comparison on Spider 2.0 and the effect vanished.\n\nNot shrank — vanished, and if anything went the other way.\n\nWhat follows is the run, what I think happened, and the experiment that would\n\nactually settle it, which I have not done yet.\n\nSpider 1.0's databases have a median of three tables. That is why my\n\nper-database numbers there are 100% and worthless: returning the entire schema\n\nalso scores 100%. I had to invent a pooled variant — every database merged into\n\none 876-table catalog — to make the task resemble retrieval at all, and an\n\ninvented variant is exactly the kind of thing a reader is right to discount.\n\nSpider 2.0-lite needs no such construction. Measured from the distribution:\n\n162 databases, 7,892 tables, a median of 15 tables per database and a maximum\n\nof 785, drawn from real BigQuery and Snowflake warehouses. The databases are\n\nalready big.\n\nTable recall, 158 questions across 103 databases. spider2-lite ships 547\n\nexamples in total, so this is a subset of the benchmark rather than the whole\n\nof it — I'll publish the harness and say exactly which subset when I push it:\n\n| top_k | all gold tables present | per-table recall | \n|---|---|---|\n| 5 | 70.9% | 81.7% | \n| 10 | 82.9% | 88.5% | \n| 20 | 86.1% | 90.5% | \n\nNothing was tuned for this. I downloaded it, pointed the same code at it, and\n\nthose are the numbers.\n\n**With the caveat the sample size demands.** n=158 gives a 95% confidence\n\ninterval of roughly [63.4, 77.4] on that 70.9%, and [76.3, 88.0] on the 82.9%.\n\nSo the honest comparison with my Spider 1.0 pooled figures is not \"within a\n\npoint or two\" — the intervals are too wide to resolve a point or two. It is\n\nthat the two are *indistinguishable at this sample size*, on databases an order\n\nof magnitude larger and questions written to be hard. That is still the\n\ninteresting result. It is just a weaker sentence than the one I wanted to\n\nwrite.\n\n| k=10, strict recall | hashed | sentence model | \n|---|---|---|\n| Spider 1.0 pooled (n=1,034) | 82.6% | **88.1%** | \n| Spider 2.0-lite (n=158) | 82.9% | 82.3% | \n\nAnd immediately, the discipline I failed to apply to my own good news this\n\nmorning: **82.9% and 82.3% differ by one question.** 131 versus 130. I cannot\n\nclaim the embedder is \"slightly worse\" from that, any more than I could have\n\nclaimed it was better. The correct statement is that there is *no measurable difference at n=158*.\n\nThat still kills \"consistent win,\" which is what I said and what was wrong.\n\nThe Spider 1.0 effect, meanwhile, is real: at n=1,034 the intervals around\n\n82.6% and 88.1% do not overlap. So the finding is not \"the embedder does\n\nnothing.\" It is *measurably useful on one benchmark and unmeasurable on the other*, and the reason matters.\n\nSpider 2.0's tables carry real descriptions, harvested from the warehouses'\n\nown data dictionaries. Spider 1.0's tables carry none — just identifiers.\n\nWhen a table has prose describing it, lexical scoring over that prose already\n\ncloses the gap between the words a user types and the words a schema uses. A\n\nquestion about \"revenue\" finds a table whose description says revenue, without\n\nany embedding involved. The sentence model was earning its keep on Spider 1.0\n\nby compensating for the absence of that text. Give the corpus the text and\n\nthere is less left to compensate for.\n\nThis is a hypothesis fitted to two data points and I am labelling it as such.\n\nA few days ago I wrote up the opposite-looking result: adding LLM-generated\n\ntable descriptions to a 1,245-object schema made retrieval *worse*, because\n\ngenerated prose inflated the document frequency of the domain's own nouns until\n\n\"contact\" scored an IDF of 0.15 and the `contacts` table fell out of the top 40.\n\nThose two results look unrelated. They are the same axis:\n\nThe value of any semantic layer — a learned encoder, or generated\n\ndescriptions — depends on how much prose the schema already has. Where prose\n\nexists, lexical scoring over it does most of the work. Where it doesn't, you\n\nneed something to bridge the vocabulary gap. And if you add prose to the same\n\nfield as the identifiers, you damage the identifiers.\n\nWhich gives a practical rule I can actually stand behind:\n\nTwo benchmarks differing in everything is a correlation, not a mechanism. The\n\ncontrolled version is one line of setup: strip the descriptions out of Spider\n\n2.0 and re-run the same comparison on the same questions and the same\n\ndatabases.\n\n| Spider 2.0-lite, k=10 | hashed | sentence model | \n|---|---|---|\n| with data-dictionary prose | 82.9% | 82.3% | \n| **descriptions removed** | ? | ? | \n\nIf the encoder's advantage reappears once the prose is gone, the hypothesis is\n\nconfirmed inside one corpus with everything else held constant. If it doesn't,\n\nthe explanation is something else — dialect, question style, table size — and I\n\nshould stop telling this story.\n\nI'll run it and post the row either way.\n\nSpider 2.0 carries `description` as a *list* for some tables. That crashed\n\nindexing with:\n\n```\nTypeError: sequence item 2: expected str instance, list found\n```\n\nNaming neither the object nor the field. A catalog of 800 tables was\n\nunindexable because one of them described itself in a list instead of a string.\n\nFixed at the boundary rather than defensively at each reader — `ObjectDoc`\n\nnormalises `description`, `hint` and column comments once on construction,\n\nand `None` stays `None`.\n\nThat bug would have hit anyone pointing this at a real data dictionary. It did\n\nnot show up across six schemas I wrote myself, 445 tests, or a live run against\n\na 127-object Oracle database, because I built all of those and I would never\n\nhave thought to put a list there. It took ninety seconds to find on someone\n\nelse's data.\n\nWhich is the actual argument for running public benchmarks, more than any\n\nnumber in the tables above. They are not there to prove you are good. They are\n\nthere to be data you did not shape.\n\n*The library is [schemagate](https://github.com/ashishsinha1602/schemagate),\nApache-2.0. The benchmark scripts are in `benchmarks/` and the data comes from\nthe original sources; the whole suite reruns in about three minutes.\nThe Spider 2.0 numbers and the corrected embedder row are not on `main` yet —\n[BENCHMARKS.md](https://github.com/ashishsinha1602/schemagate/blob/main/BENCHMARKS.md)\nstill shows the Spider 1.0 figures alone as I write this. It gets the\ncorrection, and I'd rather say that here than have you click and find the\nflattering version.*", "url": "https://wpnews.pro/news/spider-2-0-deleted-a-claim-i-made-this-morning", "canonical_source": "https://dev.to/ashish_sinha_5241c7673d93/spider-20-deleted-a-claim-i-made-this-morning-56i9", "published_at": "2026-09-13 04:09:10+00:00", "updated_at": "2026-09-13 04:26:18.850614+00:00", "lang": "en", "topics": ["natural-language-processing", "machine-learning", "ai-research", "ai-tools"], "entities": ["Spider 1.0", "Spider 2.0", "Spider 2.0-lite", "BigQuery", "Snowflake"], "alternates": {"html": "https://wpnews.pro/news/spider-2-0-deleted-a-claim-i-made-this-morning", "markdown": "https://wpnews.pro/news/spider-2-0-deleted-a-claim-i-made-this-morning.md", "text": "https://wpnews.pro/news/spider-2-0-deleted-a-claim-i-made-this-morning.txt", "jsonld": "https://wpnews.pro/news/spider-2-0-deleted-a-claim-i-made-this-morning.jsonld"}}