# Spider 2.0 deleted a claim I made this morning > Source: > Published: 2026-09-13 04:09:10+00:00 This morning I wrote that swapping the hashed vectoriser in my schema retriever for a sentence-transformer was a consistent win. I had numbers: on a pooled Spider 1.0 catalog, strict recall at k=10 went from 82.6% to 88.1%. Consistent, I said. Larger than my own benchmarks suggested. This evening I ran the same comparison on Spider 2.0 and the effect vanished. Not shrank — vanished, and if anything went the other way. What follows is the run, what I think happened, and the experiment that would actually settle it, which I have not done yet. Spider 1.0's databases have a median of three tables. That is why my per-database numbers there are 100% and worthless: returning the entire schema also scores 100%. I had to invent a pooled variant — every database merged into one 876-table catalog — to make the task resemble retrieval at all, and an invented variant is exactly the kind of thing a reader is right to discount. Spider 2.0-lite needs no such construction. Measured from the distribution: 162 databases, 7,892 tables, a median of 15 tables per database and a maximum of 785, drawn from real BigQuery and Snowflake warehouses. The databases are already big. Table recall, 158 questions across 103 databases. spider2-lite ships 547 examples in total, so this is a subset of the benchmark rather than the whole of it — I'll publish the harness and say exactly which subset when I push it: | top_k | all gold tables present | per-table recall | |---|---|---| | 5 | 70.9% | 81.7% | | 10 | 82.9% | 88.5% | | 20 | 86.1% | 90.5% | Nothing was tuned for this. I downloaded it, pointed the same code at it, and those are the numbers. **With the caveat the sample size demands.** n=158 gives a 95% confidence interval of roughly [63.4, 77.4] on that 70.9%, and [76.3, 88.0] on the 82.9%. So the honest comparison with my Spider 1.0 pooled figures is not "within a point or two" — the intervals are too wide to resolve a point or two. It is that the two are *indistinguishable at this sample size*, on databases an order of magnitude larger and questions written to be hard. That is still the interesting result. It is just a weaker sentence than the one I wanted to write. | k=10, strict recall | hashed | sentence model | |---|---|---| | Spider 1.0 pooled (n=1,034) | 82.6% | **88.1%** | | Spider 2.0-lite (n=158) | 82.9% | 82.3% | And immediately, the discipline I failed to apply to my own good news this morning: **82.9% and 82.3% differ by one question.** 131 versus 130. I cannot claim the embedder is "slightly worse" from that, any more than I could have claimed it was better. The correct statement is that there is *no measurable difference at n=158*. That still kills "consistent win," which is what I said and what was wrong. The Spider 1.0 effect, meanwhile, is real: at n=1,034 the intervals around 82.6% and 88.1% do not overlap. So the finding is not "the embedder does nothing." It is *measurably useful on one benchmark and unmeasurable on the other*, and the reason matters. Spider 2.0's tables carry real descriptions, harvested from the warehouses' own data dictionaries. Spider 1.0's tables carry none — just identifiers. When a table has prose describing it, lexical scoring over that prose already closes the gap between the words a user types and the words a schema uses. A question about "revenue" finds a table whose description says revenue, without any embedding involved. The sentence model was earning its keep on Spider 1.0 by compensating for the absence of that text. Give the corpus the text and there is less left to compensate for. This is a hypothesis fitted to two data points and I am labelling it as such. A few days ago I wrote up the opposite-looking result: adding LLM-generated table descriptions to a 1,245-object schema made retrieval *worse*, because generated prose inflated the document frequency of the domain's own nouns until "contact" scored an IDF of 0.15 and the `contacts` table fell out of the top 40. Those two results look unrelated. They are the same axis: The value of any semantic layer — a learned encoder, or generated descriptions — depends on how much prose the schema already has. Where prose exists, lexical scoring over it does most of the work. Where it doesn't, you need something to bridge the vocabulary gap. And if you add prose to the same field as the identifiers, you damage the identifiers. Which gives a practical rule I can actually stand behind: Two benchmarks differing in everything is a correlation, not a mechanism. The controlled version is one line of setup: strip the descriptions out of Spider 2.0 and re-run the same comparison on the same questions and the same databases. | Spider 2.0-lite, k=10 | hashed | sentence model | |---|---|---| | with data-dictionary prose | 82.9% | 82.3% | | **descriptions removed** | ? | ? | If the encoder's advantage reappears once the prose is gone, the hypothesis is confirmed inside one corpus with everything else held constant. If it doesn't, the explanation is something else — dialect, question style, table size — and I should stop telling this story. I'll run it and post the row either way. Spider 2.0 carries `description` as a *list* for some tables. That crashed indexing with: ``` TypeError: sequence item 2: expected str instance, list found ``` Naming neither the object nor the field. A catalog of 800 tables was unindexable because one of them described itself in a list instead of a string. Fixed at the boundary rather than defensively at each reader — `ObjectDoc` normalises `description`, `hint` and column comments once on construction, and `None` stays `None`. That bug would have hit anyone pointing this at a real data dictionary. It did not show up across six schemas I wrote myself, 445 tests, or a live run against a 127-object Oracle database, because I built all of those and I would never have thought to put a list there. It took ninety seconds to find on someone else's data. Which is the actual argument for running public benchmarks, more than any number in the tables above. They are not there to prove you are good. They are there to be data you did not shape. *The library is [schemagate](https://github.com/ashishsinha1602/schemagate), Apache-2.0. The benchmark scripts are in `benchmarks/` and the data comes from the original sources; the whole suite reruns in about three minutes. The Spider 2.0 numbers and the corrected embedder row are not on `main` yet — [BENCHMARKS.md](https://github.com/ashishsinha1602/schemagate/blob/main/BENCHMARKS.md) still shows the Spider 1.0 figures alone as I write this. It gets the correction, and I'd rather say that here than have you click and find the flattering version.*