{"slug": "i-m-afraid-of-spiders-so-i-made-ai-look-at-2k-of-them", "title": "I'm afraid of spiders. So I made AI look at 2k of them", "summary": "Gemini 3.8 Flash identified spider species correctly on 997 of 2,000 photos, a 49.85% accuracy rate that was the highest among nine AI models tested in a benchmark built from research-grade iNaturalist observations covering 671 species and subspecies. The multiple-choice test, run on the same 2,000-photo subset drawn from 2,183 filtered images, gave each model a list of 20 possible species names per photo including the correct answer. The benchmark was inspired by Piotr Migdał's post on identifying mushrooms with AI.", "body_md": "[← All posts](https://labqoat.com/blog)\n\n# I’m Afraid of Spiders. So I Made AI Look at 2,000 of Them.\n\n## In this article\n\nI’m afraid of spiders. A lot. 🕷️\n\nMy identification system currently consists of “the one with long legs” and “the short but big one.” There’s also “where the fuck did it go,” but that’s more of an emergency than a classification.\n\nSo I got curious: how well could AI actually identify them?\n\nI tested nine models on the same 2,000 spider photos to see how often they got the species right.\n\n(Looking at this many spiders was not a comfortable experience.)\n\nThis post was heavily inspired by [Piotr Migdał’s post on identifying mushrooms with AI](https://quesma.com/blog/mushroom-llm-vision/).\n\n## [First, approximately ten seconds of biology](#first-approximately-ten-seconds-of-biology)\n\nA **family** is a broad group of related spiders. A **genus** is a smaller group inside it, and a **species** is the specific kind of spider.\n\nFor example: **Araneidae → Araneus → Araneus diadematus**, the [European garden spider](https://en.wikipedia.org/wiki/Araneus_diadematus). In that last name, *Araneus* is the genus.\n\nThat’s enough biology for now.\n\n## [How the benchmark works](#how-the-benchmark-works)\n\nEach model got the same **2,000 photos**, covering **671 species and subspecies**, and a list of **20 possible names** for each photo, including the correct answer.\n\nThe task was to identify the spider’s species from the photo and return the matching name from the list.\n\nI gave each model the same possible answers so I could compare how well they distinguished the species, with fewer ambiguities in scoring. That makes this a **multiple-choice identification test**, and the choices themselves can help.\n\nI built the dataset from research-grade [iNaturalist](https://www.inaturalist.org/) observations, using the community’s species identifications as the expected answers. The species list comes from a Polish spider checklist, but the photos were taken worldwide. After filtering out label mismatches and unsuitable images, 2,183 photos remained, and I used the same [2,000-photo subset](https://github.com/qforge-dev/spider-bench/blob/main/data/benchmarks/runs/gemini-20260910-230024/tasks.jsonl) for every model.\n\n## [Let’s look at a few examples](#lets-look-at-a-few-examples)\n\nI’d call any of these “spider,” but… 😅\n\nAccording to [iNaturalist](https://www.inaturalist.org/observations/24731930), it’s a **Red-bellied Jumping Spider** (*Philaeus chrysops*).\n\n### What did the models say?\n\n9 of 9 picked the expected species.\n\n| Model answers for Philaeus chrysops |  |  | \n|---|---|---|\n| Model | Answer | Result | \n|---|---|---|\n| Gemini 3.8 Flash | Philaeus chrysopsCorrect | Correct | \n| GPT-6 Astra | Philaeus chrysopsCorrect | Correct | \n| Claude Fable 5.1 | Philaeus chrysopsCorrect | Correct | \n| Muse Spark 1.3 <sup>*</sup> | Philaeus chrysopsCorrect | Correct | \n| GLM 5.3 Flash | Philaeus chrysopsCorrect | Correct | \n| GPT-5.6 Sol | Philaeus chrysopsCorrect | Correct | \n| DeepSeek V4.1 Flash | Philaeus chrysopsCorrect | Correct | \n| GPT-5.6 Terra | Philaeus chrysopsCorrect | Correct | \n| GPT-5.6 Luna | Philaeus chrysopsCorrect | Correct | \n\nEveryone got this one right.\n\n## See the 20 choices the models received\n\n1. 1.Nigma walckenaeri\n2. 2.Zelotes aeneus\n3. 3.Attulus inexpectus\n4. 4.Philaeus chrysops\n5. 5.Gnaphosa lucifuga\n6. 6.Savignia frontata\n7. 7.Sittisax saxicola\n8. 8.Kishidaia conspicua\n9. 9.Evarcha laetabunda\n10. 10.Metellina mengei\n11. 11.Attulus rupicola\n12. 12.Araniella displicata\n13. 13.Leptorchestes berolinensis\n14. 14.Neon valentulus\n15. 15.Attulus terebratus\n16. 16.Marpissa pomatia\n17. 17.Cybaeus tetricus\n18. 18.Nuctenea umbratica\n19. 19.Euophrys frontalis\n20. 20.Larinioides ixobolus\n\n<sup>*</sup> Muse Spark 1.3 uses the Contributor tier.\n\nSelected examples; overall scores use all 2,000 photos.\n\n## [Okay, who knew the spiders?](#okay-who-knew-the-spiders)\n\n**Gemini 3.8 Flash got 997 out of 2,000 correct: 49.85%**, the highest score in these runs.\n\nGPT-6 Astra followed at **47.50%**, then Claude Fable 5.1 at **43.10%** and Muse Spark 1.3 at **39.10%**. The top two were separated by 47 photos. I’d read that as a result on this set of spiders, not a universal ranking of how much the models know about them.\n\nSame 2,000 photos for every model. Wrong and failed answers stay in the denominator.\n\n<sup>*</sup> Muse Spark 1.3 uses the discounted Contributor tier, which allows provider training/data use.\n\n## See the exact numbers\n\n| Exact-species accuracy. Each run contains 2,000 photos. |  |  | \n|---|---|---|\n| Model | Correct | Accuracy | \n|---|---|---|\n| Gemini 3.8 Flash | 997 | 49.85% | \n| GPT-6 Astra | 950 | 47.50% | \n| Claude Fable 5.1 | 862 | 43.10% | \n| Muse Spark 1.3 <sup>*</sup> | 782 | 39.10% | \n| GLM 5.3 Flash | 735 | 36.75% | \n| GPT-5.6 Sol | 673 | 33.65% | \n| DeepSeek V4.1 Flash | 528 | 26.40% | \n| GPT-5.6 Terra | 456 | 22.80% | \n| GPT-5.6 Luna | 430 | 21.50% | \n\n## [How wrong is wrong?](#how-wrong-is-wrong)\n\nGemini 3.8 Flash missed the exact species in 1,003 photos. But in **871 of those cases**, it named another spider from the same family. That’s almost **87% of its misses**. In 276 cases, it even got the genus right.\n\nGPT-6 Astra and Claude Fable 5.1 showed a similar pattern. Their exact-species scores were **47.50% and 43.10%**, but their answers belonged to the correct family **92.00% and 91.45%** of the time. Most of their mistakes happened between species within the same family.\n\nMuse Spark 1.3 identified more exact species than GLM 5.3 Flash. GLM’s answers, however, belonged to the correct family slightly more often: **89.40% versus 88.70%**.\n\nThat makes the errors more interesting than a simple correct-or-wrong score suggests. There’s a substantial difference between getting the broad group right and identifying the particular species.\n\n- Correct species\n- Same genus only\n- Same family only\n- Wrong family\n- Failed response\n\nEach answer appears in one segment. “Only” means the more specific identification was wrong. Failed responses include invalid names, refusals, empty answers, truncations, and request errors.\n\n<sup>*</sup> Muse Spark 1.3 uses the discounted Contributor tier, which allows provider training/data use.\n\n## See the exact numbers\n\n| How close were the answers?. Each run contains 2,000 photos. |  |  |  |  |  | \n|---|---|---|---|---|---|\n| Model | Correct species | Same genus only | Same family only | Wrong family | Failed response | \n|---|---|---|---|---|---|\n| Gemini 3.8 Flash | 997 | 276 | 595 | 128 | 4 | \n| GPT-6 Astra | 950 | 312 | 578 | 156 | 4 | \n| Claude Fable 5.1 | 862 | 347 | 620 | 169 | 2 | \n| Muse Spark 1.3 <sup>*</sup> | 782 | 306 | 686 | 226 | 0 | \n| GLM 5.3 Flash | 735 | 322 | 731 | 209 | 3 | \n| GPT-5.6 Sol | 673 | 264 | 684 | 378 | 1 | \n| DeepSeek V4.1 Flash | 528 | 328 | 710 | 342 | 92 | \n| GPT-5.6 Terra | 456 | 272 | 718 | 554 | 0 | \n| GPT-5.6 Luna | 430 | 248 | 671 | 606 | 45 | \n\nCounts, not cumulative percentages. Each row adds up to 2,000.\n\n## [Accuracy versus cost](#accuracy-versus-cost)\n\nMuse Spark 1.3 was the cheapest model here and still finished fourth, ahead of five more expensive models. At its Contributor pricing, the entire run cost **$0.93 for 782 correct identifications**.\n\nMoving to Gemini 3.8 Flash brought that total to 997. That’s **215 more correct answers for another $26.18**, an improvement of **10.75 percentage points** at roughly **29 times the cost**. 💸\n\nSpending beyond that didn’t improve the results. Claude Fable 5.1 cost about twice as much as Gemini, and GPT-6 Astra cost **3.6 times as much**. Both scored lower. Gemini delivered the highest accuracy here; Muse offered a cheaper trade-off if identifying fewer species was acceptable.\n\nEstimated totals for 2,000 photos. The dollar axis is logarithmic: spacing represents cost ratios.\n\n<sup>*</sup> Muse Spark 1.3 uses the discounted Contributor tier, which allows provider training/data use.\n\n## See the exact numbers\n\n| Accuracy versus estimated cost. Each run contains 2,000 photos. |  |  |  | \n|---|---|---|---|\n| Model | Correct | Accuracy | Cost | \n|---|---|---|---|\n| Gemini 3.8 Flash | 997 | 49.85% | $27.11 | \n| GPT-6 Astra | 950 | 47.50% | $97.53 | \n| Claude Fable 5.1 | 862 | 43.10% | $54.19 | \n| Muse Spark 1.3 <sup>*</sup> | 782 | 39.10% | $0.93 | \n| GLM 5.3 Flash | 735 | 36.75% | $1.46 | \n| GPT-5.6 Sol | 673 | 33.65% | $42.89 | \n| DeepSeek V4.1 Flash | 528 | 26.40% | $8.52 | \n| GPT-5.6 Terra | 456 | 22.80% | $20.53 | \n| GPT-5.6 Luna | 430 | 21.50% | $4.99 | \n\n## [Is getting half wrong actually bad?](#is-getting-half-wrong-actually-bad)\n\nGetting almost half right is respectable when the task is telling similar species apart. The gap between recognising a family and naming the exact species is where this gets difficult.\n\nTry telling  apart. For *Araniella opisthographa*, [NatureSpot’s identification guidance](https://www.naturespot.org/species/araniella-opisthographa) calls for examination at high magnification. That’s the sort of detail a photo can leave out.\n\nThe code and full benchmark results are on [GitHub](https://github.com/qforge-dev/spider-bench).\n\n*PS: If you actually know spiders, I’d love to hear what you think on [twitter](https://x.com/breeg554). How obvious are these mistakes to someone who knows what they’re looking at?*\n\nP.P.S. Thankfully, I have a cat in charge of spider security at home.\n\n[← All posts](https://labqoat.com/blog)Labqoat", "url": "https://wpnews.pro/news/i-m-afraid-of-spiders-so-i-made-ai-look-at-2k-of-them", "canonical_source": "https://labqoat.com/blog/how-well-can-ai-identify-spiders", "published_at": "2026-09-21 09:00:37+00:00", "updated_at": "2026-09-21 09:23:58.133625+00:00", "lang": "en", "topics": ["computer-vision", "ai-research", "machine-learning"], "entities": ["Gemini 3.8 Flash", "iNaturalist", "Piotr Migdał", "GPT-6 Astra", "Claude Fable 5.1", "Muse Spark 1.3", "GLM 5.3 Flash", "DeepSeek V4.1 Flash"], "alternates": {"html": "https://wpnews.pro/news/i-m-afraid-of-spiders-so-i-made-ai-look-at-2k-of-them", "markdown": "https://wpnews.pro/news/i-m-afraid-of-spiders-so-i-made-ai-look-at-2k-of-them.md", "text": "https://wpnews.pro/news/i-m-afraid-of-spiders-so-i-made-ai-look-at-2k-of-them.txt", "jsonld": "https://wpnews.pro/news/i-m-afraid-of-spiders-so-i-made-ai-look-at-2k-of-them.jsonld"}}