{"slug": "budhist-benchmark-testing-the-knowledge-of-buddhist-history-in-open-weight-llms", "title": "BudHist Benchmark: Testing the Knowledge of Buddhist History in Open-Weight LLMs", "summary": "An ongoing open-weight LLM benchmark project, BudHist, reports that qwen3.6:35b leads tested local open-weight models on Buddhist history multiple-choice questions with 210/250 (84%), while cloud-based open-weight models glm-5.2 (238/250, 95%) and minimax-m3 top the cloud tier, with deepseek-v4-pro (232/250, 93%) and gemini-3.6-flash (233/250, 93%) close behind. The project runs four-option multiple-choice tests over a growing dataset — 150 questions in the 2025-10 test, 210 in 2026-04 and 250 in 2026-08 — using Ollama for local experiments and OpenRouter for cloud tests. The authors state that at roughly 94% the best open-weight cloud models have essentially saturated the benchmark, and that closed reference models perform in the same range.", "body_md": "This ongoing project aims to develop datasets and best practices to test the knowledge of language models in the domain of Buddhist history. The focus is on open-weights models. Local, open-weights models are important for researchers because they allow researchers full control over their experiments. Experiments are repeatable without restrictions and the models can be archived without constraints.\n\n        We use a growing dataset of four-options multiple choice questions regarding different periods of Buddhist history. Currently, [Ollama](https://ollama.com/) is used as middle-ware for the local experiments, [OpenRouter](https://openrouter.ai/) for the cloud-based tests. \n\n        The aim of the project is to:\n\t\t\n\nAs of , among the tested **local** open-weight models **qwen3.6:35b** 'knows' most about Buddhist history. Among the tested **cloud-based** open-weights models **glm-5.2** and **minimax-m3** are best, but with deepseek-v4-pro and gemini-3.6-flash tied close behind. With that (~94%) the best open-weight, cloud-based models have basically saturated the benchmark. The closed, non-open-weights models we included as reference perform in the same ball park. \n\nApril 2026 - now\n\n| Model | Test 2025-10 (150 questions) | Test 2026-04 (210 questions) | Test 2026-08 (250 questions) | \n|---|---|---|---|\n| **Local** |  |  |  | \n| **deepseek-r1:14b** |  | 128/210 (61%) |  | \n| **gemma2:9b** | 81/150 (54%) |  |  | \n| **gemma3:4b** | 60/150 (40%) |  |  | \n| **gemma3:12b** | 93/150 (62%) | 134/210 (64%) | 171/250 (68%) | \n| **gemma3:27b** | 96/150 (64%) | 139/210 (66%) |  | \n| **gemma4:e4b** |  | 121/210 (58%) |  | \n| **gemma4:31b** |  | **159/210 (76%)** | 198/250 (79%) | \n| **glm-4.7-flash:q4_K_M (latest)** |  | 158/210 (75%) | 176/250 (70%) | \n| **laguna-xs-2.1:q4_K_M** |  |  | 146/250 (58%) | \n| **llama3:8b** | 84/150 (56%) |  |  | \n| **llama3.1:8b** | 85/150 (57%) |  |  | \n| **llama4:16x17b** |  | X <sup>[n]</sup> needs 60GB local memory |  | \n| **mistral:7b** | 72/150 (48%) |  |  | \n| **mistral-nemo:12b** |  | 124/210 (59%) | 154/250 (62%) | \n| **nehrdbsd/mitra-chat (Gemma 2 based)** |  |  | 162/250 (65%) | \n| **nehrdbsd/mitra-chat-qwen (qwen3.5 based)** |  |  | 189/250 (76%) <sup>[n]</sup> Mitra is optimized for translation not general knowledge. See online application here:[https://dharmamitra.org/translate](https://dharmamitra.org/translate) | \n| **mixtral:8x7b** | 98/150 (65%) | 151/210 (72%) |  | \n| **olmo-3:7b** |  |  | X <sup>[n]</sup> Olmo does not take “think=False”. Workaround slows the response time drastically. | \n| **olmo-3.1:32b** |  | X <sup>[n]</sup> Olmo does not take “think=False”. Workaround slows the response time drastically. |  | \n| **phi3:3.8b** | 75/150 (50%) |  |  | \n| **phi4:14b (latest)** | 81/150 (54%) | 132/210 (63%) | 168/250 (67%) | \n| **qwen2.5:7b** | 86/150 (57%) |  |  | \n| **qwen2.5:14b** | **114/150 (76%)** | 158/210 (75%) | 196/250 (78%) | \n| **qwen2.5:32b** | **120/150 (80%)** | **165/210 (79%)** | **200/250 (80%)** | \n| **qwen3:8b** | 102/150 (68%) |  |  | \n| **qwen3:14b** | **114/150 (76%)** | 158/210 (75%) | 194/250 (78%) | \n| **qwen3:32b** | 110/150 (73%) |  |  | \n| **qwen3.5:9b (latest)** |  | 153/210 (73%) | 185/250 (74%) | \n| **qwen3.5:27b** |  | **180/210 (86%)** | **205/250 (82%)** | \n| **qwen3.6:35b** |  |  | **210/250 (84%)** | \n| **qwen3.8:27b** |  |  | 198/250 (79%) | \n| **Cloud** |  | (Ollama Cloud) | (OpenRouter) | \n| **deepseek-v3.2** |  | **191/210 (91%)** |  | \n| **deepseek-v4-flash** |  |  | 230/250 (92%) | \n| **deepseek-v4-pro** |  |  | **232/250 (93%)** | \n| **devstral-2:123b** |  | 171/210 (81%) |  | \n| **gemini-3.5-flash-lite** |  |  | 229/250 (92%) | \n| **gemini-3.6-flash** |  |  | **233/250 (93%)** | \n| **glm-4.7** |  | 188/210 (90%) |  | \n| **glm-5.1** |  | **192/210 (91%)** |  | \n| **glm-5.2** |  |  | **238/250 (95%)** | \n| **gpt-oss-120b** |  | 77/210 (37%) | 199/250 (80%) | \n| **hy3** |  |  | 229/250 (92%) | \n| **inkling-small** |  |  | 221/250 (88%) | \n| **inkling** |  |  | 231/250 (92%) | \n| **kimi-k2-thinking** |  | **197/210 (94%)** | 231/250 (92%) | \n| **kimi-k2.5** |  | 187/210 (89%) |  | \n| **kimi-k2.6** |  |  | 230/250 (92%) | \n| **laguna-s-2.1** |  |  | X <sup>[n]</sup> No ZDR endpoint | \n| **minimax-m3** |  |  | **234/250 (94%)** | \n| **mistral-large-2512** |  | 185/210 (88%) |  | \n| **mistral-small-2603** |  |  | 202/250 (81%) | \n| **mistral-medium-3-5** |  |  | X <sup>[n]</sup> No ZDR endpoint. | \n| **qwen3.5:397b** |  | 187/210 (89%) |  | \n| **qwen3.5-9b** |  |  | 200/250 (80%) | \n| **qwen3.6-35b-a3b** |  |  | 217/250 (87%) | \n| **Closed Models**  |  |  | (OpenRouter) | \n| **claude-sonnet-4-5** |  |  | 230/250 (92%) | \n| **claude-sonnet-4-6** |  |  | **233/250 (93%)** | \n| **gpt-4.1** |  |  | 216/250 (86%) | \n| **gpt-5.6-luna** |  |  | 222/250 (89%) |", "url": "https://wpnews.pro/news/budhist-benchmark-testing-the-knowledge-of-buddhist-history-in-open-weight-llms", "canonical_source": "https://mbingenheimer.net/budhistbenchmark/", "published_at": "2026-11-03 11:38:46+00:00", "updated_at": "2026-11-03 11:51:27.802645+00:00", "lang": "en", "topics": ["large-language-models", "ai-research", "machine-learning"], "entities": ["BudHist", "qwen3.6:35b", "glm-5.2", "minimax-m3", "deepseek-v4-pro", "gemini-3.6-flash", "Ollama", "OpenRouter"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/budhist-benchmark-testing-the-knowledge-of-buddhist-history-in-open-weight-llms", "markdown": "https://wpnews.pro/news/budhist-benchmark-testing-the-knowledge-of-buddhist-history-in-open-weight-llms.md", "text": "https://wpnews.pro/news/budhist-benchmark-testing-the-knowledge-of-buddhist-history-in-open-weight-llms.txt", "jsonld": "https://wpnews.pro/news/budhist-benchmark-testing-the-knowledge-of-buddhist-history-in-open-weight-llms.jsonld"}}