# BudHist Benchmark: Testing the Knowledge of Buddhist History in Open-Weight LLMs

> Source: <https://mbingenheimer.net/budhistbenchmark/>
> Published: 2026-11-03 11:38:46+00:00

This ongoing project aims to develop datasets and best practices to test the knowledge of language models in the domain of Buddhist history. The focus is on open-weights models. Local, open-weights models are important for researchers because they allow researchers full control over their experiments. Experiments are repeatable without restrictions and the models can be archived without constraints.

        We use a growing dataset of four-options multiple choice questions regarding different periods of Buddhist history. Currently, [Ollama](https://ollama.com/) is used as middle-ware for the local experiments, [OpenRouter](https://openrouter.ai/) for the cloud-based tests. 

        The aim of the project is to:
		

As of , among the tested **local** open-weight models **qwen3.6:35b** 'knows' most about Buddhist history. Among the tested **cloud-based** open-weights models **glm-5.2** and **minimax-m3** are best, but with deepseek-v4-pro and gemini-3.6-flash tied close behind. With that (~94%) the best open-weight, cloud-based models have basically saturated the benchmark. The closed, non-open-weights models we included as reference perform in the same ball park. 

April 2026 - now

| Model | Test 2025-10 (150 questions) | Test 2026-04 (210 questions) | Test 2026-08 (250 questions) | 
|---|---|---|---|
| **Local** |  |  |  | 
| **deepseek-r1:14b** |  | 128/210 (61%) |  | 
| **gemma2:9b** | 81/150 (54%) |  |  | 
| **gemma3:4b** | 60/150 (40%) |  |  | 
| **gemma3:12b** | 93/150 (62%) | 134/210 (64%) | 171/250 (68%) | 
| **gemma3:27b** | 96/150 (64%) | 139/210 (66%) |  | 
| **gemma4:e4b** |  | 121/210 (58%) |  | 
| **gemma4:31b** |  | **159/210 (76%)** | 198/250 (79%) | 
| **glm-4.7-flash:q4_K_M (latest)** |  | 158/210 (75%) | 176/250 (70%) | 
| **laguna-xs-2.1:q4_K_M** |  |  | 146/250 (58%) | 
| **llama3:8b** | 84/150 (56%) |  |  | 
| **llama3.1:8b** | 85/150 (57%) |  |  | 
| **llama4:16x17b** |  | X <sup>[n]</sup> needs 60GB local memory |  | 
| **mistral:7b** | 72/150 (48%) |  |  | 
| **mistral-nemo:12b** |  | 124/210 (59%) | 154/250 (62%) | 
| **nehrdbsd/mitra-chat (Gemma 2 based)** |  |  | 162/250 (65%) | 
| **nehrdbsd/mitra-chat-qwen (qwen3.5 based)** |  |  | 189/250 (76%) <sup>[n]</sup> Mitra is optimized for translation not general knowledge. See online application here:[https://dharmamitra.org/translate](https://dharmamitra.org/translate) | 
| **mixtral:8x7b** | 98/150 (65%) | 151/210 (72%) |  | 
| **olmo-3:7b** |  |  | X <sup>[n]</sup> Olmo does not take “think=False”. Workaround slows the response time drastically. | 
| **olmo-3.1:32b** |  | X <sup>[n]</sup> Olmo does not take “think=False”. Workaround slows the response time drastically. |  | 
| **phi3:3.8b** | 75/150 (50%) |  |  | 
| **phi4:14b (latest)** | 81/150 (54%) | 132/210 (63%) | 168/250 (67%) | 
| **qwen2.5:7b** | 86/150 (57%) |  |  | 
| **qwen2.5:14b** | **114/150 (76%)** | 158/210 (75%) | 196/250 (78%) | 
| **qwen2.5:32b** | **120/150 (80%)** | **165/210 (79%)** | **200/250 (80%)** | 
| **qwen3:8b** | 102/150 (68%) |  |  | 
| **qwen3:14b** | **114/150 (76%)** | 158/210 (75%) | 194/250 (78%) | 
| **qwen3:32b** | 110/150 (73%) |  |  | 
| **qwen3.5:9b (latest)** |  | 153/210 (73%) | 185/250 (74%) | 
| **qwen3.5:27b** |  | **180/210 (86%)** | **205/250 (82%)** | 
| **qwen3.6:35b** |  |  | **210/250 (84%)** | 
| **qwen3.8:27b** |  |  | 198/250 (79%) | 
| **Cloud** |  | (Ollama Cloud) | (OpenRouter) | 
| **deepseek-v3.2** |  | **191/210 (91%)** |  | 
| **deepseek-v4-flash** |  |  | 230/250 (92%) | 
| **deepseek-v4-pro** |  |  | **232/250 (93%)** | 
| **devstral-2:123b** |  | 171/210 (81%) |  | 
| **gemini-3.5-flash-lite** |  |  | 229/250 (92%) | 
| **gemini-3.6-flash** |  |  | **233/250 (93%)** | 
| **glm-4.7** |  | 188/210 (90%) |  | 
| **glm-5.1** |  | **192/210 (91%)** |  | 
| **glm-5.2** |  |  | **238/250 (95%)** | 
| **gpt-oss-120b** |  | 77/210 (37%) | 199/250 (80%) | 
| **hy3** |  |  | 229/250 (92%) | 
| **inkling-small** |  |  | 221/250 (88%) | 
| **inkling** |  |  | 231/250 (92%) | 
| **kimi-k2-thinking** |  | **197/210 (94%)** | 231/250 (92%) | 
| **kimi-k2.5** |  | 187/210 (89%) |  | 
| **kimi-k2.6** |  |  | 230/250 (92%) | 
| **laguna-s-2.1** |  |  | X <sup>[n]</sup> No ZDR endpoint | 
| **minimax-m3** |  |  | **234/250 (94%)** | 
| **mistral-large-2512** |  | 185/210 (88%) |  | 
| **mistral-small-2603** |  |  | 202/250 (81%) | 
| **mistral-medium-3-5** |  |  | X <sup>[n]</sup> No ZDR endpoint. | 
| **qwen3.5:397b** |  | 187/210 (89%) |  | 
| **qwen3.5-9b** |  |  | 200/250 (80%) | 
| **qwen3.6-35b-a3b** |  |  | 217/250 (87%) | 
| **Closed Models**  |  |  | (OpenRouter) | 
| **claude-sonnet-4-5** |  |  | 230/250 (92%) | 
| **claude-sonnet-4-6** |  |  | **233/250 (93%)** | 
| **gpt-4.1** |  |  | 216/250 (86%) | 
| **gpt-5.6-luna** |  |  | 222/250 (89%) |