This ongoing project aims to develop datasets and best practices to test the knowledge of language models in the domain of Buddhist history. The focus is on open-weights models. Local, open-weights models are important for researchers because they allow researchers full control over their experiments. Experiments are repeatable without restrictions and the models can be archived without constraints.
We use a growing dataset of four-options multiple choice questions regarding different periods of Buddhist history. Currently, [Ollama](https://ollama.com/) is used as middle-ware for the local experiments, [OpenRouter](https://openrouter.ai/) for the cloud-based tests.
The aim of the project is to:
As of , among the tested local open-weight models qwen3.6:35b 'knows' most about Buddhist history. Among the tested cloud-based open-weights models glm-5.2 and minimax-m3 are best, but with deepseek-v4-pro and gemini-3.6-flash tied close behind. With that (~94%) the best open-weight, cloud-based models have basically saturated the benchmark. The closed, non-open-weights models we included as reference perform in the same ball park.
April 2026 - now
| Model | Test 2025-10 (150 questions) | Test 2026-04 (210 questions) | Test 2026-08 (250 questions) |
|---|---|---|---|
| Local | | | |
| **deepseek-r1:14b** | | 128/210 (61%) | |
| **gemma2:9b** | 81/150 (54%) | | |
| **gemma3:4b** | 60/150 (40%) | | |
| **gemma3:12b** | 93/150 (62%) | 134/210 (64%) | 171/250 (68%) |
| **gemma3:27b** | 96/150 (64%) | 139/210 (66%) | |
| **gemma4:e4b** | | 121/210 (58%) | |
| **gemma4:31b** | | **159/210 (76%)** | 198/250 (79%) |
| **glm-4.7-flash:q4_K_M (latest)** | | 158/210 (75%) | 176/250 (70%) |
| **laguna-xs-2.1:q4_K_M** | | | 146/250 (58%) |
| **llama3:8b** | 84/150 (56%) | | |
| **llama3.1:8b** | 85/150 (57%) | | |
| **llama4:16x17b** | | X <sup>[n]</sup> needs 60GB local memory | |
| **mistral:7b** | 72/150 (48%) | | |
| **mistral-nemo:12b** | | 124/210 (59%) | 154/250 (62%) |
| **nehrdbsd/mitra-chat (Gemma 2 based)** | | | 162/250 (65%) |
| nehrdbsd/mitra-chat-qwen (qwen3.5 based) | | | 189/250 (76%) <sup>[n]</sup> Mitra is optimized for translation not general knowledge. See online application here:https://dharmamitra.org/translate |
| **mixtral:8x7b** | 98/150 (65%) | 151/210 (72%) | |
| **olmo-3:7b** | | | X <sup>[n]</sup> Olmo does not take “think=False”. Workaround slows the response time drastically. |
| **olmo-3.1:32b** | | X <sup>[n]</sup> Olmo does not take “think=False”. Workaround slows the response time drastically. | |
| **phi3:3.8b** | 75/150 (50%) | | |
| **phi4:14b (latest)** | 81/150 (54%) | 132/210 (63%) | 168/250 (67%) |
| **qwen2.5:7b** | 86/150 (57%) | | |
| **qwen2.5:14b** | **114/150 (76%)** | 158/210 (75%) | 196/250 (78%) |
| **qwen2.5:32b** | **120/150 (80%)** | **165/210 (79%)** | **200/250 (80%)** |
| **qwen3:8b** | 102/150 (68%) | | |
| **qwen3:14b** | **114/150 (76%)** | 158/210 (75%) | 194/250 (78%) |
| **qwen3:32b** | 110/150 (73%) | | |
| **qwen3.5:9b (latest)** | | 153/210 (73%) | 185/250 (74%) |
| **qwen3.5:27b** | | **180/210 (86%)** | **205/250 (82%)** |
| **qwen3.6:35b** | | | **210/250 (84%)** |
| **qwen3.8:27b** | | | 198/250 (79%) |
| **Cloud** | | (Ollama Cloud) | (OpenRouter) |
| **deepseek-v3.2** | | **191/210 (91%)** | |
| **deepseek-v4-flash** | | | 230/250 (92%) |
| **deepseek-v4-pro** | | | **232/250 (93%)** |
| **devstral-2:123b** | | 171/210 (81%) | |
| **gemini-3.5-flash-lite** | | | 229/250 (92%) |
| **gemini-3.6-flash** | | | **233/250 (93%)** |
| **glm-4.7** | | 188/210 (90%) | |
| **glm-5.1** | | **192/210 (91%)** | |
| **glm-5.2** | | | **238/250 (95%)** |
| **gpt-oss-120b** | | 77/210 (37%) | 199/250 (80%) |
| **hy3** | | | 229/250 (92%) |
| **inkling-small** | | | 221/250 (88%) |
| **inkling** | | | 231/250 (92%) |
| **kimi-k2-thinking** | | **197/210 (94%)** | 231/250 (92%) |
| **kimi-k2.5** | | 187/210 (89%) | |
| **kimi-k2.6** | | | 230/250 (92%) |
| **laguna-s-2.1** | | | X <sup>[n]</sup> No ZDR endpoint |
| **minimax-m3** | | | **234/250 (94%)** |
| **mistral-large-2512** | | 185/210 (88%) | |
| **mistral-small-2603** | | | 202/250 (81%) |
| **mistral-medium-3-5** | | | X <sup>[n]</sup> No ZDR endpoint. |
| **qwen3.5:397b** | | 187/210 (89%) | |
| **qwen3.5-9b** | | | 200/250 (80%) |
| **qwen3.6-35b-a3b** | | | 217/250 (87%) |
| Closed Models | | | (OpenRouter) |
| **claude-sonnet-4-5** | | | 230/250 (92%) |
| **claude-sonnet-4-6** | | | **233/250 (93%)** |
| **gpt-4.1** | | | 216/250 (86%) |
| **gpt-5.6-luna** | | | 222/250 (89%) |