cd /news/large-language-models/astra-6-2-and-sol-6-2-flop-at-simple… · home › topics › large-language-models › article
[ARTICLE · art-173785] src=simple-bench.com ↗ pub= topic=large-language-models verified=true sentiment=· neutral

Astra 6.2 and Sol 6.2 flop at Simple Bench

A new multiple-choice text benchmark called SimpleBench, with over 200 questions on spatio-temporal reasoning, social intelligence and trick questions, found that a non-specialized human baseline of 83.7% from nine participants outperformed every tested LLM, including top model Claude Fable at 81.9%. The benchmark's authors said the results confirm that the memorized knowledge and approximate reasoning retrieval used by frontier LLMs is not always enough to answer basic questions. The models can be run on the Simple Bench app at LMcouncil.ai, with a full technical report linked.

recommendation

Include spatio-temporal, social-intelligence and adversarial trick questions in your eval set; SimpleBench found humans at 83.7% beat the top model's 81.9%.

more in playbook: Evals and testing for LLM apps
read1 min views7 publishedNov 21, 2026

We introduce SimpleBench, a multiple-choice text benchmark for LLMs where individuals with

                            unspecialized (high school) knowledge outperform SOTA models. SimpleBench includes over 200
                            questions covering spatio-temporal reasoning, social intelligence, and what we call
                            linguistic adversarial robustness (or trick questions). For the vast majority of text-based
                            benchmarks LLMs outperform a non-specialized human, and increasingly, exceed expert human
                            performance. However, on SimpleBench, a non-specialized human baseline is 83.7%, based on
                            our small sample of nine participants, ~~outperforming every tested LLM, including
                            today's top model, Claude Fable, which scored 81.9%~~. While we expect model performance to improve over time, the
                            results of SimpleBench confirm that the memorized knowledge, and approximate reasoning
                            retrieval, utilized by frontier LLMs is not always enough to answer basic questions just
                            yet. 

For a deeper dive into our results and our methods, check out the full technical report [here](https://drive.google.com/file/d/1mddNFK5UbBFVr3oDftd2Kyc6D8TFctfe/view).

                                Use all of these models on the Simple Bench app - **LMcouncil.ai**
Rank Model Score (AVG@5) Organization
── more in #large-language-models 4 stories · sorted by recency
── more on @simplebench 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/astra-6-2-and-sol-6-…] indexed:0 read:1min 2026-11-21 · —