{"slug": "astra-6-2-and-sol-6-2-flop-at-simple-bench", "title": "Astra 6.2 and Sol 6.2 flop at Simple Bench", "summary": "A new multiple-choice text benchmark called SimpleBench, with over 200 questions on spatio-temporal reasoning, social intelligence and trick questions, found that a non-specialized human baseline of 83.7% from nine participants outperformed every tested LLM, including top model Claude Fable at 81.9%. The benchmark's authors said the results confirm that the memorized knowledge and approximate reasoning retrieval used by frontier LLMs is not always enough to answer basic questions. The models can be run on the Simple Bench app at LMcouncil.ai, with a full technical report linked.", "body_md": "We introduce **SimpleBench**, a multiple-choice text benchmark for LLMs where individuals\n                            with\n                            unspecialized (high school) knowledge outperform SOTA models. SimpleBench includes over 200\n                            questions covering spatio-temporal reasoning, social intelligence, and what we call\n                            linguistic adversarial robustness (or trick questions). For the vast majority of text-based\n                            benchmarks LLMs outperform a non-specialized human, and increasingly, exceed expert human\n                            performance. However, on SimpleBench, a non-specialized human baseline is 83.7%, based on\n                            our small sample of nine participants, ~~outperforming every tested LLM, including\n                            today's top model, Claude Fable, which scored 81.9%~~. While we expect model performance to improve over time, the\n                            results of SimpleBench confirm that the memorized knowledge, and approximate reasoning\n                            retrieval, utilized by frontier LLMs is not always enough to answer basic questions just\n                            yet. \n\nFor a deeper dive into our results and our methods, check out the full technical report [here](https://drive.google.com/file/d/1mddNFK5UbBFVr3oDftd2Kyc6D8TFctfe/view).\n\n                                Use all of these models on the Simple Bench app - **LMcouncil.ai**\n\n| Rank | Model | Score (AVG@5) | Organization | \n|---|---|---|---|", "url": "https://wpnews.pro/news/astra-6-2-and-sol-6-2-flop-at-simple-bench", "canonical_source": "https://simple-bench.com/", "published_at": "2026-11-21 13:16:12+00:00", "updated_at": "2026-11-21 13:43:16.279333+00:00", "lang": "en", "topics": ["large-language-models", "ai-research", "artificial-intelligence"], "entities": ["SimpleBench", "Claude Fable", "LMcouncil.ai"], "also_reported_by": [], "takeaway": "On SimpleBench's 200-plus spatio-temporal, social-intelligence and trick questions, a nine-person non-specialized human baseline scored 83.7% versus 81.9% for the top model, Claude Fable.", "recommendation": {"advice": "Include spatio-temporal, social-intelligence and adversarial trick questions in your eval set; SimpleBench found humans at 83.7% beat the top model's 81.9%.", "playbook": "evals-testing", "playbook_url": "https://wpnews.pro/playbooks/evals-testing"}, "alternates": {"html": "https://wpnews.pro/news/astra-6-2-and-sol-6-2-flop-at-simple-bench", "markdown": "https://wpnews.pro/news/astra-6-2-and-sol-6-2-flop-at-simple-bench.md", "text": "https://wpnews.pro/news/astra-6-2-and-sol-6-2-flop-at-simple-bench.txt", "jsonld": "https://wpnews.pro/news/astra-6-2-and-sol-6-2-flop-at-simple-bench.jsonld"}}