A new arXiv study introduces the 'fragility grid,' showing that 12 open-weight instruction-tuned LLMs from 4 families score between 31 and 89 percent on the same 3,679 items from 4 benchmarks (ARC, He…
Trunchbull, a browser-based tool for running LLM benchmarks, lets users author tests and evaluate models in real time, with native support for the harbor authoring system and the Vercel AI SDK. It inc…
A new arXiv paper (2608.10288v1) introduces Power Law Graph Attention (PLGA), a learned attention mechanism that exactly generalizes scaled dot-product attention (SDPA) and is used in the Power Law De…
A study labeling 7,440 answers from four open-weight LLMs found that more than half of the hallucination labels were incorrect, and correcting them reordered the results. The author, Priyanshi Jain, r…
A new arXiv preprint (2607.28801v1) introduces a dataset-centric meta-evaluation framework that audits LLM benchmark datasets at the sample level across five dimensions, revealing internal heterogenei…
A new open-source benchmarking framework, Inspect India Evals, reveals that large language models (LLMs) tested on Indian linguistic and cultural contexts show significant performance gaps, with Sarva…
A new arXiv preprint (2607.16799v1) finds that hidden-state readouts from language models primarily encode self-judgement (SJ) rather than objective correctness (OC), with the SJ-associated direction …
Researchers have developed a lightweight method for quantifying uncertainty in neural network predictions using two key approximations: a first-order Taylor expansion and an isotropy assumption. The a…
Researchers introduced a multi-factor scoring system to evaluate large language model responses, assessing accuracy, conciseness, factual consistency, readability, and coherence. Tests on the Truthful…
A researcher found that a clean implementation of abliteration on Qwen3.5-27B costs only about 1.4 TruthfulQA points, far less than the 5.5+ points lost by HuiHui AI's crude method, confirming that mo…
Researchers found that a linear probe applied to mid-layer hidden states of quantized large language models can detect hallucinations with up to 1.000 AUROC, significantly outperforming sampling-based…
Researchers have developed SERC, a semantic error correction framework inspired by LDPC codes that treats LLM text generation as a noisy communication channel to detect and fix hallucinations. The tra…
Researchers have developed MechELK, a three-stage framework that uses mechanistic interpretability to extract hidden factual and reasoning knowledge from large language models. The framework, which co…