A new arXiv study introduces the 'fragility grid,' showing that 12 open-weight instruction-tuned LLMs from 4 families score between 31 and 89 percent on the same 3,679 items from 4 benchmarks (ARC, He…
A developer explains how the choice between `acc` and `acc_norm` in lm-eval-harness can skew LLM evaluation results due to length bias. The raw `acc` metric favors shorter answers because it sums toke…
A new method called Forward-Pass-Only MLP training (FPO) adapts large language models without a backward pass, achieving 2.7–3.2x the throughput of standard fine-tuning at ~40% less peak training memo…
A new arXiv preprint (2607.28801v1) introduces a dataset-centric meta-evaluation framework that audits LLM benchmark datasets at the sample level across five dimensions, revealing internal heterogenei…
A developer's team replaced subjective 'looks good to me' testing with an automated LLM evaluation pipeline that catches 92% of hallucinations before deployment. The system uses a judge ensemble inclu…
A PlainEnglish article warns that AI benchmark scores such as MMLU, HumanEval, and HellaSwag can overstate production readiness when leaderboard numbers are treated as proof of model quality. The comm…
Researchers introduced Pollux, a new class of decoder-only LLMs that use native Leech-lattice quantization to achieve 0.76 bits per parameter, compressing a 1B-class model into 76 MB of SRAM. The mode…
Researchers introduce the Prism Transformer, a new architecture that progressively increases head counts across layers to create a hierarchical attention processing structure. This design improves per…
Mike X Cohen's new book "50 ML Projects to Understand LLMs" uses GPT-2 as a scientific specimen, teaching readers to investigate the model through 50 hands-on projects focused on code, statistics, and…
Researchers at OpenPangu identified two failure modes in HiF8 W8A8 quantization-aware training for the OpenPangu-Embedded-1B model: amax saturation, which corrupts knowledge-sensitive representations …