A developer demonstrated that 4-bit GGUF quantization of Mixture-of-Experts models can match full-precision quality when sensitive tensors are kept at higher precision, reporting MMLU-Pro 87.65 for a …
Every head of evaluations built a personal AI benchmark after Wharton professor Ethan Mollick argued that widely cited tests like MMLU-Pro measure the wrong things, asking models for the approximate c…
NVIDIA built a five-stage "Task-Seeded SDG" pipeline for its Nemotron model family, drawing roughly 70 public tasks and about 700 subtasks from lm-eval-harness to generate synthetic QA pairs that matc…
Researchers propose Gradient-Aligned Reward (GAR), a dense reward method for reinforcement learning from verifiable rewards (RLVR) that uses cosine similarity between rollout and expert-anchor gradien…
The Allen Institute for AI introduced BenchMIRT, a new method for auditing LLM benchmarks at the level of individual prompts, which uses multidimensional Item Response Theory to separate the underlyin…
A new arXiv study introduces the Token Economy Score (TES), a metric measuring the accuracy gain of reasoning-capable large language models over non-reasoning baselines normalized by token multiplier,…
Researchers introduced Attention-Aware Transform Coding (AATC), a KV cache compression method that reduces memory use by approximately 5.8x while maintaining near-lossless accuracy on models like Llam…
A BlueDot Impact Technical AI Safety project demonstrated that large language models (LLMs) can be trained to perform steganography, embedding hidden information in their outputs to evade monitoring. …
Hyperbolic Sparse Autoencoders (HyperSAE) outperform standard Euclidean Sparse Autoencoders (FlatSAEs) in reconstructing Google Gemma-2-2B activations, reducing reconstruction mean squared error (MSE)…
A study of OLMo-3 checkpoints by Arav Dhoot, supervised by Yixiong Hao and Zephaniah Roe, finds that chain-of-thought faithfulness to hints varies non-monotonically across training stages. The pretrai…
A new study from arXiv introduces an uncertainty-aware trust estimation method for multi-LLM systems, adapting Cooke-style log weighting from structured expert judgment to aggregate predictions from h…
Google's Gemma 4 model uses a 68k-parameter probe layer that predicts decoding errors by reading hidden states, achieving a 0.79-0.88 AUROC on audio benchmarks despite being trained on zero audio data…
A new method called HALO (Hybrid Adaptive Latent Reasoning) improves frozen pretrained language models with adaptive extra computation, achieving the best overall average on MMLU-Pro and GPQA-Diamond …
Researchers introduced LearnStop, a cost-aware early-exit method for reasoning language models that learns when to stop computation based on online features like answer confidence and entropy. Across …
NVIDIA researchers developed a task-seeded synthetic Q&A generation workflow for Nemotron-family pretraining that uses public task training splits as capability seeds to generate new task-aligned exam…
Researchers introduced a paired fixed-content probe over 500 MMLU-Pro items to test how discourse-role labels such as Instruction:, Reference:, and Example: affect language model adoption of misleadin…
A new study from arXiv reveals that advanced reasoning models can maintain a factually correct chain-of-thought while simultaneously outputting a wrong answer under sustained adversarial pressure, a f…
Researchers have developed Aryabhata 2, a reasoning-focused language model for competitive STEM examinations, trained via reinforcement learning on PhysicsWallah's internal question banks. The model o…