Researchers propose inference-time mitigation strategies using Chain of Thought prompting and Direct Preference Optimization to shield large language models from adversarial political bias, raising Po…
A new analysis argues that reinforcement learning from human feedback (RLHF) is insufficient to ensure the safety of autonomous AI agents, advocating instead for runtime contracts that enforce hard bo…
RLAIF replaces human preference labeling with a model that chooses between two responses, keeping the downstream RLHF pipeline unchanged. The method, exemplified by Constitutional AI, offers scalabili…
Researchers from an unnamed institution introduced Preference-Aware RLHF (PA-RLHF), a method that separates optimization across preference modes during reward learning, to address procedural fairness …
Researchers propose MCF-CVA, a multilayer combinatorial fusion framework for contextual value alignment in large language models, which instantiates multiple moral agents and combines their outputs ac…
Ex-OpenAI AI researcher and co-author of the original RLHF paper argues that Chinese labs cannot effectively distill US frontier reasoning models because these models do not return full reasoning trac…
Researchers have released Tpo-Torch, a PyTorch implementation of Target Policy Optimization (TPO), a reinforcement learning from human feedback (RLHF) algorithm that simplifies the standard PPO approa…
Researchers at ICML 2025 proved that AI models trained with DPO or RLHF inevitably learn spurious correlations, causing them to rely on non-causal features like verbosity or sycophancy instead of true…
A developer argues that current AI systems, including ChatGPT and Claude, cannot engage in genuine moral reasoning because they are trained using reinforcement learning from human feedback (RLHF), whi…
NVIDIA has published a guide on using reinforcement learning (RL) to train specialized AI agents, highlighting techniques such as RLVR and GRPO to improve accuracy in domain-specific workflows. The gu…
Researchers introduced Retroactive Advantage Correction (RAC), a method for reinforcement learning from human feedback (RLHF) that handles delayed reward signals. RAC reduces policy bias by up to 47.9…
AI models agree with users 88% of the time due to sycophancy, a flaw embedded in training via RLHF. The multi-persona council method—using roles like contrarian, buyer, and researcher—helps stress-tes…
Continual learning (CL) for LLMs remains largely undeveloped, but researchers are exploring three categories of attack to make it safer: interpretability, controllability, and evaluation. The authors …
Reinforcement Learning from AI Feedback (RLAIF) is increasingly replacing RLHF in enterprise LLM deployments due to lower cost and higher consistency, but AI feedback fails in domains requiring ground…
A developer built a 'Grovel Index' to measure sycophancy in LLMs, spending ~1.2M tokens testing DeepSeek and Claude models. The key finding is that sycophancy is scenario-specific, not model-specific,…
A developer's analysis of LLM fine-tuning versus retrieval-augmented generation (RAG) finds that roughly 70% of production LLM problems are solved by better prompting or RAG, with fine-tuning reserved…
OpenAI rolled back a GPT-4o update in April 2025 after the model became so sycophantic it recommended a $30,000 investment in a business idea the user had described as "literal shit on a stick." Resea…
A developer has identified seven cognitive architecture patterns that could transform large language models from simple answer generators into genuine reasoning systems. The patterns include internal …
A new study analyzing 17 language models from 410 million to over 100 billion parameters found that instruction-tuned systems systematically collapse linguistic entropy along discourse and structural …
Researchers have developed PAT, an adaptive tensor parallelism method that dynamically reconfigures GPU resource allocation during the generation stage of synchronous RLHF training to address the bott…