# Rethinking Critic Learning in PPO: Understanding and Mitigating Value Flattening

> Source: <https://aiflash.com/news/121203/>
> Published: 2026-09-17 08:30:26+00:00

In reinforcement learning for large language models, Proximal Policy Optimization (PPO) commonly uses a critic to estimate state values and reduce the variance of policy updates. However, we uncover a systematic failure mode in PPO critics, which we call Value Flattening: state values, estimated fro