S3AC: Stabilized, Selective, and Scalable Actor-Critic for LLM Reasoning

Researchers present S3AC (Stabilized, Selective, and Scalable Actor-Critic), an actor-critic method for reinforcement learning with verifiable rewards (RLVR) on long chain-of-thought reasoning that the authors say addresses the lack of token-level credit assignment in critic-free methods, which typically rely on grouped rollouts to estimate prompt-level signals. The work targets the costly critic training that critic-free approaches avoid, aiming to combine token-level credit assignment with stable, selective and scalable training for LLM reasoning.

Critic-free methods are widely used in reinforcement learning with verifiable rewards RLVR for long chain-of-thought CoT reasoning, avoiding difficult and costly critic training. Common alternatives lack token-level credit assignment and typically rely on grouped rollouts to estimate prompt-spec