Progressive Content Refinement with Decaying Reward Joint LinUCB

Researchers propose a new contextual bandit algorithm that explicitly models reward decay to improve iterative refinement in Large Language Models (LLMs), addressing over-exploitation from static prompts. The method, which uses an Expectation-Maximization (EM) algorithm to estimate arm-specific and decay parameters, outperforms strong baselines on Sentiment Reversal and GSM8K benchmarks, with ablation studies confirming the importance of decay modeling.

arXiv:2608.06750v1 Announce Type: new Abstract: Iterative refinement has significantly enhanced Large Language Model LLM performance; however, existing methods ranging from feedback-based Self-Refine to traditional bandit approaches often rely on static options or overlook the saturation effect. This neglect leads to over-exploitation, where the continuous use of identical prompts or arms results in diminishing rewards over time. To address this challenge, we propose a novel contextual bandit algorithm that explicitly incorporates reward decay modeling. Utilizing an Expectation-Maximization EM algorithm, our method simultaneously estimates both arm-specific and decay parameters. Furthermore, by embedding prompts as arms, we facilitate the joint learning of arm values, distinguishing our approach from the traditional disjoint Linear Upper Confidence Bound LinUCB framework. Experimental results on Sentiment Reversal and GSM8K benchmarks demonstrate that our method achieves significant performance gains over strong baselines. Finally, our ablation study confirms that the integration of reward decay modeling within the bandit framework is crucial for mitigating over-exploitation and optimizing the iterative refinement process.