About the journal
Browse
Collections
Multimedia collections
Authors & reviewers
Data-dependent exploration for online reinforcement learning from human feedback
Zhen-Yu ZHANG , Yuting TANG , Jiandong ZHANG , Lanjihong MA , Masashi SUGIYAMA
Front. Comput. Sci. ›› 2027, Vol. 21 ›› Issue (9) : 2109357
Online reinforcement learning from human feedback (RLHF) has emerged as a promising paradigm for aligning large language models (LLMs) by continuously collecting new preference feedback during training. A foundational challenge in this setting is exploration, which requires algorithms that enable the LLMs to generate informative comparisons that improve sample-efficiency in online RLHF. Existing exploration strategies often derive bonuses via on-policy expectations, which are difficult to estimate reliably from the limited historical preference data available during training; as a result, the policy can prematurely down-weight under-explored regions that may contain high-value behaviors. In this paper, we propose data-dependent exploration for preference optimization (DEPO), a simple and scalable method that leverages historical data to construct an extra uncertainty bonus for high-uncertainty regions, encouraging exploration toward potentially high-value data. Theoretically, we provide a data-dependent regret bound for the proposed algorithm, showing that it adapts to the hardness of the learning task itself and can be tighter than worst-case bounds in practice. Empirically, the proposed method consistently outperforms strong baselines across benchmarks, demonstrating improved sample efficiency.
reinforcement learning from human feedback / preference learning / online learning
| [1] |
|
| [2] |
Touvron H, Martin L, Stone K, Albert P, Almahairi A, et al. Llama 2: open foundation and fine-tuned chat models. 2023, arXiv preprint arXiv: 2307.09288 |
| [3] |
Schulman J, Wolski F, Dhariwal P, Radford A, Klimov O. Proximal policy optimization algorithms. 2017, arXiv preprint arXiv: 1707.06347 |
| [4] |
|
| [5] |
|
| [6] |
|
| [7] |
|
| [8] |
Dwaracherla V, Asghari S M, Hao B, Van Roy B. Efficient exploration for LLMs. In: Proceedings of the 41st International Conference on Machine Learning. 2024, 12215−12227 |
| [9] |
|
| [10] |
|
| [11] |
Guo S, Zhang B, Liu T, Liu T, Khalman M, Llinares F, Rame A, Mesnard T, Zhao Y, Piot B, Ferret J, Blondel M. Direct language model alignment from online AI feedback. 2024, arXiv preprint arXiv: 2402.04792 |
| [12] |
|
| [13] |
|
| [14] |
|
| [15] |
Auer P, Cesa-Bianchi N, Fischer P. Finite-time analysis of the multiarmed bandit problem. Machine Learning, 2002, 47(2−3): 235−256 |
| [16] |
Bradley R A, Terry M E. Rank analysis of incomplete block designs: I. The method of paired comparisons. Biometrika, 1952, 39(3−4): 324−345 |
| [17] |
|
| [18] |
Rafailov R, Sharma A, Mitchell E, Ermon S, Manning C D, Finn C. Direct preference optimization: your language model is secretly a reward model. In: Proceedings of the 37th International Conference on Neural Information Processing Systems. 2023, 2338 |
| [19] |
|
| [20] |
Tuyls J, Foster D J, Krishnamurthy A, Ash J T. Representation-based exploration for language models: from test-time to post-training. 2025, arXiv preprint arXiv: 2510.11686 |
| [21] |
|
| [22] |
|
| [23] |
|
| [24] |
|
| [25] |
Lin S, Hilton J, Evans O. TruthfulQA: measuring how models mimic human falsehoods. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2022, 3214−3252 |
| [26] |
Cobbe K, Kosaraju V, Bavarian M, Chen M, Jun H, Kaiser L, Plappert M, Tworek J, Hilton J, Nakano R, Hesse C, Schulman J. Training verifiers to solve math word problems. 2021, arXiv preprint arXiv: 2110.14168 |
| [27] |
Dubois Y, Galambosi B, Liang P, Hashimoto T B. Length-controlled AlpacaEval: a simple way to debias automatic evaluators. 2024, arXiv preprint arXiv: 2404.04475 |
| [28] |
Bai G, Liu J, Bu X, He Y, Liu J, Zhou Z, Lin Z, Su W, Ge T, Zheng B, Ouyang W. MT-Bench-101: a fine-grained benchmark for evaluating large language models in multi-turn dialogues. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024, 7421−7454 |
| [29] |
|
| [30] |
|
| [31] |
Zhao Y, Joshi R, Liu T, Khalman M, Saleh M, Liu P J. SLiC-HF: sequence likelihood calibration with human feedback. 2023, arXiv preprint arXiv: 2305.10425 |
| [32] |
|
| [33] |
|
| [34] |
|
| [35] |
|
| [36] |
|
| [37] |
Li L, Chu W, Langford J, Schapire R E. A contextual-bandit approach to personalized news article recommendation. In: Proceedings of the 19th International Conference on World Wide Web. 2010, 661−670 |
| [38] |
|
| [39] |
|
| [40] |
|
Higher Education Press
/
| 〈 |
|
〉 |