Data-dependent exploration for online reinforcement learning from human feedback

Zhen-Yu ZHANG , Yuting TANG , Jiandong ZHANG , Lanjihong MA , Masashi SUGIYAMA

Front. Comput. Sci. ›› 2027, Vol. 21 ›› Issue (9) : 2109357

PDF (1395KB)
Front. Comput. Sci. ›› 2027, Vol. 21 ›› Issue (9) :2109357 DOI: 10.1007/s11704-026-60860-y
Artificial Intelligence
RESEARCH ARTICLE
Data-dependent exploration for online reinforcement learning from human feedback
Author information +
History +
PDF (1395KB)

Abstract

Online reinforcement learning from human feedback (RLHF) has emerged as a promising paradigm for aligning large language models (LLMs) by continuously collecting new preference feedback during training. A foundational challenge in this setting is exploration, which requires algorithms that enable the LLMs to generate informative comparisons that improve sample-efficiency in online RLHF. Existing exploration strategies often derive bonuses via on-policy expectations, which are difficult to estimate reliably from the limited historical preference data available during training; as a result, the policy can prematurely down-weight under-explored regions that may contain high-value behaviors. In this paper, we propose data-dependent exploration for preference optimization (DEPO), a simple and scalable method that leverages historical data to construct an extra uncertainty bonus for high-uncertainty regions, encouraging exploration toward potentially high-value data. Theoretically, we provide a data-dependent regret bound for the proposed algorithm, showing that it adapts to the hardness of the learning task itself and can be tighter than worst-case bounds in practice. Empirically, the proposed method consistently outperforms strong baselines across benchmarks, demonstrating improved sample efficiency.

Graphical abstract

Keywords

reinforcement learning from human feedback / preference learning / online learning

Cite this article

Download citation ▾
Zhen-Yu ZHANG, Yuting TANG, Jiandong ZHANG, Lanjihong MA, Masashi SUGIYAMA. Data-dependent exploration for online reinforcement learning from human feedback. Front. Comput. Sci., 2027, 21 (9) : 2109357 DOI:10.1007/s11704-026-60860-y

登录浏览全文

4963

注册一个新账户 忘记密码

References

[1]

Ouyang L, Wu J, Jiang X, Almeida D, Wainwright C L, Mishkin P, Zhang C, Agarwal S, Slama K, Ray A, Schulman J, Hilton J, Kelton F, Miller L, Simens M, Askell A, Welinder P, Christiano P, Leike J, Lowe R. Training language models to follow instructions with human feedback. In: Proceedings of the 36th International Conference on Neural Information Processing Systems. 2022, 2011

[2]

Touvron H, Martin L, Stone K, Albert P, Almahairi A, et al. Llama 2: open foundation and fine-tuned chat models. 2023, arXiv preprint arXiv: 2307.09288

[3]

Schulman J, Wolski F, Dhariwal P, Radford A, Klimov O. Proximal policy optimization algorithms. 2017, arXiv preprint arXiv: 1707.06347

[4]

Zhu Z, Tian H, Chen X, Zhang K, Yu Y . Offline model-based reinforcement learning with causal structured world models. Frontiers of Computer Science, 2025, 19( 4): 194347

[5]

Dong H, Xiong W, Pang B, Wang H, Zhao H, Zhou Y, Jiang N, Sahoo D, Xiong C, Zhang T. . RLHF workflow: from reward modeling to online RLHF.. Transactions on Machine Learning Research, 2024, 2024:

[6]

Cen S, Mei J, Goshvadi K, Dai H, Yang T, Yang S, Schuurmans D, Chi Y, Dai B. Value-incentivized preference optimization: a unified approach to online and offline RLHF. In: Proceedings of the 13th International Conference on Learning Representations. 2025

[7]

Li L F, Qian Y Y, Zhao P, Zhou Z H. Provably efficient RLHF pipeline: a unified view from contextual bandits. In: Proceedings of the 39th International Conference on Neural Information Processing Systems. 2025

[8]

Dwaracherla V, Asghari S M, Hao B, Van Roy B. Efficient exploration for LLMs. In: Proceedings of the 41st International Conference on Machine Learning. 2024, 12215−12227

[9]

Xie T, Foster D J, Krishnamurthy A, Rosset C, Awadallah A H, Rakhlin A. Exploratory preference optimization: harnessing implicit Q*-approximation for sample-efficient RLHF. In: Proceedings of the 13th International Conference on Learning Representations. 2025

[10]

Ma X, Zhao S Y, Yin Z H, Li W J . Clustered reinforcement learning. Frontiers of Computer Science, 2025, 19( 4): 194313

[11]

Guo S, Zhang B, Liu T, Liu T, Khalman M, Llinares F, Rame A, Mesnard T, Zhao Y, Piot B, Ferret J, Blondel M. Direct language model alignment from online AI feedback. 2024, arXiv preprint arXiv: 2402.04792

[12]

Wang R J, Xue K, Wang Y T, Yang P, Fu H B, Fu Q, Qian C . Diversity from human feedback. Frontiers of Computer Science, 2026, 20( 2): 2002320

[13]

Chen M, Chen Y, Sun W, Zhang X. Avoiding exp(R) scaling in RLHF through preference-based exploration. In: Proceedings of the 39th International Conference on Neural Information Processing Systems. 2025

[14]

Lai T L, Robbins H. . Asymptotically efficient adaptive allocation rules.. Advances in Applied Mathematics, 1985, 6( 1): 4–22

[15]

Auer P, Cesa-Bianchi N, Fischer P. Finite-time analysis of the multiarmed bandit problem. Machine Learning, 2002, 47(2−3): 235−256

[16]

Bradley R A, Terry M E. Rank analysis of incomplete block designs: I. The method of paired comparisons. Biometrika, 1952, 39(3−4): 324−345

[17]

Christiano P F, Leike J, Brown T B, Martic M, Legg S, Amodei D. Deep reinforcement learning from human preferences. In: Proceedings of the 31st International Conference on Neural Information Processing Systems. 2017, 4302−4310

[18]

Rafailov R, Sharma A, Mitchell E, Ermon S, Manning C D, Finn C. Direct preference optimization: your language model is secretly a reward model. In: Proceedings of the 37th International Conference on Neural Information Processing Systems. 2023, 2338

[19]

Song Y, Swamy G, Singh A, Bagnell J A, Sun W. The importance of online data: understanding preference fine-tuning via coverage. In: Proceedings of the 38th International Conference on Neural Information Processing Systems. 2024, 392

[20]

Tuyls J, Foster D J, Krishnamurthy A, Ash J T. Representation-based exploration for language models: from test-time to post-training. 2025, arXiv preprint arXiv: 2510.11686

[21]

Li P, Hastie T J, Church K W. Very sparse random projections. In: Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD). 2006, 287−296

[22]

Sherman J, Morrison W J . Adjustment of an inverse matrix corresponding to a change in one element of a given matrix. The Annals of Mathematical Statistics, 1950, 21( 1): 124–127

[23]

Hendrycks D, Burns C, Basart S, Zou A, Mazeika M, Song D, Steinhardt J. Measuring massive multitask language understanding. In: Proceedings of the 9th International Conference on Learning Representations. 2021

[24]

Rein D, Hou B L, Stickland A C, Petty J, Pang R Y, Dirani J, Michael J, Bowman S R. GPQA: a graduate-level google-proof Q&A benchmark. In: Proceedings of the 1st Conference on Language Modeling. 2024

[25]

Lin S, Hilton J, Evans O. TruthfulQA: measuring how models mimic human falsehoods. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2022, 3214−3252

[26]

Cobbe K, Kosaraju V, Bavarian M, Chen M, Jun H, Kaiser L, Plappert M, Tworek J, Hilton J, Nakano R, Hesse C, Schulman J. Training verifiers to solve math word problems. 2021, arXiv preprint arXiv: 2110.14168

[27]

Dubois Y, Galambosi B, Liang P, Hashimoto T B. Length-controlled AlpacaEval: a simple way to debias automatic evaluators. 2024, arXiv preprint arXiv: 2404.04475

[28]

Bai G, Liu J, Bu X, He Y, Liu J, Zhou Z, Lin Z, Su W, Ge T, Zheng B, Ouyang W. MT-Bench-101: a fine-grained benchmark for evaluating large language models in multi-turn dialogues. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024, 7421−7454

[29]

Jain N, Han K, Gu A, Li W D, Yan F, Zhang T, Wang S, Solar-Lezama A, Sen K, Stoica I. LiveCodeBench: holistic and contamination free evaluation of large language models for code. In: Proceedings of the 13th International Conference on Learning Representations. 2025

[30]

Azar M G, Guo Z D, Piot B, Munos R, Rowland M, Valko M, Calandriello D. A general theoretical paradigm to understand learning from human preferences. In: Proceedings of the 27th International Conference on Artificial Intelligence and Statistics. 2024, 4447−4455

[31]

Zhao Y, Joshi R, Liu T, Khalman M, Saleh M, Liu P J. SLiC-HF: sequence likelihood calibration with human feedback. 2023, arXiv preprint arXiv: 2305.10425

[32]

Zhao W, Hu Y, Sui X, Li Z, Deng Y, Zhao Y, Qin B, Che W . The gains do not make up for the losses: a comprehensive evaluation for safety alignment of large language models via machine unlearning. Frontiers of Computer Science, 2026, 20( 2): 2002319

[33]

Zhang X Y, Ye H J, Zhan D C . Trustworthy evaluation of large language models. Frontiers of Computer Science, 2026, 20( 2): 2002324

[34]

Xiong W, Shi C, Shen J, Rosenberg A, Qin Z, Calandriello D, Khalman M, Joshi R, Piot B, Saleh M, Jin C, Zhang T, Liu T. Building math agents with multi-turn iterative preference learning. In: Proceedings of the 13th International Conference on Learning Representations. 2025

[35]

Zhang S, Yu D, Sharma H, Zhong H, Liu Z, Yang Z, Wang S, Awadalla H H, Wang Z. . Self-exploring language models: active preference elicitation for online alignment. Transactions on Machine Learning Research, 2025, 2025:

[36]

Abbasi-Yadkori Y, Pál D, Szepesvári C. Improved algorithms for linear stochastic bandits. In: Proceedings of the 25th International Conference on Neural Information Processing Systems. 2011, 2312−2320

[37]

Li L, Chu W, Langford J, Schapire R E. A contextual-bandit approach to personalized news article recommendation. In: Proceedings of the 19th International Conference on World Wide Web. 2010, 661−670

[38]

Zhang Y J, Xu S A, Zhao P, Sugiyama M. Generalized linear bandits: almost optimal regret with one-pass update. In: Proceedings of the 39th International Conference on Neural Information Processing Systems. 2025

[39]

Wang J, Yu M, Zhao P, Zhou Z H. Learning with adaptive resource allocation. In: Proceedings of the 41st International Conference on Machine Learning. 2024, 52099−52116

[40]

Tropp J A . User-friendly tail bounds for sums of random matrices. Foundations of Computational Mathematics, 2012, 12( 4): 389–434

Rights & permissions

Higher Education Press

PDF (1395KB)

Supplementary files

Highlights

331

Accesses

0

Citation

Detail

Sections
Recommended

/

〈 〉