Online DPO: Online Direct Preference Optimization with Fast-Slow Chasing
Biqing Qi , Junqi Gao , Pengfei Li , Fangyuan Li , Zhixuan Su , Kaiyan Zhang , Bowen Zhou
Direct preference optimization (DPO) improves the alignment of large language models (LLMs) with human values by training directly on human preference datasets, eliminating the need for reward models. However, due to the presence of cross-domain human preferences, direct continual training can lead to catastrophic forgetting, limiting DPO’s performance and efficiency. Inspired by intraspecific competition driving species evolution, we propose an Online Fast-Slow Chasing DPO (OFS-DPO) for preference alignment, simulating competition through fast and slow chasing among models to facilitate rapid adaptation. Specifically, we first derive the regret upper bound for online learning, validating our motivation with a min-max optimization pattern. Based on this, we introduce two identical modules using Low-Rank Adaptation (LoRA) with different optimization speeds to simulate intraspecific competition, and propose a new regularization term to guide their learning. To further mitigate catastrophic forgetting in cross-domain scenarios, we extend OFS-DPO with a LoRA module combination strategy, resulting in the Cross-domain Online Fast-Slow Chasing DPO (COFS-DPO). This method leverages linear combinations of fast module parameters from different task domains, fully utilizing historical information to achieve continual value alignment. Experimental results show that OFS-DPO outperforms DPO in in-domain alignment, while COFS-DPO excels in cross-domain continual learning scenarios.
Direct preference optimization / Large language models / Online learning / Catastrophic forgetting / Continual learning
| ● OFS-DPO uses fast and slow LoRA modules with a regularization term that guides their preference-probability gap during online alignment. | |
| ● COFS-DPO linearly combines task-specific fast LoRA modules to retain previously learned preferences during cross-domain continual alignment. | |
| ● Regret and gradient analyses characterize the fast-slow competition mechanism and its effect on online learning. | |
| ● Experiments show that OFS-DPO outperforms DPO in in-domain alignment and that COFS-DPO improves cross-domain continual learning. |
| [1] |
Stiennon N, Ouyang L, Wu J, et al. Learning to summarize from human feedback. In: Proceedings of the 34th International Conference on Neural Information Processing Systems. Vancouver, Canada, 2020, 253 |
| [2] |
Christiano P F, Leike J, Brown T B, et al. Deep reinforcement learning from human preferences. In: Proceedings of the 31st International Conference on Neural Information Processing Systems. Long Beach, CA, USA, 2017, 4302−4310 |
| [3] |
Bai Y T, Jones A, Ndousse K, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint: arXiv: 2204.05862, 2022 |
| [4] |
Rafailov R, Sharma A, Mitchell E, et al. Direct preference optimization: your language model is secretly a reward model. In: Proceedings of the 37th International Conference on Neural Information Processing Systems. New Orleans, LA, USA, 2023, 2338 |
| [5] |
Guo S M, Zhang B, Liu T L, et al. Direct language model alignment from online AI feedback. arXiv preprint: arXiv: 2402.04792, 2024 |
| [6] |
|
| [7] |
Rolnick D, Ahuja A, Schwarz J, et al. Experience replay for continual learning. In: Proceedings of the 33rd International Conference on Neural Information Processing Systems. Vancouver, Canada, 2019, 32 |
| [8] |
Zhang H, Lei Y, Gui L, et al. CPPO: continual learning for reinforcement learning with human feedback. In: Proceedings of the 12th International Conference on Learning Representations. Vienna, Austria, 2024 |
| [9] |
Lee H, Phatale S, Mansoor H, et al. RLAIF vs. RLHF: scaling reinforcement learning from human feedback with AI feedback. In: Proceedings of the 41st International Conference on Machine Learning. Vienna, Austria, 2024, 26874–26901 |
| [10] |
|
| [11] |
|
| [12] |
Agarwal N, Bullins B, Hazan E, et al. Online control with adversarial disturbances. In: Proceedings of the 36th International Conference on Machine Learning. Long Beach, CA, USA, 2019, 111–119 |
| [13] |
Hazan E, Kakade S, Singh K. The nonstochastic control problem. In: Proceedings of the 31st International Conference on Algorithmic Learning Theory. San Diego, CA, USA, 2020, 408–421 |
| [14] |
|
| [15] |
Hu E J, Shen Y L, Wallis P, et al. LoRA: low-rank adaptation of large language models. In: Proceedings of the 10th International Conference on Learning Representations. Online, 2022. |
| [16] |
|
| [17] |
|
| [18] |
Qi B Q, Chen X Q, Gao J Q, et al. Interactive continual learning: fast and slow thinking. In: Proceedings of 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, WA, USA, 2024, 12882–12892 |
| [19] |
Jiang Z M, Han X T, Jin H Y, et al. Chasing fairness under distribution shift: a model weight perturbation approach. In: Proceedings of the 37th International Conference on Neural Information Processing Systems. New Orleans, LA, USA, 2023, 2793 |
| [20] |
Wallace B, Dang M H, Rafailov R, et al. Diffusion model alignment using direct preference optimization. In: Proceedings of 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, WA, USA, 2024, 8228–8238 |
| [21] |
|
| [22] |
Majumder N, Hung C Y, Ghosal D, et al. Tango 2: aligning diffusion-based text-to-audio generations through direct preference optimization. In: Proceedings of the 32nd ACM International Conference on Multimedia. Melbourne, Australia, 2024, 564–572 |
| [23] |
Zhang R H, Gui L K, Sun Z Q, et al. Direct preference optimization of video large multimodal models from language model reward. In: Proceedings of 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies. Albuquerque, New Mexico, 2025, 694–717 |
| [24] |
Yang G Y, Chen J H, Lin W Z, et al. Direct preference optimization for neural machine translation with Minimum Bayes Risk decoding. In: Proceedings of 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Mexico City, Mexico, 2024, 391–398 |
| [25] |
Zhou Z H, Liu J, Yang C, et al. Beyond one-preference-fits-all alignment: multi-objective direct preference optimization. In: Proceedings of Findings of the Association for Computational Linguistics: ACL 2024. Bangkok, Thailand, 2024, 10586–10613 |
| [26] |
Pal A, Karkhanis D, Dooley S, et al. Smaug: fixing failure modes of preference optimisation with DPO-positive. arXiv preprint: arXiv: 2402.13228, 2024 |
| [27] |
Xie J W, Yan S P, He X M. General incremental learning with domain-aware categorical representations. In: Proceedings of 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans, LA, USA, 2022, 14331–14340 |
| [28] |
Simon C, Faraki M, Tsai Y H, et al. On generalizing beyond domains in cross-domain continual learning. In: Proceedings of 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans, LA, USA, 2022, 9255–9264 |
| [29] |
|
| [30] |
Aljundi R, Babiloni F, Elhoseiny M, et al. Memory aware synapses: learning what (not) to forget. In: Proceedings of the 15th European Conference on Computer Vision. Munich, Germany, 2018, 144–161 |
| [31] |
Chaudhry A, Dokania P K, Ajanthan T, et al. Riemannian walk for incremental learning: understanding forgetting and intransigence. In: Proceedings of the 15th European Conference on Computer Vision. Munich, Germany, 2018, 556–572 |
| [32] |
|
| [33] |
Castro F M, Marín-Jiménez M J, Guil N, et al. End-to-end incremental learning. In: Proceedings of the 15th European Conference on Computer Vision. Munich, Germany, 2018, 241–257 |
| [34] |
Sun F K, Ho C H, Lee H Y. LAMOL: language modeling for lifelong language learning. In: Proceedings of the 8th International Conference on Learning Representations. Addis Ababa, Ethiopia, 2020 |
| [35] |
Liu X L, Wu C S, Menta M, et al. Generative feature replay for class-incremental learning. In: Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. Seattle, WA, USA, 2020, 915–924 915 |
| [36] |
|
| [37] |
Rosenfeld E, Ravikumar P, Risteski A. An online learning approach to interpolation and extrapolation in domain generalization. In: Proceedings of the 25th International Conference on Artificial Intelligence and Statistics. Valencia, Spain, 2022, 2641–2657 |
| [38] |
Volpi R, Larlus D, Rogez G. Continual adaptation of visual representations via domain randomization and meta-learning. In: Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Nashville, TN, USA, 2021, 4441–4451 |
| [39] |
Kundu J N, Venkatesh R M, Venkat N, et al. Class-incremental domain adaptation. In: Proceedings of the 16th European Conference on Computer Vision. Glasgow, UK, 2020, 53–69 |
| [40] |
Zhao P, Zhang L. Improved analysis for dynamic regret of strongly convex and smooth functions. In: Proceedings of the 3rd Conference on Learning for Dynamics and Control. Zurich, Switzerland, 2021, 48–59 |
| [41] |
Haghtalab N, Roughgarden T, Shetty A. Smoothed analysis of online and differentially private learning. In: Proceedings of the 34th International Conference on Neural Information Processing Systems. Vancouver, Canada, 2020, 772 |
| [42] |
Nguyen L M, Nguyen P H, van Dijk M, et al. SGD and Hogwild! Convergence without the bounded gradients assumption. In: Proceedings of the 35th International Conference on Machine Learning. Stockholm, Sweden, 2018, 3750–3758 |
| [43] |
Gorbunov E, Hanzely F, Richtárik P. A unified theory of SGD: variance reduction, sampling, quantization and coordinate descent. In: Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics. Palermo, Italy, 2020, 680–690 |
| [44] |
Jang U, Lee J D, Ryu E K. LoRA training in the NTK regime has no spurious local minima. In: Proceedings of the 41st International Conference on Machine Learning. Vienna, Austria, 2024, 21306–21328 |
| [45] |
Maas A L, Daly R E, Pham P T, et al. Learning word vectors for sentiment analysis. In: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. Portland, OR, USA, 2011, 142–150 |
| [46] |
Radford A, Wu J, Child R, et al. Language models are unsupervised multitask learners. Available online (accessed 17 Sep 2026) |
| [47] |
Ziegler D M, Stiennon N, Wu J, et al. Fine-tuning language models from human preferences. arXiv preprint: arXiv: 1909.08593, 2019 |
| [48] |
Wang B, Komatsuzaki A. GPT-J-6B: a 6 billion parameter autoregressive language model. Available online (accessed 17 Sep 2026) |
| [49] |
Chen Y, Wang R, Jiang H Y, et al. Exploring the use of large language models for reference-free text quality evaluation: an empirical study. In: Proceedings of Findings of the Association for Computational Linguistics: IJCNLP-AACL 2023. Nusa Dua, Bali, 2023, 361–374 |
| [50] |
AI@Meta. Llama 3 model card. Available online (accessed 17 Sep 2026) |
| [51] |
Gao L, Schulman J, Hilton J. Scaling laws for reward model overoptimization. In: Proceedings of the 40th International Conference on Machine Learning. Honolulu, HI, USA, 2023, 10835–10866 |
| [52] |
Lin C Y. ROUGE: a package for automatic evaluation of summaries. In: Proceedings of Text Summarization Branches Out. Barcelona, Spain, 2004, 74–81 |
| [53] |
Aljundi R, Kelchtermans K, Tuytelaars T. Task-free continual learning. In: Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, CA, USA, 2019, 11246–11255 |
| [54] |
Kaplanis C, Shanahan M, Clopath C. Policy consolidation for continual reinforcement learning. In: Proceedings of the 36th International Conference on Machine Learning. Long Beach, California, USA, 2019, 3242–3251 |
| [55] |
Schöpf P, Auddy S, Hollenstein J, et al. Hypernetwork-PPO for continual reinforcement learning. In: Proceedings of the Deep Reinforcement Learning Workshop at NeurIPS 2022. New Orleans, USA, 2022 |
| [56] |
Ramamurthy R, Ammanabrolu P, Brantley K, et al. Is reinforcement learning (not) for natural language processing: benchmarks, baselines, and building blocks for natural language policy optimization. In: Proceedings of the 11th International Conference on Learning Representations. Kigali, Rwanda, 2023 |
The Author(s). This article is published by Higher Education Press.
/
| 〈 |
|
〉 |