About the journal
Browse
Collections
Multimedia collections
Authors & reviewers
LMM-R1: empowering 3B LMMs with strong reasoning abilities with two-stage rule-based RL
Yingzhe PENG , Gongrui ZHANG , Miaosen ZHANG , Zhiyuan YOU , Jie LIU , Qipeng ZHU , Kai YANG , Xingzhong XU , Xin GENG , Xu YANG
Front. Comput. Sci. ›› 2027, Vol. 21 ›› Issue (9) : 2109353
Large multimodal models (LMMs) have demonstrated strong performance in vision-language tasks, yet their reasoning capabilities remain limited, particularly in smaller models like 3B LMMs, where model capacity restricts reasoning ability. Enhancing the reasoning of LMMs presents two challenges compared to language models: (1) high-quality multimodal reasoning data is scarcer than textual data, and (2) the alignment between vision encoders and language decoders often degrades pretrained language reasoning skills. In this work, we propose a two-stage training framework, LMM-R1, to improve the reasoning abilities of 3B LMMs. The first stage, Foundational Reasoning Enhancement (FRE), restores core language reasoning by training on abundant text-only data through rule-based reinforcement learning (RL). This phase leverages high-quality textual data to acquire reasoning skills without costly multimodal supervision. The second stage, Multimodal Generalization Training (MGT), extends these skills to the multimodal domain. Our experiments show that FRE outperforms direct RL on multimodal data across various tasks. LMM-R1 achieves a 4.5% and 4.69% average improvement in text-only and multimodal benchmarks, respectively, and a 4.6% improvement in agent-related tasks, comparable to models like GPT-4o. These results highlight the importance of enhancing foundational reasoning for effective multimodal generalization. The code is available at https://github.com/TideDra/lmm-r1, with an NPU version provided at https://github.com/GlowLED/lmm-r1-ascend.
large multimodal models / reasoning / reinforcement learning
| [1] |
DeepSeek-AI. DeepSeek-R1: incentivizing reasoning capability in LLMs via reinforcement learning. 2025, arXiv preprint arXiv: 2501.12948 |
| [2] |
OpenAI. OpenAI o1 system card. 2024, arXiv preprint arXiv: 2412.16720 |
| [3] |
Qwen Team. QwQ: reflect deeply on the boundaries of the unknown. See qwenlm.github.io/blog/qwq-32b-preview/website, 2024 |
| [4] |
Ke Z, Jiao F, Ming Y, Nguyen X P, Xu A, Long D X, Li M, Qin C, Wang P, Savarese S, Xiong C, Joty S. A survey of frontiers in LLM reasoning: inference scaling, learning to reason, and agentic systems. 2025, arXiv preprint arXiv: 2504.09037 |
| [5] |
Sun H, Haider M, Zhang R, Yang H, Qiu J, Yin M, Wang M, Bartlett P, Zanette A. Fast best-of-N decoding via speculative rejection. 2024, arXiv preprint arXiv: 2410.20290 |
| [6] |
Xie Y, Goyal A, Zheng W, Kan M Y, Lillicrap T P, Kawaguchi K, Shieh M. Monte Carlo tree search boosts reasoning via iterative preference learning. 2024, arXiv preprint arXiv: 2405.00451 |
| [7] |
Yao H, Huang J, Wu W, Zhang J, Wang Y, Liu S, Wang Y, Song Y, Feng H, Shen L, Tao D. Mulberry: empowering MLLM with o1-like reasoning and reflection via collective Monte Carlo tree search. 2024, arXiv preprint arXiv: 2412.18319 |
| [8] |
|
| [9] |
|
| [10] |
Luo L, Liu Y, Liu R, Phatale S, Guo M, Lara H, Li Y, Shu L, Zhu Y, Meng L, Sun J, Rastogi A. Improve mathematical reasoning in language models by automated process supervision. 2024, arXiv preprint arXiv: 2406.06592 |
| [11] |
|
| [12] |
Wang Y, Chen W, Han X, Lin X, Zhao H, Liu Y, Zhai B, Yuan J, You Q, Yang H. Exploring the reasoning abilities of multimodal large language models (MLLMs): a comprehensive survey on emerging trends in multimodal reasoning. 2024, arXiv preprint arXiv: 2401.06805 |
| [13] |
|
| [14] |
|
| [15] |
|
| [16] |
Zeng, Weihao and Huang, Yuzhen and Liu, Qian and Liu, Wei and He, Keqing and Ma, Zejun and He, Junxian. Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild. 2025, arXiv preprint arXiv:2503.18892 |
| [17] |
Luo M, Tan S, Wong J, Shi X, Tang W Y, Roongta M, Cai C, Luo J, Zhang T, Li L E, Popa R A, Stoica I. DeepScaleR: surpassing o1-preview with a 1.5b model by scaling RL. Technical Report, 2025 |
| [18] |
Yu X, Ding P, Zhang W, Huang S, Gao S, Qin C, Wu K, Fan Z, Qiao Z, Wang D. Unicorn: text-only data synthesis for vision language model training. 2025, arXiv preprint arXiv: 2503.22655v1 |
| [19] |
|
| [20] |
|
| [21] |
|
| [22] |
|
| [23] |
Xiang K, Liu Z, Jiang Z, Nie Y, Huang R, Fan H, Li H, Huang W, Zeng Y, Han J, Hong L, Xu H, Liang X. AtomThink: a slow thinking framework for multimodal mathematical reasoning. 2024, arXiv preprint arXiv: 2411.11930v1 |
| [24] |
|
| [25] |
|
| [26] |
Patnaik N, Nayak N, Agrawal H B, Khamaru M C, Bal G, Panda S S, Raj R, Meena V, Vadlamani K. Small vision-language models: a survey on compact architectures and techniques. 2025, arXiv preprint arXiv: 2503.10665 |
| [27] |
|
| [28] |
Masry A, Long D X, Tan J Q, Joty S, Hoque E. ChartQA: a benchmark for question answering about charts with visual and logical reasoning. In: Proceedings of the Findings of the Association for Computational Linguistics. 2022 |
| [29] |
|
| [30] |
|
| [31] |
|
| [32] |
|
| [33] |
Bai S, Chen K, Liu X, Wang J, Ge W, et al. Qwen2.5-VL technical report. 2025, arXiv preprint arXiv: 2502.13923 |
| [34] |
OpenAI. GPT-4 technical report. 2023, arXiv preprint arXiv: 2303.08774 |
| [35] |
Touvron H, Lavril T, Izacard G, Martinet X, Lachaux M A, Lacroix T, Rozière B, Goyal N, Hambro E, Azhar F, Rodriguez A, Joulin A, Grave E, Lample G. LLaMA: open and efficient foundation language models. 2023, arXiv preprint arXiv: 2302.13971 |
| [36] |
The Vicuna Team. Vicuna: an open-source chatbot impressing GPT-4 with 90%* ChatGPT quality. See lmsys.org/blog/2023-03-30-vicuna/website, 2023 |
| [37] |
Bai J, Bai S, Chu Y, Cui Z, Dang K, et al. Qwen technical report. 2023, arXiv preprint arXiv: 2309.16609 |
| [38] |
|
| [39] |
|
| [40] |
|
| [41] |
|
| [42] |
Liu H, Li C, Li Y, Li B, Zhang Y, Shen S, Lee Y J. LLaVA-NeXT: improved reasoning, OCR, and world knowledge. See llava-vl.github.io/blog/2024-01-30-llava-next/website, 2024 |
| [43] |
|
| [44] |
Ye Q, Xu H, Xu G, Ye J, Yan M, Zhou Y, Wang J, Hu A, Shi P, Shi Y, Li C, Xu Y, Chen H, Tian J, Qian Q, Zhang J, Huang F, Zhou J. mPLUG-Owl: modularization empowers large language models with multimodality. 2023, arXiv preprint arXiv: 2304.14178 |
| [45] |
|
| [46] |
|
| [47] |
|
| [48] |
Peng Z, Wang W, Dong L, Hao Y, Huang S, Ma S, Wei F. Kosmos-2: grounding multimodal large language models to the world. 2023, arXiv preprint arXiv: 2306.14824 |
| [49] |
Chen K, Zhang Z, Zeng W, Zhang R, Zhu F, Zhao R. Shikra: unleashing multimodal LLM’s referential dialogue magic. 2023, arXiv preprint arXiv: 2306.15195 |
| [50] |
Bai J, Bai S, Yang S, Wang S, Tan S, Wang P, Lin J, Zhou C, Zhou J. Qwen-VL: a versatile vision-language model for understanding, localization, text reading, and beyond. 2023, arXiv preprint arXiv: 2308.12966 |
| [51] |
Wang P, Bai S, Tan S, Wang S, Fan Z, Bai J, Chen K, Liu X, Wang J, Ge W, Fan Y, Dang K, Du M, Ren X, Men R, Liu D, Zhou C, Zhou J, Lin J. Qwen2-VL: enhancing vision-language model’s perception of the world at any resolution. 2024, arXiv preprint arXiv: 2409.12191 |
| [52] |
|
| [53] |
OpenAI. Gpt-4v(Ision) system card. 2023 |
| [54] |
Gemini Team Google. Gemini: a family of highly capable multimodal models. 2023, arXiv preprint arXiv: 2312.11805 |
| [55] |
|
| [56] |
Luo R, Zheng Z, Wang Y, Yu Y, Ni X, Lin Z, Zeng J, Yang Y. URSA: understanding and verifying chain-of-thought reasoning in multimodal mathematics. 2025, arXiv preprint arXiv: 2501.04686v1 |
| [57] |
|
| [58] |
Bai Y, Jones A, Ndousse K, Askell A, Chen A, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. 2022, arXiv preprint arXiv: 2204.05862 |
| [59] |
|
| [60] |
Kimi Team. Kimi k1.5: scaling reinforcement learning with LLMs. 2025, arXiv preprint arXiv: 2501.12599 |
| [61] |
Chen, Liang and Li, Lei and Zhao, Haozhe and Song, Yifan and Vinci and Kong, Lingpeng and Liu, Qi and Baobao Chang. RLVR in Vision Language Models: Findings, Questions and Directions, Notion Post, Feb. 2025. https://deepagent.notion.site/rlvr-in-vlms |
| [62] |
Schulman J, Wolski F, Dhariwal P, Radford A, Klimov O. Proximal policy optimization algorithms. 2017, arXiv preprint arXiv: 1707.06347 |
| [63] |
|
| [64] |
Shi W, Hu Z, Bin Y, Liu J, Yang Y, Ng S K, Bing L, Lee R K W. Math-LLaVA: bootstrapping mathematical reasoning for multimodal large language models. In: Proceedings of the Findings of the Association for Computational Linguistics. 2024 |
| [65] |
|
| [66] |
|
| [67] |
|
| [68] |
|
Higher Education Press
/
| 〈 |
|
〉 |