LMM-R1: empowering 3B LMMs with strong reasoning abilities with two-stage rule-based RL

Yingzhe PENG , Gongrui ZHANG , Miaosen ZHANG , Zhiyuan YOU , Jie LIU , Qipeng ZHU , Kai YANG , Xingzhong XU , Xin GENG , Xu YANG

Front. Comput. Sci. ›› 2027, Vol. 21 ›› Issue (9) : 2109353

PDF (1792KB)
Front. Comput. Sci. ›› 2027, Vol. 21 ›› Issue (9) :2109353 DOI: 10.1007/s11704-026-51776-8
Artificial Intelligence
RESEARCH ARTICLE
LMM-R1: empowering 3B LMMs with strong reasoning abilities with two-stage rule-based RL
Author information +
History +
PDF (1792KB)

Abstract

Large multimodal models (LMMs) have demonstrated strong performance in vision-language tasks, yet their reasoning capabilities remain limited, particularly in smaller models like 3B LMMs, where model capacity restricts reasoning ability. Enhancing the reasoning of LMMs presents two challenges compared to language models: (1) high-quality multimodal reasoning data is scarcer than textual data, and (2) the alignment between vision encoders and language decoders often degrades pretrained language reasoning skills. In this work, we propose a two-stage training framework, LMM-R1, to improve the reasoning abilities of 3B LMMs. The first stage, Foundational Reasoning Enhancement (FRE), restores core language reasoning by training on abundant text-only data through rule-based reinforcement learning (RL). This phase leverages high-quality textual data to acquire reasoning skills without costly multimodal supervision. The second stage, Multimodal Generalization Training (MGT), extends these skills to the multimodal domain. Our experiments show that FRE outperforms direct RL on multimodal data across various tasks. LMM-R1 achieves a 4.5% and 4.69% average improvement in text-only and multimodal benchmarks, respectively, and a 4.6% improvement in agent-related tasks, comparable to models like GPT-4o. These results highlight the importance of enhancing foundational reasoning for effective multimodal generalization. The code is available at https://github.com/TideDra/lmm-r1, with an NPU version provided at https://github.com/GlowLED/lmm-r1-ascend.

Graphical abstract

Keywords

large multimodal models / reasoning / reinforcement learning

Cite this article

Download citation ▾
Yingzhe PENG, Gongrui ZHANG, Miaosen ZHANG, Zhiyuan YOU, Jie LIU, Qipeng ZHU, Kai YANG, Xingzhong XU, Xin GENG, Xu YANG. LMM-R1: empowering 3B LMMs with strong reasoning abilities with two-stage rule-based RL. Front. Comput. Sci., 2027, 21 (9) : 2109353 DOI:10.1007/s11704-026-51776-8

登录浏览全文

4963

注册一个新账户 忘记密码

References

[1]

DeepSeek-AI. DeepSeek-R1: incentivizing reasoning capability in LLMs via reinforcement learning. 2025, arXiv preprint arXiv: 2501.12948

[2]

OpenAI. OpenAI o1 system card. 2024, arXiv preprint arXiv: 2412.16720

[3]

Qwen Team. QwQ: reflect deeply on the boundaries of the unknown. See qwenlm.github.io/blog/qwq-32b-preview/website, 2024

[4]

Ke Z, Jiao F, Ming Y, Nguyen X P, Xu A, Long D X, Li M, Qin C, Wang P, Savarese S, Xiong C, Joty S. A survey of frontiers in LLM reasoning: inference scaling, learning to reason, and agentic systems. 2025, arXiv preprint arXiv: 2504.09037

[5]

Sun H, Haider M, Zhang R, Yang H, Qiu J, Yin M, Wang M, Bartlett P, Zanette A. Fast best-of-N decoding via speculative rejection. 2024, arXiv preprint arXiv: 2410.20290

[6]

Xie Y, Goyal A, Zheng W, Kan M Y, Lillicrap T P, Kawaguchi K, Shieh M. Monte Carlo tree search boosts reasoning via iterative preference learning. 2024, arXiv preprint arXiv: 2405.00451

[7]

Yao H, Huang J, Wu W, Zhang J, Wang Y, Liu S, Wang Y, Song Y, Feng H, Shen L, Tao D. Mulberry: empowering MLLM with o1-like reasoning and reflection via collective Monte Carlo tree search. 2024, arXiv preprint arXiv: 2412.18319

[8]

Lightman H, Kosaraju V, Burda Y, Edwards H, Baker B, Lee T, Leike J, Schulman J, Sutskever I, Cobbe K. Let’s verify step by step. In: Proceedings of the 12th International Conference on Learning Representations. 2024

[9]

Setlur A, Nagpal C, Fisch A, Geng X, Eisenstein J, Agarwal R, Agarwal A, Berant J, Kumar A. Rewarding progress: scaling automated process verifiers for LLM reasoning. In: Proceedings of the 13th International Conference on Learning Representations. 2025

[10]

Luo L, Liu Y, Liu R, Phatale S, Guo M, Lara H, Li Y, Shu L, Zhu Y, Meng L, Sun J, Rastogi A. Improve mathematical reasoning in language models by automated process supervision. 2024, arXiv preprint arXiv: 2406.06592

[11]

Wang P, Li L, Shao Z, Xu R, Dai D, Li Y, Chen D, Wu Y, Sui Z. Math-shepherd: verify and reinforce LLMs step-by-step without human annotations. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics. 2024

[12]

Wang Y, Chen W, Han X, Lin X, Zhao H, Liu Y, Zhai B, Yuan J, You Q, Yang H. Exploring the reasoning abilities of multimodal large language models (MLLMs): a comprehensive survey on emerging trends in multimodal reasoning. 2024, arXiv preprint arXiv: 2401.06805

[13]

Wang K, Pan J, Shi W, Lu Z, Ren H, Zhou A, Zhan M, Li H. Measuring multimodal mathematical reasoning with the MATH-vision dataset. In: Proceedings of the 38th International Conference on Neural Information Processing Systems. 2024

[14]

Zhang R, Jiang D, Zhang Y, Lin H, Guo Z, Qiu P, Zhou A, Lu P, Chang K W, Qiao Y, Gao P, Li H. MATHVERSE: does your multi-modal LLM truly see the diagrams in visual math problems? In: Proceedings of the 18th European Conference on Computer Vision. 2025

[15]

He C, Luo R, Bai Y, Hu S, Thai Z, Shen J, Hu J, Han X, Huang Y, Zhang Y, Liu J, Qi L, Liu Z, Sun M. OlympiadBench: a challenging benchmark for promoting AGI with Olympiad-level bilingual multimodal scientific problems. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics. 2024

[16]

Zeng, Weihao and Huang, Yuzhen and Liu, Qian and Liu, Wei and He, Keqing and Ma, Zejun and He, Junxian. Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild. 2025, arXiv preprint arXiv:2503.18892

[17]

Luo M, Tan S, Wong J, Shi X, Tang W Y, Roongta M, Cai C, Luo J, Zhang T, Li L E, Popa R A, Stoica I. DeepScaleR: surpassing o1-preview with a 1.5b model by scaling RL. Technical Report, 2025

[18]

Yu X, Ding P, Zhang W, Huang S, Gao S, Qin C, Wu K, Fan Z, Qiao Z, Wang D. Unicorn: text-only data synthesis for vision language model training. 2025, arXiv preprint arXiv: 2503.22655v1

[19]

Lin J, Yin H, Ping W, Molchanov P, Shoeybi M, Han S. VILA: on pre-training for visual language models. In: Proceedings of 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024

[20]

Li L, Peng J, Chen H, Gao C, Yang X. How to configure good in-context sequence for visual question answering. In: Proceedings of 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024, 26710–26720

[21]

Zhang Y K, Lu S, Li Y, Ma Y, Chen Q G, Xu Z, Luo W, Zhang K, Zhan D C, Ye H J. WINGS: learning multimodal LLMs without text-only forgetting. In: Proceedings of the 38th International Conference on Neural Information Processing Systems. 2024, 31828–31853

[22]

Cheng Z, Chen Q, Zhang J, Fei H, Feng X, Che W, Li M, Qin L. CoMT: a novel benchmark for chain of multi-modal thought on large vision-language models. In: Proceedings of the 39th AAAI Conference on Artificial Intelligence. 2025

[23]

Xiang K, Liu Z, Jiang Z, Nie Y, Huang R, Fan H, Li H, Huang W, Zeng Y, Han J, Hong L, Xu H, Liang X. AtomThink: a slow thinking framework for multimodal mathematical reasoning. 2024, arXiv preprint arXiv: 2411.11930v1

[24]

Jiang D, Zhang R, Guo Z, Li Y, Qi Y, Chen X, Wang L, Jin J, Guo C, Yan S, Zhang B, Fu C, Gao P, Li H. MME-CoT: benchmarking chain-of-thought in large multimodal models for reasoning quality, robustness, and efficiency. In: Proceedings of the 42nd International Conference on Machine Learning. 2025

[25]

Li Z, Luo R, Zhang J, Qiu M, Huang X, Wei Z. VoCoT: unleashing visually grounded multi-step reasoning in large multi-modal models. In: Proceedings of 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies. 2025

[26]

Patnaik N, Nayak N, Agrawal H B, Khamaru M C, Bal G, Panda S S, Raj R, Meena V, Vadlamani K. Small vision-language models: a survey on compact architectures and techniques. 2025, arXiv preprint arXiv: 2503.10665

[27]

Lu P, Mishra S, Xia T, Qiu L, Chang K W, Zhu S C, Tafjord O, Clark P, Kalyan A. Learn to explain: multimodal reasoning via thought chains for science question answering. In: Proceedings of the 36th International Conference on Neural Information Processing Systems. 2022

[28]

Masry A, Long D X, Tan J Q, Joty S, Hoque E. ChartQA: a benchmark for question answering about charts with visual and logical reasoning. In: Proceedings of the Findings of the Association for Computational Linguistics. 2022

[29]

Liu X, Zhang T, Gu Y, Iong I L, Song X, , et al. VisualAgentBench: towards large multimodal models as visual foundation agents. In: Proceedings of the 13th International Conference on Learning Representations. 2025

[30]

Wang J, Xu H, Jia H, Zhang X, Yan M, Shen W, Zhang J, Huang F, Sang J. Mobile-agent-v2: mobile device operation assistant with effective navigation via multi-agent collaboration. In: Proceedings of the 38th Conference on Neural Information Processing Systems. 2024

[31]

Racanière S, Weber T, Reichert D P, Buesing L, Guez A, Rezende D, Puigdomènech Badia A, Vinyals O, Heess N, Li Y, Pascanu R, Battaglia P, Hassabis D, Silver D, Wierstra D. Imagination-augmented agents for deep reinforcement learning. In: Proceedings of the 31st Conference on Neural Information Processing Systems. 2017

[32]

Kurach K, Raichuk A, Stańczyk P, Zając M, Bachem O, Espeholt L, Riquelme C, Vincent D, Michalski M, Bousquet O, Gelly S. Google research football: a novel reinforcement learning environment. In: Proceedings of the 34th AAAI Conference on Artificial Intelligence. 2020

[33]

Bai S, Chen K, Liu X, Wang J, Ge W, et al. Qwen2.5-VL technical report. 2025, arXiv preprint arXiv: 2502.13923

[34]

OpenAI. GPT-4 technical report. 2023, arXiv preprint arXiv: 2303.08774

[35]

Touvron H, Lavril T, Izacard G, Martinet X, Lachaux M A, Lacroix T, Rozière B, Goyal N, Hambro E, Azhar F, Rodriguez A, Joulin A, Grave E, Lample G. LLaMA: open and efficient foundation language models. 2023, arXiv preprint arXiv: 2302.13971

[36]

The Vicuna Team. Vicuna: an open-source chatbot impressing GPT-4 with 90%* ChatGPT quality. See lmsys.org/blog/2023-03-30-vicuna/website, 2023

[37]

Bai J, Bai S, Chu Y, Cui Z, Dang K, et al. Qwen technical report. 2023, arXiv preprint arXiv: 2309.16609

[38]

Alayrac J B, Donahue J, Luc P, Miech A, Barr I, , et al. Flamingo: a visual language model for few-shot learning. In: Proceedings of the 36th International Conference on Neural Information Processing Systems. 2022

[39]

Li J, Li D, Savarese S, Hoi S. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In: Proceedings of the 40th International Conference on Machine Learning. 2023

[40]

Liu H, Li C, Wu Q, Lee Y J. Visual instruction tuning. In: Proceedings of the 37th International Conference on Neural Information Processing Systems. 2023

[41]

Liu H, Li C, Li Y, Lee Y J. Improved baselines with visual instruction tuning. In: Proceedings of 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024

[42]

Liu H, Li C, Li Y, Li B, Zhang Y, Shen S, Lee Y J. LLaVA-NeXT: improved reasoning, OCR, and world knowledge. See llava-vl.github.io/blog/2024-01-30-llava-next/website, 2024

[43]

Zhu D, Chen J, Shen X, Li X, Elhoseiny M. MiniGPT-4: enhancing vision-language understanding with advanced large language models. In: Proceedings of the 12th International Conference on Learning Representations. 2024

[44]

Ye Q, Xu H, Xu G, Ye J, Yan M, Zhou Y, Wang J, Hu A, Shi P, Shi Y, Li C, Xu Y, Chen H, Tian J, Qian Q, Zhang J, Huang F, Zhou J. mPLUG-Owl: modularization empowers large language models with multimodality. 2023, arXiv preprint arXiv: 2304.14178

[45]

Ye Q, Xu H, Ye J, Yan M, Hu A, Liu H, Qian Q, Zhang J, Huang F. mPLUG-OwI2: revolutionizing multi-modal large language model with modality collaboration. In: Proceedings of 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024

[46]

Ye J, Xu H, Liu H, Hu A, Yan M, Qian Q, Zhang J, Huang F, Zhou J. mPLUG-Owl3: towards long image-sequence understanding in multi-modal large language models. In: Proceedings of the 13th International Conference on Learning Representations. 2025

[47]

Wang W, Chen Z, Chen X, Wu J, Zhu X, Zeng G, Luo P, Lu T, Zhou J, Qiao Y, Dai J. VisionLLM: large language model is also an open-ended decoder for vision-centric tasks. In: Proceedings of the 37th International Conference on Neural Information Processing Systems. 2023

[48]

Peng Z, Wang W, Dong L, Hao Y, Huang S, Ma S, Wei F. Kosmos-2: grounding multimodal large language models to the world. 2023, arXiv preprint arXiv: 2306.14824

[49]

Chen K, Zhang Z, Zeng W, Zhang R, Zhu F, Zhao R. Shikra: unleashing multimodal LLM’s referential dialogue magic. 2023, arXiv preprint arXiv: 2306.15195

[50]

Bai J, Bai S, Yang S, Wang S, Tan S, Wang P, Lin J, Zhou C, Zhou J. Qwen-VL: a versatile vision-language model for understanding, localization, text reading, and beyond. 2023, arXiv preprint arXiv: 2308.12966

[51]

Wang P, Bai S, Tan S, Wang S, Fan Z, Bai J, Chen K, Liu X, Wang J, Ge W, Fan Y, Dang K, Du M, Ren X, Men R, Liu D, Zhou C, Zhou J, Lin J. Qwen2-VL: enhancing vision-language model’s perception of the world at any resolution. 2024, arXiv preprint arXiv: 2409.12191

[52]

Chen Z, Wu J, Wang W, Su W, Chen G, Xing S, Zhong M, Zhang Q, Zhu X, Lu L, Li B, Luo P, Lu T, Qiao Y, Dai J. Intern VL: scaling up vision foundation models and aligning for generic visual-linguistic tasks. In: Proceedings of 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024

[53]

OpenAI. Gpt-4v(Ision) system card. 2023

[54]

Gemini Team Google. Gemini: a family of highly capable multimodal models. 2023, arXiv preprint arXiv: 2312.11805

[55]

Sun H L, Sun Z, Peng H, Ye H J. Mitigating visual forgetting via take-along visual conditioning for multi-modal long cot reasoning. In: Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics. 2025

[56]

Luo R, Zheng Z, Wang Y, Yu Y, Ni X, Lin Z, Zeng J, Yang Y. URSA: understanding and verifying chain-of-thought reasoning in multimodal mathematics. 2025, arXiv preprint arXiv: 2501.04686v1

[57]

Ouyang L, Wu J, Jiang X, Almeida D, Wainwright C L, , et al. Training language models to follow instructions with human feedback. In: Proceedings of the 36th International Conference on Neural Information Processing Systems. 2022

[58]

Bai Y, Jones A, Ndousse K, Askell A, Chen A, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. 2022, arXiv preprint arXiv: 2204.05862

[59]

Sun Z, Shen S, Cao S, Liu H, Li C, Shen Y, Gan C, Gui L, Wang Y X, Yang Y, Keutzer K, Darrell T. Aligning large multimodal models with factually augmented RLHF. In: Proceedings of the Findings of the Association for Computational Linguistics. 2024

[60]

Kimi Team. Kimi k1.5: scaling reinforcement learning with LLMs. 2025, arXiv preprint arXiv: 2501.12599

[61]

Chen, Liang and Li, Lei and Zhao, Haozhe and Song, Yifan and Vinci and Kong, Lingpeng and Liu, Qi and Baobao Chang. RLVR in Vision Language Models: Findings, Questions and Directions, Notion Post, Feb. 2025. https://deepagent.notion.site/rlvr-in-vlms

[62]

Schulman J, Wolski F, Dhariwal P, Radford A, Klimov O. Proximal policy optimization algorithms. 2017, arXiv preprint arXiv: 1707.06347

[63]

Zhang M, Dai Q, Yang Y, Bao J, Chen D, Qiu K, Luo C, Geng X, Guo B. MageBench: bridging large multimodal models to agents. In: Proceedings of 2026 IEEE/CVF Winter Conference on Applications of Computer Vision. 2026

[64]

Shi W, Hu Z, Bin Y, Liu J, Yang Y, Ng S K, Bing L, Lee R K W. Math-LLaVA: bootstrapping mathematical reasoning for multimodal large language models. In: Proceedings of the Findings of the Association for Computational Linguistics. 2024

[65]

Chen L, Li J, Dong X, Zhang P, Zang Y, Chen Z, Duan H, Wang J, Qiao Y, Lin D, Zhao F. Are we on the right way for evaluating large vision-language models? In: Proceedings of the 38th International Conference on Neural Information Processing Systems. 2024

[66]

Lu P, Bansal H, Xia T, Liu J, Li C, Hajishirzi H, Cheng H, Chang K W, Galley M, Gao J. MathVista: evaluating mathematical reasoning of foundation models in visual contexts. In: Proceedings of the 12th International Conference on Learning Representations. 2024

[67]

Rein D, Hou B L, Stickland A C, Petty J, Pang R Y, Dirani J, Michael J, Bowman S R. GPQA: a graduate-level Google-proof Q&A benchmark. In: Proceedings of the Conference on Language Modeling. 2024

[68]

Zhai Y, Tong S, Li X, Cai M, Qu Q, Lee Y J, Ma Y. Investigating the catastrophic forgetting in multimodal large language model fine-tuning. In: Proceedings of the Conference on Parsimony and Learning. 2024

Rights & permissions

Higher Education Press

PDF (1792KB)

Supplementary files

Highlights

Supplementary materials

782

Accesses

0

Citation

Detail

Sections
Recommended

/

〈 〉