Empowering LVLM’s Multimodal Reasoning by Recovering Language Reasoning Ability

Abstract

Building upon powerful Large Language Models (LLMs), Large Vision-Language Models (LVLMs) have demonstrated impressive capabilities in processing visual information with natural language. To develop these multimodal abilities, current LVLMs typically undergo multi-stage vision-language training that fine-tunes the LLM backbone’s parameters. However, this vision-centric training process often prioritizes visual perception skills over advanced multimodal reasoning ability. Furthermore, our preliminary study comparing current LVLMs with their parent LLMs on language reasoning tasks reveals a significant gap, indicating that such training potentially undermines the language reasoning capability inherited from parent LLMs. This finding raises an important question: can multimodal reasoning performance be enhanced by restoring the diminished language reasoning capability? To explore this, we propose a training-free framework, BackEnsemble, which restores language reasoning ability while preserving visual perception capability through multi-stage model collaboration. We then evaluate its effectiveness on multimodal reasoning tasks to verify whether such restoration enhances multimodal reasoning performance. Specifically, given that LLMs inherently lack visual processing capability, we first construct a Backcross-LVLM through Backcross Merging that integrates an LVLM with its parent LLM to preserve critical language reasoning ability. Subsequently, during decoding, we fuse the output distributions of the LVLM and the Backcross-LVLM at each generation step, enabling a dynamic incorporation of the restored reasoning abilities without impairing the model’s visual perception. We evaluate BackEnsemble on five challenging multimodal reasoning tasks. Results demonstrate consistent improvements by effectively harnessing the parent LLM’s reasoning capability while maintaining visual perception proficiency. Furthermore, when combined with zero-shot chain-of-thought prompting, our approach continues to deliver consistent gains, demonstrating compatibility with more advanced reasoning enhancement techniques.

Keywords

Large Vision-Language Models / Multimodal Reasoning / Model Collaboration / Large Language Models

RIGHTS & PERMISSIONS

Higher Education Press 2026

Just Accepted

This article has successfully passed peer review and final editorial review, and will soon enter typesetting, proofreading and other publishing processes. The currently displayed version is the accepted final manuscript. The officially published version will be updated with format, DOI and citation information upon launch. We recommend that you pay attention to subsequent journal notifications and preferentially cite the officially published version. Thank you for your support and cooperation.