2027-09-15 2027, Volume 21 Issue 9

  • Select all
  • RESEARCH ARTICLE
    Yingzhe PENG, Gongrui ZHANG, Miaosen ZHANG, Zhiyuan YOU, Jie LIU, Qipeng ZHU, Kai YANG, Xingzhong XU, Xin GENG, Xu YANG

    Large multimodal models (LMMs) have demonstrated strong performance in vision-language tasks, yet their reasoning capabilities remain limited, particularly in smaller models like 3B LMMs, where model capacity restricts reasoning ability. Enhancing the reasoning of LMMs presents two challenges compared to language models: (1) high-quality multimodal reasoning data is scarcer than textual data, and (2) the alignment between vision encoders and language decoders often degrades pretrained language reasoning skills. In this work, we propose a two-stage training framework, LMM-R1, to improve the reasoning abilities of 3B LMMs. The first stage, Foundational Reasoning Enhancement (FRE), restores core language reasoning by training on abundant text-only data through rule-based reinforcement learning (RL). This phase leverages high-quality textual data to acquire reasoning skills without costly multimodal supervision. The second stage, Multimodal Generalization Training (MGT), extends these skills to the multimodal domain. Our experiments show that FRE outperforms direct RL on multimodal data across various tasks. LMM-R1 achieves a 4.5% and 4.69% average improvement in text-only and multimodal benchmarks, respectively, and a 4.6% improvement in agent-related tasks, comparable to models like GPT-4o. These results highlight the importance of enhancing foundational reasoning for effective multimodal generalization. The code is available at https://github.com/TideDra/lmm-r1, with an NPU version provided at https://github.com/GlowLED/lmm-r1-ascend.

  • RESEARCH ARTICLE
    Jiajun LI, Xiaojun SHI, Angeline Mary MARCHELLA, Tianze HU, Lianpeng QIAO, Kaiyu LI, Guoren WANG

    With the increase of digital information, there is a growing demand for efficiently extracting insights and answering queries from tables. However, real-world tables often feature complex structures like merged cells, multi-level headers, and block layouts. These heterogeneous tables pose significant challenges for question-answering systems. Existing research focuses primarily on normalized tables. Even state-of-the-art large language models for tabular data achieve 31%–42% lower accuracy on heterogeneous tables. Current methods either linearize tables into text and lose structural information or adopt graph-based representations that are computationally inefficient and yield suboptimal accuracy. Moreover, these methods typically require extensive manual processing to convert complex tables into specific formats, incurring additional annotation costs. To address these limitations, this paper presents DTI-HTQA (Dual-Tree-Indexing Heterogeneous Table Question Answering), an end-to-end framework for heterogeneous table question answering. It uses a graph-based representation to preserve table semantics, a LLM-driven header recognition algorithm to eliminate manual annotations, and dual-tree indices for intelligent retrieval. Four core operators guide precise cell retrieval, while a chain-of-thought reasoning strategy enhances transparency and accuracy for complex queries. Experiments on HiTab and AIT-QA show that DTI-HTQA achieves top performance in exact match and LLM-evaluated accuracy, approaching or surpassing state-of-the-art methods. The header recognition algorithm achieves an accuracy of 88%, and an upper-bound analysis confirms that this causes only a 2%–4% accuracy gap compared to gold-standard annotations, validating the feasibility of the LLM-based approach. Comprehensive ablation studies demonstrate that each component contributes meaningfully to the overall performance, with the dual-tree index providing the most substantial gains. Cost-efficiency analysis shows that DTI-HTQA achieves 52%–71% lower token consumption than comparable multi-step frameworks while delivering higher accuracy. This study tackles key challenges in heterogeneous table question answering and offers a practical solution by reducing reliance on costly manual annotations, enabling more efficient use of tabular data.

  • REVIEW ARTICLE
    Xinda JIA, Jinpeng LI, Zezhong WANG, Jingjing LI, Xingshan ZENG, Yasheng WANG, Weinan ZHANG, Yong YU, Weiwen LIU

    Large Language Models (LLMs) have demonstrated significant progress in reasoning across diverse domains. However, effective reasoning in real-world tasks requires adapting both the computational depth and the knowledge source to the demands of the problem, ranging from fast, intuitive responses to deliberate, step-by-step reasoning and from purely internal reasoning to externally tool-augmented processes. Drawing inspiration from cognitive psychology, we propose a novel taxonomy of LLM reasoning strategies along two orthogonal knowledge boundaries: (1) a fast/slow boundary separating intuitive from deliberative processes, and (2) an internal/external boundary distinguishing reasoning grounded in the model’s parameters from reasoning augmented by external tools. We systematically survey recent work on adaptive reasoning in LLMs and categorize methods based on key decision factors. We conclude by highlighting open challenges and future directions toward more adaptive, efficient, and reliable LLMs.

  • RESEARCH ARTICLE
    Yanshen YI, Chao WANG, Ying SUN, Qi ZHANG, Xunpeng HUANG, Hui XIONG

    Mutual funds are essential financial products for investors. However, existing fund recommendation systems often struggle to capture personalized, multi-dimensional needs like risk tolerance and investment goals, and face challenges from data privacy regulations and fragmented user interaction data. To address these limitations, we propose the Multi-round Adaptive Fund Recommendation Reinforcement Learning (MAFR-RL) framework. MAFR-RL innovatively employs a multi-round ratings-based scenario with a simulated user to overcome data limitations and efficiently infer feedback. We process rich multi-view fund attributes (e.g., historical profitability, risk, investment domains) using specialized encoders and self-attention mechanisms for comprehensive representation. Our actor network features a unique hierarchical Mixture-of-Experts (MoE) architecture with Mamba-based experts. This design first divides experts into shared and view-specific groups, then further employs MoE within each group for deeply adaptive feature fusion and precise policy generation. Furthermore, we introduce a novel dual-critic network, coupled with an adaptive multi-task Q-value formulation, to effectively balance fund-level appeal and constituent stock preferences. A specifically designed fund pool sampling strategy further enhances recommendation diversity and robustness. Extensive offline experiments and a small-scale online study demonstrate the effectiveness and efficiency of the MAFR-RL framework.

  • RESEARCH ARTICLE
    Zhen-Yu ZHANG, Yuting TANG, Jiandong ZHANG, Lanjihong MA, Masashi SUGIYAMA

    Online reinforcement learning from human feedback (RLHF) has emerged as a promising paradigm for aligning large language models (LLMs) by continuously collecting new preference feedback during training. A foundational challenge in this setting is exploration, which requires algorithms that enable the LLMs to generate informative comparisons that improve sample-efficiency in online RLHF. Existing exploration strategies often derive bonuses via on-policy expectations, which are difficult to estimate reliably from the limited historical preference data available during training; as a result, the policy can prematurely down-weight under-explored regions that may contain high-value behaviors. In this paper, we propose data-dependent exploration for preference optimization (DEPO), a simple and scalable method that leverages historical data to construct an extra uncertainty bonus for high-uncertainty regions, encouraging exploration toward potentially high-value data. Theoretically, we provide a data-dependent regret bound for the proposed algorithm, showing that it adapts to the hardness of the learning task itself and can be tighter than worst-case bounds in practice. Empirically, the proposed method consistently outperforms strong baselines across benchmarks, demonstrating improved sample efficiency.

  • RESEARCH ARTICLE
    Wei ZHANG, Xing LIU, Biying XU, Xinzhuo DENG, Kam-Kwai WONG, Wenjie NING, Wei CHEN

    Museums play a vital role in promoting public art education and cultural appreciation. However, traditional guidance methods often lack interactivity and fail to convey curatorial insights effectively, resulting in fragmented visitor attention and superficial understanding of artifacts. To address these challenges, we present ARtiVision, a gaze-guided augmented reality framework designed to deliver immersive and personalized museum experiences. Based on interviews with 2 curators and a visitor behavior study with 12 museum visitors, we construct a hierarchical artifact appreciation model that bridges expert knowledge with diverse visitor needs, enabling multi-level engagement with cultural exhibits. ARtiVision integrates two core components: an expert-driven annotation system for structuring interpretive content, and an AR application that presents this content interactively based on visitor preferences and attention. This design allows visitors to explore artifacts at varying depths, enhancing autonomy and comprehension. A follow-up study with 12 museum visitors and 2 curators demonstrates that ARtiVision improves engagement duration, enhances knowledge retention, and streamlines curatorial workflows. Our results highlight ARtiVision’s potential as a scalable, user-adaptive solution for enriching museum experiences.