Parameter-efficient fine-tuning (PEFT) has become a crucial paradigm for domain adaptation, achieving strong performance by updating only a small fraction of model parameters. Among various PEFT methods, low-rank adaptation (LoRA) is widely adopted due to its structural simplicity and computational efficiency. However, in multitask scenarios, LoRA often suffers from performance degradation due to task interference. Recent extensions, such as mixture-of-experts (MoE) and asymmetric LoRA variants, attempt to mitigate these issues; however, their reliance on fixed subspace-mixing strategies limits flexibility and makes the models more sensitive to input noise and data sparsity. Moreover, vanilla LoRA typically initialises low-rank matrices with Gaussian noise or zeros and optimises them in unconstrained subspaces, which may disrupt the structured representations learnt by pretrained models. In this article, we propose M3LoRA, a novel PEFT framework that leverages multiple low-rank matrices with mixture-of-subspaces and minor singular components initialisation. M3LoRA utilises multiple low-rank matrices to minimise interference between task-specific subspaces while maintaining representational capacity. Furthermore, it employs a learnable mixing matrix positioned between the down-projection and up-projection matrices to dynamically combine their subspaces, thereby enabling adaptive task-specific combination mechanisms. Additionally, it initialises these low-rank matrices within a subspace orthogonal to the principal singular components of pretrained weights-termed the minor singular components-thereby leveraging directions unexplored during pretraining to better capture task-specific features from labelled data. Extensive experiments on a wide range of benchmark datasets demonstrate that M3LoRA achieves substantial improvements over existing PEFT baselines, particularly in multi-task scenarios where task interference often degrades conventional LoRA performance.
Acknowledgements
This work was supported by the National Key Research and Development Program of China (Grant 2025YFC2511700), the National Natural Science Foundation of China (Grants 62576159 and 61402220), Zhejiang NSF (LR22F020005) and the Natural Science Foundation of Hunan Province, China (Grants 2025JJ50384 and 2022JJ30495).
Conflicts of Interest
The authors declare no conflicts of interest.
Data Availability Statement
The data that support the findings of this study are available from the network, all of them are publicly available datasets.
| [1] |
L. Zheng, W. L. Chiang, Y. Sheng, et al., Judging LLM-as-a-Judge With MT-Bench and Chatbot Arena,” Advances in Neural Information Processing Systems 36 (2023): 46595-46623.
|
| [2] |
P. Wang, L. Li, Z. Shao, et al., Math-Shepherd: Verify and Reinforce LLMS Step-by-Step Without Human Annotations,” preprint, arXiv:2312.08935 (2023).
|
| [3] |
H. Ivison, Y. Wang, V. Pyatkin, et al., Camels in a Changing Climate: Enhancing LM Adaptation With Tulu 2,” (2023), https://arxiv.org/abs/2311.10702. 2023.
|
| [4] |
E. J. Hu, Y. Shen, P. Wallis, et al., Lora: Low-Rank Adaptation of Large Language Models,” International Conference on Learning Representations 1, no. 2 (2022): 3.
|
| [5] |
X. Ma, Y. Liu, and C. Ouyang, “Capturing Semantic Features to Improve Chinese Event Detection,” CAAI Transactions on Intelligence Technology 7, no. 2 (2022): 219-227, https://doi.org/10.1049/cit2.12062.
|
| [6] |
W. Feng, C. Hao, Y. Zhang, Y. Han, and H. Wang, “Mixture-of-Loras: An Efficient Multitask Tuning for Large Language Models,” preprint, arXiv:2403.03432 (2024).
|
| [7] |
Q. Liu, X. Wu, X. Zhao, et al., When Moe Meets LLMS: Parameter Efficient Fine-Tuning for Multi-Task Medical Applications,” in Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval (2024), 1104-1114.
|
| [8] |
A. Agiza, M. Neseem, and S. Reda, “Mtlora: Low-Rank Adaptation Approach for Efficient Multi-Task Learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2024), 16196-16205.
|
| [9] |
C. Tian, Z. Shi, Z. Guo, L. Li, and C. Z. Xu, “Hydralora: An Asymmetric Lora Architecture for Efficient Fine-Tuning,” Advances in Neural Information Processing Systems 37 (2024): 9565-9584.
|
| [10] |
Y. Yang, D. Muhtar, Y. Shen, et al., MTL-LoRA: Low-Rank Adaptation for Multi-Task Learning,” Proceedings of the AAAI Conference on Artificial Intelligence 39, no. 20 (2025): 22010-22018, https://doi.org/10.1609/aaai.v39i20.35509.
|
| [11] |
L. Ren, Y. Liu, C. Ouyang, et al., DyLas: A Dynamic Label Alignment Strategy for Large-Scale Multi-Label Text Classification,” Information Fusion 120 (2025): 103081, https://doi.org/10.1016/j.inffus.2025.103081.
|
| [12] |
Y. Zhou, C. Yao, and J. Chen, “Cola: Collaborative Low-Rank Adaptation,” preprint, arXiv:2505.15471 (2025).
|
| [13] |
F. Meng, Z. Wang, and M. Zhang, “Pissa: Principal Singular Values and Singular Vectors Adaptation of Large Language Models,” Advances in Neural Information Processing Systems 37 (2024): 121038-121072.
|
| [14] |
H. Wang, Y. Li, S. Wang, G. Chen, and Y. Chen, “Milora: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuning,” preprint, arXiv:2406.09044 (2024).
|
| [15] |
H. Hajimolahoseini, M. Rezagholizadeh, V. Partovinia, M. Tahaei, O. M. Awad, and Y. Liu, “Compressing Pre-Trained Language Models Using Progressive Low Rank Decomposition,” Advances in Neural Information Processing Systems 35 (2021): 6-14.
|
| [16] |
P. Sharma, J. T. Ash, and D. Misra, “The Truth Is in There: Improving Reasoning in Language Models With Layer-Selective Rank Reduction,” preprint, arXiv:2312.13558 (2023).
|
| [17] |
Y. Li, B. Xiong, G. Chen, and Y. Chen, “Setar: Out-of-Distribution Detection With Selective Low-Rank Approximation,” Advances in Neural Information Processing Systems 37 (2024): 72840-72871.
|
| [18] |
X. Han, W. Zhao, N. Ding, Z. Liu, and M. Sun, “PTR: Prompt Tuning With Rules for Text Classification,” preprint, arXiv:2105.11259 (2021).
|
| [19] |
B. Lester, R. Al-Rfou, and N. Constant, “The Power of Scale for Parameter-Efficient Prompt Tuning,” preprint, arXiv:2104.08691 (2021).
|
| [20] |
X. L. Li and P. Liang, “Prefix-Tuning: Optimizing Continuous Prompts for Generation,” preprint, arXiv:2101.00190 (2021).
|
| [21] |
Z. Tao, C. Ouyang, Y. Liu, T. Chung, and Y. Cao, “Multi-Head Attention Graph Convolutional Network Model: End-to-End Entity and Relation Joint Extraction Based on Multi-Head Attention Graph Convolutional Network,” CAAI Transactions on Intelligence Technology 8, no. 2 (2023): 468-477, https://doi.org/10.1049/cit2.12086.
|
| [22] |
N. Houlsby, A. Giurgiu, S. Jastrzebski, et al., Parameter-Efficient Transfer Learning for NLP,” preprint, arXiv:1902.00751 (2019).
|
| [23] |
A. Rücklé, G. Geigle, M. Glockner, et al., AdapterDrop: On the Efficiency of Adapters in Transformers,” preprint, arXiv:2010.11918 (2021).
|
| [24] |
V. C. Lingam, A. Neerkaje, A. Vavre, et al., Svft: Parameter-Efficient Fine-Tuning With Singular Vectors,” Advances in Neural Information Processing Systems 37 (2024): 41425-41446.
|
| [25] |
Y. Qin, X. Wang, Y. Su, et al., Exploring Universal Intrinsic Task Subspace via Prompt Tuning,” preprint, arXiv:2110.07867 (2021).
|
| [26] |
X. Lin, Y. Wang, Z. Zhang, and M. Zhang, “Scientific Claim Recognition via Staged Fine-Tuning With LoRA,” Data Intelligence 7, no. 2 (2025): 303-335, https://doi.org/10.3724/2096-7004.di.2025.0009.
|
| [27] |
M. Valipour, M. Rezagholizadeh, I. Kobyzev, and A. Ghodsi, “DyLoRA: Parameter-Efficient Tuning of Pre-Trained Models Using Dynamic Search-Free Low-Rank Adaptation,” in Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics (2023), 3274-3287.
|
| [28] |
Q. Zhang, M. Chen, A. Bukharin, et al., AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning,” preprint, arXiv:2303.10512 (2023).
|
| [29] |
S. Y. Liu, C. Y. Wang, H. Yin, et al., DoRA: Weight-Decomposed Low-Rank Adaptation,” preprint, arXiv:2402.09353 (2024).
|
| [30] |
D. J. Kopiczko, T. Blankevoort, and Y. M. Asano, “VeRA: Vector-Based Random Matrix Adaptation,” preprint, arXiv:2310.11454 (2023).
|
| [31] |
H. Wang, T. Liu, R. Li, M. X. Cheng, T. Zhao, and J. Gao, “RoseLoRA: Row and Column-Wise Sparse Low-Rank Adaptation of Pre-Trained Language Model for Knowledge Editing and Fine-Tuning,” in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (2024), 996-1008.
|
| [32] |
Z. Gao, Q. Wang, A. Chen, et al., Parameter-Efficient Fine-Tuning With Discrete Fourier Transform,” preprint, arXiv:2405.03003 (2024).
|
| [33] |
Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-Based Learning Applied to Document Recognition,” Proceedings of the IEEE 86, no. 11 (2002): 2278-2324, https://doi.org/10.1109/5.726791.
|
| [34] |
M. Riesenhuber and T. Poggio, “Hierarchical Models of Object Recognition in Cortex,” Nature Neuroscience 2, no. 11 (1999): 1019-1025, https://doi.org/10.1038/14819.
|
| [35] |
G. E. Hinton, S. Osindero, and Y. W. Teh, “A Fast Learning Algorithm for Deep Belief Nets,” Neural Computation 18, no. 7 (2006): 1527-1554, https://doi.org/10.1162/neco.2006.18.7.1527.
|
| [36] |
K. Li, T. Zhan, K. Fu, et al., Mergenet: Knowledge Migration Across Heterogeneous Models, Tasks, and Modalities,” Proceedings of the AAAI Conference on Artificial Intelligence 39, no. 5 (2025): 4824-4832, https://doi.org/10.1609/aaai.v39i5.32510.
|
| [37] |
C. Gao, K. Chen, J. Rao, et al., Higher Layers Need More Lora Experts,” preprint, arXiv:2402.08562 (2024).
|
| [38] |
T. Wu, J. Wang, Z. Zhao, and N. Wong, “Mixture-of-Subspaces in Low-Rank Adaptation,” preprint, arXiv:2406.11909 (2024).
|
| [39] |
H. Yang, L. Wang, and M. Z. Hossain, “SRLoRA: Subspace Recomposition in Low-Rank Adaptation via Importance-Based Fusion and Reinitialization,” preprint, arXiv:2505.12433 (2025).
|
| [40] |
M. Conover, M. Hayes, A. Mathur, et al., Free Dolly: Introducing the World’s First Truly Open Instruction-Tuned LLM,” (2023), https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tuned-llm.
|
| [41] |
D. Hendrycks, C. Burns, S. Basart, et al., Measuring Massive Multitask Language Understanding,” preprint, arXiv:2009.03300 (2020).
|
| [42] |
Alignment-Lab-AI, “ Lawyer-Instruct,” (2024).
|
| [43] |
I. Chalkidis, N. Garneau, C. Goanta, D. Katz, and A. Søgaard, “LeXFiles and LegaLLaMA: Facilitating English Multinational Legal Language Model Development,” in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, Vol. 1 (2023), 15513-15535: Long Papers.
|
| [44] |
Y. Li, Z. Li, K. Zhang, R. Dan, S. Jiang, and Y. Zhang, “Chatdoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledge,” Cureus 15, no. 6 (2023), https://doi.org/10.7759/cureus.40895.
|
| [45] |
K. Cobbe, V. Kosaraju, M. Bavarian, et al., Training Verifiers to Solve Math Word Problems,” preprint, arXiv:2110.14168 (2021).
|
| [46] |
N. Wang, H. Yang, and C. D. Wang, “Fingpt: Instruction Tuning Benchmark for Open-Source Large Language Models in Financial Datasets,” preprint, arXiv:2310.04793 (2023).
|
| [47] |
W. Lian, B. Goodson, E. Pentland, A. Cook, and V. C. Teknium, “Openorca: An Open Dataset of Gpt Augmented Flan Reasoning Traces,” Hugging Face Dataset Repository (2023).
|
| [48] |
M. Suzgun, N. Scales, N. Schärli, et al., Challenging Big-Bench Tasks and Whether Chain-of-Thought Can Solve Them,” preprint, arXiv:2210.09261 (2022).
|
| [49] |
Z. Diao, M. Qiao, X. Wang, et al., Not all Data Are What You Need: A Data-Efficient Training Method Using Heterogeneous Hardware,” IEEE Transactions on Knowledge and Data Engineering 38, no. 5 (2026): 2995-3008, https://doi.org/10.1109/TKDE.2026.3668991.
|
| [50] |
Z.Hongying, A.Javed, M.Abdullah, J.Rashid, and M.Faheem , “ Large Language Models With Contrastive Decoding Algorithm for Hallucination Mitigation in Low-Resource Languages,” CAAI Transactions on Intelligence Technology 10, no. 4 (2025): 1104-1117, https://doi.org/10.1049/cit2.70004.
|
| [51] |
A. Grattafiori, A. Dubey, A. Jauhri, et al., The LLaMA 3 Herd of Models,” preprint, arXiv:2407.21783 (2024).
|
| [52] |
X. Liu, Y. Zheng, Z. Du, et al., GPT Understands, Too,” preprint, arXiv:2103.10385 (2021).
|
| [53] |
H. Liu, D. Tam, M. Muqeeth, et al., Few-Shot Parameter-Efficient Fine-Tuning Is Better and Cheaper Than in-Context Learning,” Advances in Neural Information Processing Systems 35 (2022): 1950-1965.
|
| [54] |
K. Büyükakyüz, “OLoRA: Orthonormal Low-Rank Adaptation of Large Language Models,” preprint, arXiv:2406.01775 (2024).
|
| [55] |
P. Wu, W. Wen, H. Li, Z. Li, and N. Zeng, “AI-Driven Automation of Aviation Equipment Inspection: Insights From a Complex Adaptive Systems Perspective,” Innovation 7, no. 1 (2026): 101084, https://doi.org/10.1016/j.xinn.2025.101084.
|
| [56] |
P. Wu, H. Li, X. Luo, L. Hu, R. Yang, and N. Zeng, “From Data Analysis to Intelligent Maintenance: A Survey on Visual Defect Detection in Aero-Engines,” Measurement Science and Technology 36, no. 6 (2025): 062001, https://doi.org/10.1088/1361-6501/add6c8.
|
PDF
(1797KB)
