A survey on LoRA of large language models

Abstract

Low-Rank Adaptation (LoRA), which updates the dense neural network layers with pluggable low-rank matrices, is one of the best performed parameter efficient fine-tuning paradigms. Furthermore, it has significant advantages in cross-task generalization and privacy-preserving. Hence, LoRA has gained much attention recently, and the number of related literature demonstrates exponential growth. It is necessary to conduct a comprehensive overview of the current progress on LoRA. This survey categorizes and reviews the progress from the perspectives of (1) downstream adaptation improving variants that improve LoRA’s performance on downstream tasks; (2) cross-task generalization methods that mix multiple LoRA plugins to achieve cross-task generalization; (3) efficiency-improving methods that boost the computation-efficiency of LoRA; (4) data privacy-preserving methods that use LoRA in federated learning; (5) application. Besides, this survey also discusses the future directions in this field.

Graphical abstract

Keywords

low-rank adaptation / LoRA / large language models / LLMs

References

[1] Devlin J, Chang M W, Lee K, Toutanova K. BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT. 2019, 4171−4186

[2] Chowdhery A, Narang S, Devlin J, Bosma M, Mishra G, Roberts A, Barham P, Chung H W, Sutton C, Gehrmann S, Schuh P, Shi K, Tsvyashchenko S, Maynez J, Rao A, Barnes P, Tay Y, Shazeer N, Prabhakaran V, Reif E, Du N, Hutchinson B, Pope R, Bradbury J, Austin J, Isard M, Gur-Ari G, Yin P, Duke T, Levskaya A, Ghemawat S, Dev S, Michalewski H, Garcia X, Misra V, Robinson K, Fedus L, Zhou D, Ippolito D, Luan D, Lim H, Zoph B, Spiridonov A, Sepassi R, Dohan D, Agrawal S, Omernick M, Dai A M, Pillai T S, Pellat M, Lewkowycz A, Moreira E, Child R, Polozov O, Lee K, Zhou Z, Wang X, Saeta B, Diaz M, Firat O, Catasta M, Wei J, Meier-Hellstern K, Eck D, Dean J, Petrov S, Fiedel N . PaLM: scaling language modeling with pathways. The Journal of Machine Learning Research, 2023, 24( 1): 240

[3] Chen Y, Qian S, Tang H, Lai X, Liu Z, Han S, Jia J. LongLoRA: efficient fine-tuning of long-context large language models. In: Proceedings of the 12th International Conference on Learning Representations. 2024

[4] Pan R, Liu X, Diao S, Pi R, Zhang J, Han C, Zhang T. LISA: layerwise importance sampling for memory-efficient large language model fine-tuning. 2024, arXiv preprint arXiv: 2403.17919

[5] Ding N, Qin Y, Yang G, Wei F, Yang Z, Su Y, Hu S, Chen Y, Chan C M, Chen W, Yi J, Zhao W, Wang X, Liu Z, Zheng H T, Chen J, Liu Y, Tang J, Li J, Sun M . Parameter-efficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence, 2023, 5( 3): 220–235

[6] Houlsby N, Giurgiu A, Jastrzebski S, Morrone B, de Laroussilhe Q, Gesmundo A, Attariyan M, Gelly S. Parameter-efficient transfer learning for NLP. In: Proceedings of the 36th International Conference on Machine Learning. 2019, 2790−2799

[7] Lester B, Al-Rfou R, Constant N. The power of scale for parameter-efficient prompt tuning. In: Proceedings of 2021 Conference on Empirical Methods in Natural Language Processing. 2021, 3045−3059

[8] Zaken E B, Goldberg Y, Ravfogel S. BitFit: simple parameter-efficient fine-tuning for transformer-based masked language-models. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). 2022, 1−9

[9] Hu E J, Shen Y, Wallis P, Allen-Zhu Z, Li Y, Wang S, Wang L, Chen W. LoRA: low-rank adaptation of large language models. In: Proceedings of the 10th International Conference on Learning Representations. 2022

[10] Zhao W X, Zhou K, Li J, Tang T, Wang X, Hou Y, Min Y, Zhang B, Zhang J, Dong Z, Du Y, Yang C, Chen Y, Chen Z, Jiang J, Ren R, Li Y, Tang X, Liu Z, Liu P, Nie J Y, Wen J R. A survey of large language models. 2023, arXiv preprint arXiv: 2303.18223

[11] Han Z, Gao C, Liu J, Zhang J, Zhang S Q. Parameter-efficient fine-tuning for large models: a comprehensive survey. 2024, arXiv preprint arXiv: 2403.14608

[12] Malladi S, Wettig A, Yu D, Chen D, Arora S. A kernel-based view of language model fine-tuning. In: Proceedings of the 40th International Conference on Machine Learning. 2023, 23610−23641

[13] Koubbi H, Boussard M, Hernandez L. The impact of LoRA on the emergence of clusters in transformers. 2024, arXiv preprint arXiv: 2402.15415

[14] Jang U, Lee J D, Ryu E K. LoRA training in the NTK regime has no spurious local minima. 2024, arXiv preprint arXiv: 2402.11867

[15] Zhu J, Greenewald K, Nadjahi K, de Ocáriz Borde H S, Gabrielsson R B, Choshen L, Ghassemi M, Yurochkin M, Solomon J. Asymmetry in low-rank adapters of foundation models. 2024, arXiv preprint arXiv: 2402.16842

[16] Zeng Y, Lee K. The expressive power of low-rank adaptation. In: Proceedings of the 12th International Conference on Learning Representations. 2024

[17] Lialin V, Muckatira S, Shivagunde N, Rumshisky A. ReLoRA: high-rank training through low-rank updates. In: Proceedings of the 12th International Conference on Learning Representations. 2024

[18] Jiang T, Huang S, Luo S, Zhang Z, Huang H, Wei F, Deng W, Sun F, Zhang Q, Wang D, Zhuang F. MoRA: high-rank updating for parameter-efficient fine-tuning. 2024, arXiv preprint arXiv: 2405.12130

[19] Huh M, Cheung B, Bernstein J, Isola P, Agrawal P. Training neural networks from scratch with parallel low-rank adapters. 2024, arXiv preprint arXiv: 2402.16828

[20] Liang Y S, Li W J. InfLoRA: interference-free low-rank adaptation for continual learning. 2024, arXiv preprint arXiv: 2404.00228

[21] Zhao H, Ni B, Wang H, Fan J, Zhu F, Wang Y, Chen Y, Meng G, Zhang Z. Continual forgetting for pre-trained vision models. 2024, arXiv preprint arXiv: 2403.11530

[22] Ren W, Li X, Wang L, Zhao T, Qin W. Analyzing and reducing catastrophic forgetting in parameter efficient tuning. 2024, arXiv preprint arXiv: 2402.18865

[23] Zhang H. SinkLoRA: enhanced efficiency and chat capabilities for long-context large language models. 2024, arXiv preprint arXiv: 2406.05678

[24] Xia W, Qin C, Hazan E. Chain of LoRA: efficient fine-tuning of language models via residual learning. 2024, arXiv preprint arXiv: 2401.04151

[25] Ren P, Shi C, Wu S, Zhang M, Ren Z, de Rijke M, Chen Z, Pei J. MELoRA: mini-ensemble low-rank adapters for parameter-efficient fine-tuning. 2024, arXiv preprint arXiv: 2402.17263

[26] Hao Y, Cao Y, Mou L. Flora: low-rank adapters are secretly gradient compressors. 2024, arXiv preprint arXiv: 2402.03293

[27] Zi B, Qi X, Wang L, Wang J, Wong K F, Zhang L. Delta-LoRA: fine-tuning high-rank parameters with the delta of low-rank matrices. 2023, arXiv preprint arXiv: 2309.02411

[28] Zhang Q, Chen M, Bukharin A, He P, Cheng Y, Chen W, Zhao T. Adaptive budget allocation for parameter-efficient fine-tuning. In: Proceedings of the 11th International Conference on Learning Representations. 2023

[29] Hu Y, Xie Y, Wang T, Chen M, Pan Z . Structure-aware low-rank adaptation for parameter-efficient fine-tuning. Mathematics, 2023, 11( 20): 4317

[30] Zhang F, Li L, Chen J, Jiang Z, Wang B, Qian Y. IncreLoRA: incremental parameter allocation method for parameter-efficient fine-tuning. 2023, arXiv preprint arXiv: 2308.12043

[31] Mao Y, Huang K, Guan C, Bao G, Mo F, Xu J. DoRA: enhancing parameter-efficient fine-tuning with dynamic rank distribution. 2024, arXiv preprint arXiv: 2405.17357

[32] Zhang R, Qiang R, Somayajula S A, Xie P. AutoLoRA: automatically tuning matrix ranks in low-rank adaptation based on meta learning. 2024, arXiv preprint arXiv: 2403.09113

[33] Ding N, Lv X, Wang Q, Chen Y, Zhou B, Liu Z, Sun M. Sparse low-rank adaptation of pre-trained language models. In: Proceedings of 2023 Conference on Empirical Methods in Natural Language Processing. 2023, 4133−4145

[34] Liu Z, Lyn J, Zhu W, Tian X, Graham Y. ALoRA: allocating low-rank adaptation for fine-tuning large language models. 2024, arXiv preprint arXiv: 2403.16187

[35] Valipour M, Rezagholizadeh M, Kobyzev I, Ghodsi A. DyLoRA: parameter-efficient tuning of pre-trained models using dynamic search-free low-rank adaptation. In: Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics. 2023, 3274−3287

[36] Hayou S, Ghosh N, Yu B. The impact of initialization on LoRA finetuning dynamics. 2024, arXiv preprint arXiv: 2406.08447

[37] Meng F, Wang Z, Zhang M. PiSSA: principal singular values and singular vectors adaptation of large language models. 2024, arXiv preprint arXiv: 2404.02948

[38] Wang H, Xiao Z, Li Y, Wang S, Chen G, Chen Y. MiLoRA: harnessing minor singular components for parameter-efficient LLM finetuning. 2024, arXiv preprint arXiv: 2406.09044

[39] Zhang F, Pilanci M. Riemannian preconditioned LoRA for fine-tuning foundation models. 2024, arXiv preprint arXiv: 2402.02347

[40] Hayou S, Ghosh N, Yu B. LoRA+: efficient low rank adaptation of large models. 2024, arXiv preprint arXiv: 2402.12354

[41] Shi S, Huang S, Song M, Li Z, Zhang Z, Huang H, Wei F, Deng W, Sun F, Zhang Q. ResLoRA: identity residual mapping in low-rank adaption. 2024, arXiv preprint arXiv: 2402.18039

[42] Wen Z, Zhang J, Fang Y. SIBO: a simple booster for parameter-efficient fine-tuning. 2024, arXiv preprint arXiv: 2402.11896

[43] Jin F, Liu Y, Tan Y. Derivative-free optimization for low-rank adaptation in large language models. 2024, arXiv preprint arXiv: 2403.01754

[44] Liu S Y, Wang C Y, Yin H, Molchanov P, Wang Y C F, Cheng K T, Chen M H. DoRA: weight-decomposed low-rank adaptation. 2024, arXiv preprint arXiv: 2402.09353

[45] Qiang R, Zhang R, Xie P. BiLoRA: a bi-level optimization framework for overfitting-resilient low-rank adaptation of large pre-trained models. 2024, arXiv preprint arXiv: 2403.13037

[46] Lin Y, Ma X, Chu X, Jin Y, Yang Z, Wang Y, Mei H. LoRA dropout as a sparsity regularizer for overfitting control. 2024, arXiv preprint arXiv: 2404.09610

[47] Wang S, Chen L, Jiang J, Xue B, Kong L, Wu C. LoRA meets dropout under a unified framework. 2024, arXiv preprint arXiv: 2403.00812

[48] Yang A X, Robeyns M, Wang X, Aitchison L. Bayesian low-rank adaptation for large language models. In: Proceedings of the 12th International Conference on Learning Representations. 2024

[49] Qi Z, Tan X, Shi S, Qu C, Xu Y, Qi Y. PILLOW: enhancing efficient instruction fine-tuning via prompt matching. In: Proceedings of 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track. 2023, 471−482

[50] Zhang L, Wu J, Zhou D, Xu G. STAR: constraint LoRA with dynamic active learning for data-efficient fine-tuning of large language models. 2024, arXiv preprint arXiv: 2403.01165

[51] Wang X, Aitchison L, Rudolph M. LoRA ensembles for large language model fine-tuning. 2023, arXiv preprint arXiv: 2310.00035

[52] Zhao Z, Gan L, Wang G, Zhou W, Yang H, Kuang K, Wu F. LoraRetriever: input-aware LoRA retrieval and composition for mixed tasks in the wild. 2024, arXiv preprint arXiv: 2402.09997

[53] Smith J S, Cascante-Bonilla P, Arbelle A, Kim D, Panda R, Cox D, Yang D, Kira Z, Feris R, Karlinsky L. ConStruct-VL: data-free continual structured VL concepts learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023, 14994−15004

[54] Sun Y, Li M, Cao Y, Wang K, Wang W, Zeng X, Zhao R. To be or not to be? An exploration of continuously controllable prompt engineering. 2023, arXiv preprint arXiv: 2311.09773

[55] Zhang J, Chen S, Liu J, He J. Composing parameter-efficient modules with arithmetic operations. 2023, arXiv preprint arXiv: 2306.14870

[56] Chitale R, Vaidya A, Kane A, Ghotkar A. Task arithmetic with LoRA for continual learning. 2023, arXiv preprint arXiv: 2311.02428

[57] Belofsky J. Token-Level Adaptation of LoRA adapters for downstream task generalization. In: Proceedings of the 6th Artificial Intelligence and Cloud Computing Conference. 2023, 168−172

[58] Jiang W, Lin B, Shi H, Zhang Y, Li Z, Kwok J T. Effective and parameter-efficient reusing fine-tuned models. 2023, arXiv preprint arXiv: 2310.01886

[59] Asadi N, Beitollahi M, Khalil Y, Li Y, Zhang G, Chen X. Does combining parameter-efficient modules improve few-shot transfer accuracy? 2024, arXiv preprint arXiv: 2402.15414

[60] Huang C, Liu Q, Lin B Y, Pang T, Du C, Lin M. LoraHub: efficient cross-task generalization via dynamic LoRA composition. 2023, arXiv preprint arXiv: 2307.13269

[61] Yadav P, Choshen L, Raffel C, Bansal M. ComPEFT: compression for communicating parameter efficient updates via sparsification and quantization. 2023, arXiv preprint arXiv: 2311.13171

[62] Tang A, Shen L, Luo Y, Zhan Y, Hu H, Du B, Chen Y, Tao D. Parameter-efficient multi-task model fusion with partial linearization. In: Proceedings of the 12th International Conference on Learning Representations. 2024

[63] Shen Y, Xu Z, Wang Q, Cheng Y, Yin W, Huang L. Multimodal instruction tuning with conditional mixture of LoRA. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024, 637−648

[64] Buehler E L, Buehler M J . X-LoRA: mixture of low-rank adapter experts, a flexible framework for large language models with applications in protein mechanics and molecular design. APL Machine Learning, 2024, 2( 2): 026119

[65] Yang S, Ali M A, Wang C L, Hu L, Wang D. MoRAL: MoE augmented LoRA for LLMs’ lifelong learning. 2024, arXiv preprint arXiv: 2402.11260

[66] Dou S, Zhou E, Liu Y, Gao S, Zhao J, Shen W, Zhou Y, Xi Z, Wang X, Fan X, Pu S, Zhu J, Zheng R, Gui T, Zhang Q, Huang X. LoRAMoE: alleviate world knowledge forgetting in large language models via MoE-style plugin. 2023, arXiv preprint arXiv: 2312.09979

[67] Gou Y, Liu Z, Chen K, Hong L, Xu H, Li A, Yeung D Y, Kwok J T, Zhang Y. Mixture of cluster-conditional LoRA experts for vision-language instruction tuning. 2023, arXiv preprint arXiv: 2312.12379

[68] Liu Q, Wu X, Zhao X, Zhu Y, Xu D, Tian F, Zheng Y. When MOE meets LLMs: parameter efficient fine-tuning for multi-task medical applications. In: Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2024, 1104−1114

[69] Feng W, Hao C, Zhang Y, Han Y, Wang H. Mixture-of-LoRAs: an efficient multitask tuning method for large language models. In: Proceedings of 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation. 2024, 11371−11380

[70] Wang Y, Lin Y, Zeng X, Zhang G. MultiLoRA: democratizing LoRA for better multi-task learning. 2023, arXiv preprint arXiv: 2311.11501

[71] Yang Y, Jiang P T, Hou Q, Zhang H, Chen J, Li B. Multi-task dense prediction via mixture of low-rank experts. 2024, arXiv preprint arXiv: 2403.17749

[72] Agiza A, Neseem M, Reda S. MTLoRA: low-rank adaptation approach for efficient multi-task learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2024, 16196−16205

[73] Gao C, Chen K, Rao J, Sun B, Liu R, Peng D, Zhang Y, Guo X, Yang J, Subrahmanian V S. Higher layers need more LoRA experts. 2024, arXiv preprint arXiv: 2402.08562

[74] Chen S, Jie Z, Ma L. LLaVA-MoLE: sparse mixture of LoRA experts for mitigating data conflicts in instruction finetuning MLLMs. 2024, arXiv preprint arXiv: 2401.16160

[75] Zhu Y, Wichers N, Lin C C, Wang X, Chen T, Shu L, Lu H, Liu C, Luo L, Chen J, Meng L. SiRA: sparse mixture of low rank adaptation. 2023, arXiv preprint arXiv: 2311.09179

[76] Chen Z, Wang Z, Wang Z, Liu H, Yin Z, Liu S, Sheng L, Ouyang W, Qiao Y, Shao J. Octavius: mitigating task interference in MLLMs via MoE. 2023, arXiv preprint arXiv: 2311.02684

[77] Wen Y, Chaudhuri S. Batched low-rank adaptation of foundation models. In: Proceedings of the Twelfth International Conference on Learning Representations. 2024

[78] Wu T, Wang J, Zhao Z, Wong N. Mixture-of-Subspaces in Low-Rank Adaptation. 2024, arXiv preprint arXiv:2406.11909

[79] Wu Y, Xiang Y, Huo S, Gong Y, Liang P. LoRA-SP: streamlined partial parameter adaptation for resource efficient fine-tuning of large language models. In: Proceedings of the 3rd International Conference on Algorithms, Microchips, and Network Applications. 2024, 131711Z

[80] Zhang L, Zhang L, Shi S, Chu X, Li B. LoRA-FA: memory-efficient low-rank adaptation for large language models fine-tuning. 2023, arXiv preprint arXiv: 2308.03303

[81] Liu Z, Kundu S, Li A, Wan J, Jiang L, Beerel P A. AFLoRA: adaptive freezing of low rank adaptation in parameter efficient fine-tuning of large models. 2024, arXiv preprint arXiv: 2403.13269

[82] Woo S, Park B, Kim B, Jo M, Kwon S, Jeon D, Lee D. DropBP: accelerating fine-tuning of large language models by dropping backward propagation. 2024, arXiv preprint arXiv: 2402.17812

[83] Bałazy K, Banaei M, Aberer K, Tabor J. LoRA-XS: low-rank adaptation with extremely small number of parameters. 2024, arXiv preprint arXiv: 2405.17604

[84] Zhou H, Lu X, Xu W, Zhu C, Zhao T, Yang M. LoRA-drop: efficient LoRA parameter pruning based on output evaluation. 2024, arXiv preprint arXiv: 2402.07721

[85] Zhang M, Chen H, Shen C, Yang Z, Ou L, Yu X, Zhuang B. LoRAPrune: structured pruning meets low-rank parameter-efficient fine-tuning. In: Proceedings of the Findings of the Association for Computational Linguistics. 2024, 3013−3026

[86] Chen T, Ding T, Yadav B, Zharkov I, Liang L. LoRAShear: efficient large language model structured pruning and knowledge recovery. 2023, arXiv preprint arXiv: 2310.18356

[87] Zhu Y, Yang X, Wu Y, Zhang W. Parameter-efficient fine-tuning with layer pruning on free-text sequence-to-sequence modeling. 2023, arXiv preprint arXiv: 2305.08285

[88] Kopiczko D J, Blankevoort T, Asano Y M. VeRA: vector-based random matrix adaptation. In: Proceedings of the 12th International Conference on Learning Representations. 2024

[89] Li Y, Han S, Ji S. VB-LoRA: extreme parameter efficient fine-tuning with vector banks. 2024, arXiv preprint arXiv: 2405.15179

[90] Gao Z, Wang Q, Chen A, Liu Z, Wu B, Chen L, Li J. Parameter-efficient fine-tuning with discrete Fourier transform. 2024, arXiv preprint arXiv: 2405.03003

[91] Dettmers T, Pagnoni A, Holtzman A, Zettlemoyer L. QLORA: efficient finetuning of quantized LLMs. In: Proceedings of the 37th International Conference on Neural Information Processing Systems. 2023

[92] Xu Y, Xie L, Gu X, Chen X, Chang H, Zhang H, Chen Z, Zhang X, Tian Q. QA-LoRA: quantization-aware low-rank adaptation of large language models. In: Proceedings of the 12th International Conference on Learning Representations. 2024

[93] Li Y, Yu Y, Liang C, He P, Karampatziakis N, Chen W, Zhao T. LoftQ: LoRA-fine-tuning-aware quantization for large language models. In: Proceedings of the 12th International Conference on Learning Representations. 2024

[94] Liao B, Herold C, Khadivi S, Monz C. ApiQ: finetuning of 2-bit quantized large language model. 2024, arXiv preprint arXiv: 2402.05147

[95] Jeon H, Kim Y, Kim J J. L4Q: parameter efficient quantization-aware training on large language models via LoRA-wise LSQ. 2024, arXiv preprint arXiv: 2402.04902

[96] Ye Z, Li D, Tian J, Lan T, Zuo J, Duan L, Lu H, Jiang Y, Sha J, Zhang K, Tang M. ASPEN: high-throughput LoRA fine-tuning of large language models with a single GPU. 2023, arXiv preprint arXiv: 2312.02515

[97] Chen L, Ye Z, Wu Y, Zhuo D, Ceze L, Krishnamurthy A. Punica: multi-tenant LoRA serving. In: Proceedings of the Seventh Annual Conference on Machine Learning and Systems. 2024, 1−13

[98] Sheng Y, Cao S, Li D, Hooper C, Lee N, Yang S, Chou C, Zhu B, Zheng L, Keutzer K, Gonzalez J E, Stoica I. S-LoRA: serving thousands of concurrent LoRA adapters. 2023, arXiv preprint arXiv: 2311.03285

[99] Li S, Lu H, Wu T, Yu M, Weng Q, Chen X, Shan Y, Yuan B, Wang W. CaraServe: CPU-assisted and rank-aware LoRA serving for generative LLM inference. 2024, arXiv preprint arXiv: 2401.11240

[100] Babakniya S, Elkordy A R, Ezzeldin Y H, Liu Q, Song K B, El-Khamy M, Avestimehr S. SLoRA: federated parameter efficient fine-tuning of language models. 2023, arXiv preprint arXiv: 2308.06522

[101] Yan Y, Tang S, Shi Z, Yang Q. FeDeRA: efficient fine-tuning of language models in federated learning leveraging weight decomposition. 2024, arXiv preprint arXiv: 2404.18848

[102] Sun Y, Li Z, Li Y, Ding B. Improving LoRA in privacy-preserving federated learning. In: Proceedings of the 12th International Conference on Learning Representations. 2024

[103] Wu P, Li K, Wang T, Wang F. FedMS: federated learning with mixture of sparsely activated foundations models. 2023, arXiv preprint arXiv: 2312.15926

[104] Bai J, Chen D, Qian B, Yao L, Li Y. Federated fine-tuning of large language models under heterogeneous language tasks and client resources. 2024, arXiv preprint arXiv: 2402.11505

[105] Cho Y J, Liu L, Xu Z, Fahrezi A, Barnes M, Joshi G. Heterogeneous LoRA for federated fine-tuning of on-device foundation models. In: Proceedings of the International Workshop on Federated Learning in the Age of Foundation Models in Conjunction with NeurIPS. 2023

[106] Yi L, Yu H, Wang G, Liu X, Li X. pFedLoRA: model-heterogeneous personalized federated learning with LoRA tuning. 2023, arXiv preprint arXiv: 2310.13283

[107] Huang W, Wang Y, Cheng A, Zhou A, Yu C, Wang L. A fast, performant, secure distributed training framework for large language model. 2024, arXiv preprint arXiv: 2401.09796

[108] Wang Y, Lin Y, Zeng X, Zhang G. PrivateLoRA for efficient privacy preserving LLM. 2023, arXiv preprint arXiv: 2311.14030

[109] Zhang Y, Wang M, Wu Y, Tiwari P, Li Q, Wang B, Qin J. DialogueLLM: context and emotion knowledge-tuned large language models for emotion recognition in conversations. 2024, arXiv preprint arXiv: 2310.11374

[110] Li Z, Li X, Liu Y, Xie H, Li J, Wang F L, Li Q, Zhong X. Label supervised LLaMA finetuning. 2023, arXiv preprint arXiv: 2310.01208

[111] Bornheim T, Grieger N, Blaneck P G, Bialonski S. Speaker attribution in German parliamentary debates with QLoRA-adapted large language models. 2024, arXiv preprint arXiv: 2309.09902

[112] Xue L, Zhang D, Dong Y, Tang J. AutoRE: document-level relation extraction with large language models. 2024, arXiv preprint arXiv: 2403.14888

[113] Alves D M, Guerreiro N M, Alves J, Pombal J, Rei R, de Souza J G C, Colombo P, Martins A F T. Steering large language models for machine translation with finetuning and in-context learning. In: Proceedings of the Findings of the Association for Computational Linguistics. 2023, 11127−11148

[114] Zheng J, Hong H, Wang X, Su J, Liang Y, Wu S. Fine-tuning large language models for domain-specific machine translation. 2024, arXiv preprint arXiv: 2402.15061

[115] Mujadia V, Urlana A, Bhaskar Y, Pavani P A, Shravya K, Krishnamurthy P, Sharma D M. Assessing translation capabilities of large language models involving English and Indian languages. 2023, arXiv preprint arXiv: 2311.09216

[116] Zhang Y, Wang J, Yu L C, Xu D, Zhang X. Personalized LoRA for human-centered text understanding. In: Proceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence. 2024, 19588−19596

[117] Liu Y, An C, Qiu X . Y-tuning: an efficient tuning paradigm for large-scale pre-trained models via label representation learning. Frontiers of Computer Science, 2024, 18( 4): 184320

[118] Liu S, Keung J, Yang Z, Liu F, Zhou Q, Liao Y. Delving into parameter-efficient fine-tuning in code change learning: an empirical study. In: Proceedings of the IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). 2024, 465−476

[119] Guo Y, Gao X, Jiang B. An empirical study on JIT defect prediction based on BERT-style model. 2024, arXiv preprint arXiv: 2403.11158

[120] Ayupov S, Chirkova N. Parameter-efficient finetuning of transformers for source code. 2022, arXiv preprint arXiv: 2212.05901

[121] Silva A, Fang S, Monperrus M. RepairLLaMA: efficient representations and fine-tuned adapters for program repair. 2023, arXiv preprint arXiv: 2312.15698

[122] Roberson R, Kaki G, Trivedi A. Analyzing the effectiveness of large language models on text-to-SQL synthesis. 2024, arXiv preprint arXiv: 2401.12379

[123] Pan J, Sadé A, Kim J, Soriano E, Sole G, Flamant S. SteloCoder: a decoder-only LLM for multi-language to python code translation. 2023, arXiv preprint arXiv: 2310.15539

[124] Sidahmed H, Phatale S, Hutcheson A, Lin Z, Chen Z, Yu Z, Jin J, Komarytsia R, Ahlheim C, Zhu Y, Chaudhary S, Li B, Ganesh S, Byrne B, Hoffmann J, Mansoor H, Li W, Rastogi A, Dixon L. PERL: parameter efficient reinforcement learning from human feedback. 2024, arXiv preprint arXiv: 2403.10704

[125] Santacroce M, Lu Y, Yu H, Li Y, Shen Y. Efficient RLHF: reducing the memory usage of PPO. 2023, arXiv preprint arXiv: 2309.00754

[126] Sun S, Gupta D, Iyyer M. Exploring the impact of low-rank adaptation on the performance, efficiency, and regularization of RLHF. 2023, arXiv preprint arXiv: 2309.09055

[127] Quan S. DMoERM: recipes of mixture-of-experts for effective reward modeling. 2024, arXiv preprint arXiv: 2403.01197

[128] Zhang S, Chen Z, Chen S, Shen Y, Sun Z, Gan C. Improving reinforcement learning from human feedback with efficient reward model ensemble. 2024, arXiv preprint arXiv: 2401.16635

[129] Zhai Y, Zhang H, Lei Y, Yu Y, Xu K, Feng D, Ding B, Wang H. Uncertainty-penalized reinforcement learning from human feedback with diverse reward LoRA ensembles. 2023, arXiv preprint arXiv: 2401.00243

[130] Yang A X, Robeyns M, Coste T, Shi Z, Wang J, Bou-Ammar H, Aitchison L. Bayesian reward models for LLM alignment. 2024, arXiv preprint arXiv: 2402.13210

[131] Daxberger E, Kristiadi A, Immer A, Eschenhagen R, Bauer M, Hennig P. Laplace redux-effortless bayesian deep learning. Advances in Neural Information Processing Systems. 2021

[132] Tran H, Yang Z, Yao Z, Yu H. BioInstruct: instruction tuning of large language models for biomedical natural language processing. 2023, arXiv preprint arXiv: 2310.19975

[133] Gema A P, Minervini P, Daines L, Hope T, Alex B. Parameter-efficient fine-tuning of LLaMA for the clinical domain. 2023, arXiv preprint arXiv: 2307.03042

[134] Toma A, Lawler P R, Ba J, Krishnan R G, Rubin B B, Wang B. Clinical camel: an open-source expert-level medical language model with dialogue-based knowledge encoding. 2023, arXiv preprint arXiv: 2305.12031

[135] Suri K, Mishra P, Saha S, Singh A. Suryakiran at MEDIQA-Sum 2023: leveraging LoRA for clinical dialogue summarization. In: Proceedings of the Working Notes of the Conference and Labs of the Evaluation Forum. 2023, 1720−1735

[136] Ji Y, Yu Z, Wang Y. Assertion detection large language model in-context learning LoRA fine-tuning. 2024, arXiv preprint arXiv: 2401.17602

[137] Wang R, Duan Y, Lam C, Chen J, Xu J, Chen H, Liu X, Pang P C I, Tan T. IvyGPT: InteractiVe Chinese pathway language model in medical domain. In: Proceedings of the 3rd CAAI International Conference on Artificial Intelligence. 2024, 378−382

[138] Bhatti A, Parmar S, Lee S. SM70: a large language model for medical devices. 2023, arXiv preprint arXiv: 2312.06974

[139] Konstantinidis T, Iacovides G, Xu M, Constantinides T G, Mandic D. FinLlama: financial sentiment classification for algorithmic trading applications. 2024, arXiv preprint arXiv: 2403.12285

[140] Pavlyshenko B M. Financial news analytics using fine-tuned llama 2 GPT model. 2023, arXiv preprint arXiv: 2308.13032

[141] Liu X Y, Wang G, Yang H, Zha D. FinGPT: democratizing internet-scale data for financial large language models. 2023, arXiv preprint arXiv: 2307.10485

[142] Li J, Lei Y, Bian Y, Cheng D, Ding Z, Jiang C . RA-CFGPT: Chinese financial assistant with retrieval-augmented large language model. Frontiers of Computer Science, 2024, 18( 5): 185350

[143] Zhou X, Sun Z, Li G . DB-GPT: large language model meets database. Data Science and Engineering, 2024, 9( 1): 102–111

[144] Li S. DiffStyler: diffusion-based localized image style transfer. 2024, arXiv preprint arXiv: 2403.18461

[145] Frenkel Y, Vinker Y, Shamir A, Cohen-Or D. Implicit style-content separation using B-LoRA. 2024, arXiv preprint arXiv: 2403.14572

[146] Liu Y, Yu C, Shang L, He Y, Wu Z, Wang X, Xu C, Xie H, Wang W, Zhao Y, Zhu L, Cheng C, Chen W, Yao Y, Zhou W, Xu J, Wang Q, Chen Y, Xie X, Sun B. FaceChain: a playground for human-centric artificial intelligence generated content. 2023, arXiv preprint arXiv: 2308.14256

[147] Liao Q, Xia G, Wang Z. Calliffusion: Chinese calligraphy generation and style transfer with diffusion modeling. 2023, arXiv preprint arXiv: 2305.19124

[148] Shrestha S, Sripada V S S, Venkataramanan A. Style transfer to Calvin and Hobbes comics using stable diffusion. 2023, arXiv preprint arXiv: 2312.03993

[149] Li L, Zeng H, Yang C, Jia H, Xu D. Block-wise LoRA: revisiting fine-grained LoRA for effective personalization and stylization in text-to-image generation. 2024, arXiv preprint arXiv: 2403.07500

[150] Kong Z, Zhang Y, Yang T, Wang T, Zhang K, Wu B, Chen G, Liu W, Luo W. OMG: occlusion-friendly personalized multi-concept generation in diffusion models. 2024, arXiv preprint arXiv: 2403.10983

[151] Shi J, Hua H. Space narrative: generating images and 3D scenes of Chinese garden from text using deep learning. In: Proceedings of the xArch-Creativity in the Age of Digital Reproduction Symposium. 2024, 236−243

[152] Jin Z, Song Z. Generating coherent comic with rich story using ChatGPT and stable diffusion. 2023, arXiv preprint arXiv: 2305.11067

[153] Wang H, Xiang X, Fan Y, Xue J H. Customizing 360-degree panoramas through text-to-image diffusion models. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2024, 4921−4931

[154] Guo J, Xu X, Pu Y, Ni Z, Wang C, Vasu M, Song S, Huang G, Shi H. Smooth diffusion: crafting smooth latent spaces in diffusion models. 2023, arXiv preprint arXiv: 2312.04410

[155] Cheng J, Xie P, Xia X, Li J, Wu J, Ren Y, Li H, Xiao X, Zheng M, Fu L. ResAdapter: domain consistent resolution adapter for diffusion models. 2024, arXiv preprint arXiv: 2403.02084

[156] Smith J S, Hsu Y C, Kira Z, Shen Y, Jin H. Continual diffusion with STAMINA: STack-and-mask INcremental adapters. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024, 1744−1754

[157] Sun J, Fu D, Hu Y, Wang S, Rassin R, Juan D C, Alon D, Herrmann C, van Steenkiste S, Krishna R, Rashtchian C. Dreamsync: aligning text-to-image generation with image understanding feedback. In: Proceedings of the Synthetic Data for Computer Vision Workshop@ CVPR 2024. 2023

[158] Wang Z, Wang X, Xie L, Qi Z, Shan Y, Wang W, Luo P. StyleAdapter: a single-pass LoRA-free model for stylized image generation. 2023, arXiv preprint arXiv: 2309.01770

[159] Gu Y, Wang X, Wu J Z, Shi Y, Chen Y, Fan Z, Xiao W, Zhao R, Chang S, Wu W, Ge Y, Shan Y, Shou M Z. Mix-of-show: decentralized low-rank adaptation for multi-concept customization of diffusion models. In: Proceedings of the 37th International Conference on Neural Information Processing Systems. 2023

[160] Luo S, Tan Y, Patil S, Gu D, von Platen P, Passos A, Huang L, Li J, Zhao H. LCM-LoRA: a universal stable-diffusion acceleration module. 2023, arXiv preprint arXiv: 2311.05556

[161] Golnari P A. LoRA-enhanced distillation on guided diffusion models. 2023, arXiv preprint arXiv: 2312.06899

[162] Ren Y, Zhou Y, Yang J, Shi J, Liu D, Liu F, Kwon M, Shrivastava A. Customize-A-video: one-shot motion customization of text-to-video diffusion models. 2024, arXiv preprint arXiv: 2402.14780

[163] Deng Y, Wang R, Zhang Y, Tai Y W, Tang C K. DragVideo: interactive drag-style video editing. 2023, arXiv preprint arXiv: 2312.02216

[164] Yang S, Zhou Y, Liu Z, Loy C C. Rerender A video: zero-shot text-guided video-to-video translation. In: Proceedings of the SIGGRAPH Asia 2023 Conference Papers. 2023, 95

[165] Khandelwal A. InFusion: inject and attention fusion for multi concept zero-shot text-based video editing. In: Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops. 2023, 3009−3018

[166] Blattmann A, Dockhorn T, Kulal S, Mendelevitch D, Kilian M, Lorenz D, Levi Y, English Z, Voleti V, Letts A, Jampani V, Rombach R. Stable video diffusion: scaling latent video diffusion models to large datasets. 2023, arXiv preprint arXiv: 2311.15127

[167] Guo Y, Yang C, Rao A, Liang Z, Wang Y, Qiao Y, Agrawala M, Lin D, Dai B. AnimateDiff: animate your personalized text-to-image diffusion models without specific tuning. In: Proceedings of the 12th International Conference on Learning Representations. 2024

[168] Huang T, Zeng Y, Zhang Z, Xu W, Xu H, Xu S, Lau R W H, Zuo W. DreamControl: control-based text-to-3D generation with 3D self-prior. 2023, arXiv preprint arXiv: 2312.06439

[169] Ma Y, Fan Y, Ji J, Wang H, Sun X, Jiang G, Shu A, Ji R. X-dreamer: creating high-quality 3D content by bridging the domain gap between text-to-2D and text-to-3D generation. 2023, arXiv preprint arXiv: 2312.00085

[170] Yu K, Liu J, Feng M, Cui M, Xie X. Boosting3D: high-fidelity image-to-3D by boosting 2D diffusion prior to 3D prior with progressive learning. 2023, arXiv preprint arXiv: 2311.13617

[171] Yoo S, Kim K, Kim V G, Sung M. As-plausible-as-possible: plausibility-aware mesh deformation using 2D diffusion priors. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024, 4315−4324

[172] Zhang Y, Xu Q, Zhang L. DragTex: generative point-based texture editing on 3D mesh. 2024, arXiv preprint arXiv: 2403.02217

[173] Ding H, Gao J, Yuan Y, Wang Q. SamLP: a customized segment anything model for license plate detection. 2024, arXiv preprint arXiv: 2401.06374

[174] Ye Z, Lovell L, Faramarzi A, Ninic J. SAM-based instance segmentation models for the automation of structural damage detection. 2024, arXiv preprint arXiv: 2401.15266

[175] Na S, Guo Y, Jiang F, Ma H, Huang J. Segment any cell: a SAM-based auto-prompting fine-tuning framework for nuclei segmentation. 2024, arXiv preprint arXiv: 2401.13220

[176] Chen X, Wang C, Ning H, Li S, Shen M. SAM-OCTA: prompting segment-anything for OCTA image segmentation. 2023, arXiv preprint arXiv: 2310.07183

[177] Feng W, Zhu L, Yu L. Cheap lunch for medical image segmentation by fine-tuning SAM on few exemplars. 2023, arXiv preprint arXiv: 2308.14133

[178] Zhang K, Liu D. Customized segment anything model for medical image segmentation. 2023, arXiv preprint arXiv: 2304.13785

[179] Wang A, Islam M, Xu M, Zhang Y, Ren H. SAM meets robotic surgery: an empirical study on generalization, robustness and adaptation. In: Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention. 2023, 234−244

[180] Lin L, Fan H, Zhang Z, Wang Y, Xu Y, Ling H. Tracking meets LoRA: faster training, larger model, stronger performance. 2024, arXiv preprint arXiv: 2403.05231

[181] Kong C, Li H, Wang S. Enhancing general face forgery detection via vision transformer with low-rank adaptation. In: Proceedings of the 6th International Conference on Multimedia Information Processing and Retrieval. 2023, 102−107

[182] Chen Z, Huang H, Andrusenko A, Hrinchuk O, Puvvada K C, Li J, Ghosh S, Balam J, Ginsburg B. SALM: speech-augmented language model with in-context learning for speech recognition and translation. In: Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2024, 13521−13525

[183] Dong X, Zhang P, Zang Y, Cao Y, Wang B, Ouyang L, Wei X, Zhang S, Duan H, Cao M, Zhang W, Li Y, Yan H, Gao Y, Zhang X, Li W, Li J, Chen K, He C, Zhang X, Qiao Y, Lin D, Wang J. InternLM-XComposer2: mastering free-form text-image composition and comprehension in vision-language large model. 2024, arXiv preprint arXiv: 2401.16420

[184] Ye Q, Xu H, Xu G, Ye J, Yan M, Zhou Y, Wang J, Hu A, Shi P, Shi Y, Li C, Xu Y, Chen H, Tian J, Qian Q, Zhang J, Huang F, Zhou J. mPLUG-Owl: modularization empowers large language models with multimodality. 2023, arXiv preprint arXiv: 2304.14178

[185] Lee B K, Park B, Kim C W, Ro Y M. CoLLaVO: crayon large language and vision mOdel. 2024, arXiv preprint arXiv: 2402.11248

[186] Yeo J H, Han S, Kim M, Ro Y M. Where visual speech meets language: VSP-LLM framework for efficient and context-aware visual speech processing. 2024, arXiv preprint arXiv: 2402.15151

[187] Liu Z, Li S, Luo Y, Fei H, Cao Y, Kawaguchi K, Wang X, Chua T S. MolCA: molecular graph-language modeling with cross-modal projector and uni-modal adapter. In: Proceedings of 2023 Conference on Empirical Methods in Natural Language Processing. 2023, 15623−15638

[188] Ren Y, Chen Y, Liu S, Wang B, Yu H, Cui Z. TPLLM: a traffic prediction framework based on pretrained large language models. 2024, arXiv preprint arXiv: 2403.02221

[189] Aghajanyan A, Gupta S, Zettlemoyer L. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing. 2021, 7319−7328

[190] Fomenko V, Yu H, Lee J, Hsieh S, Chen W. A note on LoRA. 2024, arXiv preprint arXiv: 2404.05086

[191] Bershatsky D, Cherniuk D, Daulbaev T, Mikhalev A, Oseledets I. LoTR: low tensor rank weight adaptation. 2024, arXiv preprint arXiv: 2402.01376

[192] Edalati A, Tahaei M, Kobyzev I, Nia V P, Clark J J, Rezagholizadeh M. KronA: parameter efficient tuning with kronecker adapter. 2022, arXiv preprint arXiv: 2212.10650

[193] He X, Li C, Zhang P, Yang J, Wang X E. Parameter-efficient model adaptation for vision transformers. In: Proceedings of the 37th AAAI Conference on Artificial Intelligence. 2023, 817−825

[194] Zhao Z, Gan L, Wang G, Hu Y, Shen T, Yang H, Kuang K, Wu F. Retrieval-augmented mixture of lora experts for uploadable machine learning. 2024 , arXiv preprint arXiv:2406.16989.

[195] Mahabadi R K, Henderson J, Ruder S. COMPACTER: efficient low-rank hypercomplex adapter layers. In: Proceedings of the 35th International Conference on Neural Information Processing Systems. 2021, 79

[196] Liao B, Meng Y, Monz C. Parameter-efficient fine-tuning without introducing new latency. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics. 2023, 4242−4260

[197] Hendrycks D, Burns C, Basart S, Zou A, Mazeika M, Song D, Steinhardt J. Measuring massive multitask language understanding. In: Proceedings of the 9th International Conference on Learning Representations. 2021

[198] He J, Zhou C, Ma X, Berg-Kirkpatrick T, Neubig G. Towards a unified view of parameter-efficient transfer learning. In: Proceedings of the 10th International Conference on Learning Representations. 2022

[199] Geshkovski B, Letrouit C, Polyanskiy Y, Rigollet P. A mathematical perspective on transformers. 2023, arXiv preprint arXiv: 2312.10794

[200] Geshkovski B, Letrouit C, Polyanskiy Y, Rigollet P. The emergence of clusters in self-attention dynamics. In: Proceedings of the 37th International Conference on Neural Information Processing Systems. 2023

[201] Sander M E, Ablin P, Blondel M, Peyré G. Sinkformers: transformers with doubly stochastic attention. In: Proceedings of the 25th International Conference on Artificial Intelligence and Statistics. 2022, 3515−3530

[202] Jacot A, Gabriel F, Hongler C. Neural tangent kernel: convergence and generalization in neural networks. In: Proceedings of the 32nd International Conference on Neural Information Processing Systems. 2018, 8580−8589

[203] Touvron H, Martin L, Stone K, Albert P, Almahairi A, Babaei Y, Bashlykov N, Batra S, Bhargava P, Bhosale S, Bikel D, Blecher L, Canton Ferrer C, Chen M, Cucurull G, Esiobu D, Fernandes J, Fu J, Fu W, Fuller B, Gao C, Goswami V, Goyal N, Hartshorn A, Hosseini S, Hou R, Inan H, Kardas M, Kerkez V, Khabsa M, Kloumann I, Korenev A, Koura P S, Lachaux M A, Lavril T, Lee J, Liskovich D, Lu Y, Mao Y, Martinet X, Mihaylov T, Mishra P, Molybog I, Nie Y, Poulton A, Reizenstein J, Rungta R, Saladi K, Schelten A, Silva R, Smith E M, Subramanian R, Tan X E, Tang B, Taylor R, Williams A, Kuan J X, Xu P, Yan Z, Zarov I, Zhang Y, Fan A, Kambadur M, Narang S, Rodriguez A, Stojnic R, Edunov S, Scialom T. Llama 2: open foundation and fine-tuned chat models. 2023, arXiv preprint arXiv: 2307.09288

[204] Chang Y, Chang Y, Wu Y. Bias-Aware Low-Rank Adaptation: Mitigating Catastrophic Inheritance of Large Language Models. 2024 , arXiv preprint arXiv:2408.04556

[205] Zhao J, Zhang Z, Chen B, Wang Z, Anandkumar A, Tian Y. Galore: memory-efficient LLM training by gradient low-rank projection. 2024, arXiv preprint arXiv: 2403.03507

[206] Biderman D, Ortiz J G, Portes J, Paul M, Greengard P, Jennings C, King D, Havens S, Chiley V, Frankle J, Blakeney C, Cunningham J P. LoRA learns less and forgets less. 2024, arXiv preprint arXiv: 2405.09673

[207] Han A, Li J, Huang W, Hong M, Takeda A, Jawanpuria P, Mishra B. SLTrain: a sparse plus low-rank approach for parameter and memory efficient pretraining. 2024, arXiv preprint arXiv: 2406.02214

[208] Sui Y, Yin M, Gong Y, Xiao J, Phan H, Yuan B. ELRT: efficient low-rank training for compact convolutional neural networks. 2024, arXiv preprint arXiv: 2401.10341

[209] Meng X, Dai D, Luo W, Yang Z, Wu S, Wang X, Wang P, Dong Q, Chen L, Sui Z. PeriodicLoRA: breaking the low-rank bottleneck in LoRA optimization. 2024, arXiv preprint arXiv: 2402.16141

[210] Frank M, Wolfe P . An algorithm for quadratic programming. Naval Research Logistics Quarterly, 1956, 3( 1-2): 95–110

[211] Rajabzadeh H, Valipour M, Zhu T, Tahaei M, Kwon HJ, Ghodsi A, Chen B, Rezagholizadeh M. Qdylora: Quantized dynamic low-rank adaptation for efficient large language model tuning. 2024 , arXiv preprint arXiv:2402.10462

[212] Elsken T, Metzen J H, Hutter F . Neural architecture search: a survey. The Journal of Machine Learning Research, 2019, 20( 1): 1997–2017

[213] Liu Y, Ott M, Goyal N, Du J, Joshi M, Chen D, Levy O, Lewis M, Zettlemoyer L, Stoyanov V. RoBERTa: a robustly optimized BERT pretraining approach. 2019, arXiv preprint arXiv: 1907.11692

[214] Wang A, Singh A, Michael J, Hill F, Levy O, Bowman S R. GLUE: a multi-task benchmark and analysis platform for natural language understanding. In: Proceedings of 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP. 2018, 353−355

[215] Renduchintala A, Konuk T, Kuchaiev O. Tied-LoRA: enhancing parameter efficiency of LoRA with weight tying. In: Proceedings of 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2024, 8694−8705

[216] Hansen N, Ostermeier A. Adapting arbitrary normal mutation distributions in evolution strategies: the covariance matrix adaptation. In: Proceedings of the IEEE International Conference on Evolutionary Computation. 1996, 312−317

[217] Ye M, Fang X, Du B, Yuen P C, Tao D . Heterogeneous federated learning: state-of-the-art and research challenges. ACM Computing Surveys, 2024, 56( 3): 79

[218] Liu X Y, Zhu R, Zha D, Gao J, Zhong S, White M, Qiu M. Differentially private low-rank adaptation of large language model using federated learning. 2023, arXiv preprint arXiv: 2312.17493

RIGHTS & PERMISSIONS

The Author(s) 2024. This article is published with open access at link.springer.com and journal.hep.com.cn