In recent years, prompt learning has shown promise in transferring pretrained vision-language models (VLMs) to downstream tasks. However, existing methods face two challenges in improving generalisation: (1) When leveraging the collaborative effect of multimodal prompts, it is often assumed that text and visual modalities share the same prompt requirements, neglecting the distinct hierarchical processing of their encoders, leading to prompt imbalance; and (2) current methods exhibit limited adaptability when facing diverse distribution shift scenarios, including class distribution shifts and image content variations. To address these challenges, we propose a diversified composite prompt learning (DCPL) framework that integrates unified and specific prompts. Specifically, to alleviate multimodal prompt imbalance, we design a shared root multimodal prompting strategy, which employs a shared root prompt and an independent derivation mechanism to generate the derived multimodal prompt (DMP), enabling independent deep prompting while maintaining implicit synergy across modalities. Furthermore, we design a dual-branch dynamic adaptive prompting strategy that produces the derived class-specific prompt (DCP) and image-specific prompt (ISP), driven by inter-class relations and image-patch context, respectively, to enhance adaptability across different distribution shifts. Extensive experiments on base-to-novel, cross-dataset, domain generalisation and few-shot learning demonstrate that the DCPL achieves superior performance, validating its robustness and generalisation.
Acknowledgements
This work was supported in part by the Research Project on Ideological and Political Work under the Philosophy and Social Science Planning Programme of Zhejiang Province (No. 25GXSZ009YB) and in part by the National Key Research and Development Programme of China (No. 2022YFC3303600).
Conflicts of Interest
The authors declare no conflicts of interest.
Data Availability Statement
The data used in this paper are obtained from publicly available datasets.
| [1] |
A. Radford, J. W. Kim, C. Hallacy, et al., “Learning Transferable Visual Models From Natural Language Supervision,” in International Conference on Machine Learning, (2021), 8748-8763.
|
| [2] |
K. Zhou, J. Yang, C. C. Loy, and Z. Liu, “Learning to Prompt for Vision-Language Models,” International Journal of Computer Vision 130, no. 9 (2021): 2337-2348.
|
| [3] |
M. U. Khattak, H. A. Rasheed, M. Maaz, S. H. Khan, and F. S. Khan, “Maple: Multi-Modal Prompt Learning,” in 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), (2023), 19113-19122.
|
| [4] |
H. Yao, R. Zhang, and C. Xu, “Tcp: Textual-Based Class-Aware Prompt Tuning for Visual-Language Model,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, (2024), 23438-23448.
|
| [5] |
K. Zhou, J. Yang, C. C. Loy, and Z. Liu, “Conditional Prompt Learning for Vision-Language Models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, (2022), 16816-16825.
|
| [6] |
C. Jia, Y. Yang, Y. Xia, et al., Scaling up Visual and Vision-Language Representation Learning With Noisy Text Supervision,” in International Conference on Machine Learning, (2021), 4904-4916.
|
| [7] |
H. Tang, S. He, and J. Qin, “Connecting Giants: Synergistic Knowledge Transfer of Large Multimodal Models for Few-Shot Learning,” arXiv preprint arXiv:2510.11115 (2025).
|
| [8] |
T. Lüddecke and A. Ecker, “Image Segmentation Using Text and Image Prompts,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, (2022), 7086-7096.
|
| [9] |
K. A. Nguyen, A. Juvekar, T. Yu, M. Wahed, and I. Lourentzou, “Calico: Part-Focused Semantic Co-Segmentation With Large Vision-Language Models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), (June 2025), 4550-4561.
|
| [10] |
X. Gu, T.-Y. Lin, W. Kuo, and Y. Cui, “Open-Vocabulary Object Detection via Vision and Language Knowledge Distillation,” arXiv preprint arXiv:2104.13921 (2021): https://doi.org/10.48550/arXiv.2104.13921.
|
| [11] |
S. Jin, X. Jiang, J. Huang, L. Lu, and S. Lu, “Llms Meet Vlms: Boost Open Vocabulary Object Detection With Fine-Grained Descriptors,” International Conference on Representation Learning 2024 (2024): 8734-8745.
|
| [12] |
Y. Chen, S. Shi, and H. Huang, “Which Is More Faithful, Seeing or Saying? Multimodal Sarcasm Detection Exploiting Contrasting Sentiment Knowledge,” CAAI Transactions on Intelligence Technology 10, no. 2 (2025): 375-386.
|
| [13] |
H. Tang, Y. Liu, S. Yan, F. Shen, S. He, and J. Qin, “Cross-Modal Proxy Evolving for Ood Detection With Vision-Language Models,” arXiv preprint arXiv:2601.08476 (2026).
|
| [14] |
P. Gao, S. Geng, R. Zhang, et al., Clip-Adapter: Better Vision-Language Models With Feature Adapters,” ArXiv, vol.abs/2110.04544 (2021): https://doi.org/10.48550/arXiv.2110.04544.
|
| [15] |
A. Vaswani, N. Shazeer, N. Parmar, et al. , “Attention Is all You Need,” Advances in Neural Information Processing Systems 30 (2017): https://dl.acm.org/doi/10.5555/3295222.3295349.
|
| [16] |
A. Dosovitskiy, “An Image Is Worth 16x16 Words: Transformers for Image Recognition at Scale,” arXiv preprint arXiv:2010.11929 (2020).
|
| [17] |
Z. Liu, Y. Lin, Y. Cao, et al., Swin Transformer: Hierarchical Vision Transformer Using Shifted Windows,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, (2021), 10012-10022.
|
| [18] |
M.-H. Guo, T.-X. Xu, J.-J. Liu, et al., Attention Mechanisms in Computer Vision: A Survey,” Computational Visual Media 8, no. 3 (2022): 331-368.
|
| [19] |
Z. Liu, Y. Liu, and Y. Fang, “Diffusion Model-Based Path Follower for a Salamander-Like Robot,” IEEE Transactions on Neural Networks and Learning Systems (2025): https://doi.org/10.1109/TNNLS.2025.3549307.
|
| [20] |
S. Luo, S. Li, T. Cai, et al., Stable, Fast and Accurate: Kernelized Attention With Relative Positional Encoding,” Advances in Neural Information Processing Systems 34 (2021): 22795-22807.
|
| [21] |
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-Training of Deep Bidirectional Transformers for Language Understanding,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, (2019), 4171-4186.
|
| [22] |
T. Brown, B. Mann, N. Ryder, et al., Language Models Are Few-Shot Learners,” Advances in Neural Information Processing Systems 33 (2020): 1877-1901.
|
| [23] |
X. Liu, K. Ji, Y. Fu, Z. Du, Z. Yang, and J. Tang, “P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-Tuning Universally Across Scales and Tasks,” ArXiv abs/2110.07602 (2021): https://doi.org/10.48550/arXiv.2110.07602.
|
| [24] |
X. L. Li and P. Liang, “Prefix-Tuning: Optimizing Continuous Prompts for Generation,” arXiv preprint arXiv:2101.00190 (2021).
|
| [25] |
B. Lester, R. Al-Rfou, and N. Constant, “The Power of Scale for Parameter-Efficient Prompt Tuning,” arXiv preprint arXiv:2104.08691 (2021).
|
| [26] |
E. Cho, J. Kim, and H. J. Kim, “Distribution-Aware Prompt Tuning for Vision-Language Models,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, (2023), 22004-22013.
|
| [27] |
B. Zhu, Y. Niu, Y. Han, Y. Wu, and H. Zhang, “Prompt-Aligned Gradient for Prompt Tuning,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, (2023), 15659-15669.
|
| [28] |
H. Yao, R. Zhang, and C. Xu, “Visual-Language Prompt Tuning With Knowledge-Guided Context Optimization,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, (2023), 6757-6767.
|
| [29] |
Y. Lu, J. Liu, Y. Zhang, Y. Liu, and X. Tian, “Prompt Distribution Learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, (2022), 5206-5215.
|
| [30] |
D. Lee, S. Song, J. Suh, J. Choi, S. Lee, and H. J. Kim, “Read-Only Prompt Optimization for Vision-Language Few-Shot Learning,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, (2023), 1401-1411.
|
| [31] |
G. Chen, W. Yao, X. Song, X. Li, Y. Rao, and K. Zhang, “Prompt Learning With Optimal Transport for Vision-Language Models,” ArXiv abs/2210.01253 (2022): https://doi.org/10.48550/arXiv.2210.01253.
|
| [32] |
M. U. Khattak, S. T. Wasim, M. Naseer, S. Khan, M.-H. Yang, and F. S. Khan, “Self-Regulating Prompts: Foundational Model Adaptation Without Forgetting,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, (2023), 15190-15200.
|
| [33] |
N. A. Talemi, H. Kashiani, and F. Afghah, “Style-Pro: Style-Guided Prompt Learning for Generalizable Vision-Language Models,” in 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) (2025), 6207-6216.
|
| [34] |
K. Li, Y. Zhang, K. Li, and Y. Fu, “Adversarial Feature Hallucination Networks for Few-Shot Learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, (2020), 13470-13479.
|
| [35] |
J. Ye, N. Xu, Y. Wang, et al., Llm-Da: Data Augmentation via Large Language Models for Few-Shot Named Entity Recognition,” ArXiv abs/2402.14568 (2024): https://doi.org/10.48550/arXiv.2402.14568.
|
| [36] |
A. Santoro, S. Bartunov, M. M. Botvinick, D. Wierstra, and T. P. Lillicrap, “One-Shot Learning With Memory-Augmented Neural Networks,” ArXiv abs/1605.06065 (2016): https://doi.org/10.48550/arXiv.1605.06065.
|
| [37] |
H. Tang, Z. Li, Z. Peng, and J. Tang, “Blockmix: Meta Regularization and Self-Calibrated Inference for Metric-Based Meta-Learning,” in Proceedings of the 28th ACM International Conference on Multimedia, (2020), 610-618.
|
| [38] |
H. Sima, M. Wang, L. Liu, Y. Zhang, and J. Sun, “Mso-Detr: Metric Space Optimization for Few-Shot Object Detection,” CAAI Transactions on Intelligence Technology 9, no. 6 (2024): 1515-1533.
|
| [39] |
H. Tang, C. Yuan, Z. Li, and J. Tang, “Learning Attention-Guided Pyramidal Features for Few-Shot Fine-Grained Recognition,” Pattern Recognition 130 (2022): 108792.
|
| [40] |
H. Tang, J. Liu, S. Yan, R. Yan, Z. Li, and J. Tang, “M3net: Multi-View Encoding, Matching, and Fusion For Few-Shot Fine-Grained Action Recognition,” in Proceedings of the 31st ACM International Conference on Multimedia, (2023), 1719-1728.
|
| [41] |
J. Chen, F. Xu, X. Lyu, T. Zeng, X. Li, and S. Chen, “Multi-View Aggregation and Multi-Relation Alignment for Few-Shot Fine-Grained Recognition,” Expert Systems with Applications 283 (2025): 127764.
|
| [42] |
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A Large-Scale Hierarchical Image Database,” in 2009 IEEE Conference on Computer Vision and Pattern Recognition, (2009), 248-255.
|
| [43] |
L. Fei-Fei, R. Fergus, and P. Perona, “Learning Generative Visual Models From Few Training Examples: An Incremental Bayesian Approach Tested on 101 Object Categories,” in 2004 Conference on Computer Vision and Pattern Recognition Workshop, (2004).
|
| [44] |
O. M. Parkhi, A. Vedaldi, A. Zisserman, and C. Jawahar, “Cats and Dogs,” in 2012 IEEE Conference on Computer Vision and Pattern Recognition, (2012), 3498-3505.
|
| [45] |
J. Krause, M. Stark, J. Deng, and L. Fei-Fei, “3d Object Representations For Fine-Grained Categorization,” in Proceedings of the IEEE International Conference on Computer Vision Workshops, (2013), 554-561.
|
| [46] |
M.-E. Nilsback and A. Zisserman, “Automated Flower Classification Over a Large Number of Classes,” in 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, (2008), 722-729.
|
| [47] |
L. Bossard, M. Guillaumin, and L. Van Gool, “Food-101-Mining Discriminative Components With Random Forests,” in European Conference on Computer Vision, (2014), 446-461.
|
| [48] |
S. Maji, E. Rahtu, J. Kannala, M. B. Blaschko, and A. Vedaldi, “Fine-Grained Visual Classification of Aircraft,” ArXiv abs/1306.5151 (2013): https://doi.org/10.48550/arXiv.1306.5151.
|
| [49] |
P. Helber, B. Bischke, A. R. Dengel, and D. Borth, “Eurosat: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification,” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 12, no. 7 (2017): 2217-2226.
|
| [50] |
K. Soomro, A. Zamir, and M. Shah, “Ucf101: A Dataset of 101 Human Actions Classes From Videos in the Wild,” ArXiv abs/1212.0402 (2012): https://doi.org/10.48550/arXiv.1212.0402.
|
| [51] |
M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi, “Describing Textures in the Wild,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (2014), 3606-3613.
|
| [52] |
J. Xiao, J. Hays, K. A. Ehinger, A. Oliva, and A. Torralba, “Sun Database: Large-Scale Scene Recognition From Abbey to Zoo,” in 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, (2010), 3485-3492.
|
| [53] |
B. Recht, R. Roelofs, L. Schmidt, and V. Shankar, “Do Imagenet Classifiers Generalize to Imagenet?,” in Proceedings of the 36th International Conference on Machine Learning, Vol. 97, (2019), 5389-5400.
|
| [54] |
H. Wang, S. Ge, E. P. Xing, and Z. C. Lipton, “Learning Robust Global Representations by Penalizing Local Predictive Power,” in Proceedings of the 33rd International Conference on Neural Information Processing Systems, (2019), 10506-10518.
|
| [55] |
D. Hendrycks, K. Zhao, S. Basart, J. Steinhardt, and D. Song, “Natural Adversarial Examples,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, (2021), 15262-15271.
|
| [56] |
D. Hendrycks, S. Basart, N. Mu, et al., The Many Faces of Robustness: A Critical Analysis of Out-Of-Distribution Generalization,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, (2021), 8340-8349.
|
| [57] |
Y. Ouali, A. Bulat, B. Matinez, and G. Tzimiropoulos, “Black Box Few-Shot Adaptation for Vision-Language Models,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, (2023), 15534-15546.
|
| [58] |
K. Goswami, S. Karanam, P. Udhayanan, K. Joseph, and B. V. Srinivasan, “Copl: Contextual Prompt Learning for Vision-Language Understanding,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 16, 2024, pp. 18090-18098.
|
| [59] |
X. Yin, J. Wu, W. Yang, X. Zhou, S. Zhang, and T. Zhang, “Hierarchy-Aware Interactive Prompt Learning for Few-Shot Classification,” IEEE Transactions on Circuits and Systems for Video Technology (2024).
|
| [60] |
L. Liu, N. Wang, D. Zhou, et al., Generalizable Prompt Learning via Gradient Constrained Sharpness-Aware Minimization,” IEEE Transactions on Multimedia 27 (2025): 1100-1113.
|
| [61] |
S. He, S. Wang, and S. Long, “A Slim Prompt-Averaged Consistency Prompt Learning for Vision-Language Model,” Knowledge-Based Systems 310 (2025): 113011.
|
| [62] |
P.-N. Bui, K.-B. Nguyen, and H. Choo, “Accelerating Conditional Prompt Learning via Masked Image Modeling for Vision-Language Models,” arXiv preprint arXiv:2508.04942 (2025).
|
| [63] |
G. Kim, S. Kim, and S. Lee, “Decoupling Augmentation Bias in Prompt Learning for Vision-Language Models,” Pattern Recognition 172 (2026): 112630.
|
| [64] |
H. Yin and Y. Zhao, “Multi-Modal Prompt Learning With Bidirectional Layer-Wise Prompt Fusion,” Information Fusion 117 (2025): 102919.
|
| [65] |
C. Xu, Y. Zhu, H. Shen, et al., Progressive Visual Prompt Learning With Contrastive Feature Re-Formation,” International Journal of Computer Vision 133, no. 2 (2025): 511-526.
|
| [66] |
S.-L. Lv, Y.-Y. Chen, Z. Zhou, M. Yang, and L.-Z. Guo, “Bmip: Bi-Directional Modality Interaction Prompt Learning for Vlm,” in Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, IJCAI-25, ed. J. Kwok, Vol. 8 (International Joint Conferences on Artificial Intelligence Organization, 2025), 5887-5895.
|
| [67] |
J. Wang, Q. Xu, B. Jiang, and B. Luo, “Integrated Structural Prompt Learning for vision-language Models,” arXiv preprint arXiv:2507.05677 (2025).
|
| [68] |
H. Zheng, S. Yang, Z. He, J. Yang, and Z. Huang, “Hierarchical cross-modal Prompt Learning for vision-language Models,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, (2025), 1891-1901.
|
PDF
(2090KB)
