Multi-view multi-label learning (MVML) aims to leverage heterogeneous features and semantic labels for robust model learning. While existing methods often integrate cross-view consensus and view-specific information at the data level, they typically overlook the rich semantics of labels and the inherent view-label correspondences, leading to sub-optimal performance. In this paper, we propose a novel Label-Driven Contrastive Fusion (LDCF) method, which explicitly embeds multi-label semantic correlations into a contrastive fusion scheme with cross-view commonalities exploitation and view-specific individualities extraction. Specifically, LDCF first disentangles consensus and view-specific features through an orthogonal constraint. Then, a label-driven feature selector is designed to construct contrastive sample pairs based on inter-instance label similarities, by which both intra-view and inter-view contrastive learning are applied, pulling semantically similar pairs closer while pushing dissimilar ones apart, thus enhancing the feature discriminability. Finally, LDCF proposes to jointly optimize the orthogonal constraint, the multi-view contrastive learning objective, and the multi-label BCE loss function through a multi-head collaborative classification framework. Extensive experiments on multiple datasets demonstrate the superior performance of the proposed method against state-of-the-art approaches.

As a pivotal technique for improving the defense of deep models, adversarial robustness transfer via distillation has demonstrated remarkable success in conventional image classification tasks. However, this paradigm encounters critical challenges when applied to vision-language models (VLM) (e.g., CLIP): constructing adversarially robust teacher for large-scale multi-modal models demands prohibitively high computational resources. We bridge this gap by revealing an interesting phenomenon: vanilla CLIP (without adversarial training) exhibits intrinsic defensive capabilities against adversarial examples generated by another CLIP with different architectures. We formally define this as proxy adversarial robustness, and naturally propose a Heterogeneous Proxy Transfer (HPT) framework that establishes cross-architectural robustness distillation channels between CLIP variants, effortlessly enabling the VLM robustness transfer from proxy to target models. Yet, such proxy transfer paradigm easily induces severe overfitting, leading to a sharp degradation in zero-shot natural generalization. To resolve that, we design Generalization-Pivot Decoupling (GPD) by leveraging the difference in learning rate scheduling. This decouples the proxy transfer process into a generalization-anchored warm-up that maintains generalization and a generalization-pulled HPT that promotes adversarial robustness, to achieve an equilibrium between natural generalization and adversarial robustness. Extensive experiments on 15 zero-shot datasets demonstrate the effectiveness of our HPT-GPD method. The code is available at the website of github.com/fxw13/HPT-GPD.