Robust semi-supervised learning in open environments

Abstract

Semi-supervised learning (SSL) aims to improve performance by exploiting unlabeled data when labels are scarce. Conventional SSL studies typically assume close environments where important factors (e.g., label, feature, distribution) between labeled and unlabeled data are consistent. However, more practical tasks involve open environments where important factors between labeled and unlabeled data are inconsistent. It has been reported that exploiting inconsistent unlabeled data causes severe performance degradation, even worse than the simple supervised learning baseline. Manually verifying the quality of unlabeled data is not desirable, therefore, it is important to study robust SSL with inconsistent unlabeled data in open environments. This paper briefly introduces some advances in this line of research, focusing on techniques concerning label, feature, and data distribution inconsistency in SSL, and presents the evaluation benchmarks. Open research problems are also discussed for reference purposes.

Graphical abstract

Keywords

machine learning / open environment / semi-supervised learning / robust SSL

1 Introduction

Semi-supervised learning (SSL) is an effective learning paradigm to improve learning performance by attempting to exploit abundant unlabeled data when labels are scarce. It has been reported that, in certain cases, such as image classification, SSL methods can achieve the performance of purely supervised learning even when a substantial portion of the labels in a given data set has been discarded [1].

It is noticeable that the current success of SSL is mostly based on the close environment assumption where important factors between the labeled and unlabeled data are consistent. For example, all the unlabeled instances should belong to the class label set in labeled data, the features describing labeled and unlabeled data should be the same, and all labeled and unlabeled data should be sampled from an identical distribution. Fig.1 illustrates those consistent factors assumed in close environment SSL studies.

Fig.1 Typical consistent factors assumed in close environment semi-supervised learning

Full size|PPT slide

However, many real-world applications involve open environments [2] where class label set, feature space, and data distribution could be inconsistent between labeled and unlabeled data. The main reason lies in the fact that the collection process of unlabeled data is different from labeled data, which lacks human supervision and easily collects data that are inconsistent with the target task. It is also impossible to manually validate the quality of unlabeled data, otherwise it goes against the SSL’s purpose of reducing human labor. It has been widely reported that SSL suffers severe performance degradation problems with inconsistent unlabeled data and could be even worse than the simple supervised learning baseline which does not exploit more unlabeled data [3-8]. Such phenomena undoubtedly violate the expectations of SSL and limit its effectiveness in more practical tasks.

It seems the robust SSL in open environments is relevant to studies like out-of-distribution (OOD) detection [9-12], or open set recognition [13,14]. However, these studies either assume that there is an accurate classification model or sufficient labeled data, which limits their application in SSL. There are also some unsupervised OOD detection studies [15] utilizing the power of the contrastive learning [16] framework to learn better representation for OOD detection. However, although these studies do not require labels, they still need a large amount of in-distribution data for training, while in open environment SSL it is difficult to get the clean in-distribution unlabeled data.

Despite the grand challenges, many research efforts have recently been devoted to robust SSL in open environments. This paper will briefly introduce some advances in this line of research, focusing on approaches concerning inconsistent labels, inconsistent features, and inconsistent distributions between labeled and unlabeled data. Moreover, we introduce the benchmark dataset and performance measures applicable to evaluate the robustness of SSL in open environments and provide a public SSL toolkit for related research. Open research problems are also discussed for reference purposes.

2 Robust SSL in open environments

In the SSL task, we are given a set of training data which includes labeled data set ${\mathcal{D}}_l$ consists of $n$ labeled instances ${\mathcal{D}}_l=\{({\mathbf{x}}_1,{\mathbf{y}}_1),\dots ,({\mathbf{x}}_n,{\mathbf{y}}_n)\}$ and unlabeled data set ${\mathcal{D}}_u$ consists of $m$ unlabeled instances ${\mathcal{D}}_u=\{{\mathbf{x}}_{n+1},\dots ,{\mathbf{x}}_{n+m}\}$. Usually, $m\gg n$, $\mathbf{x}\in \mathcal{X}\in {\mathbb{R}}^d$, $\mathbf{y}\in \mathcal{Y}=\{1,\dots ,K\}$ where $d$ is the number of feature dimension and $K$ is the number of classes. The goal of SSL is to learn a model $f(\mathbf{x};\theta ):\{\mathcal{X};\mathrm{\Theta }\}\to \mathcal{Y}$ parameterized by $\theta \in \mathrm{\Theta }$ from training data to minimize the generalization risk $R(f)={\mathbb{E}}_{(X,Y)}[\ell (f(X;\theta ),Y)]$, where $\ell :\mathcal{Y}\times \mathcal{Y}\to \mathbb{R}$ refers to certain loss function, e.g., mean squared error or cross-entropy loss.

In open environments, unlabeled data could be inconsistent with labeled data in terms of class label space, feature dimension, and data distribution. We denote the degree of inconsistency as $t\in [0,1]$. A higher $t$ indicates a greater inconsistency, i.e., more unlabeled instances that are inconsistent with the target task. The robust SSL studies in open environments aim to decrease the negative impact of inconsistent unlabeled data, on the one hand, improve the performance via exploiting unlabeled data, and on the other hand, in the worst case, the SSL performance should not be worse than the supervised learning baseline which does not exploit more unlabeled data.

3 Label inconsistent

Close-environment SSL studies typically assume that the class label of any unlabeled instances should be a member of the given label space $\mathcal{Y}$. However, this assumption does not always hold. This is because unlabeled data is much easier to collect than labeled data in real-world applications, and the collection process of unlabeled data has less human verification. Thus, it is more likely for unlabeled data to have unseen classes that are irrelevant to the target task. For example, in the image classification task, unlabeled images crawled from Internet/social networking according to keywords usually contain broader category concepts than labeled data [5,6,17,18]. We illustrate label inconsistency in Fig.2 to help understand the problem.

Fig.2 Irrelevant classes occur in the unlabeled dataset

Full size|PPT slide

Many researchers have pointed out that SSL is not robust to irrelevant unseen classes of unlabeled data, and could perform even worse than the simple supervised learning method that uses only a small number of labeled data [4,5].

The straightforward idea to deal with this problem is to detect and remove these irrelevant unseen class unlabeled instances. It is noteworthy that this problem is different from OOD detection since OOD detection approaches typically require a large corpus of in-distribution labeled data and would fail due to the scarcity of labeled data in SSL. Recently, a simple yet effective approach was proposed to tackle the semi-supervised OOD detection issue [19]. This approach learns a new representation space via a novel distance measure in which OOD samples could be separated well with limited labeled data and in-distribution data.

Some approaches try to decrease the negative impact of these unseen class unlabeled instances in the training process. Various scoring mechanisms have been proposed to evaluate how much contribution an unlabeled instance has to the model training [20-22]. If the score is higher than the threshold the instance is retained, otherwise, the instance is discarded. In addition to the hard threshold, some works try to assign soft weight to the unlabeled training instances [5,23].

Instead of treating all irrelevant classes unlabeled instances as harmful, some researchers find these instances could also be helpful for model training. One promising way is to exploit the irrelevant unlabeled instances to help learn better representations via the self-supervised learning paradigm [24].

Robust SSL focuses on how to avoid the negative impact of unseen class unlabeled data. Meanwhile, some studies focus on the setting that the unseen classes in unlabeled data also need to be classified, which is called open-world SSL [25,26]. This line of research is more like class-incremental learning [27], and different from the goal of robust SSL in open environments.

4 Feature inconsistent

Close-environment SSL studies typically assume that all unlabeled instances reside in the same feature space with the labeled data. Unfortunately, this does not always hold. For example, in the image classification task, the labeled data are all color images while the unlabeled data could contain grayscale images, resulting in the loss of two color channels. In tasks dealing with tabular data, such as financial analysis tasks or recommendation systems [28], decremental or incremental features in unlabeled data are more common. Fig.3 illustrates the feature inconsistent problem in SSL.

Fig.3 Feature inconsistent in SSL

Full size|PPT slide

As pointed out by [18], close environment SSL methods could suffer severe performance degradation when facing the feature inconsistent between labeled and unlabeled data.

Compared with the label inconsistent problem, detecting which unlabeled instances have inconsistent features with the target task is much easier since validating the feature $\mathbf{x}$ is irrelevant to the label. Therefore, the straightforward method to address the feature inconsistent problem is to detect and remove all inconsistent unlabeled instances.

However, this baseline method cannot effectively utilize the information behind these unlabeled data, resulting in limited performance improvement. Another approach that readily comes to mind is to remove all incremental features and fill in decremental features, whereas how to fill the missing feature to ensure the SSL performance will not degrade is still a challenging problem [18].

There are also studies focusing on the adversarial feature inconsistency unlabeled examples, which can be categorized into two aspects: Attack and Defense. Semi-supervised attack techniques study how to generate adversarial unlabeled examples that cause SSL predictions to be incorrect. The major techniques can be categorized as misleading sequence injection, which aims to inject a sequence of synthetically unlabeled examples into the unlabeled training data and cause the model to make wrong predictions [29] and adversarial perturbation generation, which aims to learn a feature perturbation generator for the training examples and making the model output wrong predictions when training with these perturbed examples [30,31]. The defense techniques study how to make the SSL robust to adversarial unlabeled examples. The major techniques can be categorized as robust regularization, which aims to design regularization terms to the objective of SSL directly [32,33] and the combination with classical distribution robust optimization methods [34].

Recent studies of robust SSL with feature inconsistency mainly focus on the image classification task. It is noteworthy that tabular data is also commonly encountered in real scenarios [28]. Compared with image data, the feature inconsistent problem is more commonly occurring in tabular data. Robust SSL with inconsistent features on tabular data is an important yet understudied problem.

5 Distribution inconsistent

Close-environment SSL studies typically assume that all labeled and unlabeled data are independent samples from an identical distribution (i.e., $i.i.d.$ samples). Unfortunately, this does not always hold. Taking the image classification as an example again, the labeled data may sampled from natural images. In contrast, the unlabeled data may be selected from the internet according to some keywords and may include cartoon images [35]. These problems also commonly happen in scenarios like sentiment analysis [5], remote sensing [36], legal judgment [37], etc. Fig.4 illustrates that ignoring the data distribution inconsistent mismatch may lead to seriously downgraded performance.

Fig.4 Distribution inconsistent between labeled and unlabeled data in SSL

Full size|PPT slide

There have been plentiful studies concerning distribution shifts such as prior probability shifts, covariate shifts, and concept shifts. However, the relevant studies mainly focus on the training/testing distribution change and are conducted under the umbrella of domain adaptation or transfer learning [38]. In SSL studies the distribution occurs within the training data. To be able to handle various kinds of data distribution inconsistent between labeled and unlabeled data is an important requirement for robust SSL in open environments.

The straightforward method is to treat labeled data as the target domain and unlabeled data as the source domain and then apply domain adaptation techniques to learn new representations for all training instances to eliminate the distribution mismatch [39-41]. However, due to the label scarcity in SSL, these methods can only consider the adaptation in an unsupervised manner and ignore task-related label information.

Recent work presented a theoretical framework that presents three main reasons why SSL algorithms can not perform well with inconsistent distributions: coupling between the pseudo-label predictor and the target predictor, biased pseudo labels, and restricted instance weights [42]. To address these challenges, they provided a new method called bidirectional adaptation that can adapt to the distribution of unlabeled data for debiased pseudo-label prediction and to the target distribution for debiased target prediction, thereby mitigating the above shortcomings.

Moreover, some works focus on the problem that unlabeled data distribution is long-tailed and report that SSL suffers performance degradation on tail classes [43-47]. Conventional long-tailed techniques can not be applied directly due to the label scarcity in SSL. The general principle is to design distribution alignment techniques to calibrate the distribution of pseudo-labels to align with the target distribution [43,44,46].

6 Evaluation benchmark

Conventional SSL studies mainly evaluate performance on standard image classification datasets and report classification accuracy. How to fair evaluate the robustness of SSL methods in open environments is under-considered. In this section, we briefly introduce some datasets, applicable performance measures, and an open-sourced SSL toolkit.

6.1 Datasets

Constructing open environment SSL benchmarks that contain different extents of inconsistency between labeled and unlabeled data is important for the evaluation of robust SSL algorithms. Recently, a more realistic SSL benchmark included both label, feature, and distribution inconsistent has been provided [18]. The benchmark involves various data types in machine learning, including tabular datasets from the UCI dataset, widely applied image datasets, and text datasets. Specifically, to simulate the inconsistent labels, they construct inconsistent labeled space by randomly selecting some classes and discarding the labeled data belonging to these classes. To simulate the inconsistent features, they randomly mask features for tabular data and convert the images to grayscale, resulting in the loss of two color channels for image datasets. For the text datasets, they employ text truncation, and truncated portions are filled with “$<pad>$”. To simulate the data distribution, for image and text datasets, they adopt the Image-CLEF [48] and the IMDA/Amazon [49,50] to construct the labeled and unlabeled data which are natural distribution shifts. For tabular datasets, they calculate the centroids of each class and use the distance between instances and class centroids to filter instances, thus constructing an environment with inconsistent data distribution.

6.2 Performance measures

To achieve a fair and comprehensive evaluation of robust SSL in open environments, only reporting the classification accuracy or error is not enough. A series of performance metrics tailored for robust SSL in open environments have been proposed recently [18]. These metrics begin by defining a function $Acc(t)$, which quantifies the change in classification accuracy as a function of the inconsistency level $t$. This function is used to construct the Robustness Analysis Curve (RAC) that maps the inconsistency level $t$ to the corresponding accuracy $Acc(t)$. Unlike conventional SSL evaluations that focus solely on $Acc(0)$ or a specific $Acc(t)$, various performance metrics are proposed based on the RAC that include Area Under the RAC Curve (AUC) which captures the overall robustness of SSL approaches; Expected Accuracy (EA) which describes the average performance across all inconsistency levels; Worst-Case Accuracy (WA) which identifies the lowest accuracy level, representing performance in the worst-case scenario; Expected Variation Magnitude (EVM) which captures the average magnitude of performance variation; Variation Stability (VS) which quantifies the stability of the performance variation; Robust Correlation Coefficient (RCC) which captures the overall trend of performance variation. The detailed formulation of these metrics is presented in Tab.1.

Tab.1 Performance metrics for robust SSL in open environments. $Acc(t)$ describe the change in classification accuracy with the inconsistency extent $t$, $P_T(t)$ is the distribution for $t$, $Ac{c}^{\prime }(\cdot )$ indicate the first derivative

Area Under the Curve (AUC)	$\text{AUC}(Acc)={\int }_0^{1}Acc(t)\mathrm{d}t$
Expected Accuracy (EA)	$\text{EA}(P_T,Acc)={⟨P⟩}_T,Acc⟨=⟩{\int }_0^1{P}_T(t)Acc(t)\mathrm{d}t$
Worst-Case Accuracy (WA)	$\text{WA}(Acc)=\underset{t\in [0,1]}{min}Acc(t)$
Expected Variation Magnitude (EVM)	$\text{EVM}(Acc)={\int }_0^1|Ac{c}^{\prime }(t)|\mathrm{d}t$
Variation Stability (VS)	$\text{VS}(Acc)={\int }_0^1[Ac{c}^{\prime }(t)-({\int }_0^{1}Ac{c}^{\prime }(t)\mathrm{d}t){]}^2\mathrm{d}t$
Robust Correlation Coefficient (RCC)	$\text{RCC}(Acc)=\frac{{\int }_0^{1}Acc(t)\cdot t\mathrm{d}t-{\int }_0^{1}Acc(t)\mathrm{d}t}{\sqrt{{\int }_0^1{t}^2\mathrm{d}t-1}\cdot \sqrt{{\int }_0^{1}Ac{c}^2(t)\mathrm{d}t-({\int }_0^{1}Acc(t)\mathrm{d}t{)}^2}}$

6.3 Open-sourced toolkit

To provide easier evaluation and implementation of SSL algorithms, an open-sourced SSL toolkit: LAMDA-SSL is released [51]. LAMDA-SSL incorporates more than 30 SSL algorithms, supports various data types, and is compatible with other popular machine learning toolkits such as “scikit-learn” and “pytorch”. The toolkit is available at the website of ygzwqzd.github.io/LAMDA-SSL.

7 Open challenges

Though robust SSL in open environments has attracted much attention, it is still in its infancy. We hope to propose new research directions to broaden and boost robust SSL research.

Theoretical issues. Many theoretical problems about robust SSL have not been addressed yet. For example, when the inconsistent unlabeled data is helpful or harmful, how the generalization performance varies with different inconsistent extents, etc. More efforts are desired to be devoted.

General data types. SSL studies mainly focus on homogeneous data, especially image data. Tabular data is also a commonly occurring data in practical tasks [28,52,53]. The heterogeneous property of tabular data causes the failure of SSL algorithms. For example, consistency regularization, which is the most important technique in SSL, encourages the model to have similar output distribution on an instance and its augmented variants. The notion of augmentation simply does not exist in tabular data. Therefore, there is an urgent need to develop robust SSL techniques for more general data types.

Exploiting pre-trained models. With the success of the “pre-train and fine-tune” paradigm, more and more pre-trained models have been released. Similar to the goal of SSL, selecting and adapting helpful pre-trained models can also decrease the labeled data requirement for the target task [54,55]. Thus, how to bridge the pre-trained model with SSL is a promising direction. Recently, there have been some studies that try to exploit SSL techniques with large language models [56] and vision-language models [57]. However, the robustness of these methods after exploiting more unlabeled data is still an unaddressed problem.

From perception to decision-making. Current SSL studies mainly focus on perceptual tasks such as image classification, while practical tasks often encounter decision-making tasks that involve interaction with the environment. The dynamic of environments poses significant challenges to robustness, meanwhile, high-quality data is expensive in decision-making tasks, posing a great need for SSL. Many researchers have been exploring how to utilize unlabeled data for reinforcement learning [58,59] on these tasks, including reward-free or action-free data [60,61]. Therefore, it is important to broaden robust SSL studies into decision-making tasks with interactive environments.

8 Conclusions

This paper introduces open environments SSL. We present a definition of this problem, in which unlabeled data could contain label/feature/distribution inconsistent with the target task, and briefly introduce some research advances in this line of research. Although we consider these inconsistent problems separately, in practice they often occur simultaneously. It can hardly be a thorough review of all the relevant work and is mostly a brief review of general principles and strategies rather than specific learning algorithms. The quality of unlabeled data is hard to validate and it is fundamentally important to enable SSL to achieve excellent performance in the usual case while keeping satisfactory performance no matter what unexpected unfortunate issues occur in unlabeled data. This is crucial for achieving robust SSL in practical tasks.

Lan-Zhe Guo is an assistant professor in the School of Intelligence Science and Technology at Nanjing University, China. His research interests are mainly in semi-supervised learning and robust machine learning. He has published over 30 papers in top-tier conferences and journals such as ICML, NeurIPS, ICLR, TPAMI, and received the Outstanding Doctoral Dissertation Award from CAAI

Lin-Han Jia is currently working toward a PhD degree in the School of Computer Science at Nanjing University, China. His research interests are mainly in weakly supervised learning and optimization

Jie-Jing Shao is currently working toward a PhD degree in the School of Computer Science at Nanjing University, China. His research interests are mainly in weakly supervised learning and reinforcement learning

Yu-Feng Li is a professor in the School of Artificial Intelligence at Nanjing University, China. His research interests are mainly in weakly supervised learning, statistical learning, and optimization. He has received the PAKDD Early-Career Research Award. He is/was co-chair of ACML 22/21 journal track, and Area Chair/SPC of top-tier conferences such as ICML, NeurIPS, ICLR, AAAI

References

[1] Wang Y, Chen H, Fan Y, Sun W, Tao R, Hou W, Wang R, Yang L, Zhou Z, Guo L Z, Qi H, Wu Z, Li Y, Nakamura S, Ye W, Savvides M, Raj B, Shinozaki T, Schiele B, Wang J, Xie X, Zhang Y. USB: a unified semi-supervised learning benchmark for classification. In: Proceedings of the 36th International Conference on Neural Information Processing Systems. 2022, 285

[2] Zhou Z H . Open-environment machine learning. National Science Review, 2022, 9( 8): nwac123

[3] Guo L Z, Li Y F. A general formulation for safely exploiting weakly supervised data. In: Proceedings of the 32nd AAAI conference on Artificial Intelligence. 2018, 3126−3133

[4] Oliver A, Odena A, Raffel C, Cubuk E D, Goodfellow I J. Realistic evaluation of deep semi-supervised learning algorithms. In: Proceedings of the 32nd International Conference on Neural Information Processing Systems. 2018, 3239−3250

[5] Guo L Z, Zhang Z Y, Jiang Y, Li Y F, Zhou Z H. Safe deep semi-supervised learning for unseen-class unlabeled data. In: Proceedings of the 37th International Conference on Machine Learning. 2020, 365

[6] Li Y F, Guo L Z, Zhou Z H . Towards safe weakly supervised learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019, 43( 1): 334–346

[7] Li Y F, Liang D M . Safe semi-supervised learning: a brief introduction. Frontiers of Computer Science, 2019, 13( 4): 669–676

[8] Wang C, Cao X, Guo L, Shi Z. DualMatch: robust semi-supervised learning with dual-level interaction. In: Proceedings of the Joint European Conference on Machine Learning and Knowledge Discovery in Databases. 2023, 102−119

[9] Hendrycks D, Gimpel K. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In: Proceedings of the 5th International Conference on Learning Representations. 2017

[10] Zhou Z, Guo L Z, Jia L H, Zhang D C, Li Y F. ODS: test-time adaptation in the presence of open-world data shift. In: Proceedings of the 40th International Conference on Machine Learning. 2023, 1793

[11] Zhou Z, Yang M, Shi J X, Guo L Z, Li Y F. DeCoOp: robust prompt tuning with out-of-distribution detection. In: Proceedings of the 41st International Conference on Machine Learning. 2024

[12] Shao J J, Guo L Z, Yang X W, Li Y F. LOG: active model adaptation for label-efficient OOD generalization. In: Proceedings of the 36th International Conference on Neural Information Processing Systems. 2022, 801

[13] Geng C, Huang S J, Chen S . Recent advances in open set recognition: a survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, 43( 10): 3614–3631

[14] Shao J J, Yang X W, Guo L Z . Open-set learning under covariate shift. Machine Learning, 2024, 113( 4): 1643–1659

[15] Sehwag V, Chiang M, Mittal P. SSD: a unified framework for self-supervised outlier detection. In: Proceedings of the 9th International Conference on Learning Representations. 2021

[16] Chen T, Kornblith S, Norouzi M, Hinton G. A simple framework for contrastive learning of visual representations. In: Proceedings of the 37th International Conference on Machine Learning. 2020, 149

[17] Yang H, Zhu S, King I, Lyu M R. Can irrelevant data help semi-supervised learning, why and how?. In: Proceedings of the 20th ACM International Conference on Information and Knowledge Management. 2011, 937−946

[18] Jia L H, Guo L Z, Zhou Z, Li Y F. Realistic evaluation of semi-supervised learning algorithms in open environments. In: Proceedings of the 12th International Conference on Learning Representations. 2024

[19] Zhou Z, Guo L Z, Cheng Z, Li Y, Pu S. STEP: out-of-distribution detection in the presence of limited in-distribution labeled data. In: Proceedings of the 34th International Conference on Neural Information Processing Systems. 2021, 29168−29180

[20] Chen Y, Zhu X, Li W, Gong S. Semi-supervised learning under class distribution mismatch. In: Proceedings of the 34th AAAI Conference on Artificial Intelligence. 2020, 3569−3576

[21] Yu Q, Ikami D, Irie G, Aizawa K. Multi-task curriculum framework for open-set semi-supervised learning. In: Proceedings of the 16th European Conference on Computer Vision. 2020, 438−454

[22] Saito K, Kim D, Saenko K. OpenMatch: open-set consistency regularization for semi-supervised learning with outliers. In: Proceedings of the 35th International Conference on Neural Information Processing Systems. 2021, 1987

[23] Peng A Y, Koh Y S, Riddle P, Pfahringer B. Investigating the effect of novel classes in semi-supervised learning. In: Proceedings of the 11th Asian Conference on Machine Learning. 2019, 615−630

[24] Huang J, Fang C, Chen W, Chai Z, Wei X, Wei P, Lin L, Li G. Trash to treasure: harvesting OOD data with cross-modal matching for open-set semi-supervised learning. In: Proceedings of 2021 IEEE/CVF International Conference on Computer Vision. 2021, 8290−8299

[25] Cao K, Brbic M, Leskovec J. Open-world semi-supervised learning. In: Proceedings of the 10th International Conference on Learning Representations. 2022

[26] Guo L Z, Zhang Y G, Wu Z F, Shao J J, Lit Y F. Robust semi-supervised learning when not all classes have labels. In: Proceedings of the 36th International Conference on Neural Information Processing Systems. 2022, 239

[27] Masana M, Liu X, Twardowski B, Menta M, Bagdanov A D, Van De Weijer J . Class-incremental learning: survey and performance evaluation on image classification. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, 45( 5): 5513–5533

[28] Borisov V, Leemann T, Seßler K, Haug J, Pawelczyk M, Kasneci G . Deep neural networks and tabular data: a survey. IEEE Transactions on Neural Networks and Learning Systems, 2024, 35( 6): 7499–7519

[29] Carlini N. Poisoning the unlabeled dataset of semi-supervised learning. In: Proceedings of the 30th USENIX Security Symposium. 2021, 1577−1592

[30] Yan Z, Li G, Tian Y, Wu J, Li S, Chen M, Poor H V. DeHiB: deep hidden backdoor attack on semi-supervised learning via adversarial perturbation. In: Proceedings of the 35th AAAI Conference on Artificial Intelligence. 2021, 10585−10593

[31] Liu X, Si S, Zhu X, Li Y, Hsieh C J. A unified framework for data poisoning attack to graph-based semi-supervised learning. In: Proceedings of the 33rd International Conference on Neural Information Processing Systems. 2019, 877

[32] Miyato T, Maeda S I, Koyama M, Ishii S . Virtual adversarial training: a regularization method for supervised and semi-supervised learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019, 41( 8): 1979–1993

[33] Yu B, Wu J, Ma J, Zhu Z. Tangent-normal adversarial regularization for semi-supervised learning. In: Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019, 10668−10676

[34] Najafi A, Maeda S I, Koyama M, Miyato T. Robustness to adversarial perturbations in learning from incomplete data. In: Proceedings of the 33rd International Conference on Neural Information Processing Systems. 2019, 497

[35] Zhao P, Zhang Y J, Zhang L, Zhou Z H . Adaptivity and non-stationarity: Problem-dependent dynamic regret for online convex optimization. Journal of Machine Learning Research, 2024, 25( 98): 1–52

[36] Mo S, Kim M, Lee K, Shin J. S-CLIP: semi-supervised vision-language learning using few specialist captions. In: Proceedings of the 37th International Conference on Neural Information Processing Systems. 2023, 2674

[37] Zhou Z, Shi J X, Song P X, Yang X W, Jin Y X, Guo L Z, Li Y F. LawGPT: a Chinese legal knowledge-enhanced large language model. 2024, arXiv preprint arXiv: 2406.04614

[38] Pan S J, Yang Q . A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering, 2010, 22( 10): 1345–1359

[39] Chen K, Yao L, Zhang D, Chang X, Long G, Wang S. Distributionally robust semi-supervised learning for people-centric sensing. In: Proceedings of the 33rd AAAI Conference on Artificial Intelligence. 2019, 3321−3328

[40] Huang Z, Xue C, Han B, Yang J, Gong C. Universal semi-supervised learning. In: Proceedings of the 35th International Conference on Neural Information Processing Systems. 2021, 2046

[41] Guo L Z, Zhou Z, Li Y F. RECORD: resource constrained semi-supervised learning under distribution shift. In: Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2020, 1636−1644

[42] Jia L H, Guo L Z, Zhou Z, Shao J J, Xiang Y K, Li Y F. Bidirectional adaptation for robust semi-supervised learning with inconsistent data distributions. In: Proceedings of the 40th International Conference on Machine Learning. 2023, 607

[43] Kim J, Hur Y, Park S, Yang E, Hwang S J, Shin J. Distribution aligning refinery of pseudo-label for imbalanced semi-supervised learning. In: Proceedings of the 34th International Conference on Neural Information Processing Systems. 2020, 1221

[44] Wei C, Sohn K, Mellina C, Yuille A, Yang F. CReST: a class-rebalancing self-training framework for imbalanced semi-supervised learning. In: Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021, 10852−10861

[45] Guo L Z, Zhou Z, Shao J J, Zhang Q, Kuang F, Li G L, Liu Z X, Wu G B, Ma N, Li Q, Li Y F. Learning from imbalanced and incomplete supervision with its application to ride-sharing liability judgment. In: Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. 2021, 487−495

[46] Guo L Z, Li Y F. Class-imbalanced semi-supervised learning with adaptive thresholding. In: Proceedings of the 39th International Conference on Machine Learning. 2022, 8082−8094

[47] Wei T, Liu Q Y, Shi J X, Tu W W, Guo L Z . Transfer and share: semi-supervised learning from long-tailed data. Machine Learning, 2024, 113( 4): 1725–1742

[48] Caputo B, Müller H, Martinez-Gomez J, Villegas M, Acar B, Patricia N, Marvasti N, Üsküdarlı S, Paredes R, Cazorla M, Garcia-Varea I, Morell V. ImageCLEF 2014: overview and analysis of the results. In: Proceedings of the 5th International Conference of the Cross-Language Evaluation Forum for European Languages. 2014, 192−211

[49] McAuley J, Leskovec J. Hidden factors and hidden topics: understanding rating dimensions with review text. In: Proceedings of the 7th ACM Conference on Recommender Systems. 2013, 165−172

[50] Maas A L, Daly R E, Pham P T, Huang D, Ng A Y, Potts C. Learning word vectors for sentiment analysis. In: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. 2011, 142−150

[51] Jia L H, Guo L Z, Zhou Z, Li Y F . LAMDA-SSL: a comprehensive semi-supervised learning toolkit. Science China Information Sciences, 2024, 67( 1): 117101

[52] Ye H J, Liu S Y, Cai H R, Zhou Q L, Zhan D C. A closer look at deep learning on tabular data. 2024, arXiv preprint arXiv: 2407.00956

[53] Zhou Z, Jin Y X, Li Y F . RTS: Learning robustly from time series data with noisy label. Froniters of Computer Science, 2024, 18( 6): 186332

[54] Guo L Z, Zhou Z, Li Y F, Zhou Z H. Identifying useful learnwares for heterogeneous label spaces. In: Proceedings of the 40th International Conference on Machine Learning. 2023, 12122−12131

[55] Li S Y, Zhao S J, Cao Z T, Huang S J, Chen S C . Robust domain adaptation with noisy and shifted label distribution. Froniters of Computer Science, 2025, 19( 3): 193310

[56] Huang J, Gu S, Hou L, Wu Y, Wang X, Yu H, Han J. Large language models can self-improve. In: Proceedings of 2023 Conference on Empirical Methods in Natural Language Processing. 2023, 1051−1068

[57] Zhu B, Zhang H . Debiasing vision-language models for vision tasks: A survey. Froniters of Computer Science, 2025, 19( 1): 191321

[58] Yu T, Kumar A, Chebotar Y, Hausman K, Finn C, Levine S. How to leverage unlabeled data in offline reinforcement learning. In: Proceedings of the 39th International Conference on Machine Learning. 2022, 25611−25635

[59] Shao J J, Shi H S, Guo L Z, Li Y F. Offline imitation learning with model-based reverse augmentation. In: Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2024, 2608−2617

[60] Zheng Q, Henaff M, Amos B, Grover A. Semi-supervised offline reinforcement learning with action-free trajectories. In: Proceedings of the 40th International Conference on Machine Learning. 2023, 1782

[61] Li Z, Xu T, Qin Z, Yu Y, Luo Z Q. Imitation learning from imperfection: theoretical justifications and algorithms. In: Proceedings of the 37th International Conference on Neural Information Processing Systems. 2023, 810

Acknowledgements

This research was supported by the Key Program of Jiangsu Science Foundation (BK20243012) and the National Natural Science Foundation of China (NSFC) (Grant Nos. 62306133, 62176118).

Competing interests

The authors declare that they have no competing interests or financial conflicts to disclose.

Open Access

This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.

The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

RIGHTS & PERMISSIONS

2024 The Author(s) 2024. This article is published with open access at link.springer.com and journal.hep.com.cn