Data fusing and joint training for learning with noisy labels

Abstract

It is well known that deep learning depends on a large amount of clean data. Because of high annotation cost, various methods have been devoted to annotating the data automatically. However, a larger number of the noisy labels are generated in the datasets, which is a challenging problem. In this paper, we propose a new method for selecting training data accurately. Specifically, our approach fits a mixture model to the per-sample loss of the raw label and the predicted label, and the mixture model is utilized to dynamically divide the training set into a correctly labeled set, a correctly predicted set, and a wrong set. Then, a network is trained with these sets in the supervised learning manner. Due to the confirmation bias problem, we train the two networks alternately, and each network establishes the data division to teach the other network. When optimizing network parameters, the labels of the samples fuse respectively by the probabilities from the mixture model. Experiments on CIFAR-10, CIFAR-100 and Clothing1M demonstrate that this method is the same or superior to the state-of-the-art methods.

Key words： deep learning; noisy labels; data fusing

Acknowledgements

This work was supported by SRC-Open Project of Research Center of Security Video and Image Processing Engineering Technology of Guizhou ([2020]001]), Beijing Advanced Innovation Center for Intelligent Robots and Systems (2018IRS20) and National Natural Science Foundation of China (Grant No. 61973334).

References

1 Yan Y , Rosales R , Fung G , Subramanian R , Dy J . Learning from multiple annotators with varying expertise. Machine Learning, 2014, 95( 3): 291– 327

2 Yu X, Liu T, Gong M, Tao D. Learning with biased complementary labels. In: Proceedings of the 15th European Conference on Computer Vision. 2018, 69– 85

3 Blum A , Kalai A , Wasserman H . Noise-tolerant learning, the parity problem, and the statistical query model. Journal of the ACM, 2003, 50( 4): 506– 519

4 Tanno R, Saeedi A, Sankaranarayanan S, Alexander D C, Silberman N. Learning from noisy labels by regularized estimation of annotator confusion. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2019, 1123 6– 11245

5 Zhang C, Bengio S, Hardt M, Recht B, Vinyals O. Understanding deep learning requires rethinking generalization. In: Proceedings of the 5th International Conference on Learning Representations (ICLR). 2017

6 Goldberger J, Ben-Reuven E. Training deep neural-networks using a noise adaptation layer. In: Proceedings of the ICLR. 2017

7 Liu T , Tao D . Classification with noisy labels by importance reweighting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 38( 3): 447– 461

8 Patrini G, Rozza A, Menon A K, Nock R, Qu L. Making deep neural networks robust to label noise: a loss correction approach. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2017, 2233−2241

9 Reed S E, Lee H, Anguelov D, Szegedy C, Erhan D, Rabinovich A. Training deep neural networks on noisy labels with bootstrapping. In: Proceedings of the 3rd International Conference on Learning Representations (ICLR). 2015

10 Tanaka D, Ikami D, Yamasaki T, Aizawa K. Joint optimization framework for learning with noisy labels. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2018, 5552−5560

11 Arazo E, Ortego D, Albert P, O’Connor N E, McGuinness K. Unsupervised label noise modeling and loss correction. In: Proceedings of the 36th International Conference on Machine Learning (ICML). 2019, 312– 321

12 Zhang H, Cissé M, Dauphin Y N, Lopez-Paz D. mixup: beyond empirical risk minimization. In: Proceedings of the 6th International Conference on Learning Representations (ICLR). 2018

13 Ren M, Zeng W, Yang B, Urtasun R. Learning to reweight examples for robust deep learning. In: Proceedings of the 35th International Conference on Machine Learning (ICML). 2018, 4331−4340

14 Jiang L, Zhou Z, Leung T, Li L, Fei-Fei L. MentorNet: learning data-driven curriculum for very deep neural networks on corrupted labels. In: Proceedings of the 35th International Conference on Machine Learning (ICML). 2018, 2309−2318

15 Han B, Yao Q, Yu X, Niu G, Xu M, Hu W, Tsang I W, Sugiyama M. Co-teaching: robust training of deep neural networks with extremely noisy labels. In: Proceedings of the 32nd International Conference on Neural Information Processing Systems (NeurIPS). 2018, 8536−8546

16 Yu X, Han B, Yao J, Niu G, Tsang I W, Sugiyama M. How does disagreement help generalization against label corruption? In: Proceedings of the 36th International Conference on Machine Learning (ICML). 2019, 7164−7173

17 Wei H, Feng L, Chen X, An B. Combating noisy labels by agreement: a joint training method with co-regularization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2020, 1372 3– 13732

18 Li J, Socher R, Hoi S C H. DivideMix: learning with noisy labels as semi-supervised learning. In: Proceedings of the 8th International Conference on Learning Representations. 2020

19 Li Y, Yang J, Song Y, Cao L, Luo J, Li L. Learning from noisy labels with distillation. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV). 2017, 1928−1936

20 Xiao T, Xia T, Yang Y, Huang C, Wang X. Learning from massive noisy labeled data for image classification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2015, 2691−2699

21 Vahdat A. Toward robustness against label noise in training deep discriminative neural networks. In: Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS). 2017, 5601−5610

22 Veit A, Alldrin N, Chechik G, Krasin I, Gupta A, Belongie S J. Learning from noisy large-scale datasets with minimal supervision. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2017, 6575−6583

23 Lee K H, He X, Zhang L, Yang L. CleanNet: transfer learning for scalable image classifier training with label noise. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2018, 5447−5456

24 Yi K, Wu J. Probabilistic end-to-end noise correction for learning with noisy labels. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2019, 7010−7018

25 Ma X, Wang Y, Houle M E, Zhou S, Erfani S M, Xia S, Wijewickrema S N R, Bailey J. Dimensionality-driven learning with noisy labels. In: Proceedings of the 35th International Conference on Machine Learning (ICML). 2018, 3361−3370

26 Hendrycks D, Mazeika M, Wilson D, Gimpel K. Using trusted data to train deep networks on labels corrupted by severe noise. In: Proceedings of the 32nd International Conference on Neural Information Processing Systems (NeurIPS). 2018, 1047 7– 10486

27 Thulasidasan S, Bhattacharya T, Bilmes J A, Chennupati G, Mohd-Yusof J. Combating label noise in deep learning using abstention. In: Proceedings of the 36th International Conference on Machine Learning (ICML). 2019, 6234−6243

28 Shen Y, Sanghavi S. Learning with bad training data via iterative trimmed loss minimization. In: Proceedings of the 36th International Conference on Machine Learning (ICML). 2019, 5739−5748

29 Ghosh A, Kumar H, Sastry P S. Robust loss functions under label noise for deep neural networks. In: Proceedings of the 31st AAAI Conference on Artificial Intelligence (AAAI). 2017, 1919−1925

30 Wang Y, Ma X, Chen Z, Luo Y, Yi J, Bailey J. Symmetric cross entropy for robust learning with noisy labels. In: Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV). 2019, 322– 330

31 Ding Y, Wang L, Fan D, Gong B. A semi-supervised two-stage approach to learning from noisy labels. In: Proceedings of the IEEE Winter Conference on Applications of Computer Vision (WACV). 2018, 1215−1224

32 Kong K , Lee J , Kwak Y , Kang M , Kim S G , Song W J . Recycling: semi-supervised learning with noisy labels in deep neural networks. IEEE Access, 2019, 7 : 66998– 67005

33 Berthelot D, Carlini N, Goodfellow I J, Oliver A, Papernot N, Raffel C. MixMatch: a holistic approach to semi-supervised learning. In: Proceedings of the 33rd International Conference on Neural Information Processing Systems (NeurIPS). 2019, 454

34 Arpit D, Jastrzebski S, Ballas N, Krueger D, Bengio E, Kanwal M S, Maharaj T, Fischer A, Courville A C, Bengio Y, Lacoste-Julien S. A closer look at memorization in deep networks. In: Proceedings of the 34th International Conference on Machine Learning (ICML). 2017, 233−242

35 Chen P, Liao B, Chen G, Zhang S. Understanding and utilizing deep neural networks trained with noisy labels. In: Proceedings of the 36th International Conference on Machine Learning (ICML). 2019, 1062−1070

36 Permuter H , Francos J , Jermyn I . A study of Gaussian mixture models of color and texture features for image classification and segmentation. Pattern Recognition, 2006, 39( 4): 695– 706

37 Tarvainen A, Valpola H. Mean teachers are better role models: weight-averaged consistency targets improve semi-supervised deep learning results. In: Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS). 2017, 1195−1204

38 Krizhevsky A. Learning multiple layers of features from tiny images. University of Toronto, Dissertation, 2009

39 Li J, Wong Y, Zhao Q, Kankanhalli M S. Learning to learn from noisy labeled data. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2019, 5046−5054

40 Wang Y, Liu W, Ma X, Bailey J, Zha H, Song L, Xia S T. Iterative learning with open-set noisy labels. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2018, 8688−8696

41 He K, Zhang X, Ren S, Sun J. Identity mappings in deep residual networks. In: Proceedings of the 14th European Conference on Computer Vision (ECCV). 2016, 630−645