Learning multi-tasks with inconsistent labels by using auxiliary big task

Abstract

Multi-task learning is to improve the performance of the model by transferring and exploiting common knowledge among tasks. Existing MTL works mainly focus on the scenario where label sets among multiple tasks (MTs) are usually the same, thus they can be utilized for learning across the tasks. However, the real world has more general scenarios in which each task has only a small number of training samples and their label sets are just partially overlapped or even not. Learning such MTs is more challenging because of less correlation information available among these tasks. For this, we propose a framework to learn these tasks by jointly leveraging both abundant information from a learnt auxiliary big task with sufficiently many classes to cover those of all these tasks and the information shared among those partially-overlapped tasks. In our implementation of using the same neural network architecture of the learnt auxiliary task to learn individual tasks, the key idea is to utilize available label information to adaptively prune the hidden layer neurons of the auxiliary network to construct corresponding network for each task, while accompanying a joint learning across individual tasks. Extensive experimental results demonstrate that our proposed method is significantly competitive compared to state-of-the-art methods.

Graphical abstract

Keywords

multi-task learning / inconsistent labels / auxiliary task

References

[1] Khattar A, Hegde S, Hebbalaguppe R. Cross-domain multi-task learning for object detection and saliency estimation. In: Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. 2021, 3634−3643

[2] Al-Qaisi L, Hassonah M A, Al-Zoubi M M, Al-Zoubi A M. A review of evolutionary data clustering algorithms for image segmentation. In: Aljarah I, Faris H, Mirjalili S, eds. Evolutionary Data Clustering: Algorithms and Applications. Singapore: Springer, 2021, 201−214

[3] Chaturvedi I, Su C L, Welsch R E . Fuzzy aggregated topology evolution for cognitive multi-tasks. Cognitive Computation, 2021, 13( 1): 96–107

[4] Ravanelli M, Zhong J, Pascual S, Swietojanski P, Monteiro J, Trmal J, Bengio Y. Multi-task self-supervised learning for robust speech recognition. In: Proceedings of 2020 IEEE International Conference on Acoustics, Speech and Signal Processing. 2020, 6989−6993

[5] Li C, Wang B, Zhang S, Liu Y, Song R, Cheng J, Chen X . Emotion recognition from EEG based on multi-task learning with capsule network and attention mechanism. Computers in Biology and Medicine, 2022, 143: 105303

[6] Cheng B, Zhu B, Pu S . Multi-auxiliary domain transfer learning for diagnosis of MCI conversion. Neurological Sciences, 2022, 43( 3): 1721–1739

[7] Ruder S, Bingel J, Augenstein I, Søgaard A. Sluice networks: learning what to share between loosely related tasks. 2017, arXiv preprint arXiv: 1705.08142v1

[8] Gong T, Zheng X, Lu X. Remote sensing scene classification with multi-task learning. In: Wang L, Wu Y, Gong J, eds. Proceedings of the 7th China High Resolution Earth Observation Conference (CHREOC 2020). Singapore: Springer, 2022, 403−418

[9] Islam M M, Iqbal T. Mumu: cooperative multitask learning-based guided multimodal fusion. In: Proceedings of the 36th Conference on Artificial Intelligence. 2022, 1043−1051

[10] Long M, Cao Z, Wang J, Yu P S. Learning multiple tasks with multilinear relationship networks. In: Proceedings of the 31st International Conference on Neural Information Processing Systems. 2017, 1593−1602

[11] Ma J, Zhao Z, Yi X, Chen J, Hong L, Chi E H. Modeling task relationships in multi-task learning with multi-gate mixture-of-experts. In: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2018, 1930−1939

[12] Feng Q, Yao J, Zhong Y, Li P, Pan Z . Learning twofold heterogeneous multi-task by sharing similar convolution kernel pairs. Knowledge-Based Systems, 2022, 252: 109396

[13] Wu S, Zhang H R, Ré C. Understanding and improving information transfer in multi-task learning. In: Proceedings of the 8th International Conference on Learning Representations. 2020, 26−30

[14] Evgeniou T, Pontil M. Regularized multi-task learning. In: Proceedings of the 10th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 2004, 109−117

[15] Honorio J, Samaras D. Multi-task learning of Gaussian graphical models. In: Proceedings of the 27th International Conference on Machine Learning. 2010, 447−454

[16] Liu Q, Li X, He Z, Fan N, Yuan D, Liu W, Liang Y. Multi-task driven feature models for thermal infrared tracking. In: Proceedings of the 34th AAAI Conference on Artificial Intelligence. 2020, 11604−11611

[17] Wang J, Zhang S, Wang Y, Zhu Z . Learning efficient multi-task stereo matching network with richer feature information. Neurocomputing, 2021, 421: 151–160

[18] Guo P, Deng C, Xu L, Huang X, Zhang Y. Deep multi-task augmented feature learning via hierarchical graph neural network. In: Proceedings of European Conference on Machine Learning and Knowledge Discovery in Databases. 2021, 538−553

[19] Vandenhende S, Georgoulis S, van Gansbeke W, Proesmans M, Dai D, van Gool L . Multi-task learning for dense prediction tasks: a survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44( 7): 3614–3633

[20] Shen Z, Cui C, Huang J, Zong J, Chen M, Yin Y. Deep adaptive feature aggregation in multi-task convolutional neural networks. In: Proceedings of the 29th ACM International Conference on Information & Knowledge Management. 2020, 2213−2216

[21] Yadav S, Chauhan J, Sain J P, Thirunarayan K, Sheth A, Schumm J. Identifying depressive symptoms from tweets: figurative language enabled multitask learning framework. In: Proceedings of the 28th International Conference on Computational Linguistics. 2020, 696−709

[22] Sun X, Panda R, Feris R, Saenko K. Adashare: learning what to share for efficient deep multi-task learning. In: Proceedings of the 34th Conference on Neural Information Processing Systems. 2020, 8728−8740

[23] Sun T, Shao Y, Li X, Liu P, Yan H, Qiu X, Huang X. Learning sparse sharing architectures for multiple tasks. In: Proceedings of the 34th AAAI Conference on Artificial Intelligence. 2020, 8936−8943

[24] Verboven S, Hafeez Chaudhary M, Berrevoets J, Verbeke W. HydaLearn: highly dynamic task weighting for multi-task learning with auxiliary tasks. 2008, arXiv preprint arXiv: 2008.11643

[25] Sanh V, Wolf T, Ruder S. A hierarchical multi-task approach for learning embeddings from semantic tasks. In: Proceedings of the 33rd AAAI Conference on Artificial Intelligence. 2019, 6949−6956

[26] Baxter J . A Bayesian/information theoretic model of learning to learn via multiple task sampling. Machine Learning, 1997, 28( 1): 7–39

[27] Ruder S, Bingel J, Augenstein I, Søgaard A. Latent multi-task architecture learning. In: In Proceedings of the 33rd AAAI Conference on Artificial Intelligence. 2019, 4822−4829

[28] Strezoski G, Noord N, Worring M. Many task learning with task routing. In: Proceedings of 2019 IEEE/CVF International Conference on Computer Vision. 2019, 1375−1384

[29] Fernando C, Banarse D, Blundell C, Zwols Y, Ha D, Rusu A A, Pritzel A, Wierstra D. PathNet: Evolution channels gradient descent in super neural networks. 2017, arXiv preprint arXiv: 1701.08734

[30] Pironkov G, Wood S U, Dupont S . Hybrid-task learning for robust automatic speech recognition. Computer Speech & Language, 2020, 64: 101103

[31] Cao P, Shan X, Zhao D, Huang M, Zaiane O . Sparse shared structure based multi-task learning for MRI based cognitive performance prediction of Alzheimer’s disease. Pattern Recognition, 2017, 72: 219–235

[32] Lee S, Son Y . Multitask learning with single gradient step update for task balancing. Neurocomputing, 2022, 467: 442–453

[33] Søgaard A, Goldberg Y. Deep multi-task learning with low level tasks supervised at lower layers. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. 2016, 231−235

[34] Fan J, Zhao T, Kuang Z, Zheng Y, Zhang J, Yu J, Peng J . HD-MTL: hierarchical deep multi-task learning for large-scale visual recognition. IEEE Transactions on Image Processing, 2017, 26( 4): 1923–1938

[35] Ott F, Rügamer D, Heublein L, Bischl B, Mutschler C. Joint classification and trajectory regression of online handwriting using a multi-task learning approach. In: Proceedings of 2022 IEEE/CVF Winter Conference on Applications of Computer Vision. 2022, 1244−1254

[36] Zhang C, Li Y, Du N, Fan W, Yu P S. Joint slot filling and intent detection via capsule neural networks. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2019, 5259−5267

[37] Li X C, Zhan D C. FedRS: federated learning with restricted softmax for label distribution non-IID data. In: Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. 2021, 995−1005

[38] Wang Y, Zhang Z, Hao W, Song C . Attention guided multiple source and target domain adaptation. IEEE Transactions on Image Processing, 2021, 30: 892–906

[39] Xue W . Weighted feature-task-aware regularization learner for multitask learning. Pattern Analysis and Applications, 2020, 23( 1): 253–263

[40] Zhang J, Miao J, Zhao K, Tian Y . Multi-task feature selection with sparse regularization to extract common and task-specific features. Neurocomputing, 2019, 340: 76–89

[41] Shao W, Peng Y, Zu C, Wang M, Zhang D, The Alzheimer's Disease Neuroimaging Initiative . Hypergraph based multi-task feature selection for multimodal classification of Alzheimer’s disease. Computerized Medical Imaging and Graphics, 2020, 80: 101663

[42] Li L, Pan X, Yang H, Liu Z, He Y, Li Z, Fan Y, Cao Z, Zhang L. Multi-task deep learning for fine-grained classification and grading in breast cancer histopathological images. Multimedia Tools and Applications, 2020, 79(21−22): 14509−14528

[43] Zheng Z, Wang Y, Dai Q, Zheng H, Wang D. Metadata-driven task relation discovery for multi-task learning. In: Proceedings of the 28th International Joint Conference on Artificial Intelligence. 2019, 4426−4432

[44] Yan C, Xu J, Xie J, Cai C, Lu H. Prior-aware CNN with multi-task learning for colon images analysis. In: Proceedings of the 17th IEEE International Symposium on Biomedical Imaging. 2020, 254−257

[45] Misra I, Shrivastava A, Gupta A, Hebert M. Cross-stitch networks for multi-task learning. In: Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. 2016, 3994−4003

[46] Duan R, Chen N F. Unsupervised feature adaptation using adversarial multi-task training for automatic evaluation of children’s speech. In: Proceedings of the 21st Annual Conference of the International Speech Communication Association. 2020, 3037−3041

[47] Augenstein I, Søgaard A. Multi-task learning of keyphrase boundary classification. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics. 2017, 341−346

[48] Rai P, Daumé III H. Infinite predictor subspace models for multitask learning. In: Proceedings of the 13th International Conference on Artificial Intelligence and Statistics. 2010, 613−620

[49] Zhou L, Cui Z, Xu C, Zhang Z, Wang C, Zhang T, Yang J. Pattern-structure diffusion for multi-task learning. In: Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020, 4513−4522

[50] Wang Y, Luo X, Ding L, Fu S, Hu S . Multi-task non-negative matrix factorization for visual object tracking. Pattern Analysis and Applications, 2020, 23( 1): 493–507

[51] Jeong J Y, Jun C H. Sparse tensor decomposition for multi-task interaction selection. In: Proceedings of 2019 IEEE International Conference on Big Knowledge. 2019, 105−114

[52] Huang F, Qiu Y, Li Q, Liu S, Ni F . Predicting drug-disease associations via multi-task learning based on collective matrix factorization. Frontiers in Bioengineering and Biotechnology, 2020, 8: 218

[53] Zhang Y, Zhang Y, Wang W. Deep multi-task learning via generalized tensor trace norm. 2002, arXiv preprint arXiv: 2002.04799

[54] Chen Z, Lei H, Zhao Y, Huang Z, Xiao X, Lei Y, Tan E L, Lei B. Template-oriented multi-task sparse low-rank learning for parkinson’s diseases diagnosis. In: Proceedings of the 3rd International Workshop on PRedictive Intelligence in MEdicine. 2020, 178−187

[55] Wu X, Zhang X, Cen Y . Multi-task joint sparse and low-rank representation target detection for hyperspectral image. IEEE Geoscience and Remote Sensing Letters, 2019, 16( 11): 1756–1760

[56] Zhang Y, Yang Q. A survey on multi-task learning. IEEE Transactions on Knowledge and Data Engineering, 2021

[57] He K, Chen X, Xie S, Li Y, Dollár P, Girshick R. Masked autoencoders are scalable vision learners. In: Proceedings of 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022, 15979−15988

[58] Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. In: Proceedings of the 3rd International Conference on Learning Representations. 2015, 1556

[59] Jaworek-Korjakowska J, Kleczek P, Gorgon M. Melanoma thickness prediction based on convolutional neural network with VGG-19 model transfer learning. In: Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. 2019, 2748−2756

[60] Gao Y, Ma J, Zhao M, Liu W, Yuille A L. Nddr-CNN: layerwise feature fusing in multi-task CNNs by neural discriminative dimensionality reduction. In: Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019, 3200−3209

[61] Heuer F, Mantowsky S, Bukhari S S, Schneider G. Multitask-centerNet (MCN): efficient and diverse multitask learning using an anchor free approach. In: Proceedings of 2021 IEEE/CVF International Conference on Computer Vision Workshops. 2021, 997−1005

RIGHTS & PERMISSIONS

Higher Education Press