Imbalanced multi-instance multi-label learning via tensor product-based semantic fusion

Abstract

With powerful expressiveness of multi-instance multi-label learning (MIML) for objects with multiple semantics and its great flexibility for complex object structures, MIML has been widely applied to various applications. In practical MIML tasks, the naturally skewed label distribution and label interdependence bring up the label imbalance issue and decrease model performance, which is rarely studied. To solve these problems, we propose an imbalanced multi-instance multi-label learning method via tensor product-based semantic fusion (IMIML-TPSF) to deal with label interdependence and label distribution imbalance simultaneously. Specifically, to reduce the effect of label interdependence, it models similarity between the query object and object sets of different label classes for similarity-structural features. To alleviate disturbance caused by the imbalanced label distribution, it establishes the ensemble model for imbalanced distribution features. Subsequently, IMIML-TPSF fuses two types of features by tensor product and generates the new feature vector, which can preserve the original and interactive feature information for each bag. Based on such features with rich semantics, it trains the robust generalized linear classification model and further captures label interdependence. Extensive experimental results on several datasets validate the effectiveness of IMIML-TPSF against state-of-the-art methods.

Graphical abstract

Keywords

multi-instance multi-label learning / tensor product fusion / similarity-based learning / imbalanced learning / feature mapping

References

[1] Yang S J, Jiang Y, Zhou Z H. Multi-instance multi-label learning with weak label. In: Proceedings of the 23rd International Joint Conference on Artificial Intelligence. 2013, 1862–1868

[2] Chen A, Dhingra B. Hierarchical multi-instance multi-label learning for detecting propaganda techniques. In: Proceedings of the 8th Workshop on Representation Learning for NLP. 2023, 155–163

[3] Julia, Da Silva E. A deep learning system to perform multi-instance multi-label event classification in video game footage. Universidade Federal de Uberlandia, Dissertation, 2022

[4] Pan Z, Wang B, Zhang R, Wang S, Li Y, Li Y. MIML-GAN: a GAN-based algorithm for multi-instance multi-label learning on overlapping signal waveform recognition. IEEE Transactions on Signal Processing, 2023, 71: 859−872

[5] Loukas C, Sgouros N P . Multi-instance multi-label learning for surgical image annotation. The International Journal of Medical Robotics and Computer Assisted Surgery, 2020, 16( 2): e2058

[6] Lai Q, Zhou J, Gan Y, Vong C M, Chen C L P. Single-stage broad multi-instance multi-label learning (BMIML) with diverse inter-correlations and its application to medical image classification. IEEE Transactions on Emerging Topics in Computational Intelligence, 2024, 8(1): 828−839

[7] Zhou Z H, Zhang M L, Huang S J, Li Y F . Multi-instance multi-label learning. Artificial Intelligence, 2012, 176( 1): 2291–2320

[8] Wu J S, Huang S J, Zhou Z H . Genome-wide protein function prediction through multi-instance multi-label learning. IEEE/ACM Transactions on Computational Biology and Bioinformatics, 2014, 11( 5): 891–902

[9] Liu J, Tang X, Cui S, Guan X . Predicting the function of rice proteins through multi-instance multi-label learning based on multiple features fusion. Briefings in Bioinformatics, 2022, 23( 3): bbac095

[10] Li Y F, Hu J H, Jiang Y, Zhou Z H. Towards discovering what patterns trigger what labels. In: Proceedings of the 26th AAAI Conference on Artificial Intelligence. 2012, 1012–1018

[11] Yang M, Tang W T, Min F. Multi-instance multi-label learning based on parallel attention and local label manifold correlation. In: Proceedings of the 9th IEEE International Conference on Data Science and Advanced Analytics. 2022, 1–10

[12] Qiu S, Wang M, Yang Y, Yu G, Wang J, Yan Z, Domeniconi C, Guo M . Meta multi-instance multi-label learning by heterogeneous network fusion. Information Fusion, 2023, 94: 272–283

[13] Su C, Yan Z, Yu G . Cost-effective multi-instance multilabel active learning. International Journal of Intelligent Systems, 2021, 36( 12): 7177–7203

[14] Yu G, Xing Y, Wang J, Domeniconi C, Zhang X . Multiview multi-instance multilabel active learning. IEEE Transactions on Neural Networks and Learning Systems, 2022, 33( 9): 4311–4321

[15] Sánchez J, Perronnin F, Mensink T, Verbeek J . Image classification with the fisher vector: theory and practice. International Journal of Computer Vision, 2013, 105( 3): 222–245

[16] Newton J. Statistical analysis of finite mixture distributions. Journal of the International Biometric Society, 1986, 42(3): 679−680

[17] Ma Z, Chen S . A similarity-based framework for classification task. IEEE Transactions on Knowledge and Data Engineering, 2023, 35( 5): 5438–5443

[18] Poladi I, Ishwardas H . Review paper on error correcting output code based on multiclass classification. International Journal of Scientific Research, 2013, 2( 2): 134–136

[19] Zadeh A, Chen M, Poria S, Cambria E, Morency L P. Tensor fusion network for multimodal sentiment analysis. In: Proceedings of 2017 Conference on Empirical Methods in Natural Language Processing. 2017, 1103–1114

[20] Liu J, Ji S, Ye J. SLEP: Sparse Learning with Efficient Projections. Phoenix City: Arizona State University, 2009

[21] Zhou Z L, Zhang M L. Multi-instance multi-label learning with application to scene classification. In: Proceedings of the 20th Annual Conference on Neural Information Processing Systems. 2006, 1609–1616

[22] Zhang M L, Zhou Z H. M3MIML: a maximum margin method for multi-instance multi-label learning. In: Proceedings of the 8th IEEE International Conference on Data Mining. 2008, 688–697

[23] Maron O, Ratan A L. Multiple-instance learning for natural scene classification. In: Proceedings of the 15th International Conference on Machine Learning. 1998, 341–349

[24] Andrews S, Tsochantaridis I, Hofmann T. Support vector machines for multiple-instance learning. In: Proceedings of the 15th International Conference on Neural Information Processing Systems. 2002, 577–584

[25] Briggs F, Fern X Z, Raich R. Rank-loss support instance machines for MIML instance annotation. In: Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 2012, 534–542

[26] Frey P W, Slate D J . Letter recognition using Holland-style adaptive classifiers. Machine Learning, 1991, 6( 2): 161–182

[27] Lin T Y, Maire M, Belongie S, Hays J, Perona P, Ramanan D, Dollár P, Zitnick C L. Microsoft COCO: common objects in context. In: Proceedings of the 13th European Conference on Computer Vision. 2014, 740–755

[28] Luo T, Zhang W, Qiu S, Yang Y, Yi D, Wang G, Ye J, Wang J. Functional annotation of human protein coding isoforms via non-convex multi-instance learning. In: Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 2017, 345–354

[29] Panwar B, Menon R, Eksi R, Li H D, Omenn G S, Guan Y . Genome-wide functional annotation of human protein-coding splice variants using multiple instance learning. Journal of Proteome Research, 2016, 15( 6): 1747–1753

[30] The ENCODE Project Consortium . An integrated encyclopedia of DNA elements in the human genome. Nature, 2012, 489( 7414): 57–74

[31] Charte F, Rivera A, Del Jesus M J, Herrera F. A first approach to deal with imbalance in multi-label datasets. In: Proceedings of the 8th International Conference on Hybrid Artificial Intelligent Systems. 2013, 150–160

[32] Charte F, Rivera A, Del Jesus M J, Herrera F. Concurrence among imbalanced labels and its influence on multilabel resampling algorithms. In: Proceedings of the 9th International Conference on Hybrid Artificial Intelligence Systems. 2014, 110–121

[33] Liu Z, Wei P, Jiang J, Cao W, Bian J, Chang Y. MESA: boost ensemble imbalanced learning with meta-sampler. In: Proceedings of the 34th International Conference on Neural Information Processing Systems. 2020, 1212

[34] Shan J, Hou C, Tao H, Zhuge W, Yi D . Randomized multi-label subproblems concatenation via error correcting output codes. Neurocomputing, 2020, 410: 317–327

[35] Liu X Y, Wang S T, Zhang M L . Transfer synthetic over-sampling for class-imbalance learning with limited minority class data. Frontiers of Computer Science, 2019, 13( 5): 996–1009

[36] Sağlam F, Cengiz M A . A novel SMOTE-based resampling technique trough noise detection and the boosting procedure. Expert Systems with Applications, 2022, 200: 117023

[37] Yu D, Wang L, Chen X, Chen J . Using BiLSTM with attention mechanism to automatically detect self-admitted technical debt. Frontiers of Computer Science, 2021, 15( 4): 154208

[38] Liu J, Feng R, Chen P, Wang X, Ni Y . Dynamic loss reweighting method based on cumulative classification scores for long-tailed remote sensing image classification. Remote Sensing, 2023, 15( 2): 394

[39] Ji Z, Ni J, Liu X, Pang Y . Teachers cooperation: team-knowledge distillation for multiple cross-domain few-shot learning. Frontiers of Computer Science, 2023, 17( 2): 172312

[40] Wu Y, Dong G, Liang L, Zhao Y, Zhang K . Group-wise co-salient object detection via multi-view self-labeling novel class discovery. Frontiers of Computer Science, 2024, 18( 2): 182709

[41] Guo W, Zhuang F Z, Zhang X, Tong Y Q, Dong J . A comprehensive survey of federated transfer learning: challenges, methods and applications. Frontiers of Computer Science, 2024, 18( 6): 186356

RIGHTS & PERMISSIONS

Higher Education Press