Non-negative matrix factorization based modeling and training algorithm for multi-label learning

Abstract

Multi-label learning is more complicated than single-label learning since the semantics of the instances are usually overlapped and not identical. The effectiveness of many algorithms often fails when the correlations in the feature and label space are not fully exploited. To this end, we propose a novel non-negative matrix factorization (NMF) based modeling and training algorithm that learns from both the adjacencies of the instances and the labels of the training set. In the modeling process, a set of generators are constructed, and the associations among generators, instances, and labels are set up, with which the label prediction is conducted. In the training process, the parameters involved in the process of modeling are determined. Specifically, an NMF based algorithm is proposed to determine the associations between generators and instances, and a non-negative least square optimization algorithm is applied to determine the associations between generators and labels. The proposed algorithm fully takes the advantage of smoothness assumption, so that the labels are properly propagated. The experimentswere carried out on six set of benchmarks. The results demonstrate the effectiveness of the proposed algorithms.

Keywords

multi-label learning / non-negative least square optimization / non-negative matrix factorization / smoothness assumption

References

[1] Zhang M L, Zhou Z H. A review on multi-label learning algorithms. IEEE Transactions on Knowledge and Data Engineering, 2014, 26(8): 1819–1837

[2] Zhang M L, Zhou Z H. Milti-label neural networks with applications to functional genomics and text categorization. IEEE Transactions on Knowledge and Data Engineering, 2006, 18(10): 1338–1351

[3] Lo H, Wang J, Wang H, Lin S. Cost sensitive multi-label learning for audio tag annotation and retrieval. IEEE Transactions on Multimedia, 2011, 13(3): 518–529

[4] Sanden C, Zhang J. Enhancing multi-label music genre classification through ensemble techniques. In: Proceedings of the 34th International ACMSIGIR Conference on Research and Development in Information Retrieval. 2011, 705–714

[5] Tang L, Rajan S, Narayanan V. Large scale multi-label classification via metalabeler. In: Proceedings of the 19th International Conference on World Wide Web. 2009, 211–220

[6] Gopal S, Yang Y. Multi-label classification with meta-level features. In: Proceedings of the 33rd International ACM SIGIR Conference on Research & Development in Information Retrieval. 2010, 315–322

[7] Zhu X, Ghahramani Z. Learning from labeled and unlabeled data with label propagation. Technical Report, 2002

[8] Read J, Martino L, Olmos P M, Luengo D. Scalable multi-output label prediction: from classifier chains to classifier trellises. Pattern Recognition, 2015, 48(6): 2096–2109

[9] Madjarov G, Gjorgjevikj D, Dzeroski S. Two stage architecture for multi-label learning. Pattern Recognition, 2012, 45(3): 1019–1034

[10] Teisseyre P. Feature ranking for multi-label classification usingMarkov networks. Neurocomputing, 2015, 205: 439–454

[11] Lee J, Kim D W. Memetic feature selection algorithm for multi-label classification. Information Sciences, 2015, 293: 80–96

[12] Zhu X, Lafferty J, Rosenfeld R. Semi-supervised learning with graphs. Carnegie Mellon University, Doctor Thesis, 2005

[13] Chapelle O, Schlkopf B, Zien A. Semi-Supervised Learning. MA: The MIT Press, 2006

[14] Hou P, Geng X, Zhang M L. Multi-label manifold learning. In: Proceedings of the 30th AAAI Conference on Artificial Intelligence. 2016, 1680–1686

[15] Gao N N, Huang S J, Chen S C. Multi-label active learning by model guided distribution matching. Frontiers of Computer Science, 2016, 10(5): 845–855

[16] Kong X, Ng M K, Zhou Z H. Transductive multi-label learning via label set propagation. IEEE Transactions on Knowledge and Data Engineering, 2013, 25(3): 704–719

[17] Huang S J, Yu Y, Zhou Z H. Multi-label hypothesis reuse. In: Proceedings of the 18th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2012, 525–533

[18] Huang S J, Zhou Z H. Multi-label learning by exploiting label correlations locally. In: Proceedings of the 26th AAAI Conference on Artificial Intelligence. 2012, 949–955

[19] Lo H, Lin S, Wang H. Generalized k-label sets ensemble for multi-label and cost-sensitive classification. IEEE Transactions on Knowledge and Data Engineering, 2014, 26(7): 1679–1691

[20] Lee J, Kim K, Kim N, Lee J H. An approach for multi-label classification by directed acyclic graph with label correlation maximization. Information Sciences, 2016, 351: 101–114

[21] Li P, Li H, Wu M. Multi-label ensemble based on variable pairwise constraint projection. Information Sciences, 2013, 222: 269–281

[22] Wang B, Tsotsos J. Dynamic label propagation for semi-supervised multi-class multi-label classification. Pattern Recognition, 2016, 52: 75–84

[23] Wang S, Wang J, Wang Z, Ji Q. Enhancing multi-label classification by modeling dependencies among labels. Pattern Recognition, 2014, 47(10): 3405–3413

[24] Pillai I, Fumera G, Roli F. Multi-label classification with a reject option. Pattern Recognition, 2013, 46(8): 2256–2266

[25] Sun F, Tang J, Li H, Qi G, Huang T S. Multi-label image categorization with sparse factor representation. IEEE Transactions on Image Processing, 2014, 23(3): 1028–1037

[26] Zhang Y, Zhou Z H. Multi-label dimensionality reduction via dependence maximization. ACM Transactions on Knowledge Discovery from Data, 2010, 4(3): 14

[27] Zhang M L, Zhang K. Multi-label learning by exploiting label dependency. In: Proceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 2010, 999–1007

[28] Zhang M L, Wu L. Lift: multi-label learning with label specific features. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 37(1): 107–120

[29] Triguero I, Vens C. Labelling strategies for hierarchical multi-label classification techniques. Pattern Recognition, 2016, 56: 170–183

[30] Quevedo J R, Luaces O, Bahamonde A. Multilabel classifiers with a probabilistic thresholding strategy. Pattern Recognition, 2012, 45(2): 876–883

[31] Zhang N, Ding S, Zhang J. Multi layer ELM-RBF for multi-label learning. Applied Soft Computing, 2016, 43: 535–545

[32] Huang Y, Wang W, Wang L. Unconstrained multimodal multi-label learning. IEEE Transactions on Multimedia, 2015, 17(11): 1923–1935

[33] Xu J, Jagadeesh V, Manjunath B S. Multi-label learning with fused multimodal Bi-relational graph. IEEE Transactions on Multimedia, 2014, 16(2): 403–412

[34] Gao W, Zhou Z. On the consistency of multi-label learning. Artificial Intelligence, 2013, 199(1): 22–44

[35] Madjarov G, Kocev D, Gjorgjevikj D, Dzeroski S. An extensive experimental comparison of methods for multi-label learning. Pattern Recognition, 2012, 45(9): 3084–3104

[36] Lee D D, Seung H S. Learning the parts of objects by non-negative matrix factorization. Nature, 1999, 401(6755): 788–791

[37] Lawson C L, Hanson R J. Solving Least Squares Problems. New Jersey: Prentice-Hall, Inc., 1974

[38] Joachims T. Transductive inference for text classification using support vector machines. In: Proceedings of the 16th International Conference on Machine Learning. 1999, 200–209

[39] Vapnik V N. Statistical Learning Theory. New York: Wiley, 1998

[40] Belkin M, Niyogi P, Sindhwani V. Manifold regularization: a geometric framework for learning from examples. Journal of Machine Learning Research, 2006, 7: 2399–2434

[41] Xu M, Jin R, Zhou Z H. Speedup matrix completion with side information: application to multi-label learning. In: Proceedings of the 27th Annual Conference on Neural Information Processing Systems. 2013, 2301–2309

[42] Read J, Pfahringer B, Holmes G, Frank E. Classifier chains for multilabel classification. Machine Learning, 2011, 85(3): 333–359

[43] Wang J, Zhao Y, Wu X, Hua X S. A transductive multi-label learning approach for video concept detection. Pattern Recognition, 2011, 44(10–11): 2274–2286

[44] Furnkranz J, Hullermeier E, Mencia E L, Brinker K. Multilabel classification via calibrated label ranking. Machine Learning, 2008, 73(2): 133–153

[45] Mencia E L, Park S H, Furnkranz J. Efficient voting prediction for pairwise multi-label classification. Neurocomputing, 2010, 73(7–9): 1164–1176

RIGHTS & PERMISSIONS

Higher Education Press and Springer-Verlag GmbH Germany, part of Springer Nature