Weakly-supervised instance co-segmentation via tensor-based salient co-peak search

Abstract

Instance co-segmentation aims to segment the co-occurrent instances among two images. This task heavily relies on instance-related cues provided by co-peaks, which are generally estimated by exhaustively exploiting all paired candidates in point-to-point patterns. However, such patterns could yield a high number of false-positive co-peaks, resulting in over-segmentation whenever there are mutual occlusions. To tackle with this issue, this paper proposes an instance co-segmentation method via tensor-based salient co-peak search (TSCPS-ICS). The proposed method explores high-order correlations via triple-to-triple matching among feature maps to find reliable co-peaks with the help of co-saliency detection. The proposed method is shown to capture more accurate intra-peaks and inter-peaks among feature maps, reducing the false-positive rate of co-peak search. Upon having accurate co-peaks, one can efficiently infer responses of the targeted instance. Experiments on four benchmark datasets validate the superior performance of the proposed method.

Graphical abstract

Keywords

weakly-supervised / co-segmentation / co-peak / tensor matching / deep network / instance segmentation

Wuxiu Quan is currently pursuing the MS degree with the School of Computer Science and Engineering, South China University of Technology, China. His current research interests include deep learning, multiview clustering

Yu Hu is currently pursuing the PhD degree with School of Computer Science and Engineering, South China University of Technology, China. His research interests include machine learning, multi-view clustering, and deep clustering

Tingting Dan is currently pursuing the PhD degree with the School of Computer Science and Engineering, South China University of Technology, China. Her current research interests cover image processing and manifold learning

Junyu Li is currently pursuing the PhD degree in Computer Science and Engineering from South China University of Technology, China. His research interests include machine learning and image processing

Yue Zhang received the PhD degree in Computer Science from Hong Kong Baptist University, China in 2017. She is an Associate Professor with the School of Computer Science, Guangdong Polytechnic Normal University, China. Her research interests include bioinformatics and big data mining

Hongmin Cai is a Professor at the School of Computer Science and Engineering, South China University of Technology, China. He received the BS and MS degrees from the Harbin Institute of Technology, China in 2001 and 2003, respectively, and the PhD from Hong Kong University, China in 2007. His areas of research interests include biomedical image processing and omics data integration

References

[1] Chen H Y, Lin Y Y, Chen B Y . Co-segmentation guided Hough transform for robust feature matching. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 37( 12): 2388–2401

[2] Subramaniam A, Nambiar A, Mittal A. Co-segmentation inspired attention networks for video-based person re-identification. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019, 562−572

[3] Mustafa A, Hilton A. Semantically coherent co-segmentation and reconstruction of dynamic scenes. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017, 5583−5592

[4] Rother C, Minka T, Blake A, Kolmogorov V. Cosegmentation of image pairs by histogram matching - incorporating a global constraint into MRFs. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. 2006, 993−1000

[5] Wang F, Huang Q, Guibas L J. Image co-segmentation via consistent functional maps. In: Proceedings of the IEEE International Conference on Computer Vision. 2013, 849−856

[6] Taniai T, Sinha S N, Sato Y. Joint recovery of dense correspondence and cosegmentation in two images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016, 4246−4255

[7] Li L, Liu Z, Zhang J . Unsupervised image co-segmentation via guidance of simple images. Neurocomputing, 2018, 275: 1650–1661

[8] Tao G, Li H, Huang J, Han C, Chen J, Ruan G, Huang W, Hu Y, Dan T, Zhang B, He S, Liu L, Cai H . SeqSeg: A sequential method to achieve nasopharyngeal carcinoma segmentation free from background dominance. Medical Image Analysis, 2022, 78: 102381

[9] Li Y, Dan T, Li H, Chen J, Peng H, Liu L, Cai H . NPCNet: Jointly segment primary nasopharyngeal carcinoma tumors and metastatic lymph nodes in MR images. IEEE Transactions on Medical Imaging, 2022, 41( 7): 1639–1650

[10] Li Y, Peng H, Dan T, Hu Y, Tao G, Cai H. Coarse-to-fine nasopharyngeal carcinoma segmentation in MRI via multi-stage rendering. In: Proceedings of the IEEE International Conference on Bioinformatics and Biomedicine. 2020, 623−628

[11] Han J, Quan R, Zhang D, Nie F . Robust object co-segmentation using background prior. IEEE Transactions on Image Processing, 2018, 27( 4): 1639–1651

[12] Li W, Hosseini Jafari O, Rother C. Deep object co-segmentation. In: Proceedings of the 14th Asian Conference on Computer Vision. 2018, 638−653

[13] Zhang K, Chen J, Liu B, Liu Q. Deep object co-segmentation via spatial-semantic network modulation. In: Proceedings of the 34th AAAI Conference on Artificial Intelligence. 2020, 12813−12820

[14] Hsu K J, Lin Y Y, Chuang Y Y. DeepCO3: Deep instance co-segmentation by co-peak search and co-saliency detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019, 8838−8847

[15] Papandreou G, Chen L C, Murphy K P, Yuille A L. Weakly-and semi-supervised learning of a deep convolutional network for semantic image segmentation. In: Proceedings of the IEEE International Conference on Computer Vision. 2015, 1742−1750

[16] Lin D, Dai J, Jia J, He K, Sun J. ScribbleSup: Scribble-supervised convolutional networks for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016, 3159−3167

[17] Roy A, Todorovic S. Combining bottom-up, top-down, and smoothness cues for weakly supervised image segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017, 7282−7291

[18] Lai B, Gong X. Saliency guided dictionary learning for weakly-supervised image parsing. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016, 3630−3639

[19] Kolesnikov A, Lampert C H. Seed, expand and constrain: three principles for weakly-supervised image segmentation. In: Proceedings of the 14th European Conference on Computer Vision. 2016, 695−711

[20] Zitnick C L, Dollár P. Edge boxes: Locating object proposals from edges. In: Proceedings of the 13th European Conference on Computer Vision. 2014, 391−405

[21] Pont-Tuset J, Arbeláez P, Barron J T, Marques F, Malik J . Multiscale combinatorial grouping for image segmentation and object proposal generation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39( 1): 128–140

[22] Wan F, Wei P, Jiao J, Han Z, Ye Q. Min-entropy latent model for weakly supervised object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2018, 1297−1306

[23] Wei Y, Feng J, Liang X, Cheng M M, Zhao Y, Yan S. Object region mining with adversarial erasing: A simple classification to semantic segmentation approach. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017, 6488−6496

[24] Khan R A, Meyer A, Konik H, Bouakaz S . Saliency-based framework for facial expression recognition. Frontiers of Computer Science, 2019, 13( 1): 183–198

[25] Li T, Zhang K, Shen S, Liu B, Liu Q, Li Z . Image co-saliency detection and instance co-segmentation using attention graph clustering based graph convolutional network. IEEE Transactions on Multimedia, 2021, 24: 492–505

[26] Meng F, Luo K, Li H, Wu Q, Xu X . Weakly supervised semantic segmentation by a class-level multiple group cosegmentation and foreground fusion strategy. IEEE Transactions on Circuits and Systems for Video Technology, 2020, 30( 12): 4823–4836

[27] Gong X, Liu X, Li Y, Li H . A novel co-attention computation block for deep learning based image co-segmentation. Image and Vision Computing, 2020, 101: 103973

[28] Wei X S, Zhang C L, Li Y, Xie C W, Wu J, Shen C, Zhou Z H. Deep descriptor transforming for image co-localization. In: Proceedings of the 26th International Joint Conference on Artificial Intelligence. 2017, 3048−3054

[29] Wei X S, Zhang C L, Wu J, Shen C, Zhou Z H . Unsupervised object discovery and co-localization by deep descriptor transformation. Pattern Recognition, 2019, 88: 113–126

[30] Zhou Y, Zhu Y, Ye Q, Qiu Q, Jiao J. Weakly supervised instance segmentation using class peak response. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2018, 3791−3800

[31] Fan R, Cheng M M, Hou Q, Mu T J, Wang J, Hu S M. S4Net: single stage salient-instance segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019, 6096−6105

[32] Li G, Xie Y, Lin L, Yu Y. Instance-level salient object segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017, 247−256

[33] Aumüller M, Bernhardsson E, Faithfull A. ANN-benchmarks: a benchmarking tool for approximate nearest neighbor algorithms. In: Proceedings of the 10th International Conference on Similarity Search and Applications. 2017, 34−49

[34] Zass R, Shashua A. Probabilistic graph and hypergraph matching. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2008, 1−8

[35] Zhang D, Han J, Zhang Y. Supervision by fusion: Towards unsupervised learning of deep salient object detector. In: Proceedings of the IEEE International Conference on Computer Vision. 2017, 4068−4076

[36] Tang K, Joulin A, Li L J, Li F F. Co-localization in real-world images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2014, 1464−1471

[37] Cho M, Kwak S, Schmid C, Ponce J. Unsupervised object discovery and localization in the wild: part-based matching with bottom-up region proposals. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2015, 1201−1210

[38] Collins E, Achanta R, Susstrunk S. Deep feature factorization for concept discovery. In: Proceedings of the 15th European Conference on Computer Vision. 2018, 352−368

[39] Luo Z, Mishra A, Achkar A, Eichel J, Li S, Jodoin P M. Non-local deep features for salient object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017, 6593−6601

[40] Li X, Yang F, Cheng H, Liu W, Shen D. Contour knowledge transfer for salient object detection. In: Proceedings of the 15th European Conference on Computer Vision. 2018, 370−385

[41] Liu N, Zhao W, Shao L, Han J . SCG: Saliency and contour guided salient instance segmentation. IEEE Transactions on Image Processing, 2021, 30: 5862–5874

[42] Wang X, Feng J, Hu B, Ding Q, Ran L, Chen X, Liu W. Weakly-supervised instance segmentation via class-agnostic learning with salient images. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021, 10220−10230

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China (Grant Nos. U21A20520, 62172112), the Key-Area Research and Development of Guangdong Province (2022A0505050014, 2020B1111190001), the National Key Research and Development Program of China (2022YFE0112200), and the Key-Area Research and Development Program of Guangzhou City (202206030009).

RIGHTS & PERMISSIONS

2024 Higher Education Press