Weakly-supervised instance co-segmentation via tensor-based salient co-peak search

Abstract

Instance co-segmentation aims to segment the co-occurrent instances among two images. This task heavily relies on instance-related cues provided by co-peaks, which are generally estimated by exhaustively exploiting all paired candidates in point-to-point patterns. However, such patterns could yield a high number of false-positive co-peaks, resulting in over-segmentation whenever there are mutual occlusions. To tackle with this issue, this paper proposes an instance co-segmentation method via tensor-based salient co-peak search (TSCPS-ICS). The proposed method explores high-order correlations via triple-to-triple matching among feature maps to find reliable co-peaks with the help of co-saliency detection. The proposed method is shown to capture more accurate intra-peaks and inter-peaks among feature maps, reducing the false-positive rate of co-peak search. Upon having accurate co-peaks, one can efficiently infer responses of the targeted instance. Experiments on four benchmark datasets validate the superior performance of the proposed method.

Graphical abstract

Keywords

weakly-supervised / co-segmentation / co-peak / tensor matching / deep network / instance segmentation

References

[1] Chen H Y, Lin Y Y, Chen B Y . Co-segmentation guided Hough transform for robust feature matching. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 37( 12): 2388–2401

[2] Subramaniam A, Nambiar A, Mittal A. Co-segmentation inspired attention networks for video-based person re-identification. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019, 562−572

[3] Mustafa A, Hilton A. Semantically coherent co-segmentation and reconstruction of dynamic scenes. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017, 5583−5592

[4] Rother C, Minka T, Blake A, Kolmogorov V. Cosegmentation of image pairs by histogram matching - incorporating a global constraint into MRFs. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. 2006, 993−1000

[5] Wang F, Huang Q, Guibas L J. Image co-segmentation via consistent functional maps. In: Proceedings of the IEEE International Conference on Computer Vision. 2013, 849−856

[6] Taniai T, Sinha S N, Sato Y. Joint recovery of dense correspondence and cosegmentation in two images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016, 4246−4255

[7] Li L, Liu Z, Zhang J . Unsupervised image co-segmentation via guidance of simple images. Neurocomputing, 2018, 275: 1650–1661

[8] Tao G, Li H, Huang J, Han C, Chen J, Ruan G, Huang W, Hu Y, Dan T, Zhang B, He S, Liu L, Cai H . SeqSeg: A sequential method to achieve nasopharyngeal carcinoma segmentation free from background dominance. Medical Image Analysis, 2022, 78: 102381

[9] Li Y, Dan T, Li H, Chen J, Peng H, Liu L, Cai H . NPCNet: Jointly segment primary nasopharyngeal carcinoma tumors and metastatic lymph nodes in MR images. IEEE Transactions on Medical Imaging, 2022, 41( 7): 1639–1650

[10] Li Y, Peng H, Dan T, Hu Y, Tao G, Cai H. Coarse-to-fine nasopharyngeal carcinoma segmentation in MRI via multi-stage rendering. In: Proceedings of the IEEE International Conference on Bioinformatics and Biomedicine. 2020, 623−628

[11] Han J, Quan R, Zhang D, Nie F . Robust object co-segmentation using background prior. IEEE Transactions on Image Processing, 2018, 27( 4): 1639–1651

[12] Li W, Hosseini Jafari O, Rother C. Deep object co-segmentation. In: Proceedings of the 14th Asian Conference on Computer Vision. 2018, 638−653

[13] Zhang K, Chen J, Liu B, Liu Q. Deep object co-segmentation via spatial-semantic network modulation. In: Proceedings of the 34th AAAI Conference on Artificial Intelligence. 2020, 12813−12820

[14] Hsu K J, Lin Y Y, Chuang Y Y. DeepCO3: Deep instance co-segmentation by co-peak search and co-saliency detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019, 8838−8847

[15] Papandreou G, Chen L C, Murphy K P, Yuille A L. Weakly-and semi-supervised learning of a deep convolutional network for semantic image segmentation. In: Proceedings of the IEEE International Conference on Computer Vision. 2015, 1742−1750

[16] Lin D, Dai J, Jia J, He K, Sun J. ScribbleSup: Scribble-supervised convolutional networks for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016, 3159−3167

[17] Roy A, Todorovic S. Combining bottom-up, top-down, and smoothness cues for weakly supervised image segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017, 7282−7291

[18] Lai B, Gong X. Saliency guided dictionary learning for weakly-supervised image parsing. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016, 3630−3639

[19] Kolesnikov A, Lampert C H. Seed, expand and constrain: three principles for weakly-supervised image segmentation. In: Proceedings of the 14th European Conference on Computer Vision. 2016, 695−711

[20] Zitnick C L, Dollár P. Edge boxes: Locating object proposals from edges. In: Proceedings of the 13th European Conference on Computer Vision. 2014, 391−405

[21] Pont-Tuset J, Arbeláez P, Barron J T, Marques F, Malik J . Multiscale combinatorial grouping for image segmentation and object proposal generation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39( 1): 128–140

[22] Wan F, Wei P, Jiao J, Han Z, Ye Q. Min-entropy latent model for weakly supervised object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2018, 1297−1306

[23] Wei Y, Feng J, Liang X, Cheng M M, Zhao Y, Yan S. Object region mining with adversarial erasing: A simple classification to semantic segmentation approach. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017, 6488−6496

[24] Khan R A, Meyer A, Konik H, Bouakaz S . Saliency-based framework for facial expression recognition. Frontiers of Computer Science, 2019, 13( 1): 183–198

[25] Li T, Zhang K, Shen S, Liu B, Liu Q, Li Z . Image co-saliency detection and instance co-segmentation using attention graph clustering based graph convolutional network. IEEE Transactions on Multimedia, 2021, 24: 492–505

[26] Meng F, Luo K, Li H, Wu Q, Xu X . Weakly supervised semantic segmentation by a class-level multiple group cosegmentation and foreground fusion strategy. IEEE Transactions on Circuits and Systems for Video Technology, 2020, 30( 12): 4823–4836

[27] Gong X, Liu X, Li Y, Li H . A novel co-attention computation block for deep learning based image co-segmentation. Image and Vision Computing, 2020, 101: 103973

[28] Wei X S, Zhang C L, Li Y, Xie C W, Wu J, Shen C, Zhou Z H. Deep descriptor transforming for image co-localization. In: Proceedings of the 26th International Joint Conference on Artificial Intelligence. 2017, 3048−3054

[29] Wei X S, Zhang C L, Wu J, Shen C, Zhou Z H . Unsupervised object discovery and co-localization by deep descriptor transformation. Pattern Recognition, 2019, 88: 113–126

[30] Zhou Y, Zhu Y, Ye Q, Qiu Q, Jiao J. Weakly supervised instance segmentation using class peak response. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2018, 3791−3800

[31] Fan R, Cheng M M, Hou Q, Mu T J, Wang J, Hu S M. S4Net: single stage salient-instance segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019, 6096−6105

[32] Li G, Xie Y, Lin L, Yu Y. Instance-level salient object segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017, 247−256

[33] Aumüller M, Bernhardsson E, Faithfull A. ANN-benchmarks: a benchmarking tool for approximate nearest neighbor algorithms. In: Proceedings of the 10th International Conference on Similarity Search and Applications. 2017, 34−49

[34] Zass R, Shashua A. Probabilistic graph and hypergraph matching. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2008, 1−8

[35] Zhang D, Han J, Zhang Y. Supervision by fusion: Towards unsupervised learning of deep salient object detector. In: Proceedings of the IEEE International Conference on Computer Vision. 2017, 4068−4076

[36] Tang K, Joulin A, Li L J, Li F F. Co-localization in real-world images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2014, 1464−1471

[37] Cho M, Kwak S, Schmid C, Ponce J. Unsupervised object discovery and localization in the wild: part-based matching with bottom-up region proposals. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2015, 1201−1210

[38] Collins E, Achanta R, Susstrunk S. Deep feature factorization for concept discovery. In: Proceedings of the 15th European Conference on Computer Vision. 2018, 352−368

[39] Luo Z, Mishra A, Achkar A, Eichel J, Li S, Jodoin P M. Non-local deep features for salient object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017, 6593−6601

[40] Li X, Yang F, Cheng H, Liu W, Shen D. Contour knowledge transfer for salient object detection. In: Proceedings of the 15th European Conference on Computer Vision. 2018, 370−385

[41] Liu N, Zhao W, Shao L, Han J . SCG: Saliency and contour guided salient instance segmentation. IEEE Transactions on Image Processing, 2021, 30: 5862–5874

[42] Wang X, Feng J, Hu B, Ding Q, Ran L, Chen X, Liu W. Weakly-supervised instance segmentation via class-agnostic learning with salient images. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021, 10220−10230

RIGHTS & PERMISSIONS

Higher Education Press