ICA-Net: improving class activation for weakly supervised semantic segmentation via joint contrastive and simulation learning

Abstract

In the field of optoelectronics, certain types of data may be difficult to accurately annotate, such as high-resolution optoelectronic imaging or imaging in certain special spectral ranges. Weakly supervised learning can provide a more reliable approach in these situations. Current popular approaches mainly adopt the classification-based class activation maps (CAM) as initial pseudo labels to solve the task. However, they may fail to estimate the complete object regions, especially in cases of multiple categories existing in one image or tight intersections existing among multiple categories. To solve the problem, we propose a two-branch framework with joint contrastive learning and simulation learning to mine more object regions and produce more complete CAM. Specifically, a contrastive learning branch is designed to learn class-independent activation maps from both foreground and background information, and an original CAM branch is served as supervision to provide accurate discriminative regions. Through simulation learning between the two branches, the enhanced activation maps, which are more complete to cover the objects, are achieved to generate accurate pixel-level pseudo labels. In addition, in order to actively provide important features for contrastive learning, we enhance the backbone in the contrastive learning branch via spatial and channel attention mechanisms. Extensive experiments on the PASCAL VOC 2012 and CUB-200-2011 benchmarks demonstrate that the proposed ICA-Net outperforms many state-of-the-art methods and achieves leading performance.

References

[1] Zhang J H, Gui M P, Wang Q C, et al.. Hierarchical topic model based object association for semantic SLAM. IEEE transactions on visualization and computer graphics, 2019, 25(11): 3052-3062 J]

[2] Zhang J H, Liu J L, Chen K Q, et al.. Map recovery and fusion for collaborative augment reality of multiple mobile devices. IEEE transactions on industrial informatics, 2020, 17(3): 2081-2089 J]

[3] Jiang P T, Yang Y Q. A simple local-to-global knowledge transfer framework for weakly supervised semantic segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 19–24, 2022, New Orleans, LA, USA, 2022, New York, IEEE 16886-16896 [C]

[4] Kim B Y, Han S, et al.. Discriminative region suppression for weakly-supervised semantic segmentation. Proceedings of the AAAI Conference on Artificial Intelligence, February 2–9, 2021, Virtual, 2021, Washington, AAAI Press 1754-1761 [C]

[5] Lee J B, Kim E J. Anti adversarially manipulated attributions for weakly and semisupervised semantic segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 19–25, 2021, Virtual, 2021, New York, IEEE 4071-4080 [C]

[6] Kim D, Lee S, Choe J, et al.. Weakly supervised semantic segmentation for driving scenes. Proceedings of the AAAI Conference on Artificial Intelligence, February 20–27, 2024, Vancouver, Canada, 2024, New York, IEEE 2741-2749 [C]

[7] Zhang D, Zhang H W. Causal intervention for weakly supervised semantic segmentation. Advances in Neural Information Processing Systems, December 6–12, 2020, Virtual, 2020, Cambridge, MIT Press 655-666 [C]

[8] Jo S, Yu I J, Kim K S. MARS: model-agnostic biased object removal without additional supervision for weakly-supervised semantic segmentation. Proceedings of the IEEE/CVF International Conference on Computer Vision, October 2–6, 2023, Paris, France, 2023, New York, IEEE 00063 [C]

[9] Zhang Z, Ren Z, Tao C, et al.. GRASS: contrastive learning with gradient-guided sampling strategy for remote sensing image semantic segmentation. IEEE transactions on geoscience and remote sensing, 2023, 61: 1-14 [J]

[10] Krishna C, Ertunc E, Neerav K, et al.. Contrastive learning of global and local features for medical image segmentation with limited annotations. Advances in Neural Information Processing Systems, December 6–12, 2020, Virtual, 2020, Cambridge, MIT Press 12546-12558 [C]

[11] Sun G L, Wang W G. Mining cross-image semantics for weakly supervised semantic segmentation. European Conference on Computer Vision, August 23–28, 2020, Virtual, 2020, Heidelberg, Springer 347-365 [C]

[12] Xu L, Ouyang W L, Mohammed B, et al.. Leveraging auxiliary tasks with affinity learning for weakly supervised semantic segmentation. Proceedings of the IEEE/CVF International Conference on Computer Vision, October 11–17, 2021, Virtual, 2021, New York, IEEE 6984-6993 [C]

[13] Yao Y Z, Chen T, Xie G S, et al.. Nonsalient region object mining for weakly supervised semantic segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 19–25, 2021, Virtual, 2021, New York, IEEE 2623-2632 [C]

[14] Wu T, Huang J S, Gao G Y, et al.. Embedded discriminative attention mechanism for weakly supervised semantic segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 19–25, 2021, Virtual, 2021, New York, IEEE 16765-16774 [C]

[15] Xie J H, Hou X X, Ye K, et al.. CLIMS: cross language image matching for weakly supervised semantic segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 19–24, 2022, New Orleans, LA, USA, 2022, New York, IEEE 4483-4492 [C]

[16] Chen Q, Yang L X, Lai J H, et al.. Self-supervised image-specific prototype exploration for weakly supervised semantic segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 19–24, 2022, New Orleans, LA, USA, 2022, New York, IEEE 4288-4298 [C]

[17] Li YI, Duan Y Q, Kuang Z H, et al.. Uncertainty estimation via response scaling for pseudo-mask noise mitigation in weakly supervised semantic segmentation. Proceedings of the AAAI Conference on Artificial Intelligence, February 21–28, 2022, Virtual, 2022, Washington, AAAI Press 1447-1455 [C]

[18] Qin J, Wu J, Xiao X F, et al.. Activation modulation and recalibration scheme for weakly supervised semantic segmentation. Proceedings of the AAAI Conference on Artificial Intelligence, February 21–28, 2022, Virtual, 2022, Washington, AAAI Press 2117-2125 [C]

[19] Lee M H, Kim D S, Shim H J. Threshold matters in WSSS: manipulating the activation for the robust and accurate segmentation model against thresholds. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 19–24, 2022, New Orleans, LA, USA, 2022, New York, IEEE 4330-4339 [C]

[20] He K M, Zhang X Y, Ren S Q, et al.. Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, June 27–30, 2016, Las Vegas, NV, USA, 2016, New York, IEEE 770-778 [C]

[21] Jiwoon A, Sunghyun C, Suha K. Weakly supervised learning of instance segmentation with inter-pixel relations. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 16–20, 2019, Los Angeles, California, USA, 2019, New York, IEEE 2209-2218 [C]

[22] Wang Y D, Zhang J, Kan M N, et al.. Self-supervised equivariant attention mechanism for weakly supervised semantic segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 16–18, 2020, Virtual, 2020, New York, IEEE 12275-12284 [C]

[23] Chang Y T, Wang Q S, Hung W C, et al.. Weakly supervised semantic segmentation via sub-category exploration. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 14–19, 2020, Virtual, 2020, New York, IEEE 8991-9000 [C]