Underwater ecological assessment via intelligent monitoring robot based on semi-supervised semantic segmentation

Abstract

Efficient and accurate underwater target recognition is crucial for assessing the stability of marine ecosystems. However, traditional manual diving surveys are costly, time-consuming and limited in coverage. With the advent of new AI technologies, intelligent underwater robots have emerged as a novel solution to this problem, but their perceptual capabilities usually rely on extensive and accurate annotated data, which is extremely challenging for underwater datasets with poor imaging quality and scarce pixel-level annotated samples. To address the above issues, this paper proposes a semi-supervised semantic segmentation framework specifically used for underwater ecological monitoring. Specifically, a multi-modal perturbation consistency method based on a multi-branch structure is constructed to simultaneously process multiple input modalities and extract intermediate features. A novel pseudo-label screening strategy based on semantic similarity is proposed, which utilizes the mapping relationship between labeled and unlabeled images to screen reliable pseudo-labels. Experiments were conducted on three underwater scene datasets, and tests were carried out on our self-developed underwater platform. The mIoU values on the SUIM dataset under 1/2 and 1/4 splits reached 72.06% and 71.45% respectively, outperforming the existing mainstream methods. In the future, this method can be deployed on underwater platforms to provide technical support for the implementation of automated and large-scale marine ecological monitoring systems. The codes are available at: https://github.com/A0268/uwea.

Keywords

Marine ecological monitoring / Semi-supervised semantic segmentation / Underwater image processing / Underwater perception robot

References

[1] H.N. Pham, K.B. Dang, T.V. Nguyen, N.C. Tran, X.Q. Ngo, D.A. Nguyen, T.T.H. Phan, T.T. Nguyen, W. Guo, H.H. Ngo, A new deep learning approach based on bilateral semantic segmentation models for sustainable estuarine wetland ecosystem management, Sci. Total Environ. 838 (2022) 155826.

[2] Y. Lu, J. Yuan, D. Du, B. Sun, X. Yi, Monitoring long-term ecological impacts from release of fukushima radiation water into ocean, Geogr. Sustain. 2 (2) (2021) 95-98.

[3] C. Zhuang, T. Ma, B. Xuan, C. Chang, B. An, M. Yin, H. Sun, Deep learning-based semantic segmentation of human features in bath scrubbing robots, Biomim. Intell. Robot. 4 (1) (2024) 100143.

[4] J. XIE, J. Di, Y. QIN, Application of deep learning in underwater imaging (invited), Acta Photonica Sin. 51 (11) (2022) 1101001.

[5] X. Zhao, L. Lai, Y. Li, X. Zhou, X. Cheng, Y. Chen, H. Huang, J. Guo, G. Wang, A lightweight bladder tumor segmentation method based on attention mechanism, Med. Biol. Eng. Comput. 62 (5) (2024) 1519-1534.

[6] X. Yang, Z. Song, I. King, Z. Xu, A survey on deep semi-supervised learning, IEEE Trans. Knowl. Data Eng. 35 (9) (2022) 8934-8954.

[7] L. Chen, H. Peng, D. Yang, T. Wang, An attachment recognition method based on semi-supervised video segmentation for tidal stream turbines, Ocean Eng. 293 (2024) 116466.

[8] H. Wang, Q. Zhang, Y. Li, X. Li, AllSpark: Reborn Labeled Features from Unlabeled in Transformer for Semi-Supervised Semantic Segmentation, in: 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, 2024, pp. 3627-3636.

[9] X. Lu, L. Jiao, L. Li, F. Liu, X. Liu, S. Yang, Z. Feng, P. Chen, Weak-to-strong consistency learning for semisupervised image segmentation, IEEE Trans. Geosci. Remote Sens. 61 (2023) 1-15.

[10] H. Mai, R. Sun, T. Zhang, F. Wu, RankMatch: Exploring the Better Consistency Regularization for Semi-supervised Semantic Segmentation, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 3391-3401.

[11] Y. Wang, H. Wang, Y. Shen, J. Fei, W. Li, G. Jin, L. Wu, R. Zhao, X. Le, Semi-supervised semantic segmentation using unreliable pseudo-labels, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 4248-4257.

[12] H. Hu, F. Wei, H. Hu, Q. Ye, J. Cui, L. Wang, Semi-supervised semantic segmentation via adaptive equalization learning, Adv. Neural Inf. Process. Syst. 34 (2021) 22106-22118.

[13] D. Guan, J. Huang, A. Xiao, S. Lu, Unbiased subclass regularization for semi-supervised semantic segmentation, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 9968-9978.

[14] K. Sohn, D. Berthelot, N. Carlini, Z. Zhang, H. Zhang, C.A. Raffel, E.D. Cubuk, A. Kurakin, C.-L. Li, Fixmatch: Simplifying semi-supervised learning with consistency and confidence, Adv. Neural Inf. Process. Syst. 33 (2020) 596-608.

[15] S. An, H. Zhu, J. Zhang, J. Ye, S. Wang, J. Yin, H. Zhang, Deep tri-training for semi-supervised image segmentation, IEEE Robot. Autom. Lett. 7 (4) (2022) 10097-10104.

[16] J. Peng, G. Estrada, M. Pedersoli, C. Desrosiers, Deep co-training for semi-supervised image segmentation, Pattern Recognit. 107 (2020) 107269.

[17] Y. Liu, Y. Tian, Y. Chen, F. Liu, V. Belagiannis, G. Carneiro, Perturbed and Strict Mean Teachers for Semi-supervised Semantic Segmentation, in: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, 2022, pp. 4248-4257.

[18] R. He, J. Yang, X. Qi, Re-distributing biased pseudo labels for semi-supervised semantic segmentation: A baseline investigation, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 6930-6940.

[19] B. Zoph, G. Ghiasi, T.-Y. Lin, Y. Cui, H. Liu, E.D. Cubuk, Q. Le, Rethinking pre-training and self-training, Adv. Neural Inf. Process. Syst. 33 (2020) 3833-3845.

[20] L. Yang, W. Zhuo, L. Qi, Y. Shi, Y. Gao, St++: Make self-training work better for semi-supervised semantic segmentation, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 4268-4277.

[21] K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 770-778.

[22] E. Xie, W. Wang, Z. Yu, A. Anandkumar, J.M. Alvarez, P. Luo, SegFormer: Simple and efficient design for semantic segmentation with transformers, Adv. Neural Inf. Process. Syst. 34 (2021) 12077-12090.

[23] E.M. Ditria, C.A. Buelow, M. Gonzalez-Rivero, R.M. Connolly, Artificial intelligence and automated monitoring for assisting conservation of marine ecosystems: A perspective, Front. Mar. Sci. 9 (2022) 918104.

[24] Y. Liu, D. An, Y. Ren, J. Zhao, C. Zhang, J. Cheng, J. Liu, Y. Wei, DP-FishNet: Dual-path pyramid vision transformer-based underwater fish detection network, Expert Syst. Appl. 238 (2024) 122018.

[25] J. Liu, L. Zhang, Y. Li, H. Liu, Deep residual convolutional neural network based on hybrid attention mechanism for ecological monitoring of marine fishery, Ecol. Inform. 77 (2023) 102204.

[26] H. Zhou, M. Kong, H. Yuan, Y. Pan, X. Wang, R. Chen, W. Lu, R. Wang, Q. Yang, Real-time underwater object detection technology for complex underwater environments based on deep learning, Ecol. Inform. 82 (2024) 102680.

[27] S. Yuan, Y. Li, F. Bao, H. Xu, Y. Yang, Q. Yan, S. Zhong, H. Yin, J. Xu, Z. Huang, et al., Marine environmental monitoring with unmanned vehicle platforms: Present applications and future prospects, Sci. Total Environ. 858 (2023) 159741.

[28] S. Lian, H. Li, R. Cong, S. Li, W. Zhang, S. Kwong, Watermask: Instance segmentation for underwater imagery, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 1305-1315.

[29] M.J. Islam, C. Edge, Y. Xiao, P. Luo, M. Mehtaz, C. Morse, S.S. Enan, J. Sattar, Semantic segmentation of underwater imagery: Dataset and benchmark, 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS, IEEE, 2020, pp. 1769-1776.

[30] A. Abdullah, T. Barua, R. Tibbetts, Z. Chen, M.J. Islam, I. Rekleitis, Caveseg: Deep semantic segmentation and scene parsing for autonomous underwater cave exploration, 2024 IEEE International Conference on Robotics and Automation, ICRA, IEEE, 2024, pp. 3781-3788.

[31] O. Álvarez-Tuñón, L.R. Marnet, M. Aubard, L. Antal, M. Costa, Y. Brodskiy, SubPipe: A submarine pipeline inspection dataset for segmentation and visual-inertial localization, OCEANS 2024-Singapore, IEEE, 2024, pp. 1-7.

[32] Z. Fu, R. Chen, Y. Huang, E. Cheng, X. Ding, K.-K. Ma, Masnet: A robust deep marine animal segmentation network, IEEE J. Ocean. Eng., (2023).

[33] X. Luo, M. Hu, T. Song, G. Wang, S. Zhang, Semi-supervised medical image segmentation via cross teaching between cnn and transformer, International Conference on Medical Imaging with Deep Learning, PMLR, 2022, pp. 820-833.

[34] R. Liu, T. Luo, S. Huang, Y. Wu, Z. Jiang, H. Zhang, CrossMatch: Cross-view matching for semi-supervised remote sensing image segmentation, IEEE Trans. Geosci. Remote Sens. 62 (2024) 1-15.

[35] Y. Chen, X. Ouyang, K. Zhu, G. Agam, ComplexMix: Semi-supervised semantic segmentation via mask-based data augmentation, 2021 IEEE International Conference on Image Processing, ICIP, 2021, pp. 2264-2268.

[36] Y. Ouali, C. Hudelot, M. Tami, Semi-supervised semantic segmentation with cross-consistency training, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 12674-12684.

[37] L. Yang, L. Qi, L. Feng, W. Zhang, Y. Shi, Revisiting weak-to-strong consistency in semi-supervised semantic segmentation, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 7236-7246.

[38] X. Chen, Y. Yuan, G. Zeng, J. Wang, Semi-supervised semantic segmentation with cross pseudo supervision, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 2613-2622.

[39] L. Wu, L. Fang, X. He, M. He, J. Ma, Z. Zhong, Querying labeled for unlabeled: Cross-image semantic consistency guided semi-supervised semantic segmentation, IEEE Trans. Pattern Anal. Mach. Intell. 45 (7) (2023) 8827-8844.

[40] C.O. Ancuti, C. Ancuti, C. De Vleeschouwer, P. Bekaert, Color balance and fusion for underwater image enhancement, IEEE Trans. Image Process. 27 (1) (2017) 379-393.

[41] N. Qin, J. Wu, X. Liu, Z. Lin, Z. Wang, Mcrnet: Underwater image enhancement using multi-color space residual network, Biomim. Intell. Robot. 4 (3) (2024) 100169.

[42] K. He, J. Sun, X. Tang, Single image haze removal using dark channel prior, IEEE Trans. Pattern Anal. Mach. Intell. 33 (12) (2010) 2341-2353.

[43] J. Zhou, L. Pang, D. Zhang, W. Zhang, Underwater image enhancement method via multi-interval subhistogram perspective equalization, IEEE J. Ocean. Eng. 48 (2) (2023) 474-488.

[44] Z. Zhang, C. Lan, W. Zeng, Z. Chen, Densely semantically aligned person re-identification, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 667-676.

[45] K. Zhu, H. Guo, S. Liu, J. Wang, M. Tang, Learning semantics-consistent stripes with self-refinement for person re-identification, IEEE Trans. Neural Netw. Learn. Syst. 34 (11) (2023) 8531-8542.

[46] J. Snell, K. Swersky, R. Zemel, Prototypical networks for few-shot learning, Adv. Neural Inf. Process. Syst., 30 (2017).

[47] I. Kabir, S. Shaurya, V. Maigur, N. Thakurdesai, M. Latnekar, M. Raunak, D. Crandall, M.A. Reza, Few-Shot Segmentation and Semantic Segmentation for Underwater Imagery, in: 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS, 2023, pp. 11451-11457.

[48] L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, H. Adam, Encoder-decoder with atrous separable convolution for semantic image segmentation, in: Proceedings of the European Conference on Computer Vision, ECCV, 2018, pp. 801-818.

[49] B. Cheng, I. Misra, A.G. Schwing, A. Kirillov, R. Girdhar, Masked-attention mask transformer for universal image segmentation, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 1290-1299.

[50] W. Zhang, J. Pang, K. Chen, C.C. Loy, K-net: Towards unified image segmentation, Adv. Neural Inf. Process. Syst. 34 (2021) 10326-10338.

[51] X. Wang, B. Zhang, L. Yu, J. Xiao, Hunting Sparsity: Density-Guided Contrastive Learning for Semi-Supervised Semantic Segmentation, in: 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, 2023, pp. 3114-3123.