Remote sensing image semantic segmentation algorithm based on improved DeepLabv3+

Abstract

The convolutional neural network (CNN) method based on DeepLabv3+ has some problems in the semantic segmentation task of high-resolution remote sensing images, such as fixed receiving field size of feature extraction, lack of semantic information, high decoder magnification, and insufficient detail retention ability. A hierarchical feature fusion network (HFFNet) was proposed. Firstly, a combination of transformer and CNN architectures was employed for feature extraction from images of varying resolutions. The extracted features were processed independently. Subsequently, the features from the transformer and CNN were fused under the guidance of features from different sources. This fusion process assisted in restoring information more comprehensively during the decoding stage. Furthermore, a spatial channel attention module was designed in the final stage of decoding to refine features and reduce the semantic gap between shallow CNN features and deep decoder features. The experimental results showed that HFFNet had superior performance on UAVid, LoveDA, Potsdam, and Vaihingen datasets, and its cross-linking index was better than DeepLabv3+ and other competing methods, showing strong generalization ability.

Keywords

semantic segmentation / high-resolution remote sensing image / deep learning / transformer model / attention mechanism / feature fusion / encoder / decoder

References

[1] LI D R, WANG M, JIANG J. China’s high-resolution optical remote sensing satellites and their mapping applications. Geo-spatial Information Science, 2021, 24(1): 85-94.

[2] TIAN X, WANG L, DING Q. Review of image semantic segmentation based on deep learning. Journal of Software, 2019, 30(2): 440-468.

[3] AL-AMRI S S, KALYANKAR N V, KHAMITKAR S D, et al. Image segmentation by using threshold techniques. 2010: 1005.4020.

[4] COATES A, NG A Y. Learning feature representations with k-means. Neural Networks: Tricks of the Trade: Second Edition. Berlin, Heidelberg: Springer Berlin Heidelberg, 2012: 561-580.

[5] AL-AMRI M S S, KALYANKAR D N, KHAMITKAR S D DR. Image segmentation by using edge detection. International Journal on Computer Science and Engineering, 2010, 2(3): 804.

[6] PENG B, ZHANG L, ZHANG D. A survey of graph theoretical approaches to image segmentation. Pattern Recognition, 2013, 46(3): 1020-1038.

[7] CHEN L C, PAPANDREOU G, KOKKINOS I, et al. Semantic image segmentation with deep convolutional nets and fully connected CRFs. 2014: 1412.7062.

[8] CHEN L C, PAPANDREOU G, KOKKINOS I, et al. DeepLab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 40(4): 834-848.

[9] CHEN L C, PAPANDREOU G, SCHROFF F, et al. Rethinking atrous convolution for semantic image segmentation. 2017: 1706.05587.

[10] CHEN L C, ZHU Y K, PAPANDREOU G, et al. Encoder-decoder with atrous separable convolution for semantic image segmentation//Computer Vision-ECCV 2018, September 8-14, 2018, Munich, Germany, Cham: Springer International Publishing, 2018: 833-851.

[11] ZHENG S X, LU J C, ZHAO H S, et al. Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 20-25, 2021, Nashville, TN, USA. New York: IEEE, 2021: 6877-6886.

[12] DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An image is worth 16×16 words: Transformers for image recognition at scale.

[13] LIU Z, LIN Y T, CAO Y, et al. Swin transformer: hierarchical vision transformer using shifted windows//2021 IEEE/CVF International Conference on Computer Vision, October 10-17, 2021, Montreal, QC, Canada. New York: IEEE, 2021: 9992-10002.

[14] WANG L B, LI R, WANG D Z, et al. Transformer meets convolution: a bilateral awareness network for semantic segmentation of very fine resolution urban scene images. Remote Sensing, 2021, 13(16): 3065.

[15] GAO L, LIU H, YANG M H, et al. STransFuse: fusing swin transformer and convolutional neural network for remote sensing image semantic segmentation. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2021, 14: 10990-11003.

[16] XIE C X, XIA C Q, MA M C, et al. Pyramid grafting network for one-stage high resolution saliency detection//2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 18-24, 2022, New Orleans, LA, USA. New York: IEEE, 2022: 11707-11716.

[17] YANG J, JIN Y X, LIU Y B, et al. Research on fabric material recognition method based on improved transformer. Journal of North University of China (Natural Science Edition), 2023, 44(2): 138-145.

[18] HU J, SHEN L, SUN G. Squeeze-and-excitation networks//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 18-23, 2018, Salt Lake City, UT, USA. New York: IEEE, 2018: 7132-7141.

[19] LI R, ZHENG S Y, ZHANG C, et al. ABCNet: Attentive bilateral contextual network for efficient semantic segmentation of fine-resolution remotely sensed imagery. ISPRS Journal of Photogrammetry and Remote Sensing, 2021, 181: 84-98.

[20] LI R, ZHENG S Y, ZHANG C, et al. Multiattention network for semantic segmentation of fine-resolution remote sensing images. IEEE Transactions on Geoscience and Remote Sensing, 2021, 60: 5607713.

[21] WANG L B, LI R, ZHANG C, et al. UNetFormer: a UNet-like transformer for efficient semantic segmentation of remote sensing urban scene imagery. ISPRS Journal of Photogrammetry and Remote Sensing, 2022, 190: 196-214.

[22] HE J N, LI W J, QU J T. Shifted window SegFormer for remote sensing image segmentation//2023 8th International Conference on Intelligent Informatics and Biomedical Sciences, November 23-25, 2023, Okinawa, Japan. New York: IEEE, 2023: 117-121.

[23] LI Y X, HOU Q B, ZHENG Z H, et al. Large selective kernel network for remote sensing object detection//2023 IEEE/CVF International Conference on Computer Vision, October 1-6, 2023, Paris, France. New York: IEEE, 2023: 16748-16759.

[24] CHEN Y, FANG P, YU J, et al. Hi-ResNet: A High-Resolution Remote Sensing Network for Semantic Segmentation.