DEGRNet: An innovative dual-branch edge-based graph reasoning network for the few-shot segmentation of industrial surface defects

Abstract

Pixel-wise segmentation techniques based on deep learning have been widely applied in the inspection of product surface defects to ensure product quality. However, existing models based on deep learning primarily focus on separate specific defect types, which introduces challenges in generalizing them to the detection of diverse product defects. The relatively low occurrence probability of certain defects also sets obstacles in obtaining sufficient defect samples for effective model training. Herein, an innovative dual-branch edge-based graph reasoning network (DEGRNet) is demonstrated for the few-shot segmentation (FSS) of industrial surface defects, which can be easily generalized to various defects with minimal labeled defect samples. DEGRNet mainly consists of a background eliminating (BE) module, an edge reinforcement (ER) module, and a graph reasoning module. The BE module can effectively fuse the feature information from the support image and support mask to reduce background interference, while the ER module works to accurately extract edge contour information. The rough segmentation maps from BE and the boundary-enhanced segmentation maps from ER are simultaneously input into the dual-branch graph reasoning module to enhance the modeling capability of long-distance feature information and refine segmentation boundaries. This feature enables our model to fully utilize global image features and gain robust generalization capabilities for unknown defect types. The results of multiple experiments validate the effectiveness of the as-proposed modules. Our model achieves state-of-the-art performance metrics in few-shot defect segmentations. Specifically, our model exhibits 2.61% and 3.50% improvements in mean intersection over union under the 1-shot and 5-shot conditions, respectively, compared with existing state-of-the-art FSS models.

Graphical abstract

Keywords

few-shot segmentation / defect inspection / DEGRNet / graph reasoning / background eliminating / edge reinforcement

References

[1] Luo Q W , Fang X X , Liu L , Yang C H , Sun Y C . Automated visual defect detection for flat steel surface: A survey. IEEE Transactions on Instrumentation and Measurement, 2020, 69(3): 626–644

[2] Xing J J , Jia M P . A convolutional neural network-based method for workpiece surface defect detection. Measurement, 2021, 176: 109185

[3] Jian C X , Gao J , Ao Y H . Automatic surface defect detection for mobile phone screen glass based on machine vision. Applied Soft Computing, 2017, 52: 348–358

[4] Feng H , Song K C , Cui W Q , Zhang Y M , Yan Y H . Cross position aggregation network for few-shot strip steel surface defect segmentation. IEEE Transactions on Instrumentation and Measurement, 2023, 72: 1–10

[5] Zuo L , Su S Q , Fan S H , Li H , Wen L , Li X Y , Xiong K G . A new dual-branch network with global information for the surface defect detection on solar PV wafer. IEEE Sensors Journal, 2024, 24(6): 9197–9207

[6] Yang X Q , Wen G J , Mei S , Dong H B , Liu X Y . DBHF: A double fusion backbone and bidirectional feature hybrid fusion detector for high-precision inspection of avionics solder joint defects. Measurement, 2024, 237: 115221

[7] Luo A S , Wen G J , Cheng Y H , Mei S , Dong H B , Liu X Y . DMMGNet: A discrimination mapping and memory bank mean guidance-based network for high-performance few-shot industrial anomaly detection. Neurocomputing, 2024, 610: 128622

[8] Yang X Q , Liu X Y , Wu Q , Wen G J , Mei S , Liao G L , Shi T L . VMMAO-YOLO: An ultra-lightweight and scale-aware detector for real-time defect detection of avionics thermistor wire solder joints. Frontiers of Mechanical Engineering, 2024, 19(3): 21

[9] LiuZLinY TCaoYHuHWeiY XZhangZLinSGuoB N. Swin transformer: Hierarchical vision transformer using shifted windows. In: 2021 IEEE/CVF International Conference on Computer Vision (ICCV). Montreal: IEEE, 2021, 9992–10002

[10] Wu Q W , Li H , Tian C Y , Wen L , Li X Y . AEKD: Unsupervised auto-encoder knowledge distillation for industrial anomaly detection. Journal of Manufacturing Systems, 2024, 73: 159–169

[11] Luo Q W , Fang X X , Sun Y C , Liu L , Ai J Q , Yang C H , Simpson O . Surface defect classification for hot-rolled steel strips by selectively dominant local binary patterns. IEEE Access: Practical Innovations, Open Solutions, 2019, 7: 23488–23499

[12] Xiao W W , Song K C , Liu J , Yan Y H . Graph embedding and optimal transport for few-shot classification of metal surface defect. IEEE Transactions on Instrumentation and Measurement, 2022, 71: 1–10

[13] NandG KNoopurNNeogi. Defect detection of steel surface using entropy segmentation. In: 2014 Annual IEEE India Conference (INDICON). Pune: IEEE, 2014, 1–6

[14] SimonyanKZissermanA. Very deep convolutional networks for large-scale image recognition. 2015, arXiv preprint arXiv:1409.1556

[15] LiC YLiL LJiangH LWengK HGengY FLiLKeZ DLiQ YChengMNieW QLiY DZhangBLiangY FZhouL YXuX MChuX XWeiX MWeiX L. YOLOv6: A single-stage object detection framework for industrial applications. 2022, arXiv preprint arXiv:2209.02976

[16] AboahAWangBBagciUAdu-GyamfiY. Real-time multi-class helmet violation detection using few-shot data sampling technique and YOLOv8. In: 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Vancouver: IEEE, 2023, 5350–5358

[17] Yu R Y , Guo B Y , Yang K . Selective prototype network for few-shot metal surface defect segmentation. IEEE Transactions on Instrumentation and Measurement, 2022, 71: 1–10

[18] LiuZMaoH ZWuC YFeichtenhoferCDarrellTXieS N. A convnet for the 2020s. In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). New Orleans: IEEE, 2022, 11966–11976

[19] KirillovAWuY XHeK MGirshickR. PointRend: Image segmentation as rendering. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Seattle: IEEE, 2020, 9796–9805

[20] ChenL CZhuY KPapandreouGSchroffFAdamH. Encoder-decoder with atrous separable convolution for semantic image segmentation. In: Ferrari V, Hebert M, Sminchisescu C, Weiss Y (eds). Computer Vision – ECCV 2018. Lecture Notes in Computer Science, vol 11211. Cham: Springer, 2018, 833–851

[21] ShabanABansalSLiuZEssaIBootsB. One-shot learning for semantic segmentation. 2017, arXiv preprint arXiv:1709.03410

[22] ZhangCLinG SLiuF YYaoRShenC H. CANet: Class-agnostic segmentation networks with iterative refinement and attentive few-shot learning. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach: IEEE, 2019, 5212–5221

[23] Tian Z T , Zhao H S , Shu M , Yang Z C , Li R Y , Jia J Y . Prior guided feature enrichment network for few-shot segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(2): 1050–1065

[24] WangK XLiewJ HZouY TZhouD QFengJ S. PANet: Few-shot image semantic segmentation with prototype alignment. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV). Seoul: IEEE, 2019, 9196–9205

[25] Zhang X L , Wei Y C , Yang Y , Huang T S . Sg-one: Similarity guidance network for one-shot semantic segmentation. IEEE Transactions on Cybernetics, 2020, 50(9): 3855–3865

[26] Bao Y Q , Song K C , Liu J , Wang Y Y , Yan Y H , Yu H , Li X J . Triplet-graph reasoning network for few-shot metal generic surface defect segmentation. IEEE Transactions on Instrumentation and Measurement, 2021, 70: 1–11

[27] LiXYangY BZhaoQ JShenT CLinZ CLiuH. Spatial pyramid based graph reasoning for semantic segmentation. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Seattle: IEEE, 2020, 8947–8956

[28] BianXLimS NZhouN. Multiscale fully convolutional network with application to industrial inspection. In: 2016 IEEE Winter Conference on Applications of Computer Vision (WACV). Lake Placid: IEEE, 2016, 1–8

[29] Song G R , Song K C , Yan Y H . EDRNet: Encoder–decoder residual network for salient object detection of strip steel surface defects. IEEE Transactions on Instrumentation and Measurement, 2020, 69(12): 9709–9719

[30] Cheng J T , Wen G J , He X , Liu X Y , Hu Y , Mei S . Achieving the defect transfer detection of semiconductor wafer by a novel prototype learning based semantic segmentation network. IEEE Transactions on Instrumentation and Measurement, 2024, 73: 1–12

[31] YangB YLiuCLiB HJiaoJ BYeQ X. Prototype mixture models for few-shot semantic segmentation. In: Vedaldi A, Bischof H, Brox T, Frahm J M (eds). Computer Vision – ECCV 2020. Lecture Notes in Computer Science, vol 12353. Cham: Springer, 2020, 763–778

[32] ZhangB FXiaoJ MQinT. Self-guided and cross-guided learning for few-shot segmentation. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Nashville: IEEE, 2021, 8308–8317

[33] KipfT NWellingM. Semi-supervised classification with graph convolutional networks. 2017, arXiv preprint arXiv:1609.02907

[34] ZhangCLinG SLiuF YGuoJ SWuQ YYaoR. Pyramid graph networks with connection attentions for region-based one-shot semantic segmentation. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV). Seoul: IEEE, 2019, 9586–9594

[35] HuH ZCuiJ SZhaH B. Boundary-aware graph convolution for semantic segmentation. In: the 25th International Conference on Pattern Recognition (ICPR). Milan: IEEE, 2021, 1828–1835

[36] Russakovsky O , Deng J , Su H , Krause J , Satheesh S , Ma S , Huang Z H , Karpathy A , Khosla A , Bernstein M , Berg A C , Li F F . Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 2015, 115(3): 211–252

[37] WooSParkJLeeJ YKweonI S. CBAM: Convolutional block attention module. In: Ferrari V, Hebert M, Sminchisescu C, Weiss Y (eds). Computer Vision – ECCV 2018. Lecture Notes in Computer Science, vol 11211. Cham: Springer, 2018, 3–19

[38] LuYChenY RZhaoD BChenJ X. Graph-FCN for image semantic segmentation. In: Lu H C, Tang H J, Wang Z S (eds). Advances in Neural Networks – ISNN 2019. Lecture Notes in Computer Science, vol 11554. Cham: Springer, 2019, 97–105

[39] Yang E Q , Zhou W J , Qian X H , Lei J S , Yu L . DRNet: Dual-stage refinement network with boundary inference for RGB-D semantic segmentation of indoor scenes. Engineering Applications of Artificial Intelligence, 2023, 125: 106729

[40] ZhangLLiX TArnabAYangK YTongY HTorrP H S. Dual graph convolutional network for semantic segmentation. 2020, arXiv preprint arXiv:1909.06121

[41] LiuW DWuZ HDingH HLiuF YLinJLinG SZhouW. Few-shot segmentation with global and local contrastive learning. 2025, arXiv preprint arXiv:2108.05293

RIGHTS & PERMISSIONS

Higher Education Press