3D vision-based anomaly detection in manufacturing: A survey

Abstract

Surface quality monitoring of manufacturing products is critical for manufacturing industries to ensure product quality and production efficiency. With the rapid development of 3D scanning technology, high-density 3D point cloud data can be generated by 3D scanners in complex manufacturing systems. However, due to the challenges of complex surface modeling and various types, it lacks effective surface anomaly detection methods that can meet the practical requirements regarding detection accuracy and speed. This survey aims to review the surface anomaly detection methodology of manufacturing products based on 3D machine vision. Specifically, the machine learning methodologies will be systematically reviewed for 3D point cloud data modeling and anomaly detection. Related public data sets for this research are also summarized. Finally, the future research directions are pointed out.

Graphical abstract

Keywords

anomaly detection / 3D Vision / manufacturing / machine learning

References

[1] BahreiniFHammad A (2021). Point cloud semantic segmentation of concrete surface defects using dynamic graph CNN. In: Proceedings of the International Symposium on Automation and Robotics in Construction, 38:379–386, IAARC Publications

[2] Bello S A, Yu S, Wang C, Adam J M, Li J, (2020). Deep learning on 3D point clouds. Remote Sensing, 12( 11): 1729

[3] BergmannPJin XSattleggerDStegerC (2022). The MVTec 3D-AD dataset for unsupervised 3D anomaly detection and localization: In: Proceedings of the 17th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, 202–213

[4] BergmannPSattlegger D (2023). Anomaly detection in 3D point clouds using deep geometric descriptors. In: 2023 IEEE/CVF Winter Conference on Applications of Computer Vision, 2612–2622

[5] Bolourian N, Nasrollahi M, Bahreini F, Hammad A, (2023). Point cloud–based concrete surface defect semantic segmentation. Journal of Computing in Civil Engineering, 37( 2): 04022056

[6] CaoXTaoC DuJ (2024a). 3D-CSAD: Untrained 3D anomaly detection for complex manufacturing surfaces. arXiv Preprint arXiv:2404.07748

[7] Cao Y, Xu X, Shen W, (2024b). Complementary pseudo multimodal feature for point cloud anomaly detection. Pattern Recognition, 156: 110761

[8] CaoYXuX ZhangJCheng YHuangXPangGShenW (2024c). A survey on visual anomaly detection: Challenge, approach, and prospect. arXiv Preprint arXiv:2401.16402

[9] ChangA XFunkhouser TGuibasLHanrahanPHuangQ LiZSavareseS SavvaMSong SSuHXiaoJYiL YuF (2015). ShapeNet: An information-rich 3D model repository: arXiv:1512.03012

[10] CharlesR QSu HKaichunMGuibasL J (2017). PointNet: Deep learning on point sets for 3D classification and segmentation. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition, 77–85

[11] Chen L, Yao X, Xu P, Moon S K, Bi G, (2021). Rapid surface defect identification for additive manufacturing with in-situ point cloud processing and machine learning. Virtual and Physical Prototyping, 16( 1): 50–67

[12] ChenRXie GLiuJWangJLuoZ WangJZheng F (2023). EasyNet: An easy network for 3D industrial anomaly detection. In: Proceedings of the 31st ACM International Conference on Multimedia, 7038–7046

[13] Cheng L, Tsung F, Wang A, (2017). A statistical transfer learning perspective for modeling shape deviations in additive manufacturing. IEEE Robotics and Automation Letters, 2( 4): 1988–1993

[14] Cheng L, Wang A, Tsung F, (2018). A prediction and compensation scheme for in-plane shape deviation of additive manufacturing with information on process parameters. IISE Transactions, 50( 5): 394–406

[15] ChuY MLiu CHsiehT IChenH TLiuT L (2023). Shape-guided dual-memory learning for 3D anomaly detection. In: Proceedings of the 40 th International Conference on Machine Learning

[16] Cohen J, Ni J, (2022). Semi-supervised learning for anomaly classification using partially labeled subsets. Journal of Manufacturing Science and Engineering, 144( 6): 061008

[17] Decker N, Wang Y, Huang Q, (2020). Efficiently registering scan point clouds of 3D printed parts for shape accuracy assessment and modeling. Journal of Manufacturing Systems, 56: 587–597

[18] Dong Q, Wang S, Chen X, Jiang W, Li R, Gu X, (2023). Pavement crack detection based on point cloud data and data fusion. Philosophical Transactions of the Royal Society of London. Series A, Mathematical and Physical Sciences, 381( 2254): 20220165

[19] Du J, Yan H, Chang T S, Shi J, (2022). A tensor voting-based surface anomaly classification approach by using 3D point cloud data. Journal of Manufacturing Science and Engineering, 144( 5): 051005

[20] Gao Y, Cao Z, Qin Y, Ge X, Lian L, Bai J, Yu H, (2023). Railway fastener anomaly detection via multi-sensor fusion and self-driven loss reweighting. IEEE Sensors Journal, 24( 2): 1812–1825

[21] Guo Y, Sohel F, Bennamoun M, Lu M, Wan J, (2013). Rotational projection statistics for 3D local surface description and object recognition. International Journal of Computer Vision, 105( 1): 63–86

[22] HaurumJAllahham MLyngeMHenriksenKNikolovI MoeslundT (2021). Sewer defect classification using synthetic point clouds. In: Proceedings of the 16th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, 891–900

[23] HeKZhangX RenSSunJ (2016). Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition, 770–778

[24] He Y, Ma W, Li Y, Hao C, Wang Y, Wang Y, (2023). An octree-based two-step method of surface defects detection for remanufacture. International Journal of Precision Engineering and Manufacturing-Green Technology, 10( 2): 311–326

[25] Hitchcox T, Zhao Y F, (2018). Random walks for unorganized point cloud segmentation with application to aerospace repair. Procedia Manufacturing, 26: 1483–1491

[26] HorwitzEHoshen Y (2023). Back to the feature: Classical 3D features are (almost) all you need for 3D anomaly detection. In: 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2968–2977

[27] HuQYangB XieLRosaS GuoYWangZ TrigoniNMarkham A (2020). RandLA-Net: Efficient semantic segmentation of large-scale point clouds. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 11105–11114

[28] Hu Q, Yang B, Xie L, Rosa S, Guo Y, Wang Z, Trigoni N, Markham A, (2021). Learning semantic egmentation of large-scale point clouds with random sampling. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44( 11): 8338–8354

[29] HuoLLiuY YangYZhuang ZSunM (2023). Review: Research on product surface quality inspection technology based on 3D point cloud. Advances in Mechanical Engineering, 15(3): 16878132231159523

[30] HuttonK T (2023). Anomaly Detection on Partial Point Clouds for the Purpose of Identifying Damage on the Exterior of Spacecrafts. Master’s thesis, The University of Western Ontario, Canada

[31] Jovančević I, Pham H H, Orteu J J, Gilblas R, Harvent J, Maurice X, Brèthes L, (2017). 3D point cloud analysis for detection and characterization of defects on airplane exterior surface. Journal of Nondestructive Evaluation, 36: 74

[32] Kaji F, Nguyen-Huu H, Budhwani A, Narayanan J A, Zimny M, Toyserkani E, (2022). A deep-learning-based in-situ surface anomaly detection methodology for laser directed energy deposition via powder feeding. Journal of Manufacturing Processes, 81: 624–637

[33] Koenderink J J, Van Doorn A J, (1992). Surface shape and curvature scales. Image and Vision Computing, 10( 8): 557–564

[34] Lee E T, Fan Z, Sencer B, (2023). A new approach to detect surface defects from 3D point cloud data with surface normal Gabor filter. Journal of Manufacturing Processes, 92: 196–205

[35] Li R, Jin M, Paquit V C, (2021). Geometrical defect detection for additive manufacturing with machine learning models. Materials & Design, 206: 109726

[36] LiWXuX (2023). Towards scalable 3D anomaly detection and localization: A benchmark via 3D anomaly synthesis and s self-supervised learning network. arXiv Preprint arXiv:2311.14897

[37] LiYZhangR LiHShaoX (2020). Dynamic attention graph convolution neural network of point cloud segmentation for defect detection. In: 2020 IEEE International Conference on Artificial Intelligence and Information Systems, 18–23

[38] Lin C H, Chen J Y, Su P L, Chen C H, (2014). Eigen-feature analysis of weighted covariance matrices for LiDAR point cloud classification. ISPRS Journal of Photogrammetry and Remote Sensing, 94: 70–79

[39] Liu J, Xie G, Chen R, Li X, Wang J, Liu Y, Wang C, Zheng F, (2024). Real3d-ad: A dataset of point cloud anomaly detection. Advances in Neural Information Processing Systems, 36: 30402–30415

[40] Madrigal C, Branch J, Restrepo A, Mery D, (2017). A method for automatic surface inspection using a model-based 3D descriptor. Sensors, 17( 10): 2262

[41] Miao Y, Fu R, Wu H, Hao M, Li G, Hao J, Zhou D, (2022). Pipeline of turbine blade defect detection based on local geometric pattern analysis. Engineering Failure Analysis, 133: 105965

[42] PangYWang WTayF E HLiuWTianY YuanL (2022). Masked autoencoders for point cloud self-supervised learning. In S. Avidan, G. Brostow, M. Cissé G. M. Farinella, & T. Hassner (Eds.), Computer Vision – ECCV 2022 (604–621) Springer Nature, Switzerlanddoi

[43] Qi C R, Yi L, Su H, Guibas L J, (2017). PointNet++: Deep hierarchical feature learning on point sets in a metric space. Advances in Neural Information Processing Systems, 30: 5105–5114

[44] RaniAOrtiz-Arroyo DDurdevicP (2024). Advancements in point cloud-based 3d defect detection and classification for industrial systems: A comprehensive survey. arXiv Preprint arXiv:2402.12923

[45] RothKPemula LZepedaJScholkopfBBroxT GehlerP (2022). Towards total recall in industrial anomaly detection. In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 14298–14308

[46] RudolphMWehrbein TRosenhahnBWandtB (2023). Asymmetric student-teacher networks for industrial anomaly detection. In: 2023 IEEE/CVF Winter Conference on Applications of Computer Vision, 2591–2601

[47] RusuR BBlodow NBeetzM (2009). Fast point feature histograms (FPFH) for 3D registration. In: 2009 IEEE International Conference on Robotics and Automation, 3212–3217

[48] RusuR BBlodow NMartonZ CBeetzM (2008). Aligning point cloud views using persistent feature histograms. In: 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems, 3384–3391

[49] ShaoZHao KWeiBTangX S (2021). Solder joint defect detection based on depth image CNN for 3D shape classification. In: 2021 CAA Symposium on Fault Detection, Supervision, and Safety for Technical Processes (SAFEPROCESS), 1–6

[50] SunYAbidiM A (2001). Surface matching by 3D point’s fingerprint. In: Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001, 2, 263–269

[51] Tao C, Du J, (2023). PointSGRADE: Sparse learning with graph representation for anomaly detection by using unstructured 3D point cloud data. IISE Transactions, 57( 2): 131–144

[52] Tao C, Du J, Chang T S, (2023). Anomaly detection for fabricated artifact by using unstructured 3D point cloud data. IISE Transactions, 55( 11): 1174–1186

[53] Vaswani A, (2017). Attention is all you need. Advances in Neural Information Processing Systems, 30: 6000–6010

[54] von Enzberg S, Al-Hamadi A, (2016). A multiresolution approach to model-based 3-D surface quality inspection. IEEE Transactions on Industrial Informatics, 12( 4): 1498–1507

[55] Wang J, Song K, Zhang D, Niu M, Yan Y, (2022a). Collaborative learning attention network based on RGB image and depth image for surface defect inspection of no-service rail. IEEE/ASME Transactions on Mechatronics, 27( 6): 4874–4884

[56] WangJZhang SXiaoYSongR (2021). A review on graph neural network methods in financial applications. arXiv Preprint arXiv:2111.15367

[57] WangLHuang YHouYZhangSShanJ (2019a). Graph attention convolution for point cloud semantic segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 10296–10305

[58] Wang Q, Wang X, He Q, Huang J, Huang H, Wang P, Yu T, Zhang M, (2024). 3D tensor-based point cloud and image fusion for robust detection and measurement of rail surface defects. Automation in Construction, 161: 105342

[59] WangYPeng JZhangJYiRWangY WangC (2023a). Multimodal industrial anomaly detection via hybrid fusion. In: 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 8032–8041

[60] Wang Y, Sun W, Jin J, Kong Z J, Yue X, (2023b). MVGCN: Multi-view graph convolutional neural network for surface defect identification using three-dimensional point cloud. Journal of Manufacturing Science and Engineering, 145( 3): 031004

[61] Wang Y, Sun Y, Liu Z, Sarma S E, Bronstein M M, Solomon J M, (2019b). Dynamic graph cnn for learning on point clouds. ACM Transactions on Graphics, 38( 5): 1–12

[62] Wang Z, Zhang Y, Lv T, Luo L, (2022b). GTAINet: Graph neural network-based two-stage anomaly identification for locking wire point clouds using hierarchical attentive edge convolution. International Journal of Applied Earth Observation and Geoinformation, 115: 103106

[63] Wei S, Chengyao S, Shilin C, Kailin Z, (2021). A microhardness indentation point cloud segmentation method based on voxel cloud connectivity segmentation. Measurement: Sensors, 18: 100124

[64] Xu Z, Lin Y, Chen D, Yuan M, Zhu Y, Ai Z, Yuan Y, (2024). Wood broken defect detection with laser profilometer based on Bi-LSTM network. Expert Systems with Applications, 242: 122789

[65] Yacob F, Semere D, Nordgren E, (2019). Anomaly detection in Skin Model Shapes using machine learning classifiers. International Journal of Advanced Manufacturing Technology, 105( 9): 3677–3689

[66] Yang J, Zhou W, Wu R, Fang M, (2023). CSANet: Contour and semantic feature alignment fusion network for rail surface defect detection. IEEE Signal Processing Letters, 30: 972–976

[67] ZavrtanikVKristan MSkočajD (2024). Cheating depth: enhancing 3D surface anomaly detection via depth simulation. In: 2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2153–2161

[68] Zhang D, Zou Q, Lin H, Xu X, He L, Gui R, Li Q, (2018). Automatic pavement defect detection using 3D laser profiling technology. Automation in Construction, 96: 350–365

[69] Zhang S, Tong H, Xu J, Maciejewski R, (2019). Graph convolutional networks: A comprehensive review. Computational Social Networks, 6: 11

[70] ZhaoHJiang LJiaJTorrP HKoltunV (2021). Point transformer. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, 16259–16268

[71] Zhao X, Del Castillo E, (2021a). An intrinsic geometrical approach for statistical process control of surface and manifold data. Technometrics, 63( 3): 295–312

[72] ZhaoXDel Castillo E (2021b). Registration-free localization of defects in 3-D parts from mesh metrology data using functional maps. arXiv Preprint arXiv:2112.14870

[73] Zhao X, Li Q, Xiao M, He Z, (2023). Defect detection of 3D printing surface based on geometric local domain features. International Journal of Advanced Manufacturing Technology, 125( 1–2): 183–194

[74] Zhou Y, Ji A, Zhang L, (2022). Sewer defect detection from 3D point clouds using a transformer-based deep learning model. Automation in Construction, 136: 104163

RIGHTS & PERMISSIONS

Higher Education Press