Bridging the gap: Neuro-Symbolic Computing for advanced AI applications in construction

Abstract

Deep Learning (DL) has revolutionized the field of Artificial Intelligence (AI) in various domains such as computer vision (CV) and natural language processing. However, DL models have limitations including the need for large labeled datasets, lack of interpretability and explainability, potential bias and fairness issues, and limitations in common sense reasoning and contextual understanding. On the other side, DL has shown significant potential in construction for safety and quality inspection tasks using CV models. However, current CV approaches may lack spatial context and measurement capabilities, and struggle with complex safety and quality requirements. The integration of Neuro-Symbolic Computing (NSC), an emerging field that combines DL and symbolic reasoning, has been proposed as a potential solution to address these limitations. NSC has the potential to enable more robust, interpretable, and accurate AI systems in construction by harnessing the strengths of DL and symbolic reasoning. The combination of symbolism and connectionism in NSC can lead to more efficient data usage, improved generalization ability, and enhanced interpretability. Further research and experimentation are needed to effectively integrate NSC with large models and advance CV technologies for precise reporting of safety and quality inspection results in construction.

Keywords

advanced AI in construction / safety and quality inspection / Neuro-Symbolic Computing / Deep Learning / computer vision

References

[1] Badreddine, S d’Avila, Garcez A Serafini, L Spranger, M (2022). Logic tensor networks. Artificial Intelligence, 303: 103649

[2] Braun, A Tuttas, S Borrmann, A Stilla, U (2020). Improving progress monitoring by fusing point clouds, semantic data and computer vision. Automation in Construction, 116: 103210

[3] Brilakis, I Park, M W Jog, G (2011). Automated vision tracking of project related entities. Advanced Engineering Informatics, 25( 4): 713–724

[4] BrownT BMannBRyderNSubbiahMKaplanJ DDhariwalPNeelakantanAShyamPSastryGAskellA, . (2020). Language models are few-shot learners. In: Proceedings of the 34th International Conference on Neural Information Processing Systems. Vancouver, BC: Curran Associates Inc., 1877–1901

[5] CarionNMassaFSynnaeveGUsunierNKirillovAZagoruykoS (2020). End-to-end object detection with transformers. In: Proceedings of the 16th European Conference. Glasgow: Springer-Verlag, 213–229

[6] Chi, S Caldas, C H (2011). Automated object identification using optical video cameras on construction sites. Computer-Aided Civil and Infrastructure Engineering, 26( 5): 368–380

[7] Chi, S Caldas, C (2012). Image-based safety assessment: Automated spatial safety risk identification of earthmoving and surface mining activities. Journal of Construction Engineering and Management, 138( 3): 341–351

[8] Chi, S Caldas, C H Kim, D Y (2009). A methodology for object identification and tracking in construction based on spatial modeling and image matching techniques. Computer-Aided Civil and Infrastructure Engineering, 24( 3): 199–211

[9] CohenW WYangFMazaitisK R (2017). TensorLog: Deep learning meets probabilistic DBs. arXiv preprint, arXiv:1707.05390

[10] Cussens, J (2001). Parameter estimation in stochastic logic programs. Machine Learning, 44( 3): 245–271

[11] d’Avila, Garcez A Lamb, L C (2023). Neurosymbolic AI: The 3rd wave. Artificial Intelligence Review, 56( 11): 12387–12406

[12] DanieleASerafiniL (2019). Knowledge enhanced neural networks. In: Proceedings of the 16th Pacific Rim International Conference on Artificial Intelligence: Trends in Artificial Intelligence. Cuvu, Yanuca Island: Springer Cham, 542–554

[13] Ding, L Fang, W Luo, H Love, P E D Zhong, B Ouyang, X (2018). A deep hybrid learning model to detect unsafe behavior: Integrating convolution neural networks and long short-term memory. Automation in Construction, 86: 118–124

[14] DuYFuZLiuQWangY (2022). Visual grounding with transformers. In: Proceedings of the IEEE International Conference on Multimedia and Expo (ICME). Taipei: IEEE, 1–6

[15] Fang, Q Li, H Luo, X Ding, L Luo, H Rose, T M An, W (2018a). Detecting non-hardhat-use by a deep learning method from far-field surveillance videos. Automation in Construction, 85: 1–9

[16] Fang, Q Li, H Luo, X Ding, L Rose, T M An, W Yu, Y (2018b). A deep learning-based method for detecting non-certified work on construction sites. Advanced Engineering Informatics, 35: 56–68

[17] Fang, W Ding, L Luo, H Love, P E D (2018c). Falls from heights: A computer vision-based approach for safety harness detection. Automation in Construction, 91: 53–61

[18] Fang, W Love, P E D Ding, L Xu, S Kong, T Li, H (2021). Computer vision and deep learning to manage safety in construction: Matching images of unsafe behavior and semantic rules. IEEE Transactions on Engineering Management, 70( 12): 4120–4132

[19] Garnelo, M Shanahan, M (2019). Reconciling deep learning with symbolic artificial intelligence: Representing objects and relations. Current Opinion in Behavioral Sciences, 29: 17–23

[20] GirdharRGkioxariGTorresaniLPaluriMTranD (2018). Detect-and-track: Efficient pose estimation in videos. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT: IEEE, 350–359

[21] GirshickR (2015). Fast R-CNN. In: Proceedings of the IEEE International Conference on Computer Vision. Santiago: IEEE, 1440–1448

[22] Gong, J Caldas, C H (2010). Computer vision-based video interpretation model for automated productivity analysis of construction operations. Journal of Computing in Civil Engineering, 24( 3): 252–263

[23] HeKGkioxariGDollárPGirshickR (2017). Mask R-CNN. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV). Venice: IEEE, 2980–2988

[24] Hochreiter, S Schmidhuber, J (1997). Long short-term memory. Neural Computation, 9( 8): 1735–1780

[25] HuQYangBXieLRosaSGuoYWangZTrigoniNMarkhamA (2020). Randla-net: Efficient semantic segmentation of large-scale point clouds. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Seattle, WA: IEEE, 11105–11114

[26] Huang, Y Wu, Q Wang, W Wang, L (2020). Image and sentence matching via semantic concepts and order learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42( 3): 636–650

[27] JainAGkanatsiosNMedirattaIFragkiadakiK (2022). Bottom up top down detection transformers for language grounding in images and point clouds. In: Proceedings of the 17th European Conference on Computer Vision. Tel Aviv: Springer-Verlag, 417–433

[28] KahnemanDHintonGBengioYLeCunYRossiF (2020). AAAI-20 fireside chat with Daniel Kahneman

[29] Kim, J Chung, D Kim, Y Kim, H (2022). Deep learning-based 3D reconstruction of scaffolds using a robot dog. Automation in Construction, 134: 104092

[30] Kim, P S Lee, D G Lee, S W (2018). Discriminative context learning with gated recurrent unit for group activity recognition. Pattern Recognition, 76: 149–161

[31] Kimmig, A Mihalkova, L Getoor, L (2015). Lifted graphical models: A survey. Machine Learning, 99( 1): 1–45

[32] KimmigAvan den BroeckGde RaedtL (2011). An algebraic prolog for reasoning about possible worlds. In: Proceedings of the 25th AAAI Conference on Artificial Intelligence. San Francisco, CA: AAAI Press, 209–214

[33] KrizhevskyASutskeverIHintonG E (2012). ImageNet classification with deep convolutional neural networks. In: Proceedings of the 25th International Conference on Neural Information Processing Systems. Lake Tahoe, NV: Curran Associates Inc., 1097–1105

[34] LeCun, Y Bengio, Y Hinton, G (2015). Deep learning. Nature, 521( 7553): 436–444

[35] Lei, L Zhou, Y Luo, H Love, P E D (2019). A CNN-based 3D patch registration approach for integrating sequential models in support of progress monitoring. Advanced Engineering Informatics, 41: 100923

[36] Li, Y Wei, H Han, Z Huang, J Wang, W (2020). Deep learning-based safety helmet detection in engineering management based on convolutional neural networks. Advances in Civil Engineering, 9703560

[37] Luo, X Li, H Cao, D Dai, F Seo, J Lee, S (2018). Recognizing diverse construction activities in site images via relevance networks of construction-related objects detected by convolutional neural networks. Journal of Computing in Civil Engineering, 32( 3): 04018012

[38] Luo, X Li, H Yang, X Yu, Y Cao, D (2019). Capturing and understanding workers’ activities in far-field surveillance videos with deep action recognition and Bayesian nonparametric learning. Computer-Aided Civil and Infrastructure Engineering, 34( 4): 333–351

[39] Luo, X Li, H Yu, Y Zhou, C Cao, D (2020). Combining deep features and activity context to improve recognition of activities of workers in groups. Computer-Aided Civil and Infrastructure Engineering, 35( 9): 965–978

[40] Manhaeve, R Dumančić, S Kimmig, A Demeester, T de, Raedt L (2021). Neural probabilistic logic programming in DeepProbLog. Artificial Intelligence, 298: 103504

[41] MaoJGanCKohliPTenenbaumJ BWuJ (2019). The neuro-symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision. In: Proceedings of the 7th International Conference on Learning Representations. New Orleans, LA: OpenReview.net

[42] QuMTangJ (2019). Probabilistic logic neural networks for reasoning. In: Proceedings of the 33rd International Conference on Neural Information Processing Systems. Vancouver, BC: Curran Associates Inc., 7712–7722

[43] RadfordANarasimhanKSalimansTSutskeverI (2018). Improving language understanding by generative pre-training. OpenAI

[44] Ren, S He, K Girshick, R Sun, J (2017). Faster R-CNN: Towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39( 6): 1137–1149

[45] Richardson, M Domingos, P (2006). Markov logic networks. Machine Learning, 62( 1–2): 107–136

[46] RiegelRGrayALuusFKhanNMakondoNYunus AkhalwayaIQianHFaginRBarahonaFSharmaUIkbalSKaranamHNeelamSLikhyaniASrivastavaS (2020). Logical neural networks. arXiv preprint, arXiv:2006.13155

[47] Rodríguez-Gonzálvez, P Rodríguez-Martín, M Ramos, L F González-Aguilera, D (2017). 3D reconstruction methods and quality assessment for visual inspection of welds. Automation in Construction, 79: 49–58

[48] Seo, J Han, S Lee, S Kim, H (2015). Computer vision techniques for construction safety and health monitoring. Advanced Engineering Informatics, 29( 2): 239–251

[49] Sourek, G Aschenbrenner, V Zelezny, F Schockaert, S Kuzelka, O (2018). Lifted relational neural networks: Efficient learning of latent relational structures. Journal of Artificial Intelligence Research, 62: 69–100

[50] Teizer, J (2015). Status quo and open challenges in vision-based sensing and tracking of temporary resources on infrastructure construction sites. Advanced Engineering Informatics, 29( 2): 225–238

[51] Teizer, J Allread, B S Fullerton, C E Hinze, J (2010). Autonomous pro-active real-time construction worker and equipment operator proximity safety alert system. Automation in Construction, 19( 5): 630–640

[52] Tran, S N d’Avila, Garcez A S (2018). Deep logic networks: Inserting and extracting knowledge from deep belief networks. IEEE Transactions on Neural Networks and Learning Systems, 29( 2): 246–258

[53] Vahdatikhaki, F Hammad, A (2015). Risk-based look-ahead workspace generation for earthwork equipment using near real-time simulation. Automation in Construction, 58: 207–220

[54] Wang, Q (2019). Automatic checks from 3D point cloud data for safety regulation compliance for scaffold work platforms. Automation in Construction, 104: 38–51

[55] Wang, Q Mao, Z Wang, B Guo, L (2017). Knowledge graph embedding: A survey of approaches and applications. IEEE Transactions on Knowledge and Data Engineering, 29( 12): 2724–2743

[56] WuFJinGGaoMHeZYangY (2019). Helmet detection based on improved YOLO v3 deep model. In: Proceedings of the 16th International Conference on Networking, Sensing and Control (ICNSC). Banff, AB: IEEE, 363–368

[57] Wu, H Zhong, B Li, H Love, P E D Pan, X Zhao, N (2021). Combining computer vision with semantic reasoning for on-site safety management in construction. Journal of Building Engineering, 42: 103036

[58] Yang, J Park, M W Vela, P A Golparvar-Fard, M (2015). Construction performance monitoring via still images, time-lapse photos, and video streams: Now, tomorrow, and the future. Advanced Engineering Informatics, 29( 2): 211–224

[59] Yang, X Li, H Yu, Y Luo, X Huang, T Yang, X (2018). Automatic pixel-level crack detection and measurement using fully convolutional network. Computer-Aided Civil and Infrastructure Engineering, 33( 12): 1090–1109

[60] Zhang, K Zhang, Z Li, Z Qiao, Y (2016). Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Processing Letters, 23( 10): 1499–1503

[61] Zhong, B Li, H Luo, H Zhou, J Fang, W Xing, X (2020). Ontology-based semantic modeling of knowledge in construction: Classification and identification of hazards implied in images. Journal of Construction Engineering and Management, 146( 4): 04020013

[62] ZhuXSuWLuLLiBWangXDaiJ (2020). Deformable DETR: Deformable transformers for end-to-end object detection. arXiv preprint, arXiv:2010.04159

Competing Interests

The authors declare that they have no competing interests.

RIGHTS & PERMISSIONS

2023 Higher Education Press