Knowledge Graph for Identifying Geological Disasters by Integrating Computer Vision with Ontology

Abstract

The occurrence of geological disasters can have a large impact on urban safety. Protecting people’s safety is the most important concern when disasters occur. Safety improvement requires a large amount of comprehensive and representative risk analysis and a large collection of information related to geological hazards, including unstructured knowledge and experience. To address the relevant information and support safety risk analysis, a geological hazard knowledge graph is developed automatically based on computer vision and domain-geoscience ontology to identify geological hazards from input images while obeying safety rules and regulations, even when affected by changes. In the implementation of the knowledge graph, we design an ontology schema of geological disasters based on a top-down approach, and by organizing knowledge as a logical semantic expression, it can be shared using ontology technologies and therefore enable semantic interoperability. Computer vision approaches are then used to automatically detect a set of entities and attributes, using the data from input images, and object types and their attributes are identified so that they can be stored in Neo4j for reasoning and searching. Finally, a reasoning model for geological hazard identification was developed using the Neo4j database to create nodes, relationships, and their properties for modeling, and geological hazards in the images can be automatically identified by searching the Neo4j database. An application on geological hazard is presented. The results show the effectiveness of the proposed approach in terms of identifying possible potential hazards in geological hazards and assisting in formulating targeted preventive measures.

Keywords

geological hazard / computer vision / knowledge graph / city safety / ontology

References

[1] Andrés S, Arvor D, Mougenot I, . Ontology-Based Classification of Remote Sensing Images Using Spectral Rules. Computers & Geosciences, 2017, 102: 158-166.

[2] Ashish, V., Noam, S., Niki, P., et al., 2017. Attention is All You Need. NIPS, 5998–6008

[3] Bharambe U, Durbha S S. Adaptive Pareto-Based Approach for Geo-Ontology Matching. Computers & Geosciences, 2018, 119: 92-108.

[4] Bittner T, Donnelly M, Smith B. A Spatio-Temporal Ontology for Geographic Information Integration. International Journal of Geographical Information Science, 2009, 23(6): 765-798.

[5] Buccella A, Cechich A, Gendarmi D, . Building a Global Normalized Ontology for Integrating Geographic Data Sources. Computers & Geosciences, 2011, 37(7): 893-916.

[6] Carion N, Massa F, Synnaeve G, . End-to-End Object Detection with Transformers. ECCV, 2020, 1: 213-229.

[7] Dai, Z. G., Cai, B. L., Lin, Y. G., et al., 2021. UP-DETR: Unsupervised Pre-Training for Object Detection with Transformers. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). June 20–25, 2021, Nashville. https://doi.org/10.1109/cvpr46437.2021.00165

[8] Dosovitskiy, A., Beyer, L., Kolesnikov, A., et al., 2020. An Image is Worth 16 × 16 Words: Transformers for Image Recognition at Scale. arXiv: 2010.11929. https://doi.org/10.48550/arXiv.2010.11929

[9] Fan C, Esparza M, Dargin J, . Spatial Biases in Crowdsourced Data: Social Media Content Attention Concentrates on Populous Areas in Disasters. Computers, Environment and Urban Systems, 2020, 83: 101514

[10] Fan R Y, Wang L Z, Yan J N, . Deep Learning-Based Named Entity Recognition and Knowledge Graph Construction for Geological Hazards. ISPRS International Journal of Geo-Information, 2019, 9(1): 15

[11] Fang W L, Ding L Y, Luo H B, . Falls from Heights: A Computer Vision-Based Approach for Safety Harness Detection. Automation in Construction, 2018, 91 53-61.

[12] Fang W L, Ding L Y, Zhong B T, . Automated Detection of Workers and Heavy Equipment on Construction Sites: A Convolutional Neural Network Approach. Advanced Engineering Informatics, 2018, 37 139-149.

[13] Fang W L, Zhong B T, Zhao N, . A Deep Learning-Based Approach for Mitigating Falls from Height with Computer Vision: Convolutional Neural Network. Advanced Engineering Informatics, 2019, 39: 170-177.

[14] Fang W L, Ma L, Love P E D, . Knowledge Graph for Identifying Hazards on Construction Sites: Integrating Computer Vision with Ontology. Automation in Construction, 2020, 119: 103310

[15] Garcia L F, Abel M, Perrin M, . The GeoCore Ontology: A Core Ontology for General Use in Geology. Computers & Geosciences, 2020, 135 104387

[16] Girshick, R. B., 2015. Fast R-CNN. IEEE International Conference on Computer Vision (ICCV), December 7–13, Santiago. https://doi.org/10.1109/iccv.2015.169

[17] Girshick, R., Donahue, J., Darrell, T., et al., 2014. Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation. 2014 IEEE Conference on Computer Vision and Pattern Recognition. June 23–28, 2014, Columbus. https://doi.org/10.1109/cvpr.2014.81

[18] Goh Y M, Chua D K H. Case-Based Reasoning Approach to Construction Safety Hazard Identification: Adaptation and Utilization. Journal of Construction Engineering and Management, 2010, 136(2): 170-178.

[19] Guarino N. Understanding, Building and Using Ontologies. International Journal of Human-Computer Studies, 1997, 46(2/3): 293-310.

[20] Guia, J., Soares, V. G., Bernardino, J., 2017. Graph Databases: Neo4j Analysis. 351–356. https://doi.org/10.5220/0006356003510356

[21] Guo B H W, Goh Y M. Ontology for Design of Active Fall Protection Systems. Automation in Construction, 2017, 82 138-153.

[22] He K M, Zhang X Y, Ren S Q, . Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 37(9): 1904-1916.

[23] He, K. M., Gkioxari, G., Piotr, D., et al., 2017. Mask R-CNN. arXiv: 1703.06870. https://doi.org/10.48550/arXiv.1703.06870

[24] Huang Q Y, Cervone G, Zhang G M. A Cloud-Enabled Automatic Disaster Analysis System of Multi-Sourced Data Streams: an Example Synthesizing Social Media, Remote Sensing and Wikipedia Data. Computers, Environment and Urban Systems, 2017, 66: 23-37.

[25] Janowicz K, Raubal M, Kuhn W. The Semantics of Similarity in Geographic Information Retrieval. Journal of Spatial Information Science, 2011, 2: 29-57.

[26] Jasper R R, Uijlings K E A, van de Sande T G, . Smeulders: Selective Search for Object Recognition. Int. J. Comput. Vis., 2013, 104(2): 154-171.

[27] Jiao L C, Zhang F, Liu F, . A Survey of Deep Learning-Based Object Detection. IEEE Access, 2019, 7: 128837-128868.

[28] Johnpaul, C. I., Mathew, T., 2017. A Cypher Query Based NoSQL Data Mining on Protein Datasets Using Neo4j Graph Database. 2017 4th International Conference on Advanced Computing and Communication Systems (ICACCS). January 6–7, 2017, Coimbatore, India. IEEE: 1–6. https://doi.org/10.1109/icaccs.2017.8014558

[29] Kim D, Liu M Y, Lee S, . Remote Proximity Monitoring between Mobile Construction Resources Using Camera-Mounted UAVs. Automation in Construction, 2019, 99: 168-182.

[30] Li L, Liu Y, Zhu H H, . A Bibliometric and Visual Analysis of Global Geo-Ontology Research. Computers & Geosciences, 2017, 99: 1-8.

[31] Liu L, Ouyang W, Wang X G, . Deep Learning for Generic Object Detection: A Survey. International Journal of Computer Vision, 2020, 128(2): 261-318.

[32] Liu, W., Anguelov, D., Erhan, D., et al., 2016. Single Shot Multibox Detector. arXiv:1512.02325. https://arxiv.org/abs/1512.02325

[33] Lu P, Qin Y Y, Li Z B, . Landslide Mapping from Multi-Sensor Data through Improved Change Detection-Based Markov Random Field. Remote Sensing of Environment, 2019, 231: 111235

[34] Lu P, Bai S B, Tofani V, . Landslides Detection through Optimized Hot Spot Analysis on Persistent Scatterers and Distributed Scatterers. ISPRS Journal of Photogrammetry and Remote Sensing, 2019, 156 147-159.

[35] Ma X G. Linked Geoscience Data in Practice: Where W3C Standards Meet Domain Knowledge, Data Visualization and OGC Standards. Earth Science Informatics, 2017, 10(4): 429-441.

[36] Ma X G, Carranza E J M, Wu C L, . Ontology-Aided Annotation, Visualization, and Generalization of Geological Time-Scale Information from Online Geological Map Services. Computers & Geosciences, 2012, 40: 107-119.

[37] Ma X G, Ma C, Wang C B. A New Structure for Representing and Tracking Version Information in a Deep Time Knowledge Graph. Computers & Geosciences, 2020, 145: 104620

[38] Mao S, Zhao Y M, Chen J H, . Development of Process Safety Knowledge Graph: A Case Study on Delayed Coking Process. Computers & Chemical Engineering, 2020, 143 107094

[39] Qiu Q J, Xie Z, Wu L, . Geoscience Keyphrase Extraction Algorithm Using Enhanced Word Embedding. Expert Systems with Applications, 2019, 125 157-169.

[40] Redmon, J., Farhadi, A., 2018. YOLOv3: An Incremental Improvement. arXiv: 1804.02767. https://arxiv.org/abs/1804.02767

[41] Redmon, J., Farhadi, A., 2016. YOLO9000: Better, Faster, Stronger. arXiv: 1612.08242. https://arxiv.org/abs/1612.08242

[42] Redmon, J., Divvala, S., Girshick, R., et al., 2015. You Only Look Once: Unified, Real-Time Object Detection. arXiv: 1506.02640. https://arxiv.org/abs/1506.02640

[43] Ren S Q, He K M, Girshick R, . Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(6): 1137-1149.

[44] Steiner, T., Verborgh, R., Troncy, R., et al., 2012. Adding Realtime Coverage to the Google Knowledge Graph. In: 11th International Semantic Web Conference (ISWC 2012). Citeseer

[45] UNDRR (United Nations Office for Disaster Risk Reduction), 2020. UNDRR Annual Report. https://www.undrr.org/about-undrr (2020)

[46] Wang C B, Ma X G, Chen J G. Ontology-Driven Data Integration and Visualization for Exploring Regional Geologic Time and Paleontological Information. Computers & Geosciences, 2018, 115: 12-19.

[47] Wang C B, Ma X G, Chen J G, . Information Extraction and Knowledge Graph Construction from Geoscience Literature. Computers & Geosciences, 2018, 112: 112-120.

[48] Wang J J, He Z C, Weng W G. A Review of the Research into the Relations between Hazards in Multi-Hazard Risk Analysis. Natural Hazards, 2020, 104(3): 2003-2026.

[49] Weber, E., Kané, H., 2020. Building Disaster Damage Assessment in Satellite Imagery with Multi-Temporal Fusion. arXiv: 2004.05525. https://arxiv.org/abs/2004.05525

[50] Xiao, T. T., Liu, Y. C., Zhou, B. L., et al., 2018. Unified Perceptual Parsing for Scene Understanding. arXiv: 1807.10221. https://arxiv.org/abs/1807.10221

[51] Zheng, M. H., Gao, P., Zhang, R. R., et al., 2020. End-to-End Object Detection with Adaptive Clustering Transformer. arXiv: 2011.09315. https://arxiv.org/abs/2011.09315

[52] Zheng X, Wang B, Zhao Y M, . A Knowledge Graph Method for Hazardous Chemical Management: Ontology Design and Entity Identification. Neurocomputing, 2021, 430: 104-111.

[53] Zhu, X. Z., Su, W. J., Lu, L. W., et al., 2020. Deformable DETR: Deformable Transformers for End-to-End Object Detection. arXiv: 2010.04159. https://arxiv.org/abs/2010.04159