Hastily formed knowledge networks and distributed situation awareness for collaborative robotics

Abstract

In the context of collaborative robotics, distributed situation awareness is essential for supporting collective intelligence in teams of robots and human agents where it can be used for both individual and collective decision support. This is particularly important in applications pertaining to emergency rescue and crisis management. During operational missions, data and knowledge are gathered incrementally and in different ways by heterogeneous robots and humans. We describe this as the creation of Hastily Formed Knowledge Networks (HFKNs). The focus of this paper is the specification and prototyping of a general distributed system architecture that supports the creation of HFKNs by teams of robots and humans. The information collected ranges from low-level sensor data to high-level semantic knowledge, the latter represented in part as RDF Graphs. The framework includes a synchronization protocol and associated algorithms that allow for the automatic distribution and sharing of data and knowledge between agents. This is done through the distributed synchronization of RDF Graphs shared between agents. High-level semantic queries specified in SPARQL can be used by robots and humans alike to acquire both knowledge and data content from team members. The system is empirically validated and complexity results of the proposed algorithms are provided. Additionally, a field robotics case study is described, where a 3D mapping mission has been executed using several UAVs in a collaborative emergency rescue scenario while using the full HFKN Framework.

Keywords

Multi-robot collaboration / Unmanned aerial vehicles / Distributed knowledge representation / Distributed situation awareness / Semantic web technology / Knowledge synchronization / Multi-agent human/robot interaction

References

[1] DenningP. J.. Hastily formed networks. Commun. ACM, 2006, 49(4):15-20 https://doi.org/10.1145/1121949.1121966

[2] DohertyP., KvarnströmJ., RudolP., WzorekM., ConteG., BergerC., HinzmannT., StastnyT.. BaldoniM., ChopraA. K., SonT. C., HirayamaK., TorroniP.. A Collaborative Framework for 3D Mapping using Unmanned Aerial Vehicles. PRIMA 2016: Principles and Practice of Multi-Agent Systems, Lecture Notes in Computer Science, vol. 9862, 2016 Phuket Springer Publishing Company 110-130 https://doi.org/10.1007/978-3-319-44832-9_7

[3] E. Horvitz, Mixed-initiative interaction (IEEE Intelligent Systems, 1999). https://doi.org/10.1109/5254.796083. https://www.microsoft.com/en-us/research/publication/mixed-initiative-interaction/.

[4] J. Ji, P. Fazli, S. Liu, T. Pereira, D. Lu, J. Liu, M. Veloso, X. Chen, in Proceedings of ICSR’16, the International Conference on Social Robots. Help me! sharing of instructions between remote and heterogeneous robots (Limassol, 2016), pp. 786–795. https://doi.org/10.1007/978-3-319-47437-3_77.

[5] KvarnströmJ.. BacchusF., DomshlakC., EdelkampS., HelmertM.. Planning for Loosely Coupled Agents Using Partial Order Forward-Chaining. Proceedings of the 21st International Conference on Automated Planning and Scheduling (ICAPS), 2011 Freiburg AAAI Press 138-145

[6] S. Lemaignan, M. Warnier, E. Sisbot, A. Clodic, R. Alami, Artificial cognition for social human-robot interaction: An implementation. Artificial Intelligence. 247: (2016). https://doi.org/10.1016/j.artint.2016.07.002.

[7] LuoL., ChakrabortyN., SycaraK.. Distributed algorithms for multirobot task assignment with task deadline constraints. IEEE Trans. Autom. Sci. Eng. Spec. Issue Networked Coop. Auton. Syst., 2015, 12(3):876-888 https://doi.org/10.1109/TASE.2015.2438032

[8] M. M. Veloso, The Increasingly Fascinating Opportunity for Human-Robot Interaction: The CoBot Mobile Service Robots. ACM Trans. Hum.-Robot Interact. (2018). https://doi.org/10.1145/3209541.

[9] DohertyP., HeintzF., KvarnströmJ.. High-level mission specification and planning for collaborative unmanned aircraft systems using delegation. Unmanned Syst., 2013, 1(1):75-119 https://doi.org/10.1142/S2301385013500052

[10] DohertyP., KvarnströmJ., SzalasA.. Temporal composite actions with constraints. Proceedings of the 13th International Conference on Principles of Knowledge Representation and Reasoning (KR), 2012 Kolkata AAAI Press 478-488 https://doi.org/10.1007/978-3-642-25920-3_4

[11] DohertyP., KvarnstromJ., WzorekM., RudolP., HeintzF., ConteG.. ValavanisK. P., VachtsevanosG. J.. HDRC3: A distributed hybrid deliberative/reactive architecture for unmanned aircraft systems. Handbook of Unmanned Aerial Vehicles, 2015 Dordrecht Springer Netherlands 849-952 https://doi.org/10.1007/978-90-481-9707-1_118

[12] M. Quigley, B. Gerkey, K. Conley, J. Faust, T. Foote, J. Leibs, E. Berger, R. Wheeler, A. Ng, in Proc. of the IEEE Intl. Conf. on Robotics and Automation (ICRA) Workshop on Open Source Robotics. Kobe, (2009).

[13] P. Doherty, J. J. C. Meyer, in The Goals of Cognition: Essays in honour of Cristiano Castelfranchi, ed. by F. Paglieri, L. Tummolini, R. Falcone, and M. Miceli. On the logic of delegation - relating theory and practice (College Publications, 2012), pp. 467–496.

[14] T. Hinzmann, T. Stastny, G. Conte, P. Doherty, P. Rudol, M. Wzorek, E. Galceran, R. Siegwart, I. Gilitschenski, in International Symposium on Experimental Robotics. Collaborative 3D reconstruction using heterogeneous uavs: System and experiments (Tokyo, 2016), pp. 43–56. https://doi.org/10.1007/978-3-319-50115-4_5.

[15] P. Doherty, F. Heintz, D. Landén, in Proceedings of the International conference on principles and practice of multi-agent systems (PRIMA), Lecture Notes in Computer Science, vol. 7057. A distributed task specification language for mixed-initiative delegation (Kuching, 2012), pp. 42–57. https://doi.org/10.1007/978-3-642-25920-3_4.

[16] GandonF., SchreiberG.. RDF 1.1 xml syntax. recommendation, 2014 Cambridge World Wide Web Consortium (W3C) https://www.w3.org/TR/rdf-syntax-grammar/

[17] M. Duerst, M. Suignard, RFC 3987: Internationalized Resource Identifiers (IRIs). RFC 3987 (Proposed Standard) (2005). http://www.ietf.org/rfc/rfc3987.txt.

[18] BrickleyD., GuhaR. V., McBrideB.. RDF schema 1.1. recommendation, 2014 Cambridge World Wide Web Consortium (W3C)

[19] McGuinnessD. L., Van HarmelenF., et al.. OWL 2 web ontology language document overview, 2012 Cambridge World Wide Web Consortium (W3C) https://www.w3.org/TR/owl2-overview/

[20] C. Berger, P. Doherty, P. Rudol, M. Wzorek, Hastily formed knowledge networks and distributed situation awareness for collaborative robotics, (2021). Preprint at https://arxiv.org/abs/2103.14078.

[21] SeaborneA., ManjunathG., BizerC., BreslinJ., DasS., DavisI., HarrisS., IdehenK., CorbyO., KjernsmoK., et al.. SPARQL/Update: A language for updating RDF graphs. Tech. rep., 2008 Cambridge World Wide Web (W3C)

[22] S. S. Sahoo, W. Halb, S. Hellmann, K. Idehen, T. Thibodeau, S. Auer, J. Sequeda, A. Ezzat, A survey of current approaches for mapping of relational databases to RDF. Tech. rep., W3C RDB2RDF Incubator Group Report, (2009).

[23] C. Stadler, J. Unbehauen, P. Westphal, M. A. Sherif, J. Lehmann, in Proceedings of the 8th Workshop on Linked Data on the Web (LDOW). Simplified RDB2RDF mapping (Rio de Janeiro, 2015).

[24] BizerC., SchultzA.. The berlin SPARQL benchmark. Int. J. Semantic Web Inf. Syst. (IJSWIS), 2009, 5(2):1-24 https://doi.org/10.4018/978-1-60960-593-3.ch004

[25] FowlerM.. Patterns of Enterprise Application Architecture, 2012 San Francisco Addison-Wesley

[26] C. Schlenoff, E. Prestes, R. Madhavan, P. Goncalves, H. Li, S. Balakirsky, T. Kramer, E. Migueláñez, in 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems. An ieee standard ontology for robotics and automation (Vilamoura, 2012), pp. 1337–1342. https://doi.org/10.1109/IROS.2012.6385518.

[27] M. Wylot, M. Hauswirth, P. Cudré-Mauroux, S. Sakr, Rdf data storage and query processing schemes: A survey. ACM Comput. Surv.51(4) (2018). https://doi.org/10.1145/3177850.

[28] A. Seaborne, A. Polleres, L. Feigenbaum, G. T. Williams, SPARQL 1.1 federated query. recommendation, World Wide Web Consortium (W3C), (Cambridge, 2013). https://www.w3.org/TR/sparql11-federated-query/.

[29] BattleR., KolasD.. Enabling the geospatial semantic web with parliament and geosparql. Semantic Web, 2012, 3(4):355-370 https://doi.org/10.3233/sw-2012-0065

[30] YoavShoham, Leyton-BrownK.. Multiagent Systems - Algorthmic, Game-Theoretic, and Logical Foundations, 2009 Cambridge Cambridge University Press

[31] WooldridgeM.. An Introduction to MultiAgent Systems, 2009 Hoboken Wiley

[32] FaginR., HalpernJ. Y., MosesY., VardiM. Y.. Reasoning about Knowledge, 1995 Cambridge MIT Press

[33] de BruijnJ., HeymansS.. RDF and logic: Reasoning and extension. 18th International Conference on Database and Expert Systems Applications (DEXA 2007), 2007 Regensburg IEEE 460-464 https://doi.org/10.1109/dexa.2007.8

[34] de BruijnJ., HeymansS.. Logical foundations of rdf(s) with datatypes. J. Artif. Intell. Res., 2010, 38: 535-569 https://doi.org/10.1613/jair.3088

[35] HayesP., Patel-SchneiderP. F.. RDF semantics. recommendation, 2004 Cambridge World Wide Web Consortium (W3C) https://www.w3.org/TR/rdf11-mt/

[36] GrecoS., MolinaroC.. Datalog and Logic Databases. Synthesis Lectures on Data Management, 2016 San Rafael Morgan & Claypool https://doi.org/10.2200/s00648ed1v01y201505dtm041

[37] E. Sirin, B. Parsia, in Proceedings of the 3rd International Workshop on OWL: Experiences and Directions. SPARQL-DL: SPARQL query for OWL DL (Innsbruck, 2007).

[38] S. Poslad, Specifying protocols for multi-agent systems interaction. ACM Trans. Auton. Adaptive Syst. (TAAS). 2(4) (2007). https://doi.org/10.1145/1293731.1293735.

[39] BratmanM.. Intention, Plans, and Practical Reason, 1987 Cambridge Harvard University Press https://doi.org/10.2307/2215590

[40] RaoA. S., GeorgeffM. P.. AllenR. F. J., SandewallE.. Modeling rational agents within a BDI-architecture. Proceedings of the 2nd International Conference on Principles of Knowledge Representation and Reasoning (KR’91), 1991 Cambridge Morgan Kaufman 473-484 https://doi.org/10.5555/3087158.3087205

[41] J. R. Chen, S. R. Wolfe, S. D. Wragg, in Proceedings of the Ninth International Conference on Information and Knowledge Management. A distributed multi-agent system for collaborative information management and sharing (McLean, 2000), pp. 382–388. https://doi.org/10.1145/354756.354844.

[42] KehoeB., PatilS., AbbeelP., GoldbergK.. A survey of research on cloud robotics and automation. IEEE Trans. Autom. Sci. Eng., 2015, 12(2):398-409 https://doi.org/10.1109/TASE.2014.2376492

[43] G. Bent, P. Dantressangle, D. Vyvyan, A. Mowshowitz, V. Mitsou, in Proceedings of the 2nd Annual Conference of the International Technology Alliance. A dynamic distributed federated database (London, 2008).

[44] ShethA. P., LarsonJ. A.. Federated database systems for managing distributed, heterogeneous, and autonomous databases. ACM Comput. Surv. (CSUR), 1990, 22(3):183-236 https://doi.org/10.1145/96602.96604

[45] ParentC., SpaccapietraS.. Issues and approaches of database integration. Commun. ACM, 1998, 41(5es):166-178 https://doi.org/10.1145/276404.276408

[46] J. C. Corbett, J. Dean, M. Epstein, A. Fikes, C. Frost, J. J. Furman, S. Ghemawat, A. Gubarev, C. Heiser, P. Hochschild, W. Hsieh, S. Kanthak, E. Kogan, H. Li, A. Lloyd, S. Melnik, D. Mwaura, D. Nagle, S. Quinlan, R. Rao, L. Rolig, Y. Saito, M. Szymaniak, C. Taylor, R. Wang, D. Woodford, Spanner: Google’s globally distributed database. ACM Trans. Comput. Syst.31(3) (2013). https://doi.org/10.1145/2491245.

[47] MoizS. A., SailajaP., VenkataswamyG., PalS. N.. Database replication: A survey of open source and commercial tools. Int. J. Comput. Appl., 2011, 13(6):1-8 https://doi.org/10.5120/1788-2469

[48] ÖzsuM. T., ValduriezP.. Principles of Distributed Database Systems, 2020 Berlin Springer https://doi.org/10.1007/978-3-030-26253-2

[49] DeCandiaG., HastorunD., JampaniM., KakulapatiG., LakshmanA., PilchinA., SivasubramanianS., VosshallP., VogelsW.. Dynamo: Amazon’s highly available key-value store. ACM SIGOPS Oper. Syst. Rev., 2007, 41(6):205-220 https://doi.org/10.1145/1323293.1294281

[50] AllemangD., HendlerJ., GandonF.. Semantic Web for the Working Ontologist: Effective Modeling for Linked Data, RDFS, and OWL, 2020 New York Association for Computing Machinery https://doi.org/10.1145/3382097

[51] Berners-LeeT., HendlerJ., LassilaO.. The semantic web. Sci. Am., 2001, 284(5):34-43 http://www.jstor.org/stable/26059207

[52] HendlerJ.. Agents and the semantic web. IEEE Intell. Syst., 2001, 16(2):30-37 https://doi.org/10.1109/5254.920597. http://dx.doi.org/10.1109/5254.920597

[53] F. Baader, D. Calvanese, D. Mcguinness, D. Nardi, P. Patel-Schneider, The Description Logic Handbook: Theory, Implementation, and Applications, 2edn. (Cambridge University Press, 2007). https://doi.org/10.1017/CBO9780511711787.

[54] WoodD., ZaidmanM., RuthL., HausenblasM.. Linked Data: Structured Data on the Web, 2014 Shelter Island Manning Publications

[55] S. Lemaignan, R. Ros, L. Mösenlechner, R. Alami, M. Beetz, in IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Oro, a knowledge management platform for cognitive architectures in robotics (Taipei, 2010), pp. 3548–3553. https://doi.org/10.1109/IROS.2010.5649547.

[56] LimG. H., SuhI. H., SuhH.. Ontology-based unified robot knowledge for service robots in indoor environments. IEEE Trans. Syst. Man Cybern. Syst. Hum., 2011, 41(3):492-509 https://doi.org/10.1109/TSMCA.2010.2076404

[57] NeuhausH., ComptonM.. The semantic sensor network ontology. AGILE workshop on challenges in geospatial data harmonisation, 2009 Germany Hannover 1-33

[58] J. Persson, A. Gallois, A. Bjoerkelund, L. Hafdell, M. Haage, J. Malec, K. Nilsson, P. Nugues, in International Symposium on Robotics (ISR). A knowledge integration framework for robotics (Munich, 2010), pp. 1–8.

[59] StenmarkM., MalecJ., NilssonK., RobertssonA.. On distributed knowledge bases for robotized small-batch assembly. IEEE Trans. Autom. Sci. Eng., 2015, 12(2):519-528 https://doi.org/10.1109/TASE.2015.2408264

[60] J. L. Carbonera, S. R. Fiorini, E. Prestes, V. A. M. Jorge, M. Abel, R. Madhavan, A. Locoro, P. Gonçalves, T. Haidegger, M. E. Barreto, C. Schlenoff, in Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Defining positioning in a core ontology for robotics (Tokyo, 2013), pp. 1867–1872. https://doi.org/10.1109/IROS.2013.6696603.

[61] C. Schlenoff, T. Hong, C. Liu, R. Eastman, S. Foufou, in Proceedings of the IEEE International Symposium on Robotic and Sensors Environments (ROSE). A literature review of sensor ontologies for manufacturing applications, (2013), pp. 96–101. https://doi.org/10.1109/ROSE.2013.6698425.

[62] BozcuogluA. K., KazhoyanG., FurutaY., StelterS., BeetzM., OkadaK., InabaM.. The exchange of knowledge using cloud robotics. Robot. Autom. Lett., 2018, 3(2):1072-1079 https://doi.org/10.1109/LRA.2018.2794626

[63] WaibelM., BeetzM., D’AndreaR., JanssenR., TenorthM., CiveraJ., ElfringJ., Gálvez-LópezD., HäussermannK., MontielJ., PerzyloA., SchießleB., ZweigleO., van de MolengraftR.. RoboEarth - A World Wide Web for Robots. Robot. Autom. Mag., 2011, 18(2):69-82

[64] M. Beetz, D. Beßler, A. Haidu, M. Pomarlan, A. Bozcuoglu, G. Bartels, in 2018 IEEE International Conference on Robotics and Automation (ICRA). Know rob 2.0 — a 2nd generation knowledge processing framework for cognition-enabled robotic agents (Brisbane, 2018), pp. 512–519. https://doi.org/10.1109/ICRA.2018.8460964.

[65] M. Tenorth, M. Beetz, Representations for robot knowledge in the knowrob framework. Artif. Intell. (2015). https://doi.org/10.1016/j.artint.2015.05.010.

[66] S. Blumenthal, H. Bruyninckx, W. Nowak, E. Prassler, in 2013 IEEE International Conference on Robotics and Automation (ICRA). A scene graph based shared 3D world model for robotic applications (Karlsruhe, 2013), pp. 453–460. https://doi.org/10.1109/icra.2013.6630614.

[67] S. Blumenthal, N. Hochgeschwender, E. Prassler, H. Voos, H. Bruyninckx, in 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). An approach for a distributed world model with QoS-based perception algorithm adaptation (Hamburg, p. 2015. https://doi.org/10.1109/iros.2015.7353612.

[68] YazdaniF., BlumenthalS., HuebelN., BozcuogluA. K., BeetzM., BruyninckxH.. Query-based integration of heterogeneous knowledge bases for search and rescue tasks. Robot. Autom. Mag., 2019, 117: 80-91 https://doi.org/10.1016/j.robot.2019.03.013. http://www.sciencedirect.com/science/article/pii/S0921889018303488

[69] BaileyT., Durrant-WhyteH.. Simultaneous localization and mapping (SLAM): part II. IEEE Robot. Autom. Mag., 2006, 13(3):108-117 https://doi.org/10.1109/MRA.2006.1678144

[70] Durrant-WhyteH., BaileyT.. Simultaneous localization and mapping: part I. IEEE Robot. Autom. Mag., 2006, 13(2):99-110 https://doi.org/10.1109/MRA.2006.1638022

[71] Vidal-CallejaT. A., BergerC., SolàJ., LacroixS.. Large scale multiple robot visual mapping with heterogeneous landmarks in semi-structured terrain. Robot. Auton. Syst., 2011, 59: 654-674 https://doi.org/10.1016/j.robot.2011.05.008

[72] A. Cunningham, K. M. Wurm, W. Burgard, F. Dellaert, in Proceedings of the IEEE International Conference on Robotics and Automation (ICRA). Fully distributed scalable smoothing and mapping with robust multi-robot data associationSaint Paul, 2012). https://doi.org/10.1109/ICRA.2012.6225356.

[73] T. Cieslewski, S. Lynen, M. Dymczyk, R. Siegwart, in Proceedings of the IEEE International Conference on Robotics and Automation (ICRA). Map API - scalable decentralized map building for robots (Seattle, 2015), pp. 6241–6247. https://doi.org/10.1109/ICRA.2015.7140075.

[74] R. Vincent, D. Fox, J. Ko, K. Konolige, B. Limketkai, B. Morisset, C. Ortiz, D. Schulz, B. Stewart, Distributed multirobot exploration, mapping, and task allocation. Ann. Math. Artif. Intell. (2008). https://doi.org/10.1007/s10472-009-9124-y.

[75] CoxA. B., GiffordF.. An overview to geographic information systems. J. Acad. Librariansh., 1997, 23(6):449-461 https://doi.org/10.1016/S0099-1333(97)90169-5

[76] CuraR., PerretJ., PaparoditisN.. Point cloud server (PCS) : point clouds IN-base management and processing. ISPRS Ann. Photogramm. Remote. Sens. Spat. Inf. Sci., 2015, II-3/W5: 531-539 https://doi.org/10.5194/isprsannals-ii-3-w5-531-2015. https://doi.org/10.5194%2Fisprsannals-ii-3-w5-531-2015

[77] F. Poux, P. Hallot, R. Neuville, R. Billen, in 3D Geoinfo Conference. Model for semantically rich point cloud data (Melbourne, 2017). https://doi.org/10.5194/isprs-annals-iv-4-w5-107-2017.

[78] CongaltonR. G.. Remote sensing and geographic information system data integration: error sources and research issues. Photogramm. Eng. Remote Sens., 1991, 57(6):677-687

[79] J. I. Meguro, K. Ishikawa, T. Hashizume, J. Takiguchi, I. Noda, M. Hatayama, in Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Disaster information integration into geographic information system using rescue robots (Beijing, 2006). https://doi.org/10.1109/IROS.2006.281636.

[80] MalleaA., ArenasM., HoganA., PolleresA.. On blank nodes. International semantic web conference, 2011 Bonn Springer 421-437 https://doi.org/10.1007/978-3-642-25073-6_27

[81] T. Foote, in 2013 IEEE Conference on Technologies for Practical Robot Applications (TePRA). tf: The transform library (Woburn, 2013), pp. 1–6. https://doi.org/10.1109/TePRA.2013.6556373.

[82] MacomberM. M.. World geodetic system 1984. Tech. rep., 1984 Fort Belvoir Defense Mapping Agency Washington DC https://doi.org/10.21236/ada147409

[83] WilliamsonR. S., RappR.. The development of the joint nasa gsfc and the national imagery and mapping agency (nima) geopotential model egm96. Tech. rep., 1998 Greenbelt Goddard Space Flight Center

[84] P. Ramsey, in FOSS4G. Lidar in postgresql with pointcloud (Nottingham, 2013).

Funding

stiftelsen f?r?strategisk forskning(RIT15-0097); elliit network organization for information and communication technology, sweden; Sichuan Province International Science and Technology Innovation Cooperation Project Grant 2020YFH0160; rexperts program grant 2020a1313030098 from the guangdong department of science and technology