WARA-PS: a research arena for public safety demonstrations and autonomous collaborative rescue robotics experimentation

Abstract

A research arena (WARA-PS) for sensing, data fusion, user interaction, planning and control of collaborative autonomous aerial and surface vehicles in public safety applications is presented. The objective is to demonstrate scientific discoveries and to generate new directions for future research on autonomous systems for societal challenges. The enabler is a computational infrastructure with a core system architecture for industrial and academic collaboration. This includes a control and command system together with a framework for planning and executing tasks for unmanned surface vehicles and aerial vehicles. The motivating application for the demonstration is marine search and rescue operations. A state-of-art delegation framework for the mission planning together with three specific applications is also presented. The first one concerns model predictive control for cooperative rendezvous of autonomous unmanned aerial and surface vehicles. The second project is about learning to make safe real-time decisions under uncertainty for autonomous vehicles, and the third one is on robust terrain-aided navigation through sensor fusion and virtual reality tele-operation to support a GPS-free positioning system in marine environments. The research results have been experimentally evaluated and demonstrated to industry and public sector audiences at a marine test facility. It would be most difficult to do experiments on this large scale without the WARA-PS research arena. Furthermore, these demonstrator activities have resulted in effective research dissemination with high public visibility, business impact and new research collaborations between academia and industry.

Keywords

Autonomous systems / Intelligent system architectures / Research demonstration arena / Autonomous drones / Autonomous marine vessels / Public safety and security / Collaborative robotics

References

[1] WASP, WASP Home Page (2021). https://wasp-sweden.org/. [Online; accessed June 2021].

[2] WASP, WARA-PS Arena (2021). https://wasp-sweden.org/research/research-arenas/wara-ps-public-safety/. [Online; accessed June 2021].

[3] Wikipedia, Wikipedia Pubic Safety and Security Home Page (2021). https://en.wikipedia.org/wiki/Public_security. [Online; accessed July 2021].

[4] WASP, WASP Publications (2021). https://wasp-sweden.org/research/publications/f. [Online; accessed July 2021].

[5] SaabNaval, Saab Naval Home Page (2021). https://www.saab.com/products/naval. [Online; accessed July 2021].

[6] SaabAir, Saab Air Home Page (2021). https://www.saab.com/products/air. [Online; accessed July 2021].

[7] Combitech, Combitech Home Page (2021). https://www.combitech.se/. [Online; accessed July 2021].

[8] SaabAB, Saab AB Home Page (2021). https://www.saab.com/. [Online; accessed July 2021].

[9] Axis, Axis Communications Home Page (2021). https://www.axis.com/en-us/. [Online; accessed July 2021].

[10] Skeldar, UMS Skeldar Home Page (2021). https://umsskeldar.aero/. [Online; accessed July 2021].

[11] AIICS, Artificial Intelligence and Integrated Computer Systems Division, IDA, LiU Home Page (2021). https://www.ida.liu.se/divisions/aiics/. [Online; accessed June 2021].

[12] SMaRC, Swedish Maritime Robotics Centre Home Page (2021). https://smarc.se/. [Online; accessed July 2021].

[13] SSRS, SSRS Home Page (2021). https://www.sjoraddning.se/. [Online; accessed July 2021].

[14] M. Dastani, J. -J. C. Meyer, in Proceedings of the AAMAS07 Workshop on Programming Multi-Agent Systems (ProMAS2007). A practical agent programming language, (2007), pp. 72–87.

[15] FirbyR. J.. An investigation into reactive planning in complex domains. Proceedings of the National Conference on Artificial Intelligence, 1987 Seattle AAAI Press 202-206

[16] KonoligeK.. Colbert: A language for reactive control in Sapphira. KI-97: Advances in Artificial Intelligence, 1997 Berlin Heidelberg Springer 31-52

[17] MacKenzieD. C., ArkinR., CameronJ. M.. Multiagent mission specification and execution. Auton. Robots, 1997, 4(1):29-52

[18] R. Simmons, D. Apfelbaum, in Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). A task description language for robot control, (1998).

[19] P. Ulam, Y. Endo, A. Wagner, R. C. Arkin, in ICRA. Integrated mission specification and task allocation for robot teams – design and implementation, (2007).

[20] A. Marzinotto, M. Colledanchise, C. Smith, P. Ögren, Towards a unified behavior trees framework for robot control, (2014).

[21] P. Doherty, D. Landén, F. Heintz, in Proceedings of the International Conference on Principles and Practice of Multi-Agent Systems (PRIMA). A distributed task specification language for mixed-initiative delegation, (2010).

[22] P. Doherty, J. Kvarnström, A. Szalas, in Proceedings of the International Conference on Principles of Knowledge Representation and Reasoning (KR). Temporal composite actions with constraints, (2012), pp. 478–488.

[23] DohertyP., HeintzF., KvarnströmJ.. High-level Mission Specification and Planning for Collaborative Unmanned Aircraft Systems using Delegation. Unmanned Syst., 2013, 1: 75-119

[24] DohertyP., KvarnströmJ., WzorekM., RudolP., HeintzF., ConteG.. ValavanisK. P., VachtsevanosG. J.. HDRC3 - A Distributed Hybrid Deliberative/Reactive Architecture for Unmanned Aircraft Systems. Handbook of Unmanned Aerial Vehicles, 2014 Netherlands Springer 849-952 This article exists in revised form in the 2nd edition of the handbook 2016

[25] MarconiL., MelchiorriC., BeetzM., PangercicD., SiegwartR., LeuteneggerS., CarloniR., StramigioliS., BruyninckxH., DohertyP., KleinerA., LippielloV., FinziA., SicilianoB., SalaA., TomatisN.. The SHERPA project: Smart collaboration between humans and ground-aerial robots for improving rescuing activities in alpine environments. Proceedings of the IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR), 2012 USA IEEE 1-4 https://doi.org/10.1109/SSRR.2012.6523905

[26] DohertyP., RudolP.. A UAV search and rescue scenario with human body detection and geolocalization. AI’07: Proceedings of the Australian Joint Conference on Advances in Artificial Intelligence, 2007 Berlin Heidelberg Springer 1-13

[27] P. Doherty, J. Kvarnström, P. Rudol, M. Wzorek, G. Conte, C. Berger, T. Hinzmann, T. Stastny, in PRIMA 2016: Principles and Practice of Multi-Agent Systems, 9862. A Collaborative Framework for 3D Mapping using Unmanned Aerial Vehicles, (2016).

[28] T. Hinzmann, T. Stastny, G. Conte, P. Doherty, P. Rudol, M. Wzorek, I. Gilitschenski, E. Galceran, R. Siegwart, in International Symposium on Robotics. Collaborative 3D reconstruction using heterogeneous unmanned aerial vehicles, (2016).

[29] BergerC., WzorekM., KvarnströmJ., ConteG., DohertyP., ErikssonA.. Area Coverage with Heterogeneous UAVs using Scan Patterns. 2016 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR), 2016 USA IEEE Robotics and Automation Society 342-349

[30] DohertyP., KvarnströmJ., HeintzF.. A Temporal Logic-based Planning and Execution Monitoring Framework for Unmanned Aircraft Systems. Auton. Agent. Multi-Agent Syst., 2009, 19(3):332-377 https://doi.org/10.1007/s10458-009-9079-8

[31] DohertyP., MeyerJ. -J. C.. PaglieriF., TummoliniL., FalconeR., MiceliM.. On the logic of delegation - relating theory and practice. The Goals of Cognition: Essays in Honour of Cristiano Castelfranchi, 2012 London College Publications Ltd 467-496

[32] Foundation for Intelligent Physical Agents, FIPA Communicative Act Library Specification (2002). http://www.fipa.org.

[33] Foundation for Intelligent Physical Agents, FIPA Contract Net Interaction Protocol Specification (2002). http://www.fipa.org.

[34] CORBA, CORBA Home Page (2021). https://www.corba.org/. [Online; accessed May 2021].

[35] ROS, ROS Homepage (2021). http://wiki.ros.org/. [Online; accessed May 2021].

[36] ROS-Industrial, ROS Industrial Home Page (2021). https://rosindustrial.org. [Online; accessed May 2021].

[37] MQTT, http://mqtt.org/. [Online; accessed May 2021] (2021). MQTT Home Page.

[38] WARA-Portal, WARA-PS Research Portal (2021). https://portal.waraps.org/. [Online; accessed June 2021].

[39] Airpelago, Airpelago Home Page (2021). https://www.airpelago.com/. [Online; accessed July 2021].

[40] L. Persson, B. Wahlberg, in AIAA Scitech 2019 Forum. Model predictive control for autonomous ship landing in a search and rescue scenario, (2019). https://doi.org/10.2514/6.2019-1169. https://arc.aiaa.org/doi/abs/10.2514/6.2019-1169.

[41] L. Persson, B. Wahlberg, in 2018 European Control Conference (ECC). Verification of cooperative maneuvers in flightgear using MPC and backwards reachable sets, (2018), pp. 1411–1416. https://doi.org/10.23919/ECC.2018.8550247.

[42] Bereza-JarocinskiR., PerssonL., WahlbergB.. Distributed model predictive control for cooperative landing. IFAC-PapersOnLine, 2019, 53(2):15180-15185 https://doi.org/10.1016/j.ifacol.2020.12.2290

[43] L. Persson, B. Wahlberg, in 2021 IEEE Aerospace Conference. Variable prediction horizon control for cooperative landing on moving target, (2021).

[44] L. Persson, A. Hansson, B. Wahlberg, in (Submitted to Control Engineering Practice). A computationally fast variable horizon MPC algorithm with application in rendezvous of autonomous unmanned vehicles, (2021).

[45] StellatoB., BanjacG., GoulartP., BemporadA., BoydS.. OSQP: an operator splitting solver for quadratic programs. Math. Program. Comput., 2020, 12(4):637-672 https://doi.org/10.1007/s12532-020-00179-2

[46] D. P. Bertsekas, Dynamic Programming and Optimal Control. (3rd edn., ed.), vol. I (Athena Scientific, Belmont, 2005).

[47] AnderssonO.. Learning to make safe real-time decisions under uncertainty for autonomous robots. PhD thesis, 2020 Sweden Linköping University https://doi.org/10.3384/diss.diva-163419

[48] O. Andersson, M. Wzorek, P. Rudol, P. Doherty, in 2016 IEEE International Conference on Robotics and Automation (ICRA). Model-predictive control with stochastic collision avoidance using bayesian policy optimization, (2016), pp. 4597–4604. https://doi.org/10.1109/ICRA.2016.7487661.

[49] DomahidiA., ZgraggenA., ZeilingerM. N., MorariM., JonesC. N.. Efficient interior point methods for multistage problems arising in receding horizon control. IEEE Conference on Decision and Control (CDC), 2012 Maui IEEE 668-674

[50] HouskaB., FerreauH. J., DiehlM.. ACADO toolkit–An open-source framework for automatic control and dynamic optimization. Optim. Control Appl. Methods, 2011, 32(3):298-312

[51] GelbartM., SnoekJ., AdamsR.. Bayesian optimization with unknown constraints. Proceedings of the Thirtieth Conference Annual Conference on Uncertainty in Artificial Intelligence (UAI-14), 2014 Corvallis, Oregon AUAI Press 250-259

[52] J. M. Hernández-Lobato, M. A. Gelbart, M. W. Hoffman, R. P. Adams, Z. Ghahramani, in Proceedings of the 32nd International Conference on Machine Learning (ICML). Predictive entropy search for bayesian optimization with unknown constraints, (2015), pp. 1699–1707.

[53] AnderssonO., WzorekM., DohertyP.. Deep Learning Quadcopter Control Via Risk-Aware Active Learning. Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence (AAAI), 2017 California AAAI Press 3812-3818

[54] AnderssonO., LjungqvistO., TigerM., AxehillD., HeintzF.. Receding-horizon lattice-based motion planning with dynamic obstacle avoidance. 2018 IEEE Conference on Decision and Control (CDC), 2018 USA IEEE 4467-4474

[55] AnderssonO., SidénP., DahlinJ., DohertyP., VillaniM.. Real-time robotic search using structural spatial point processes. Proceedings of the Thirty-Fifth Conference on Uncertainty in Artificial Intelligence, UAI 2019, Tel Aviv, Israel, July 22-25, 2019, 2019 Arlington AUAI Press

[56] I. Stenius, P. Sigray, G. Linn, Smarc mid-term evauation report. Technical Report TRITA-SCI-RAP-2020.006, KTH Royal Institute of Technology, 54 (2020). https://doi.org/10.30746/TRITA-SCI-RAP-2020:006.

[57] M. Lager, E. A. Topp, J. Malec, in Multisensor Fusion and Integration for Intelligent Systems. Underwater terrain navigation during realistic scenarios (Springer International Publishing, 2017), pp. 186–209. https://doi.org/10.1007/978-3-319-90509-9_11.

[58] LagerM., ToppE. A., MalecJ.. Robust terrain-aided navigation through sensor fusion. 2020 IEEE 23rd International Conference on Information Fusion (FUSION), 2020 USA IEEE 1-8 https://doi.org/10.23919/fusion45008.2020.9190578

[59] M. Lager, E. A. Topp, J. Malec, in Proceedings of the 1st International Workshop on Virtual, Augmented, and Mixed Reality for HRI (VAM-HRI). Remote operation of unmanned surface vessel through virtual reality-a low cognitive load approach, (2018). https://doi.org/10.6084/m9.figshare.13537019.

[60] LagerM., ToppE. A., MalecJ.. VR teleoperation to support a GPS-free positioning system in a marine environment. Int. J. Mar. Navig. Saf. Sea Transp., 2020, 14(4):789-798

[61] T. Porathe, in Usability Professionals’ Association Conference. User-centered map design, (2007).

[62] Unity, Unity (2021). https://unity.com/. [Online; accessed May 2021].

[63] J. Behley, M. Garbade, A. Milioto, J. Quenzel, S. Behnke, C. Stachniss, J. Gall, Semantickitti: A dataset for semantic scene understanding of lidar sequences, (2019).

[64] GengK., DongG., YinG., HuJ.. Deep dual-modal traffic objects instance segmentation method using camera and lidar data for autonomous driving. Remote Sens., 2020, 12(20):3274

[65] PhanD., RosenqvistS.. Semi-Automatic Annotation and Gathering of Marine Images. Student Paper, 2021 Sweden Lund University https://lup.lub.lu.se/student-papers/search/publication/9064600

[66] I. H. Organization, Standards and Specifications (2021). https://iho.int/en/standards-and-specifications.

[67] IALAAISM (International Association of Marine Aids to Navigation and Lighthouse Authorities), IALA maritime buoyage system. https://www.iala-aism.org/.

[68] US department of Defense, Global Positioning System Standard Positioning Service Performance Standard (4th Edition, September 2008). https://www.gps.gov/technical/ps/2008-SPS-performance-standard.pdf.

[69] A. Bochkovskiy, C. -Y. Wang, H. -Y. M. Liao, Yolov4: Optimal speed and accuracy of object detection. arXiv preprint arXiv:2004.10934 (2020). https://arxiv.org/abs/2004.10934.

[70] R. Girshick, J. Donahue, T. Darrell, J. Malik, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Rich feature hierarchies for accurate object detection and semantic segmentation, (2014), pp. 580–587.

[71] GrabnerH., LeistnerC., BischofH.. Semi-supervised on-line boosting for robust tracking. European Conference on Computer Vision, 2008 Berlin Heidelberg Springer 234-247

[72] BabenkoB., YangM. -H., BelongieS.. Visual tracking with online multiple instance learning. 2009 IEEE Conference on Computer Vision and Pattern Recognition, 2009 USA IEEE 983-990

[73] LiY., ZhuJ.. A scale adaptive kernel correlation filter tracker with feature integration. European Conference on Computer Vision, 2014 Cham Springer 254-265

[74] KalalZ., MikolajczykK., MatasJ.. Tracking-learning-detection. IEEE Trans. Pattern Anal. Mach. Intell., 2011, 34(7):1409-1422

[75] KalalZ., MikolajczykK., MatasJ.. Forward-backward error: Automatic detection of tracking failures. 2010 20th International Conference on Pattern Recognition, 2010 USA IEEE 2756-2759

[76] BolmeD. S., BeveridgeJ. R., DraperB. A., LuiY. M.. Visual object tracking using adaptive correlation filters. 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2010 USA IEEE 2544-2550

[77] A. Lukezic, T. Vojir, L. Čehovin Zajc, J. Matas, M. Kristan, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Discriminative correlation filter with channel and spatial reliability, (2017), pp. 6309–6318.

[78] Tzutalin, Git code. LabelImg. GitHub (2015). https://github.com/tzutalin/labelImg.

[79] AnderssonJ. A. E., GillisJ., HornG., RawlingsJ. B., DiehlM.. CasADi – A software framework for nonlinear optimization and optimal control. Math. Program. Comput., 2018, 11: 1-36

Funding

knut och alice wallenbergs stiftelse(WASP); vetenskapsr?det; stiftelsen f?r?strategisk forskning(RIT15-0097)