A Framework for Quantifying Autonomy in Robotic Systems

Abstract

Although autonomous functioning facilitates the deployment of robotic systems in operating domains that support limited to no human oversight, establishing correspondence between task requirements and a system’s autonomous performance is still an open challenge. Several techniques for characterizing operating domains and/or quantifying autonomy have been proposed over the last three decades, however, to our knowledge, these have no discernment of sub-mode features of variation of autonomy, and some are based on metrics that are susceptible to the Goodhart’s law. This paper introduces a capability-based quantitative autonomy assessment framework for fully autonomous systems. The formulation of the framework started by establishing robot task characteristics from which three autonomy metrics, namely an essential capability set, reliability, and responsiveness, were derived. The characteristics were founded on the realization that robots ultimately replace human skilled workers, from which a relationship between human job and robot task characteristics was established. Additionally, mathematical formulations relating metrics to autonomy are also presented. To emphasize the fact that autonomy is not just a question of existence, but also one of performance of a capability, the framework represents it as a two-part measure, of level and degree of autonomy. Usage of the framework has been demonstrated on two case studies, namely an autonomous vehicle at an on-road dynamic driving task and the DARPA Subterranean Challenge analysis. The framework provides not only a tool for quantifying autonomy and monitoring the integrity of systems, but also a regulatory interface and common language for autonomous systems’ developers and users.

Keywords

Autonomy framework / Autonomy metrics / Degree of autonomy / Level of autonomy / Integrity monitoring

References

[1] Singh S. Critical Reasons for Crashes Investigated in the National Motor Vehicle Crash Causation Survey; National Highway Traffic Safety Administration (NHTSA), US Department of Transportation: Washington, DC, USA, 2015.

[2] Rankin W. Meda investigation process. Boeing Commer. Aero 2007, 2, 15-21. Available online: http://pages.suddenlink.net/reed/aero.pdf (accessed on 5 November 2022).

[3] Clough BT. Metrics, Schmetrics! How the Heck Do You Determine a Uav’s Autonomy Anyway; Air Force Research Laboratory: Wright—Patterson AFB, OH, USA, 2002.

[4] Jackson E, Williams O, Buchan K. Achieving robot autonomy. In Proceedings of the 3rd Conference on Military Robotic Applications, Medicine Hat, AB, Canada, 9—12 September 1991.

[5] Lampe A, Chatila R. Performance measure for the evaluation of mobile robot autonomy. In Proceedings of the 2006 IEEE International Conference on Robotics and Automation, Orlando, FL, USA, 15—19 May 2006; pp. 4057-4062. DOI: 10.1109/ROBOT.2006.1642325

[6] Insaurralde CC, Lane DM. Autonomy—assessment criteria for underwater vehicles. In Proceedings of the 2012 IEEE/OES Autonomous Underwater Vehicles (AUV), Southampton, UK, 24—27 September 2012; pp. 1-8. DOI: 10.1109/AUV.2012.6380746

[7] Kendoul F. Towards a unified framework for uas autonomy and technology readiness assessment (atra). In Autonomous Control Systems and Vehicles; Springer: Tokyo, Japan, 2013; pp. 55-71. DOI: 10.1007/978—4—431—54276—6_4

[8] Clothier R, Williams B, Perez T. A review of the concept of autonomy in the context of the safety regulation of civil unmanned aircraft systems. In Proceedings of the Australian System Safety Conference 2013 (ASSC 2013) [Conferences in Research and Practice in Information Technology (CRPIT), Conferences in Research and Practice in Information Technology]; Australian Computer Society Inc.: Sydney, Australia, 2014; pp. 15-27.

[9] Elbanhawi M, Mohamed A, Clothier R, Palmer JL, Simic M, Watkins S. Enabling technologies for autonomous mav operations. Prog. Aerosp. Sci. 2017, 91, 27-52. DOI: 10.1016/j.paerosci.2017.03.002

[10] Huang H—M, Messina E, Albus J. Autonomy Levels for Unmanned Systems (Alfus) Framework, Volume II: Framework Models Version 1.0. 2007. Available online: https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication1011—II—1.0.pdf (accessed on 13 April 2022). DOI: 10.6028/NIST.SP.1011—II—1.0

[11] On—Road Automated Driving (ORAD) Committee. Taxonomy and Definitions for Terms Related to Driving Automation Systems for On—Road Motor Vehicles; SAE International: Warrendale, PA, USA, 2021. DOI: 10.4271/J3016_202104

[12] California Department of Motor Vehicles. California Code of Regulations Title 13, Division 1, Chapter 1, Article 3.7— Testing of Autonomous Vehicles. 2025. Available online: https://www.dmv.ca.gov/portal/uploads/2020/06/Adopted— Regulatory—Text—2019.pdf (accessed on 17 September 2022).

[13] Chrystal KA, Mizen PD, Mizen P. Goodhart’s law: Its origins, meaning and implications for monetary policy. Cent. Bank. Monet. Theory Pract. Essays Honour Charles Goodhart 2003, 1, 221-243. DOI: 10.4337/9781781950777

[14] Doboli A, Curiac D, Pescaru D, Doboli S, Tang W, Volosencu C, et al. Cities of the future: Employing wireless sensor networks for efficient decision making in complex environments. arXiv 2018, arXiv:1808.01169. DOI: 10.48550/arXiv.1808.01169

[15] Curtin TB, Crimmins DM, Curcio J, Benjamin M, Roper C. Autonomous underwater vehicles: Trends and transformations. Mar. Technol. Soc. J. 2005, 39, 65-75. DOI: 10.4031/002533205787442521

[16] Russell SJ. Artificial Intelligence a Modern Approach; Pearson Education, Inc.: Hoboken, NJ, USA, 2010.

[17] Hasslacher B, Tilden MW. Living machines. Robot. Auton. Syst. 1995, 15, 143-169. DOI: 10.1016/0921—8890(95)00019—C

[18] Lytle CW. Job Evaluation Methods; Ronald Press: Santa Fe, NM, USA, 1946; pp. 56-58.

[19] Gorelik G. Bremermann’s limit in cgh—physics. arXiv 2009, arXiv:0910.3424. DOI: 10.48550/arXiv.0910.3424

[20] Ogata K. Discrete—Time Control Systems; Prentice—Hall, Inc.: Upper Saddle River, NJ, USA, 1995; p. 9.

[21] ICAO. International standards and recommended practices. In Annex 10 to the Convention on International Civil Aviation; International Civil Aviation Organization: Montreal, QC, Canada, 2018; Volume 1.

[22] International Organization for Standardization. ISO 26262—9:2018 Road Vehicles—Functional Safety—Part 9: Automotive Safety Integrity Level (ASIL)—Oriented and Safety—Oriented Analyses; International Organization for Standardization: Geneva, Switzerland, 2018.

[23] Wang X, Chen T, Wang R, Lu J, Dou G. Review of state estimation methods for autonomous ground vehicles: Perspectives on estimation objects, vehicle characteristics, and key algorithms. Sensors 2025, 25, 3927. DOI: 10.3390/s25133927

[24] Paz D, Lai PJ, Chan N, Jiang Y, Christensen HI. Autonomous vehicle benchmarking using unbiased metrics. In Proceedings of the 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Las Vegas, NV, USA, 24 October 2020—24 January 2021; pp. 6223-6228. DOI: 10.1109/IROS45743.2020.9340902

[25] Reid TG, Houts SE, Cammarata R, Mills G, Agarwal S, Vora A, et al. Localization requirements for autonomous vehicles. SAE Int. J. Connect. Autom. Veh. 2019, 2, 173-190. DOI: 10.4271/12—02—03—0012

[26] Darpa Subterranean Challenge: Competition Rules Final Event. Available online: https://github.com/subtchallenge/official_docs/blob/main/SubT_Challenge_Finals_Rules.pdf (accessed on 1 October 2023).