Improving maintenance efficiency and safety through a human-centric approach

Abstract

This paper presents an adaptive human-machine interface (HMI) that can provide appropriate sets of digital maintenance information and guidance to an operator during maintenance. It takes into consideration the expertise level of the operator and the maintenance context and progress. The proposed human-centric methodology considers the heart rate, intention, and expertise level of the operator, which can be captured using sensors during maintenance. A set of rules is formulated based on the sensor data to infer the state of the operator during a maintenance task. Based on the operator state, the adaptive HMI can augment the operator’s senses using a scheme that combines visual, audio, and haptic guidance cues during maintenance to enhance the operator’s ability to perceive information and perform maintenance tasks. Various schemes of visual, audio, and haptic cues are developed based on a comparison of the best practices obtained from experienced operators.

Keywords

Augmented reality (AR) / Ergonomics / Product development / Maintenance

References

[1.] Rødseth H, Schjølberg P, Marhaug A. Deep digital maintenance. Adv Manuf, 2017, 5: 299-310.

[2.] Wang X, Ong SK, Nee AYC. A comprehensive survey of augmented reality assembly research. Adv Manuf, 2016, 4: 1-22.

[3.] Khalek IA, Chalhoub JM, Ayer SK. Augmented reality for identifying maintainability concerns during design. Adv Civil Eng, 2019, 3: 8547928.

[4.] Gattullo M, Scurati GW, Evangelista A et al (2020) Informing the use of visual assets in industrial augmented reality. In: Proceedings of the international conference on design tools and methods in industrial engineering, 9–10 September, Modena, Italy, pp 106–117

[5.] Martinettia A, Rajabalinejada M, van Dongena L. Shaping the future maintenance operations: reflections on the adoptions of augmented reality through problems and opportunities. Proc CIRP, 2017, 59: 4-17.

[6.] Abramovici M, Wolf M, Adwernat S, et al. Context-aware maintenance support for augmented reality assistance and synchronous multi-user collaboration. Proc CIRP, 2017, 59: 18-22.

[7.] Gattullo M, Dalena V, Evangelista A et al (2019) A context-aware technical information manager for presentation in augmented reality. In: 2019 IEEE conference on virtual reality and 3D user interfaces, 23–27 March, Osaka, Japan, pp 939–940

[8.] Caudell TP, Mizell DW (1992) Augmented reality: an application of heads-up display technology to manual manufacturing processes. In: Proceedings of the 25th international conference on system sciences, Kauai, Hawaii, 7–10 January, pp 659–669

[9.] Friedrich W, Jahn D, Schmidt L (2002) ARVIKA: augmented reality for development, production and service. In: The 1st international symposium on mixed and augmented reality (ISMAR), Darmstadt, 30 September 1 October, pp 3–4. https://doi.org/10.1109/ismar.2002.1115059

[10.] De Groote P. Maintenance performance analysis: a practical approach. J Qual Maint Eng, 1995, 1(2): 4-24.

[11.] Dini G, Mura MD. Application of augmented reality techniques in through-life engineering services. Proc CIRP, 2015, 38: 14-23.

[12.] Regenbrecht HT, Specht R (2000) A mobile passive augmented reality device-Mpard. In: ISAR 2000, Munich, Germany, 5–6 October, pp 81–84. https://doi.org/10.1109/isar.2000.880926

[13.] Ayala GA, Bobadilla IG, Figueroa GA, et al. Virtual reality training system for maintenance and operation of high-voltage overhead power lines. Virtual Reality, 2016, 20(1): 1-14.

[14.] Jetter J, Eimecke J, Rese A. Augmented reality tools for industrial applications: what are potential key performance indicators and who benefits?. Comput Hum Behav, 2018, 87: 18-33.

[15.] Oliveira R, Farinha T, Raposo H et al (2014) Augmented reality and the future of maintenance. In: Proceedings of maintenance performance measurement and management (MPMM) conference, 4–5 September, Coimbra, pp 81–88

[16.] Gattullo M, Uva AE, Fiorentino M, et al. Legibility in industrial AR: text style, color coding, and illuminance. IEEE Comput Gr Appl, 2015, 35(2): 52-61.

[17.] Dünser A, Grasset R, Seichter H et al (2007) Applying HCI principles to AR systems design. In: Workshop proceedings of mixed reality user interfaces: specification, authoring, adaptation (MRUI’07), 11 Mar, Charlotte, NC, USA, pp 37–42

[18.] Müller T (2015) Towards a framework for information presentation in augmented reality for the support of procedural tasks. In: International conference on augmented and virtual reality, 31 August 3 September, Lecce, Italy, pp 490–497

[19.] Aschenbrenner D, Latoschik ME, Schilling K (2016) Industrial maintenance with augmented reality: two case studies. In: Proceedings of the 22nd ACM conference on virtual reality software and technology (VRST16). ACM Press, Garching, Germany, pp 341–342

[20.] Mourtzisa D, Zogopoulos V, Vlachou E. Augmented reality application to support remote maintenance as a service in the robotics industry. Proc CIRP, 2017, 63: 46-51.

[21.] Kammerer K, Pryss R, Sommer K et al (2018) Towards context-aware process guidance in cyber-physical systems with augmented reality. In: The 4th international workshop on requirements engineering for self-adaptive, collaborative, and cyber physical systems, 20 Auguest, Banff, Canada, pp 44–51

[22.] Zhu J, Ong SK, Nee AYC. A context-aware augmented reality assisted maintenance system. Int J Comput Integr Manuf, 2015, 28(2): 213-225.

[23.] Erkoyuncu JA, del Amo IF, Dalle MM, et al. Improving efficiency of industrial maintenance with context aware adaptive authoring in augmented reality. Ann CIRP, 2017, 66: 465-468.

[24.] Hervás R, Bravo J, Fontecha J, et al. Achieving adaptive augmented reality through ontological context-awareness applied to AAL scenarios. J Univ Comput Sci, 2013, 19(9): 1334-1349.

[25.] Holm M, Danielsson O, Syberfeldt A, et al. Adaptive instructions to novice shop-floor operators using augmented reality. J Ind Prod Eng, 2017, 34(5): 362-374.

[26.] Syberfeldt A, Danielsson O, Holm M, et al. Dynamic operator instructions based on augmented reality and rule-based expert systems. Proc CIRP, 2016, 41: 346-351.

[27.] Lages WS, Bowman DA (2019) Walking with adaptive augmented reality workspaces: design and usage patterns. In: The 24th international conference on intelligent user interfaces, 17–20 March, CM, New York, USA, pp 356–366

[28.] Petrangeli S, Simon G, Wang H et al (2019) Dynamic adaptive streaming for augmented reality applications. In: IEEE international symposium on multimedia (ISM), 9–11 December, San Diego, California, USA, pp 56–63

[29.] Tsai CH, Huang JY. Augmented reality display based on user behavior. Comput Stand Interfaces, 2018, 55: 171-181.

[30.] Healey JA, Picard RW. Detecting stress during real-world driving tasks using physiological sensors. IEEE Trans Intelligent Transp Syst, 2005, 6(2): 156-166.

[31.] Knopfle C, Weidenhausen J, Chauvigne L et al (2005) Template based authoring for AR based service scenarios. In: IEEE proceedings of virtual reality, 12–16 March, Bonn, Germany

[32.] Gattullo M, Uva AE, Fiorentino M, et al. From paper manual to AR manual: do we still need text?. Proc Manuf, 2017, 11: 1303-1310.

[33.] Masoni R, Ferrise F, Bordegoni M, et al. Supporting remote maintenance in Industry 4.0 through augmented reality. Proc Manuf, 2017, 11: 1296-1302.

[34.] Jo GS, Oh KJ, Ha I et al (2014) A unified framework for augmented reality and knowledge-based systems in maintaining aircraft. In: Proceedings of the national conference on artificial intelligence, 27–31 July, Quebec, Canada, pp 2990–2997

[35.] Palmarini R, Erkoyuncu JA, Roy R, et al. A systematic review of augmented reality applications in maintenance. Robot Comput Integr Manuf, 2018, 49: 215-228.