Holorailway: an augmented reality system to support assembly operations in the railway industry

Abstract

During the last two decades, industrial applications of augmented reality (AR) have been incorporated in sectors such as automotive or aeronautics in tasks including manufacturing, maintenance, and assembly. However, AR’s potential has yet to be demonstrated in the railway sector due to its complexity and difficulties in automating tasks. This work aims to present an AR system based on HoloLens 2 to assist the assembly process of insulation panels in the railway sector significantly decreasing the time required to perform the assembly. Along with the technical description of the system, an exhaustive validation process is provided where the assembly using the developed system is compared to the traditional assembly method as used by a company that has facilitated a case study. The results obtained show that the system presented outperforms the traditional solution by 78% in the time spent in the localization subtask, which means a 47% decrease in the global assembly time. Additionally, it decreases the number of errors in 88% of the cases, obtaining a more precise and almost error-free assembly process. Finally, it is also proven that using AR removes the dependence on users’ prior knowledge of the system to facilitate assembly.

Keywords

Augmented reality (AR) / Railway industry / Assembly / HoloLens 2

References

[1] Schwab K (2017) The fourth industrial revolution, Hardcover edn, p. 192. Currency, Geneva

[2] Lasi H, Fettke P, Kemper H-G, Feld T, Hoffmann M. Industry 4.0. Bus Inf Syst Eng, 2014, 6(4): 239-242.

[3] Grieves M. Digital twin: manufacturing excellence through virtual factory replication. White Pap, 2014, 1: 1-7.

[4] Qi Q, Tao F. Digital twin and big data towards smart manufacturing and industry 4.0: 360 degree comparison. IEEE Access, 2018, 6: 3585-3593.

[5] Carmigniani J. Furht B (2011) Augmented reality: an overview. Handb Augment Real, 2011, 66: 3-46.

[6] De Pace F, Manuri F, Sanna A. Augmented reality in industry 4.0. Am J Comput Sci Inf Technol, 2018, 6(1): 1-7.

[7] Santi GM, Ceruti A, Liverani A, Osti F. Augmented reality in industry 4.0 and future innovation programs. Technologies, 2021, 9(2): 33.

[8] Bongomin O, Yemane A, Kembabazi B, Malanda C, Chikonkolo Mwape M, Sheron Mpofu N, Tigalana D. Industry 4.0 disruption and its neologisms in major industrial sectors: a state of the art. J Eng, 2020, 2020: 66

[9] Langley A, Lawson G, Hermawati S, D’cruz M, Apold J, Arlt F, Mura K. Establishing the usability of a virtual training system for assembly operations within the automotive industry. Hum Factors Ergon Manuf Serv Ind, 2016, 26(6): 667-679.

[10] Fraga-Lamas P, Fernández-Caramés TM, Blanco-Novoa O, Vilar-Montesinos MA. A review on industrial augmented reality systems for the industry 4.0 shipyard. IEEE Access, 2018, 6: 13358-13375.

[11] Regenbrecht H, Baratoff G, Wilke W. Augmented reality projects in the automotive and aerospace industries. IEEE Comput Graph Appl, 2005, 25(6): 48-56.

[12] Popov O, Iatsyshyn A, Sokolov D, Dement M, Neklonskyi I, Yelizarov A. Zaporozhets A, Artemchuk V. Application of virtual and augmented reality at nuclear power plants. Systems, decision and control in energy II, 2021, Cham: Springer 243-260.

[13] de Souza Cardoso LF, Mariano FCMQ, Zorzal ER. A survey of industrial augmented reality. Comput Ind Eng, 2020, 139.

[14] Geng J, Song X, Pan Y, Tang J, Liu Y, Zhao D, Ma Y. A systematic design method of adaptive augmented reality work instruction for complex industrial operations. Comput Ind, 2020, 119.

[15] Dalle Mura M, Dini G (2021) Augmented reality in assembly systems: state of the art and future perspectives. In: International precision assembly seminar. Springer, pp 3–22

[16] Ashwini KB, Patil PN et al (2020) Tracking methods in augmented reality–explore the usage of marker-based tracking. In: Proceedings of the 2nd international conference on IoT, social, mobile, analytics & cloud in computational vision & bio-engineering (ISMAC-CVB 2020)

[17] Masood T, Egger J. Augmented reality in support of industry 4.0—implementation challenges and success factors. Robot Comput Integr Manuf, 2019, 58: 181-195.

[18] Georgel PF (2011) Is there a reality in industrial augmented reality? In: 2011 10th IEEE international symposium on mixed and augmented reality. IEEE, pp 201–210

[19] Ortega M, Ivorra E, Juan A, Venegas P, Martínez J, Alcañiz M. Mantra: an effective system based on augmented reality and infrared thermography for industrial maintenance. Appl Sci, 2021, 11(1): 385.

[20] Ungureanu D, Bogo F, Galliani S et al (2020) Hololens 2 research mode as a tool for computer vision research. arXiv preprint arXiv:2008.11239

[21] Vidal-Balea A, Blanco-Novoa O, Fraga-Lamas P et al (2020) Creating collaborative augmented reality experiences for Industry 4.0 training and assistance applications: performance evaluation in the shipyard of the future. Appl Sci 10(24):9073. https://doi.org/10.3390/app10249073

[22] Psarommatis F, May G, Dreyfus P-A, Kiritsis D. Zero defect manufacturing: state-of-the-art review, shortcomings and future directions in research. Int J Prod Res, 2020, 58(1): 1-17.

[23] Psarommatis F, Sousa J, Mendonça JP, Kiritsis D. Zero-defect manufacturing the approach for higher manufacturing sustain-ability in the era of Industry 4.0: a position paper. Int J Prod Res, 2022, 60(1): 73-91.

[24] Wohlgemuth W, Triebfürst G (2000) Arvika: augmented reality for development, production and service. In: Proceedings of DARE 2000 on designing augmented reality environments, pp 151–152

[25] Lima JP, Roberto R, Simões F, Almeida M, Figueiredo L, Teixeira JM, Teichrieb V. Markerless tracking system for augmented reality in the automotive Industry. Expert Syst Appl, 2017, 82: 100-114.

[26] Egger J, Masood T. Augmented reality in support of intelligent manufacturing—a systematic literature review. Comput Ind Eng, 2020, 140.

[27] Oh YJ, Park KY, Kim EK (2014) Mobile augmented reality system for design drawing visualization. In: 16th International conference on advanced communication technology. IEEE, pp 1296–1300

[28] Masoni R, Ferrise F, Bordegoni M, Gattullo M, Uva AE, Fiorentino M, Carrabba E, Di Donato M. Supporting remote maintenance in industry 4.0 through augmented reality. Procedia Manuf, 2017, 11: 1296-1302.

[29] Funk M, Kosch T, Schmidt A (2016) Interactive worker assistance: comparing the effects of in-situ projection, head-mounted displays, tablet, and paper instructions. In: Proceedings of the 2016 ACM international joint conference on pervasive and ubiquitous computing, pp 934–939

[30] Davies R (2015) Industry 4.0: digitalisation for productivity and growth. EPRS: European Parliamentary Research Service. https://policycommons.net/artifacts/1335939/industry-40/1942749/

[31] Masood T, Egger J. Adopting augmented reality in the age of industrial digitalization. Comput Ind, 2020, 115.

[32] Doshi A, Smith RT, Thomas BH, Bouras C. Use of projector based augmented reality to improve manual spot-welding precision and accuracy for automotive manufacturing. Int J Adv Manuf Technol, 2017, 89(5): 1279-1293.

[33] Mourtzis D, Zogopoulos V, Vlachou E. Augmented reality supported product design towards Industry 4.0: a teaching factory paradigm. Procedia Manuf, 2018, 23: 207-212.

[34] Alvarez H, Aguinaga I, Borro D (2011) Providing guidance for maintenance operations using automatic markerless augmented reality system. In: 2011 10th IEEE international symposium on mixed and augmented reality. IEEE, pp 181–190

[35] Fiorentino M, Uva AE, Gattullo M, Debernardis S, Monno G. Augmented reality on large screen for interactive maintenance instructions. Comput Ind, 2014, 65(2): 270-278.

[36] Mourtzis D, Zogopoulos V, Vlachou E. Augmented reality application to support remote maintenance as a service in the robotics Industry. Procedia Cirp, 2017, 63: 46-51.

[37] Hořejší P. Augmented reality system for virtual training of parts assembly. Procedia Eng, 2015, 100: 699-706.

[38] Füchter SK, Schlichting MS, Salazar G. Aeronautic pilot training and augmented reality. Acta Imeko, 2021, 10(3): 66-71.

[39] Fründ J, Gausemeier J, Matysczok C et al (2005) Using augmented reality technology to support the automobile development. In: computer supported cooperative work in design I: 8th international conference, CSCWD 2004, Xiamen, 26–28 May, Xiamen, China.

[40] Ceruti A, Marzocca P, Liverani A, Bil C. Maintenance in aeronautics in an Industry 4.0 context: the role of augmented reality and additive manufacturing. J Comput Des Eng, 2019, 6(4): 516-526.

[41] Mourtzis D, Zogopoulos V, Xanthi F. Augmented reality application to support the assembly of highly customized products and to adapt to production re-scheduling. Int J Adv Manuf Technol, 2019, 105(9): 3899-3910.

[42] Frigo MA, da Silva E, Barbosa GF. Augmented reality in aerospace manufacturing: a review. J Ind Intell Inf, 2016, 4(2): 66

[43] Jeffri NFS, Rambli DRA. A review of augmented reality systems and their effects on mental workload and task performance. Heliyon, 2021, 7(3): 06277.

[44] de Oliveira ME, Corrˆea CG (2020) Virtual reality and augmented reality applications in agriculture: a literature review. In: 2020 22nd symposium on virtual and augmented reality (SVR). IEEE, pp 1–9

[45] Woll R, Damerau T, Wrasse K et al (2011) Augmented reality in a serious game for manual assembly processes. In: 2011 IEEE international symposium on mixed and augmented reality-arts, media, and humanities. IEEE, pp 37–39

[46] Barfield W, Caudell T (2001) Boeing’s wire bundle assembly project. In: fundamentals of wearable computers and augmented reality. CRC Press, pp 462–482

[47] Liu Y, Li S, Wang J, Zeng H, Lu J. A computer vision-based assistant system for the assembly of narrow cabin products. Int J Adv Manuf Technol, 2015, 76(1): 281-293.

[48] Lai Z-H, Tao W, Leu MC, Yin Z. Smart augmented reality instructional system for mechanical assembly towards worker-centered intelligent manufacturing. J Manuf Syst, 2020, 55: 69-81.

[49] Richardson T, Gilbert S, Holub J et al (2014) Fusing self-reported and sensor data from mixed-reality training

[50] Garrido-Jurado S, Muñoz-Salinas R, Madrid-Cuevas FJ et al (2016) Generation of fiducial marker dictionaries using mixed integer linear programming, pattern recognition. 51:481–491. https://doi.org/10.1016/j.patcog.2015.09.023

[51] Evans G, Miller J, Pena MI et al (2017) Evaluating the microsoft hololens through an augmented reality assembly application. In: degraded environments: sensing, processing, and display 2017, vol 10197. SPIE, pp 282–297

[52] Ariansyah D, Erkoyuncu JA, Eimontaite I, Johnson T, Oostveen A-M, Fletcher S, Sharples S. A head mounted augmented reality design practice for maintenance assembly: toward meeting perceptual and cognitive needs of ar users. Appl Ergon, 2022, 98.

[53] Vorraber W, Gasser J, Webb H, Neubacher D, Url P. Assessing augmented reality in production: remote-assisted maintenance with hololens. Procedia CIRP, 2020, 88: 139-144.

[54] Brunzini A, Mandolini M, Caragiuli M, Germani M et al (2021) Hololens 2 for maxillofacial surgery: a preliminary study. In: international conference on design, simulation, manufacturing: the innovation exchange. Springer, pp 133–140

[55] Leu MC, ElMaraghy HA, Nee AY, Ong SK, Lanzetta M, Putz M, Zhu W, Bernard A. Cad model based virtual assembly simulation, planning and training. CIRP Ann, 2013, 62(2): 799-822.

[56] Webel S, Becker M, Stricker D et al (2007) Identifying differences between cad and physical mock-ups using ar. In: 2007 6th IEEE and ACM international symposium on mixed and augmented reality. IEEE, pp 281–282

[57] Azpiazu J, Siltanen S , Multanen P et al (2011) Remote support for maintenance tasks by the use of augmented reality: the ManuVAR project. CARVI 2011: IX congress on virtualreality applications, 11-12 November, Alava, Spain

[58] Kwon H-J, Lee S-I, Park J-H, Kim C-S. Design of augmented reality train-ing content for railway vehicle maintenance focusing on the axle-mounted disc brake system. Appl Sci, 2021, 11(19): 9090.

[59] Yi B, Sun R, Long L, Song Y, Zhang Y. From coarse to fine: an augmented reality-based dynamic inspection method for visualized railway routing of freight cars. Meas Sci Technol, 2022, 33(5): .

[60] Seeliger A, Netland T, Feuerriegel S. Aug-mented reality for machine setups: Task performance and usability evaluation in a field test. Procedia CIRP, 2022, 107: 570-575.

[61] Teruggi S, Fassi F (2022) Hololens 2 spatial map-ping capabilities in vast monumental heritage environments. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XLVI-2/W1-2022, pp. 489–496

[62] Unger M, Heinrich S, Rick M, Halama D, Chalopin C. Hologram accuracy evaluation of hololens 2 for thermal imaging in medical applications. Curr Direct Biomed Eng, 2022, 8(2): 193-196.

[63] Curless B, Levoy M (1996) A volumetric method for building complex models from range images. In: proceedings of the 23rd annual conference on computer graphics and interactive techniques, pp 303–312

[64] Marino E, Barbieri L, Colacino B, Fleri AK, Bruno F. An augmented reality inspection tool to support workers in Industry 4.0 environments. Comput Ind, 2021, 127: 103412.

[65] Lu Y, Zheng H, Chand S, Xia W, Liu Z, Xu X, Wang L, Qin Z, Bao J. Outlook on human-centric manufacturing towards Industry 5.0. J Manuf Syst, 2022, 62: 612-627.

[66] Cao A, Chintamani KK, Pandya AK, Ellis RD. Nasa tlx: software for assessing subjective mental workload. Behav Res Methods, 2009, 41(1): 113-117.

[67] Loizeau Q, Danglade F, Ababsa F et al (2019) Evaluating added value of augmented reality to assist aeronautical maintenance workers—experimentation on on-field use case. In: international conference on virtual reality and augmented reality. Springer, pp 151–169

[68] Re GM, Oliver J, Bordegoni M. Impact of monitor-based augmented reality for on-site industrial manual operations. Cognit Technol Work, 2016, 18(2): 379-392.

[69] Jordan PW, Thomas B, McClelland IL, Weerdmeester B. Usability evaluation in industry, 1996, Boca Raton: CRC Press.

[70] Fischini A, Ababsa F, Grasser M (2018) Usability of augmented reality in aeronautic maintenance, repair and overhaul. International conference on artificial reality and telexistence and Eurographics symposium on virtual environments, Nov 2018, Limassol, Cyprus.

Funding

H2020 Future and Emerging Technologies http://dx.doi.org/10.13039/100010664(H2020-FETPROACT-EIC-07-2020-101017727)

Universitat Politècnica de València