Gesture recognition for engaging spatial experiences in healthcare: Co-design of intelligent interactive illuminative textiles

Abstract

The integration of artificial intelligence (AI) into textile design enhances functionality, automation, and user interaction. While gesture recognition has been explored in smart textiles, contactless interactive systems for healthcare remain underdeveloped. This study presents a human-centered co-design approach to the development of an AI-integrated gesture recognition system embedded in illuminative textile wall panels, aimed at enhancing spatial engagement in healthcare environments. The research was conducted in three key stages. First, a co-design workshop was conducted to explore user preferences in textile materials, graphic design, and gesture interaction. Second, intelligent illuminative textiles were developed by knitting polymeric optical fiber into base wool yarns to enable illumination. A camera was embedded and integrated with a computer vision-based deep learning model for detecting landmarks on the hands, shoulders, and head. The recognized gestures and body movements triggered specific pre-programmed color changes on the textile surface through edge-integrated light-emitting diodes. Finally, a prototype was fabricated and installed in a government-established District Health Centre in Hong Kong to support physical activity and rehabilitation for elderly users. Semi-structured interviews with stakeholders - including co-designers, users, and occupational therapists - were conducted to evaluate usability and inform design refinements. Stakeholders reported high levels of satisfaction, emphasizing the system’s ability to enhance community connection, therapeutic engagement, intuitive usability, and compelling visual feedback. These findings suggest that AI-driven interactive textiles present promising opportunities for rehabilitation, therapeutic environments, and the promotion of elderly well-being.

Keywords

Interactive textiles / Illuminative textiles / Gesture recognition / Human-artificial intelligence interaction / Deep learning / Healthcare

Acknowledgments

We sincerely appreciate the valuable support and guidance provided by Dr. Elsa Lee and her team at the WTSDHC, whose assistance was instrumental in facilitating the placement of the textile wall panels within the center.

Funding

This research is funded by the Laboratory for AI in Design (Project Code: RP3-5) under the InnoHK Research Clusters, Hong Kong Special Administrative Region.

Conflict of interest

Jeanne Tan is an Editorial Board Member of this journal, but was not in any way involved in the editorial and peer-review process conducted for this paper, directly or indirectly. Separately, other authors declared that they have no known competing financial interests or personal relationships that could have influenced the work reported in this paper.

Author contributions

Conceptualization: Jeanne Tan, Hiu Ting Tang

Investigation: Ching Lee, Hiu Ting Tang, Jun Jong Tan

Methodology: Ching Lee, Hiu Ting Tang, Jun Jong Tan

Supervision: Jeanne Tan

Writing - original draft: Jeanne Tan, Ching Lee

Writing - review & editing: Jeanne Tan, Ching Lee, Hiu Ting Tang, Wing Ki Yip, Ka Wing Tse

Ethics approval and consent to participate

The study received ethical approval from the Hong Kong Polytechnic University’s Institutional Review Board (Approval no.: HSEARS20200123003). All participants provided informed consent before the commencement of the interviews.

Consent for publication

Written informed consent for publication of the data was obtained from all participants involved in the study.

Availability of data

Data used in this work are available from the corresponding author upon reasonable request.

References

[1] Islam S, Shekhar R. Smart Textiles and Wearable Technology: Opportunities and Challenges in the Production and Distribution. Singapore: Springer Nature; 2025. p. 267-303.

[2] Scataglini S, Moorhead A, Feletti F. A systematic review of smart clothing in sports: Possible applications to extreme sports. Muscles Ligaments Tendons J. 2020; 10(2):333-342. doi: 10.32098/MLTJ.02.2020.19

[3] Wang C, Fu L, Ametefe DS, Wang S, John D. E-textiles in healthcare: A systematic literature review of wearable technologies for monitoring and enhancing human health. Neural Comput Appl. 2025; 37(4):2089-2111. doi: 10.1007/s00521-024-10947-z

[4] Sajovic I, Kert M, Boh PB. Smart textiles: A review and bibliometric mapping. Appl Sci. 2023; 13(18):10489. doi: 10.3390/app131810489

[5] Shajari S, Kuruvinashetti K, Komeili A, Sundararaj U. The emergence of ai-based wearable sensors for digital health technology: A review. Sensors (Basel). 2023; 23(23):9498. doi: 10.3390/s23239498

[6] Sarker MTH, Ahmed I, Rahaman MA. Ai-based smart textile wearables for remote health surveillance and critical emergency alerts: A systematic literature review. Am J Scholarly Res Innov. 2023; 2(2):1-29. doi: 10.63125/ceqapd08

[7] Yadav A, Yadav K. Transforming healthcare and fitness with AI powered next-generation smart clothing. Discov Electrochem. 2025; 2(1):2. doi: 10.1007/s44373-025-00015-z

[8] Avellar L, Stefano FC, Delgado G, Frizera A, Rocon E, Leal JA. AI-enabled photonic smart garment for movement analysis. Sci Rep. 2022; 12(1):4067. doi: 10.1038/s41598-022-08048-9

[9] Akter A, Apu MMH, Veeranki YR, Baroud TN, Posada- Quintero HF. Recent studies on smart textile-based wearable sweat sensors for medical monitoring: A systematic review. J Sens Actuator Netw. 2024; 13(4):40. doi: 10.3390/jsan13040040

[10] Tan J, Shao L, Lam NYK, Toomey A, Ge L. Intelligent textiles: Designing a gesture-controlled illuminated textile based on computer vision. Text Res J. 2022; 92(17-18):3034-3048. doi: 10.1177/00405175211034245

[11] World Health Organization. Decade of Healthy Ageing: Baseline Report. Geneva, Switzerland: World Health Organization; 2021.

[12] Department of Economic and Social Affairs, United Nations. World Population Prospects. New York, NY: United Nations; 2019.

[13] Committee on the Future Health Care Workforce for Older Americans. Retooling for an Aging America: Building the Health Care Workforce. Washington, DC: National Academies Press; 2008.

[14] Shah MN, Bazarian JJ, Lerner EB, et al. The epidemiology of emergency medical services use by older adults: An analysis of the national hospital ambulatory medical care survey. Acad Emerg Med. 2007; 14(5):441-447. doi: 10.1111/j.1553-2712.2007.tb01804.x

[15] United Nations Economic and Social Commission for Asia and the Pacific (UN ESCAP). Asia-Pacific Report on Population Ageing 2022: Trends, Policies and Good Practices Regarding Older Persons and Population Ageing. Bangkok, Thailand: UN ESCAP; 2022.

[16] United Nations Department of Economic and Social Affairs (UN DESA).Countries Forecast to Have the Highest Share of 65-Year-Old People Worldwide in 2050. New York, NY: UN DESA; 2023.

[17] Wilmoth JR, Bas D, Mukherjee S, Hanif N. World Social Report: Leaving no One Behind in an Ageing World. New York, NY: United Nations; 2023.

[18] McClellan CB. Health care utilization and expenditures in health professional shortage areas. Med Care Res Rev. 2024; 81(4):335-345. doi: 10.1177/10775587241235705

[19] Research Office,Legislative Council Secretariat. The 2025- 2026 Budget. Hong Kong: Legislative Council; 2025.

[20] Wu X, Law CK, Yip PSF. A projection of future hospitalisation needs in a rapidly ageing society: A Hong Kong experience. Int J Environ Res Public Health. 2019; 16(3):473. doi: 10.3390/ijerph16030473

[21] Choy R. Implementation of community care policy for older adults in Hong Kong. In: Law VTS, Fong BYF,editors. Ageing with Dignity in Hong Kong and Asia: Holistic and Humanistic Care. Ch. 3. Singapore: Springer Nature Singapore; 2022.

[22] Government of the Hong Kong Special Administrative Region (HKSAR). LCQ11:District Health Centres and District Health Centre Expresses. Press Releases from the Government of the Hong Kong Special Administrative Region. Available from: https://www.info.gov.hk/gia/general/202403/27/P2024032700591.htm [Last accessed on 2024 Apr 25].

[23] Wong Tai Sin District Health Centre, Key functions and features of DHC.Available from: https://www.dhc.gov.hk/en/healthcare-service-providers.html#key-functions-and-features-of-dhc [Last accessed on 2024 Apr 25].

[24] Ta AWA, Goh HL, Ang C, Koh LY, Poon K, Miller SM. Two Singapore public healthcare AI applications for national screening programs and other examples. Health Care Sci. 2022; 1(2):41-57. doi: 10.1002/hcs2.10

[25] Ang A. Behind Singapore’s Widespread AI Adoption in Public Health. Healthc IT News. Available from: https://www.healthcareitnews.com/news/asia/behind-singapores-widespread-ai-adoption-public-health [Last accessed on 2024 Apr 25].

[26] Raghavan A, Demircioglu MA, Taeihagh A. Public health innovation through cloud adoption: A comparative analysis of drivers and barriers in Japan, South Korea, and Singapore. Int J Environ Res Public Health. 2021; 18(1):334. doi: 10.3390/ijerph18010334

[27] Wright J. Inside Japan’s long experiment in automating elder care. MIT Technol Rev. [Internet]. 2023. Available from: https://www.technologyreview.com/2023/01/09/1065135/japan-automating-eldercare-robots [Last accessed 2024 Apr 25].

[28] Li LW, Ma CC. Application of AI in addressing challenges of primary healthcare in Hong Kong. In: Ageing with Dignity in Hong Kong and Asia: Holistic and Humanistic Care. Singapore: Springer Nature Singapore; 2025. p. 589-609.

[29] Legislative Council. Promotion of Smart Healthcare in Selected Places. Hong Kong: Legislative Council; 2022.

[30] Köttl H, Gallistl V, Rohner R, Ayalon L. But at the age of 85? Forget it!: Internalized ageism, a barrier to technology use. J Aging Stud. 2021; 59:100971. doi: 10.1016/j.jaging.2021.100971

[31] Zhang J, Wang H, Li Q, Luximon Y. What is the real-life experience of older adults on smart healthcare technologies? An exploratory interview study. Gerontology. 2024; 70(9):978-990. doi: 10.1159/000539539

[32] Millward P. The’grey digital divide’: Perception, exclusion and barriers of access to the internet for older people. First Monday. First Monday. Available from: https://firstmonday.org/ojs/index.php/fm/article/view/1066 [Last accessed on 2024 Apr 25].

[33] Rice RE, Katz JE. Comparing internet and mobile phone usage: Digital divides of usage, adoption, and dropouts. Telecomm Policy. 2003; 27(8-9): 597-623. doi: 10.1016/S0308-5961(03)00068-5

[34] Brown T. Design thinking. Harv Bus Rev. 2008; 86(6):84-92.

[35] Sanders EBN, Stappers PJ. Co-creation and the new landscapes of design. CoDesign. 2008; 4(1):5-18. doi: 10.1080/15710880701875068

[36] Steen M, Manschot M, De KN. Benefits of co-design in service design projects. Int J Des. 2011; 5(2):53-60.

[37] Bate P, Robert G. Experience-based design: from redesigning the system around the patient to co-designing services with the patient. Qual Saf Health Care. 2006; 15(5):307-310. doi: 10.1136/qshc.2005.016527

[38] Robert G, Cornwell J, Locock L, Purushotham A, Sturmey G, Gager M. Patients and staff as codesigners of healthcare services. BMJ. 2015;350:g7741. doi: 10.1136/bmj.g7714

[39] Locock L, Robert G, Boaz A, et al. Testing accelerated experience-based co-design: A qualitative study of using a national archive of patient experience narrative interviews to promote rapid patient-centred service improvement. Health Serv Deliv Res. 2014; 2(4):1-122. doi: 10.3310/hsdr02040

[40] Norman DA, Verganti R. Incremental and radical innovation: Design research vs. Technology and meaning change. Des Issues. 2014; 30(1):78-96. doi: 10.1162/desi-a-00250

[41] Rafael S, Santiago E, Rebelo F, Noriega P, Vilar E. Bio-centred interaction design:A new paradigm for human-system interaction. In: International Conference on Human-Computer Interaction. Cham: Springer International Publishing; 2022. p. 69-79.

[42] Woods B, O’Philbin L, Farrell EM, Spector AE, Orrell M.Reminiscence therapy for dementia. Cochrane Database Syst Rev. 2018; 2018(3): CD001120. doi: 10.1002/14651858.cd001120.pub3

[43] Lazar A, Demiris G, Thompson HJ. Evaluation of a multifunctional technology system in a memory care unit: Opportunities for innovation in dementia care. Inform Health Soc Care. 2016; 41(4):373-386. doi: 10.3109/17538157.2015.1064428

[44] Poupyrev I, Gong NW, Fukuhara S, Karagozler ME, Schwesig C, Robinson KE. Project jacquard:Interactive digital textiles at scale. In: Proceedings of the 2016 Chi Conference on Human Factors in Computing Systems. New York: Association for Computing Machinery; 2016. p. 4216-4227

[45] Gorka R, Subramaniyan AK, Velu R. Integrating advanced technologies in post-operative rehabilitation:3D-knitting, 3D-printed electronics, and sensor-embedded textiles. In: Zheng Y, Chow H,editors. Digital Health Innovation. Ch. 10. Singapore: Springer Nature; 2023.

[46] Caldani L, Pacelli M, Farina D, Paradiso R. E-Textile platforms for rehabilitation. In: 2010 Annual International Conference of the IEEE Engineering in Medicine and Biology Society. Piscataway, NJ: IEEE; 2010. p. 5181-5184.

[47] Meena JS, Choi SB, Jung SB, Kim JW. Electronic textiles: New age of wearable technology for healthcare and fitness solutions. Mater Today Bio. 2023; 19:100565. doi: 10.1016/j.mtbio.2023.100565

[48] Khatwani P, Desai K.Interactive smart textile fabrics. Interactive smart textile fabrics. In: Functional and Technical Textiles. United Kingdom: Woodhead Publishing; 2023. p. 293-311.

[49] Bai ZQ, Tan J, Johnston CF, Tao XM. Connexion development of interactive soft furnishings with polymeric optical fibre (POF) textiles. Int J Cloth Sci Technol. 2015; 27(6):870-894. doi: 10.1108/IJCST-05-2014-0058

[50] Cochrane C, Mordon SR, Lesage JC, Koncar V. New design of textile light diffusers for photodynamic therapy. Mater Sci Eng C Mater Biol Appl. 2013; 33(3):1170-1175. doi: 10.1016/j.msec.2012.12.007

[51] Schrank V, Beer M, Beckers M, Gries T. Polymer-optical fibre (POF) integration into textile fabric structures. In: Polymer Optical Fibres. United Kingdom: Woodhead Publishing; 2017. p. 337-348.

[52] Avellar L, Frizera A, Leal JA. POF smart pants: A fully portable optical fiber-integrated smart textile for remote monitoring of lower limb biomechanics. Biomed Opt Express. 2023; 14(7):3689-3704. doi: 10.1364/BOE.492796

[53] Gong Z, Xiang Z, OuYang X, et al. Wearable fiber optic technology based on smart textile: A review. Materials. 2019; 12(20):3311. doi: 10.3390/ma12203311

[54] Tan J, Shao L, Lam NYK, et al. Evaluating the usability of a prototype gesture-controlled illuminative textile. J Text Inst. 2024; 115(3):350-356. doi: 10.1080/00405000.2023.2193790

[55] Lam NYK, Tan J, Toomey A, Cheuk KCJ. Illuminative knitted textiles: Machine knitting with polymeric optical fibres (POFs). Res J Text Appare. 2024; 28(2):317-335. doi: 10.1108/RJTA-12-2021-0144

[56] Chen A, Tan J, Henry P, Tao X. The design and development of an illuminated polymeric optical fibre (POF) knitted garment. J Text Inst. 2020; 111(5):745-755. doi: 10.1080/00405000.2019.1661937

[57] Tam EY, Chi CM. The hong kong geriatrics society annual scientific meeting 2024. Asian J Gerontol Geriatr. 2024; 19(2):50-51.

[58] Park M, Park T, Park S, Yoon SJ, Koo SH, Park YL. Stretchable glove for accurate and robust hand pose reconstruction based on comprehensive motion data. Nat Commun. 2024; 15(1):5821. doi: 10.1038/s41467-024-50101-w

[59] Glauser O, Wu S, Panozzo D, Hilliges O, Sorkine-Hornung O. Interactive hand pose estimation using a stretch-sensing soft glove. ACM Trans Graph. 2019; 38(4):162. doi: 10.1145/3306346.3322957

[60] Panagiotou C, Faliagka E, Antonopoulos CP, Voros N. Multidisciplinary ML techniques on gesture recognition for people with disabilities in a smart home environment. AI. 2025; 6(1):17. doi: 10.3390/ai6010017

[61] Guo K, Orban M, Lu J, Al-Quraishi MS, Yang H, Elsamanty M. Empowering hand rehabilitation with AI-powered gesture recognition: A study of an sEMG-based system. Bioengineering (Basel). 2023; 10(5):557. doi: 10.3390/bioengineering10050557

[62] Oudah M, Al-Naji A, Chahl J. Elderly care based on hand gestures using kinect sensor. Computers. 2020; 10(1):5. doi: 10.3390/computers10010005

[63] Munoz NM, Kristoffersen MB, Sunnerhagen KS, Naber A, Alt MM, Ortiz CM. Upper limb stroke rehabilitation using surface electromyography: A systematic review and meta-analysis. Front Hum Neurosci. 2022; 16:897870. doi: 10.3389/fnhum.2022.897870

[64] Jaramillo YA, Benalcázar ME, Mena ME. Real-time hand gesture recognition using surface electromyography and machine learning: A systematic literature review. Sensors (Basel). 2020; 20(9):2467. doi: 10.3390/s20092467

[65] MediaPipe. MediaPipe - Hand Landmarker and Pose Landmarker. Google AI; 2023. Available from https://ai.google.dev/edge/mediapipe/solutions/vision/hand-landmarker [Last accessed on 2024 Apr 28].

[66] Anthropic. Claude: Helpful, Honest, and Harmless AI Assistant. Anthropic Website; 2023. Available from: https://www.anthropic.com [Last accessed on 2024 Apr 27].

[67] IRIDA. IRIDA Project - AI-Driven Mental Health Support Platform. IRIDA Website; 2023.Available from: https:// irida.health [last accessed on 2024 Apr 27].

[68] Koulieris GA, Kannengiesser N, Noristani A. Affect recognition through multimodal fusion of face, body pose, and physiological data using deep learning. Neurocomputing. 2023; 540:127171. doi: 10.1016/j.neucom.2023.127171