A wearable assistive system for the visually impaired using object detection, distance measurement and tactile presentation

Abstract

With the current development of society, ensuring traffic and walking safety for the visually impaired is becoming increasingly important. We propose a wearable system based on a system previously developed by us that uses object recognition, a distance measurement function, and the corresponding vibration pattern presentation to support the mobility of the visually impaired. The system recognizes obstacles in front of a user in real time, measures their distances, processes the information, and then presents safety actions through vibration patterns from a tactile glove woven with shape memory alloy (SMA) actuators. The deep learning model is compressed to achieve real-time recognition using a microcomputer while maintaining recognition accuracy. Measurements of the distances to multiple objects are realized using a stereo camera, and vibration patterns are presented through a tactile glove in response to these distances. Experiments are conducted to verify the system performance to provide safe navigation depending on the positions and the distances of multiple obstacles in front of the user.

Keywords

SMA / tactile display / wearable device / visually impaired / object detection / distance measurement / deep learning / model compression

References

[1] Number of visually impaired people to "triple by 2050" worldwide. Available online: https://www.bbc.com/japanese/40810904 [Last accessed on 8 Sep 2023]

[2] Dionisi A,Serpelloni M.Wearable object detection system for the blind. 2012 IEEE International Instrumentation and Measurement Technology Conference Proceedings, Graz, Austria; 2012, pp. 1255-8.

[3] Wagner CR,Howe RD.A tactile shape display using RC servomotors. Proceedings 10th Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems. HAPTICS 2002, Orlando, FL, USA; 2002, pp. 354-5.

[4] Mukhiddinov M.Smart glass system using deep learning for the blind and visually impaired.Electronics2021;10:2756

[5] Shen J,Sawada H.A Wearable assistive device for blind pedestrians using real-time object detection and tactile presentation.Sensors2022;22:4537. PMCID:PMC9229985

[6] Ashveena A,Mary SP.Portable camera based identification system for visually impaired people. 2023 7th International Conference on Trends in Electronics and Informatics (ICOEI). Tirunelveli, India; 2023, pp. 1444-50.

[7] Redmon J,Girshick R.You only look once: unified, real-time object detection. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR); 2016, pp. 779-88.

[8] Liu Z,Shen Z,Yan S.Learning efficient convolutional networks through network slimming. In Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV); 2017, pp. 2755-63.

[9] Intel neural compute stick 2 (in Japanese). Available online: https://www.intel.co.jp/content/www/jp/ja/products/sku/140109/intel-neural-compute-stick-2/specifications.html [Last accessed on 5 Sep 2023]

[10] Kolkur S,Shimpi P,Jatakia J.Human skin detection using RGB, HSV and YCbCr color models. Proceedings of the International Conference on Communication and Signal Processing 2016 (ICCASP 2016); 2016, pp. 324-32.

[11] Zou L.A method of stereo vision matching based on OpenCV. 2010 International Conference on Audio, Language and Image Processing; 2010, pp. 185-90.

[12] Extract skin color regions using HSV color space in Python, OpenCV (in Japanese). Available online: https://www.udemy.com/course/computervision_mediapipe/ [Last accessed on 8 Sep 2023]

[13] Jiang C, Uchida K, Sawada H. Research and development of vision based tactile display system using shape memory alloys (in Japanese). IJICIC 2014;10:837-50. Available online: https://waseda.elsevierpure.com/ja/publications/research-and-development-of-vision-based-tactile-display-system-u [Last accessed on 8 Sep 2023]

[14] Sawada H.Tactile pad for the presentation of tactile sensation from moving pictures. In Proceedings of the 2014 7th International Conference on Human System Interactions (HSI). Costa da Caparica, Portugal; 2014, pp. 135-40.

[15] Zhao F,Sawada H.Compact Braille display using SMA wire array. RO-MAN 2009- The 18th IEEE International Symposium on Robot and Human Interactive Communication; 2009, pp. 28-33.