Assistive assessment of neurological dysfunction via eye movement patterns and ocular biometrics

Abstract

Aim: Nerve dysfunction often manifests as abnormal eye behaviors, necessitating accurate and objective neurological assessment. Current deep learning-based facial analysis methods lack adaptability to inter-patient variability, making it difficult to capture subtle and rapid ocular dynamics such as incomplete eyelid closure or asymmetric eye movement. To address this, we propose a precise deep learning system for quantitative ocular state analysis, providing objective support for the evaluation of neurological dysfunction.

Methods: We propose the Ocular-enhanced Face Keypoints Net (OFKNet). It incorporates three key innovations: (1) a 40-point anatomically informed ocular landmark design enabling dense modeling of eyelid contours, canthus structure, and pupil dynamics; (2) a MobileNetV3-based region enhancement module that amplifies feature responses within clinically critical areas such as the internal canthus; (3) and an improved Path Aggregation Network combined with Squeeze-and-Excitation modules that enables adaptive multi-scale fusion and enhances sensitivity to subtle ocular deformations.

Results: Using clinically acquired data, OFKNet demonstrates substantial performance gains over state-of-the-art baselines. It achieves a 65.3% reduction in normalized mean error on the 40-point dataset (0.029 vs. 0.084) and a 38.4% reduction on the 14-point dataset, with all improvements statistically significant (P < 0.001). Despite operating on high-resolution inputs, the system maintains real-time capability and provides stable frame-level landmark localization, enabling precise capture of dynamic ocular motion patterns.

Conclusion: OFKNet provides a reliable tool for real-time monitoring of eye movement patterns in patients with neurological disorders. By visualizing time-series graphs of bilateral eye openness, the system enables a more comprehensive understanding of ocular dynamics and supports timely clinical decision-making and treatment adjustment.

Keywords

Computer vision / nerve dysfunction / facial paralysis / convolutional neural networks / and eye movement tracking

References

[1] Skaramagkas V,Kefalopoulou Z.Multi-modal deep learning diagnosis of Parkinson’s disease-a systematic review.IEEE Trans Neural Syst Rehabil Eng2023;31:2399-423

[2] Lou J,Wang FY.A review on automated facial nerve function assessment from visual face capture.IEEE Trans Neural Syst Rehabil Eng2020;28:488-97

[3] Liu X,Li X,Guo Y.Facial paralysis evaluation based on improved residual network. In: 2023 2nd International Conference on Advanced Sensing, Intelligent Manufacturing (ASIM); 2023 May 12-14; Changsha City, China. New York: IEEE; 2023. pp. 36-40.

[4] Liu X,Yu H,Jian M.Region based parallel hierarchy convolutional neural network for automatic facial nerve paralysis evaluation.IEEE Trans Neural Syst Rehabil Eng2020;28:2325-32

[5] Ge X,Wang P,Liu X.ALGRNet: multi-relational adaptive facial action unit modelling for face representation and relevant recognitions.IEEE Trans Biom Behav Identity Sci2023;5:566-78

[6] Zhang Y,Xu Z.The feasibility of an automatical facial evaluation system providing objective and reliable results for facial palsy.IEEE Trans Neural Syst Rehabil Eng2023;31:1680-6

[7] Zhang Y,Yu H,Xia Y.Artificial intelligence-based facial palsy evaluation: a survey.IEEE Trans Neural Syst Rehabil Eng2024;32:3116-34

[8] Xia Y,Yap Kannan R,Enrique Berner J.AFLFP: a database with annotated facial landmarks for facial palsy.IEEE Trans Comput Soc Syst2023;10:1975-85

[9] Jin B,Gonçalves N.Pseudo RGB-D face recognition.IEEE Sensors J2022;22:21780-94

[10] Jin B,Cruz L,Yu Y.Simulated multimodal deep facial diagnosis.Expert Syst Appl2024;252:123881

[11] Sachs NA,Vyas N,Weiland JD.Electrical stimulation of the paralyzed orbicularis oculi in rabbit.IEEE Trans Neural Syst Rehabil Eng2007;15:67-75

[12] Linhares CDG,Ponciano JR.ClinicalPath: a visualization tool to improve the evaluation of electronic health records in clinical decision-making.IEEE Trans Vis Comput Graph2023;29:4031-46

[13] Liu Z,Wu CY,Darrell T. A ConvNet for the 2020s. In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); 2022 Jun 19-24; New Orleans, LA, USA. New York: IEEE; 2022. pp. 11966-76. Available from: https://openaccess.thecvf.com/content/CVPR2022/html/Liu_A_ConvNet_for_the_2020s_CVPR_2022_paper.html [accessed 9 December 2025].

[14] Howard A,Chen B. Searching for MobileNetV3. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV); 2019 Oct 27-Nov 2; Seoul, South Korea. New York: IEEE; 2019. pp. 1314-24. Available from: https://openaccess.thecvf.com/content_ICCV_2019/html/Howard_Searching_for_MobileNetV3_ICCV_2019_paper.html [accessed 9 December 2025].

[15] Paszke A,Massa F. Pytorch: an imperative style, high-performance deep learning library. In: Wallach H, Larochelle H, Beygelzimer A, d’Alché-Buc F, Fox E, Garnett R, Editors. Advances in Neural Information Processing Systems 32. NeurIPS 2019; 2019 Dec 8-14; Vancouver, Canada. NeurIPS; 2019. Available from: https://proceedings.neurips.cc/paper/2019/hash/bdbca288fee7f92f2bfa9f7012727740-Abstract.html [accessed 9 December 2025].

[16] Liu S,Qin H,Jia J. Path aggregation network for instance segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR); 2018 Jun 18-22; Salt Lake City, UT, USA. New York: IEEE; 2018. pp. 8759-68. Available from: https://openaccess.thecvf.com/content_cvpr_2018/html/Liu_Path_Aggregation_Network_CVPR_2018_paper.html [accessed 9 December 2025].

[17] Hu J,Sun G. Squeeze-and-excitation networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR); 2018 Jun 18-22; Salt Lake City, UT, USA. New York: IEEE; 2018. pp. 7132-41. Available from: https://openaccess.thecvf.com/content_cvpr_2018/html/Hu_Squeeze-and-Excitation_Networks_CVPR_2018_paper.html [accessed 9 December 2025].

[18] Moncaliano MC,Goshe JM,Ciolek PJ.Clinical features, evaluation, and management of ophthalmic complications of facial paralysis: a review.J Plast Reconstr Aesthet Surg2023;87:361-8

[19] Ding L.Features versus context: an approach for precise and detailed detection and delineation of faces and facial features.IEEE Trans Pattern Anal Mach Intell2010;32:2022-38 PMCID:PMC3657115

[20] Lugaresi C,Nash H. MediaPipe: a framework for perceiving and processing reality. In: Third workshop on computer vision for AR/VR at IEEE computer vision and pattern recognition (CVPR); 2019 Jun 16-20; Long Beach, CA, USA. New York: IEEE; 2019. Available from: https://static1.squarespace.com/static/5c3f69e1cc8fedbc039ea739/t/5e130ff310a69061a71cbd7c/1578307584840/NewTitle_May1_MediaPipe_CVPR_CV4ARVR_Workshop_2019.pdf [accessed 9 December 2025].

[21] King DE. Dlib-ml: a machine learning toolkit. J Mach Learn Res 2009;10:1755-8. Available from: https://www.jmlr.org/papers/volume10/king09a/king09a.pdf [accessed 9 December 2025].

[22] Deng J,An X,Zafeiriou S. Masked face recognition challenge: the insightface track report. In: IEEE/CVF International Conference on Computer Vision (ICCV) Workshops; 2021 Oct 11-17; Virtual. New York: IEEE; 2021. pp. 1437-44. Available from: https://openaccess.thecvf.com/content/ICCV2021W/MFR/html/Deng_Masked_Face_Recognition_Challenge_The_InsightFace_Track_Report_ICCVW_2021_paper.html [accessed 9 December 2025].

[23] Wagner K.Proportionate-type normalized least mean square algorithms with gain allocation motivated by mean-square-error minimization for white input.IEEE Trans Signal Process2011;59:2410-5

[24] Mandal B,Wang GS.Towards detection of bus driver fatigue based on robust visual analysis of eye state.IEEE Trans Intell Transport Syst2017;18:545-57

[25] Chen S.Efficient and robust pupil size and blink estimation from near-field video sequences for human-machine interaction.IEEE Trans Cybern2014;44:2356-67

[26] Aloudat M,El-Sayed A.Automated vision-based high intraocular pressure detection using frontal eye images.IEEE J Transl Eng Health Med2019;7:3800113 PMCID:PMC6537927

[27] Liu S,Yan D.Alterations in patients with first-episode depression in the eyes-open and eyes-closed conditions: a resting-state EEG study.IEEE Trans Neural Syst Rehabil Eng2022;30:1019-29

[28] Chiranjeevi P,Moogi P.Neutral face classification using personalized appearance models for fast and robust emotion detection.IEEE Trans Image Process2015;24:2701-11