Unlocking the head: unleashing deep learning and depth camera for free head movement gaze estimation

Abstract

Gaze estimation has applications such as visual attention analysis and human-computer interaction. However, performance under free head movement conditions can be further enhanced. This study aims to improve the accuracy and robustness of appearance-based gaze estimation without head fixation. We develop a gaze-tracking system using a consumer depth camera and a deep-learning-based gaze estimation model [Gaze-Point-Net (GPN)]. A customized YOLOv5-based detector is developed to simultaneously localize the eyes and mouth, providing reliable facial landmarks for gaze estimation. The detected regions and depth image were used to calculate a vector of location and posture. In GPN, a double-channel convolutional neural network with a Squeeze-and-Excitation module extracts features from binocular images, and these features are concatenated with the location-and-posture vector to complete the multimodal gaze-position prediction task. Our GPN presents competitive performance in four experiments: (1) The error distribution in the sequential point test ranges from 4 to 10 degrees; (2) In the random point test, the calibrated and filtered GPN achieved a pixel error of 185.19 ± 116.57 pixels and an angular error of 4.52° ± 2.87°, demonstrating superior performance over counterpart methods; (3) In the trajectory tracking test, approximately 81.91% of gaze points were located within a 200-pixel tolerance radius of the target trajectory; (4) In the browsing test, the generated gaze heat and trajectory maps showed satisfactory results. The developed eye-tracking system, integrating a depth camera and deep learning models, demonstrates competitive performance and strong potential for several eye-tracking applications.

Keywords

Gaze estimation / eye movement / deep learning / depth camera / free head movement

References

[1] Casado-Aranda L,Ibáñez-Zapata J.Evaluating communication effectiveness through eye tracking: benefits, state of the art, and unresolved questions.Int J Bus Commun2023;60:24-61

[2] Liu J,Yang H.In the eye of the beholder: a survey of gaze tracking techniques.Pattern Recognit2022;132:108944

[3] Fan K,Meng Z.Predicting the reader’s English level from reading fixation patterns using the siamese convolutional neural network.IEEE Trans Neural Syst Rehabil Eng2022;30:1071-80

[4] Wang FS,Meboldt M.An algorithmic approach to determine expertise development using object-related gaze pattern sequences.Behav Res Methods2022;54:493-507 PMCID:PMC8863757

[5] Fan X,Song D,Liu J.GazMon: eye gazing enabled driving behavior monitoring and prediction.IEEE Trans Mob Comput2021;20:1420-33

[6] Čegovnik T,Jakus G.An analysis of the suitability of a low-cost eye tracker for assessing the cognitive load of drivers.Appl Ergon2018;68:1-11

[7] Li P,Duan X,Song G.Appearance-based gaze estimator for natural interaction control of surgical robots.IEEE Access2019;7:25095-110

[8] Chhimpa GR,Garhwal S.Development of a real-time eye movement-based computer interface for communication with improved accuracy for disabled people under natural head movements.J Real Time Image Proc2023;20:81

[9] Ramirez Gomez, A.; Lankes, M. Eyesthetics: making sense of the aesthetics of playing with gaze.Proc ACM Hum Comput Interact2021;5:1-24

[10] Papavlasopoulou S,Melhart D.Investigating gaze interaction to support children’s gameplay.Int J Child Comput Interact2021;30:100349

[11] Deane O,Yeo SH.Deep-SAGA: a deep-learning-based system for automatic gaze annotation from eye-tracking data.Behav Res Methods2023;55:1372-91 PMCID:PMC10126076

[12] Yuan G,Yan H.Self-calibrated driver gaze estimation via gaze pattern learning.Knowl Based Syst2022;235:107630

[13] Namnakani O,Grizou J.Comparing dwell time, pursuits and gaze gestures for gaze interaction on handheld mobile devices. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery; 2023. pp. 1-17.

[14] Hacques G,Komar J.Visual control during climbing: variability in practice fosters a proactive gaze pattern.PLoS One2022;17:e0269794 PMCID:PMC9187105

[15] Fischer T,Demiris Y.RT-GENE: real-time eye gaze estimation in natural environments. In Computer Vision - ECCV 2018. Cham: Springer International Publishing; 2018. pp. 339-57.

[16] Wu Z,Van As T,Rabinovich A.EyeNet: a multi-task deep network for off-axis eye gaze estimation. In 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), Seoul, Korea. Ocr 27-28, 2019. IEEE; 2019. pp. 3683-7.

[17] Wang Y,Li J.Contrastive regression for domain adaptation on gaze estimation. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, USA. Jun 18-24, 2022. IEEE; 2022. pp. 19354-63.

[18] Nonaka S,Nishino K.Dynamic 3D gaze from afar: deep gaze estimation from temporal eye-head-body coordination. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, USA. Jun 18-24, 2022. IEEE; 2022. pp. 2182-91.

[19] Fatahipour H,Fariborz J.Uncalibrated eye gaze estimation using SE-ResNext with unconstrained head movement and ambient light change.Research Square2023;

[20] Ma C,Choi K.Improved remote gaze estimation using corneal reflection-adaptive geometric transforms.Opt Eng2014;53:053112

[21] Shin Y,Kim S.A novel single IR light based gaze estimation method using virtual glints.IEEE Trans Consum Electron2015;61:254-60

[22] Chi J,Wang F,Hou Z.3-D gaze-estimation method using a multi-camera-multi-light-source system.IEEE Trans Instrum Meas2020;69:9695-708

[23] Liu J,Hu W.3D model-based gaze tracking via iris features with a single camera and a single light source.IEEE Trans Human Mach Syst2021;51:75-86

[24] Qin J,Sugano Y.Learning-by-novel-view-synthesis for full-face appearance-based 3D gaze estimation. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), New Orleans, USA. Jun 19-20, 2022. IEEE; 2022. pp. 4977-87.

[25] Li J,Zhong Y.Appearance-based gaze estimation for ASD diagnosis.IEEE Trans Cybern2022;52:6504-17

[26] Wang Q,Dang R.Style transformed synthetic images for real world gaze estimation by using residual neural network with embedded personal identities.Appl Intell2023;53:2026-41

[27] Khanam R. What is YOLOv5: a deep look into the internal features of the popular object detector. arXiv 2024, arXiv:2407.20892. Available online: https://doi.org/10.48550/arXiv.2407.20892. (accessed 2026-09-11)

[28] Keselman L,Grunnet-Jepsen A.Intel(R) realSense(TM) stereoscopic depth cameras. In 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Honolulu, USA. Jul 21-26, 2017. IEEE; 2017. pp. 1267-76.

[29] Hu J,Albanie S,Wu E.Squeeze-and-excitation networks.IEEE Trans Pattern Anal Mach Intell2020;42:2011-23

[30] Park S,Molchanov P,Hilliges O.Few-shot adaptive gaze estimation. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Korea. Oct 27 - Nov 02, 2019. IEEE; 2019. pp. 9367-76.

[31] Valliappan N,Steinberg E.Accelerating eye movement research via accurate and affordable smartphone eye tracking.Nat Commun2020;11:4553 PMCID:PMC7486382

[32] Criminisi A,Konukoglu E.Decision forests: a unified framework for classification, regression, density estimation, manifold learning and semi-supervised learning.Found Trends Comput Graph Vis2012;7:81-227

[33] Zhang Y,Cui C,Zhao D.Predicting breast cancer response to neoadjuvant therapy by integrating radiomic and deep-learning features from early-and-peak phases of DCE-MRI.BMC Cancer2025;25:1747 PMCID:PMC12604257

[34] Martin D,Tal D.A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proceedings of 8th IEEE International Conference on Computer Vision. ICCV 2001, Vancouver, Canada. Jul 07-14, 2001. IEEE; 2001. pp. 416-423.

[35] Doukas MC,Sharmanska V.Free-HeadGAN: neural talking head synthesis with explicit gaze control.IEEE Trans Pattern Anal Mach Intell2023;45:9743-56

[36] Arar NM,Thiran J.A regression-based user calibration framework for real-time gaze estimation.IEEE Trans Circuits Syst Video Technol2017;27:2623-38

[37] Bao Y,Wang H.Generalizing gaze estimation with rotation consistency. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleeans, USA. Jun 18-24, 2022. IEEE; 2022. pp. 4197-206.

[38] Zhang X,Beeler T,Tang S.ETH-XGaze: a large scale dataset for gaze estimation under extreme head pose and gaze variation. In Computer Vision - ECCV 2020. Cham: Springer International Publishing; 2020. pp. 365-81.

[39] Kim JH.Gaze in the dark: gaze estimation in a low-light environment with generative adversarial networks.Sensors2020;20:4935 PMCID:PMC7506593

[40] Ren Z,Hou G,Niu R.Appearance-based gaze estimation with feature fusion of multi-level information elements.J Comput Des Eng2023;10:1080-109

[41] Li Y,Chen J,Tan B.Appearance-based gaze estimation method using static transformer temporal differential network.Mathematics2023;11:686