AiCareNeonates: artificial intelligence powered adaptation of transfer learning models to classify neonate’s sleep-wake states for pediatricians in the loop^✩

Abstract

This study monitors neonatal sleep patterns using artificial intelligence through an innovative transfer learning approach integrating video and Electroencephalogram (EEG) data. Leveraging non-intrusive, camera-based technology, this method offers healthcare professionals an effective tool to evaluate and understand sleep quality in newborns. Such monitoring is essential for pediatricians, providing critical insights into the health and development of infants. Automated sleep/wake staging tools aid healthcare professionals in analyzing infant sleep patterns. Among these methods, camera-based approaches have gained prominence due to their non-intrusive and user-friendly characteristics, making them applicable for home monitoring. This study introduces a novel transfer learning technique for classifying neonatal sleep/wake stages. Our approach utilizes multiple color palettes, such as thermal, amber, grayscale, high contrast, hot metal, and red-blue, captured through a Fluke® (TiX 580) camera system. Continuous monitoring of neonatal sleep is essential for pediatricians to assess neonatal sleep quality comprehensively. Automated sleep/wake staging tools assist healthcare professionals in evaluating infant sleep patterns. Camera-based approaches have garnered significant attention among the various methods due to their non-intrusive and user-friendly nature, making them suitable for home use. In this paper, we propose a novel transfer learning approach for classifying neonatal sleep/wake staging using a combination of multiple color palettes, including thermal, amber, grayscale, high contrast, hot metal, and red-blue, recorded through a Fluke® (TiX 580) camera. The proposed method leverages the retraining of the last fully connected layer of well-established deep neural networks such as Visual Geometry Group 16 and 19, AlexNet, Inception-V3, ResNet-18, ResNet-50, and GoogLeNet to perform accurate sleep and wake stage classification. To enhance the precision of our approach, we integrate EEG data with video data obtained from neonatal subjects. The performance of transfer learning networks is validated using a leave-one-subject-out cross-validation strategy, ensuring robustness in classifying wake and sleep stages. In particular, the Inception-V3 model, when applied to the red-blue color palette video frames, demonstrates an impressive classification accuracy rate of 85.9%. Furthermore, we assessed the impact of including EEG data alongside video data, and even in this context, our approach maintains the same high accuracy of 85.9%. These findings underscore the robustness and effectiveness of our proposed method for neonatal sleep/wake staging classification, which is suitable for home-based monitoring and enhances its practicality and accessibility for pediatricians and caregivers.

Keywords

Deep convolutional neural network / Neonatal sleep classification / Transfer learning / Neonatal Red-Green-Blue (RGB) face database / Quadrupole exciton / Polariton

CRediT authorship contribution statement

Muhammad Awais: Validation, Software, Project administration, Methodology. Hemant Ghayvat: Writing – review & editing, Writing – original draft, Supervision, Conceptualization. Rebakah Geddam: Writing – review & editing, Writing – original draft, Validation, Resources, Data curation. Lewis Nkenyereye: Writing – review & editing, Writing – original draft. Kapal Dev: Writing – review & editing, Writing – original draft, Supervision.

Ethical and informed consent for data used

The project’s ethical approval was obtained from the Suryam Newborn Care and Children Hospital, Ahmedabad, Gujarat 38004, India, with the ethical reference number 13012024. During the data collection and trial, the ethics protocol was followed.

Declaration of competing interest

We hereby acknowledge that the disclosure of any conflicts of interest is undertaken with the intention of fostering transparency and does not inherently imply any unethical conduct or bias in the conduct of this study. Diligent efforts have been made to effectively address any potential conflicts of interest encountered during the study process, with the explicit aim of upholding the integrity and impartiality of our work. The findings and conclusions delineated in this paper are derived from a meticulous application of scientific methodologies and substantiated by compelling empirical evidence.

Acknowledgements

We acknowledge doctors, nurses, and caregivers at Suryam Newborn Care and Children Hospital for supporting the data collection protocol.

Data availability

The majority of data were synthetic, even though we secured ethical approval. The datasets employed, comprising both synthetically generated data for training and evaluating machine learning models, are openly available for use by other researchers.

References

[1] S.N. Graven, J.V. Browne, Sleep and brain development: the critical role of sleep in fetal and early neonatal brain development, Newborn Infants Nurs. Rev. 8 (4) (2008) 173-179.

[2] S.M. Ludington—Hoe, M.W. Johnson, K. Morgan, T. Lewis, J. Gutman, P.D. Wilson, M.S. Scher, Neurophysiologic assessment of neonatal sleep organization: preliminary results of a randomized, controlled trial of skin contact with preterm infants, Pediatrics 117 (5) (2006) e909-e923.

[3] M. Awais, X. Long, B. Yin, S.F. Abbasi, S. Akbarzadeh, C. Lu, X. Wang, L. Wang, J. Zhang, J. Dudink, et al., A hybrid dcnn—svm model for classifying neonatal sleep and wake states based on facial expressions in video, IEEE J. Biomed. Health Inform. 25 (5) (2021) 1441-1449.

[4] M. Grigg—Damberger, D. Gozal, C.L. Marcus, S.F. Quan, C.L. Rosen, R.D. Chervin, M. Wise, D.L. Picchietti, S.H. Sheldon, C. Iber, The visual scoring of sleep and arousal in infants and children, J. Clin. Sleep Med. 3 (02) (2007) 201-240.

[5] J. Werth, L. Atallah, P. Andriessen, X. Long, E. Zwartkruis—Pelgrim, R.M. Aarts, Unobtrusive sleep state measurements in preterm infants—a review, Sleep Med. Rev. 32 (2017) 109-122.

[6] J.E. Linde, J. Schulz, J.M. Perlman, K. Øymar, F. Francis, J. Eilevstjønn, H.L. Ersdal, Normal newborn heart rate in the first five minutes of life assessed by dry—electrode electrocardiography, Neonatology 110 (3) (2016) 231-237.

[7] W.K. Lee, H. Yoon, C. Han, K.M. Joo, K.S. Park, Physiological signal monitoring bed for infants based on load—cell sensors, Sensors 16 (3) (2016) 409.

[8] R. de Goederen, S. Pu, M.S. Viu, D. Doan, S. Overeem, W.A. Serdijn, K.F. Joosten, X. Long, J. Dudink, Radar—based sleep stage classification in children undergoing polysomnography: a pilot—study, Sleep Med. 82 (2021) 1-8.

[9] S. Montazeri Moghadam, P. Nevalainen, N.J. Stevenson, S. Vanhatalo, Sleep state trend (sst), a bedside measure of neonatal sleep state fluctuations based on single eeg channels, Clin. Neurophysiol. 143 (2022) 75-83.

[10] D. Holditch—Davis, M. Scher, T. Schwartz, D. Hudson—Barr, Sleeping and waking state development in preterm infants, Early Hum. Dev. 80 (1) (2004) 43-64.

[11] X. Long, R. Otte, E.v.d. Sanden, J. Werth, T. Tan, Video—based actigraphy for monitoring wake and sleep in healthy infants: a laboratory study, Sensors 19 (5) (2019) 1075.

[12] M.A. Khan, S.A. Iqbal, M.S. Khan, M.G. Hafez, Factor—bridging algorithm for the prediction of job satisfaction: developing country perspective, J. King Saud Univ, Comput. Inf. Sci. 35 (9) (2023) 101743.

[13] V. Bertelle, A. Sevestre, K. Laou—Hap, M. Nagahapitiye, J. Sizun, Sleep in the neonatal intensive care unit, J. Perinat. Neonatal Nurs. 21 (2) (2007) 140-148.

[14] M. Awais, X. Long, B. Yin, C. Chen, S. Akbarzadeh, S.F. Abbasi, M. Irfan, C. Lu, X. Wang, L. Wang, et al., Can pre—trained convolutional neural networks be directly used as a feature extractor for video—based neonatal sleep and wake classification?, BMC Res. Notes 13 (1) (2020) 1-6.

[15] G. Zamzmi, R. Kasturi, D. Goldgof, R. Zhi, T. Ashmeade, Y. Sun, A review of automated pain assessment in infants: features, classification tasks, and databases, IEEE Rev. Biomed. Eng. 11 (2017) 77-96.

[16] S. Adhikary, K. Jain, B. Saha, D. Chowdhury, Optimized eeg based mood detection with signal processing and deep neural networks for brain—computer interface, Biomed. Phys. Eng. Express 9 (3) (2023) 035014.

[17] B. Rim, N.—J. Sung, S. Min, M. Hong, Deep learning in physiological signal data: a survey, Sensors 20 (4) (2020) 969.

[18] M. Awais, C. Chen, X. Long, B. Yin, A. Nawaz, S.F. Abbasi, S. Akbarzadeh, L. Tao, C. Lu, L. Wang, et al., Novel framework: face feature selection algorithm for neonatal facial and related attributes recognition, IEEE Access 8 (2020) 59100-59113.

[19] A. Ansari, et al., Quiet sleep detection in preterm infants using deep convolutional neural networks, J. Neural Eng. 15 (2018) 066006.

[20] M. Awais, H. Ghayvat, W. Chen, Face and its features detection during nap, in: 2018 IEEE 23rd International Conference on Digital Signal Processing (DSP), IEEE, 2018, pp. 1-5.

[21] A. Sadeh, P. Lavie, A. Scher, E. Tirosh, R. Epstein, Actigraphic home—monitoring sleep—disturbed and control infants and young children: a new method for pediatric assessment of sleep—wake patterns, Pediatrics 87 (4) (1991) 494-499.

[22] S.F. Dodge, L.J. Karam, Quality robust mixtures of deep neural networks, IEEE Trans. Image Process. 27 (11) (2018) 5553-5562.

[23] A. Heinrich, X. Aubert, G. de Haan, Body movement analysis during sleep based on video motion estimation, in: 2013 IEEE 15th International Conference on e—Health Networking, Applications and Services (Healthcom 2013), IEEE, 2013, pp. 539-543.

[24] N. Koolen, L. Oberdorfer, Z. Rona, V. Giordano, T. Werther, K. Klebermass—Schrehof, N. Stevenson, S. Vanhatalo, Automated classification of neonatal sleep states using eeg, Clin. Neurophysiol. 128 (6) (2017) 1100-1108.

[25] K. Palmu, et al., Sleep—wake cycling in early preterm infants: comparison of polysomnographic recordings with a novel eeg—based index, Clin. Neurophysiol. 124 (9) (2013) 1807-1814.