Global-local combined features to detect pain intensity from facial expression images with attention mechanism

Abstract

The estimation of pain intensity is critical for medical diagnosis and treatment of patients. With the development of image monitoring technology and artificial intelligence, automatic pain assessment based on facial expression and behavioral analysis shows a potential value in clinical applications. This paper reports a framework of convolutional neural network with global and local attention mechanism (GLA-CNN) for the effective detection of pain intensity at four-level thresholds using facial expression images. GLA-CNN includes two modules, namely global attention network (GANet) and local attention network (LANet). LANet is responsible for extracting representative local patch features of faces, while GANet extracts whole facial features to compensate for the ignored correlative features between patches. In the end, the global correlational and local subtle features are fused for the final estimation of pain intensity. Experiments under the UNBC-McMaster Shoulder Pain database demonstrate that GLA-CNN outperforms other state-of-the-art methods. Additionally, a visualization analysis is conducted to present the feature map of GLA-CNN, intuitively showing that it can extract not only local pain features but also global correlative facial ones. Our study demonstrates that pain assessment based on facial expression is a non-invasive and feasible method, and can be employed as an auxiliary pain assessment tool in clinical practice.

Keywords

Attention / Convolutional neural network / Facial expression / Pain intensity

Author contributions

J. Wu and Y. Shi contributed equally to this study. J. Wu contributed to the conceptualization, methodology, software, writing-original draft of this study; Y. Shi contributed to the data analysis, writing-original draft; Sh. Yan performed the visualization, investigation; H.-M Yan contributed to supervision, writing-review.

Funding

This work was supported by the National Natural Science Foundation of China under Grant No. 62276051 and Natural Science Foundation of Sichuan Province under Grant No. 2023NSFSC0640, and Medical Industry Information Integration Collaborative Innovation Project of Yangtze Delta Region Institute under Grant No. U0723002.

Institutional review board statement

The study was approved by researchers at the McMaster University and University of Northern British Columbia. In addition, we also signed the requisite agreement, ensuring that all methods employed in this research were conducted in accordance with relevant guidelines and regulations.

Informed consent statement

In the manuscript, the facial images used in Fig. 1, Fig. 3, Fig. 6 are also part of the publicly available dataset in the UNBC-McMaster Shoulder Pain Expression Archive Database. When presenting these images, mosaic is partly overlapped on the faces to safeguard the privacy and anonymity of the individuals.

Data availability statement

The data used in the paper was sourced from a publicly available data that can be downloaded from the link http://www.pitt.edu/˜jeffcohn/PainArchive/.

Declaration of competing interest

The authors declare no conflicts of interest.

References

[1] M. Lynch, Pain as the fifth vital sign, J. Intraven. Nurs. 24 (2) (Mar. 2001) 85-94.

[2] D. Lopez-Martinez, K. Peng, S.C. Steele, A.J. Lee, D. Borsook, R. Picard, Multi-task multiple kernel machines for personalized pain recognition from functional near-infrared spectroscopy brain signals, in: Proc. of the 24th Intl. Conf. on Pattern Recognition (ICPR), (2018), pp. 2320-2325. Beijing, China.

[3] T.D. Wager, L.Y. Atlas, M.A. Lindquist, M. Roy, C.W. Woo, E. Kross, An fMRI-based neurologic signature of physical pain, New Engl. J. Med. 368 (15) (Apr. 2013) 1388-1397.

[4] S.M.G. Zwakhalen, K.A.J. Van Dongen, J.P.H. Hamers, H.H. Abu-Saad, Pain assessment in intellectually disabled people: non-verbal indicators, J. Adv. Nurs. 45 (3) (Feb. 2004) 236-245.

[5] M. Kunz, S. Scharmann, U. Hemmeter, K. Schepelmann, S. Lautenbacher, The facial expression of pain in patients with dementia, PAIN® 133 (1–3) (Dec. 2007) 221-228.

[6] F.X. Lesage, S. Berjot, F. Deschamps, Clinical stress assessment using a visual analogue scale, Occup. Med.-Oxford 62 (8) (Sept. 2012) 600-605.

[7] H. Liu, C. Zhang, Y.-J. Deng, T.-T. Liu, Z.-L. Zhang, Y.-F. Li, Orientation cues-aware facial relationship representation for head pose estimation via transformer, IEEE T. Image Process. 32 (Nov. 2023) 6289-6302.

[8] H. Liu, T.-T. Liu, Y. Chen, Z.-L. Zhang, Y.-F. Li, EHPE: Skeleton cues-based Gaussian coordinate encoding for efficient human pose estimation, IEEE T. Multimedia. (Aug. 2022), 1-12 https://doi.org/10.1109/TMM.2022.3197364.

[9] H. Liu, T.-T. Liu, Z.-L. Zhang, A.K. Sangaiah, B. Yang, Y.-F. Li, ARHPE: Asymmetric relation-aware representation learning for head pose estimation in industrial human-computer interaction, IEEE T. Ind. Inform. 18 (10) (Oct. 2022) 7107-7117.

[10] G. Bargshady, X.-J. Zhou, R.C. Deo, J. Soar, F. Whittaker, H. Wang, Ensemble neural network approach detecting pain intensity from facial expressions, Artif. Intell. Med. 109 (Sept. 2020) 1-12. 101954.

[11] N. Ahmed, T. Natarajan, K.R. Rao, Discrete cosine transform, IEEE T. Comput. C-23 (1) (Jan. 1974) 90-93.

[12] T. Ahonen, A. Hadid, M. Pietikäinen, Face recognition with local binary patterns, in: Proc. of the 8th European Conf. on Computer Vision, (2004), pp. 469-481. Prague, Czech Republic.

[13] N. Dalal, B. Triggs, Histograms of oriented gradients for human detection, in: Proc. of the IEEE Computer Society Conf, on Computer Vision and Pattern Recognition, San Diego, USA, (2005), pp. 886-893.

[14] Y. LeCun, Y. Bengio, G. Hinton, Deep learning, Nature 521 (7553) (May 2015) 436-444.

[15] A. Kamilaris, F.X. Prenafeta-Boldú, Deep learning in agriculture: a survey, Comput. Electron. Agr. 147 (Apr. 2018) 70-90.

[16] X.-W. Xin, X.-Y. Lin, S.-F. Yang, X. Zheng, Pain intensity estimation based on a spatial transformation and attention CNN, PLoS One 15 (8) (Aug. 2020) 1-15. e0232412.

[17] A. Graves, Long short-term memory, A. Graves (Ed.), Supervised Sequence Labelling with Recurrent Neural Networks, Springer, Berlin, Heidelberg, Germany, (2012), pp. 37-45.

[18] Z. Hammal, J.F. Cohn, Automatic detection of pain intensity, in: Proc. of the 14th ACM Intl. Conf. on Multimodal Interaction, (2012), pp. 47-52. Santa Monica, USA.

[19] J. Zhou, X.-P. Hong, F. Su, G.-Y. Zhao, Recurrent convolutional neural network regression for continuous pain intensity estimation in video, in: Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition Workshops, Las Vegas, USA, (2016), pp. 1535-1543.

[20] E. Friesen, P. Ekman, Facial action coding system: A technique for the measurement of facial movement, Palo Alto 3 (2) (1978) 5.

[21] K.M. Prkachin, P.E. Solomon, The structure, reliability and validity of pain expression: evidence from patients with shoulder pain, Pain 139 (2) (Oct. 2008) 267-274.

[22] P. Lucey, J.F. Cohn, K.M. Prkachin, P.E. Solomon, I. Matthews, Painful data: the UNBC-McMaster shoulder pain expression archive database, in: Proc. of the IEEE Intl, Conf. on Automatic Face & Gesture Recognition (FG), Santa Barbara, USA, (2011), pp. 57-64.

[23] R.-J. Yang, X.-P. Hong, J.-Y. Peng, X.-Y. Feng, G.-Y. Zhao, Incorporating high-level and low-level cues for pain intensity estimation, in: Proc. of the 24th Intl. Conf. on Pattern Recognition (ICPR), (2018), pp. 3495-3500. Beijing, China.

[24] A.B. Ashraf, S. Lucey, J.F. Cohn, et al., The painful face: pain expression recognition using active appearance models, in: Proc. of the 9th Intl. Conf. on Multimodal Interfaces, Nagoya, Japan, (2007), pp. 9-14.

[25] S. Kaltwang, O. Rudovic, M. Pantic, Continuous pain intensity estimation from facial expressions, in: Proc. of the 8th Intl. Symposium on Visual Computing, Rethymnon, Crete, Greece, (2012), pp. 368-377.

[26] P. Rodriguez, G. Cucurull, J. Gonzàlez, et al., Deep pain: exploiting long short-term memory networks for facial expression classification, IEEE T. Cybernetics 52 (5) (May 2022) 3314-3324.

[27] N. Zakaria, Y.M.M. Hassim, A review study of the visual geometry group approaches for image classification, J. of Appl. Sci., Tech. and Comput. 1 (1) (Feb. 2024) 14-28.

[28] M. Tavakolian, A. Hadid, A spatiotemporal convolutional neural network for automatic pain intensity estimation from facial dynamics, Int. J. Comput. Vis. 127 (10) (Oct. 2019) 1413-1425.

[29] L. Itti, C. Koch, Computational modelling of visual attention, Nat. Rev. Neurosci. 2 (3) (Mar. 2001) 194-203.

[30] H. Liu, C. Zheng, T.-C. Li, et al., EDMF: efficient deep matrix factorization with review feature learning for industrial recommender system, IEEE T. Ind. Inform. 18 (7) (Jul. 2022) 4361-4371.

[31] H. Liu, C. Zheng, T.-C. Li, et al., Multi-perspective social recommendation method with graph representation learning, Neurocomputing 468 (Jan. 2022) 469-481.

[32] H. Liu, C. Zhang, Y.-J. Deng, B.-C. Xie, T.-T. Liu, Y.-F. Li, TransIFC: invariant cues-aware feature concentration learning for efficient fine-grained bird image classification, IEEE T. Multimedia, (2023),10.1109/TMM.2023.3238548.

[33] H. Liu, S. Fang, Z.-L. Zhang, C.H. Li, K. Lin, J.-Z.H. Wang, MFDNet: Collaborative poses perception and matrixfisher distribution for head pose estimation, IEEE T. on Multimedia 24 (May 2021) 2449-2460.

[34] Y. Li, J.-B. Zeng, S.-G. Shan, X.-L. Chen, Occlusion aware facial expression recognition using CNN with attentionmechanism, IEEE T. Image Process 28 (5) (May 2019) 2439-2450.

[35] H. Zhang, W. Su, Z. Wang, Weakly supervised local-global attention network for facial expression recognition, 8, IEEE Access, (2020), pp. 37976-37987.

[36] C. Liu, K. Hirota, J. Ma, et al., Facial expression recognition using hybrid features of pixel and geometry, IEEE Access 9 (2021) 18876-18889.

[37] X. Liu, J. Li, T.-T. Duan, J.-T. Li, Y. Wang, DSMA: reference-based image super-resolution method based on dual-view supervised learning and multi-attention mechanism, IEEE Access 10 (May 2022) 54649-54659.

[38] M. Jaderberg, K. Simonyan, A. Zisserman, K. Kavukcuoglu, Spatial transformer networks, in: Proc. of the 28th Intl. Conf. on Neural Information Processing Systems, (2015), pp. 2017-2025. Montreal, Canada.

[39] K.-P. Zhang, Z.-P. Zhang, Z.-F. Li, Y. Qian, Joint face detection and alignment using multitask cascaded convolutional networks, IEEE Signal Proc. Let., 23 (10) (Oct. 2016) 1499-1503.

[40] E.E. Benarroch, Pain-autonomic interactions: a selective review, Clin. Auton. Res. 11 (6) (Dec. 2001) 343-349.

[41] G. Bargshady, X.-J. Zhou, R.C. Deo, J. Soar, F. Whittaker, H. Wang, Enhanced deep learning algorithm development to detect pain intensity from facial expression images, Expert Syst. Appl. 149 (Jul. 2020) 113305. 1–10.

[42] S. Walter, S. Gruss, H. Ehleiter, et al., The biovid heat pain database data for the advancement and systematic validation of an automated pain recognition system, in: Proc. of the IEEE Intl. Conf. on Cybernetics (CYBCO), (2013), pp. 128-131. Lausanne, Switzerland.

[43] R.-J. Yang, S.-J. Tong, M. Bordallo, et al., On pain assessment from facial videos using spatio-temporal local descriptors, in: Proc. of the Sixth Intl. Conf. on Image Processing Theory, Tools and Applications (IPTA), (2016), pp. 1-6. Oulu Finland.

[44] D. Huang, Z.-Q. Xia, L. Li, K.-W. Wang, X.-Y. Feng, Pain-awareness multistream convolutional neural network for pain estimation, J. Electron. Imag. 28 (4) (Jul. 2019) 1-10. 043008.

[45] L. Pham, T.H. Vu, T.A. Tran, Facial expression recognition using residual masking network, in: Proc. of the 25th Intl. Conf. on Pattern Recognition (ICPR), (2021), pp. 4513-4519. Milan, Italy.

[46] Z.-Q. Zhao, Q.-S. Liu, S.-M. Wang, Learning deep global multi-scale and local attention features for facial expressionrecognition in the wild, IEEE T. Image Process, 30 (Jul. 2021) 6544-6556.

[47] Z. Liu, J. Ning, Y. Cao, et al., Video swin transformer, in: Proc. of the IEEE/CVF Conf, on Computer Vision and Pattern Recognition, New Orleans, USA, (2022), pp. 3192-3201.

[48] J.-W. Mao, R. Xu, X.-S. Yin, Y.-Q. Chang, B.-L. Nie, A.-B. Huang, POSTER++: A simpler and stronger facial expression recognition network [Online]. https://arxiv.org/abs/2301.12149, February 2023

[49] R.R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, D. Batra, Grad-CAM: Visual explanations from deep networks via gradient-based localization, in: Proc. of the IEEE Intl. Conf. on Computer Vision, (2017), pp. 618-626. Venice, Italy.