Faster R-CNN-MobileNetV3 Based Micro Expression Detection for Autism Spectrum Disorder

Abstract

Autism spectrum disorder (ASD) is a neuropathic disease which is characterized by deficits in social interaction and communication. Therefore, the ASD patients have weak ability to express themselves or let others know about their thoughts. As society pays more attention to ASD patients, early intervention programs, behavioral therapy and technological assistance have emerged to help ASD patients improve their quality of lives. This paper aims to propose an improved object detection algorithm based on Faster R-CNN-MobileNetV3 to analyze the micro expressions of ASD patients. The data set includes 1358 face images of ASD patients built from 12 ASD movies with the method of Cinemetrics. Through the training and testing of the ASD data set with the improved model, the overall precision rate has reached 0.9 and mean Average Precision also has significant improvement. As a result, the improved Faster R-CNN-MobileNetV3 model achieves a good performance to recognize micro expressions and emotions of ASD patients.

Keywords

autism spectrum disorder / micro expressions / Cinemetrics / object detection / Faster R-CNN / MobileNetV3

References

[1] Hirota, T.; King, B.H. Autism Spectrum Disorder: A Review. J. Am. Med. Assoc. 2023, 329, 157-168. https://doi.org/10.1001/jama.2022.23661.

[2] National Institute of Mental Health. Autism Spectrum Disorder—National Institute of Mental Health (NIMH). Available online: https://www.nih.gov/ (accessed on 12 February 2024).

[3] Mayo, Clinic. Autism Spectrum Disorder—Symptoms and Causes—Mayo Clinic. Available online: https://www.mayoclinic.org/ (accessed on 6 January 2018).

[4] Wood, J.J.; Kendall, P.C.; Wood, K.S.; et al. Cognitive Behavioral Treatments for Anxiety in Children with Autism Spectrum Disorder: A Randomized Clinical Trial. JAMA Psychiatry 2019, 77, 474-483. https://doi.org/10.1001/jamapsychiatry.

[5] Sharma, S.R.; Gonda, X.; Tarazi, F.I. Autism Spectrum Disorder: Classification, diagnosis and therapy. Pharmacol. Ther. 2018, 190, 91-104. https://doi.org/10.1016/j.pharmthera.2018.05.007.

[6] Porter, S.; Ten Brinke, L. Reading between the lies: Identifying concealed and falsified emotions in universal facial expressions. Psychol. Sci. 2008, 19, 508-514.

[7] Ekman, P.; Martin, C.W. Lie catching and microexpressions. In The Philosophy of Deception; Oxford University Press: Oxford, UK, 2009; pp. 118-121.

[8] Wang, C.; Peng, M.; Bi, T.; et al. Micro-attention for micro-expression recognition. Neurocomputing 2020, 410, 354-362.

[9] Li, J.; Yap, M.H.; Cheng, W.H.; et al. FME’22: 2nd Workshop on Facial Micro-Expression: Advanced Techniques for Multi-Modal Facial Expression Analysis. In Proceedings of the 30th ACM International Conference on Multimedia, New York, NY, USA, 10-14 October 2022; pp. 7397-7399.

[10] Guerdelli, H.; Ferrari, C.; Barhoumi, W.; et al. Macro-and micro-expressions facial datasets: A survey. Sensors 2022, 22, 1524.

[11] Li, Y.; Wei, J.; Liu, Y.; et al. Deep learning for micro-expression recognition: A survey. IEEE Trans. Affect. Comput. 2022, 13, 2028-2046.

[12] Sayette, M.A.; Cohn, J.F.; Wertz, J.M.; et al. A psychometric evaluation of the facial action coding system for assessing spontaneous expression. J. Nonverbal Behav. 2001, 25, 167-185.

[13] Jeffrey, F. Cohn, Takeo Kanade. Cohn-Kanade Dataset (CK), Available online: https://www.kaggle.com/datasets/davilsena/ckdataset (accessed on 20 September 2024).

[14] Zarins, Uldis. Anatomy of Facial Expression. Anatomy Next, Inc.: Beacon, NY 12508, USA, 2019.

[15] Fan, B.; Guo, L. Stylistic Transfer between Intergenerational Directors:A Metrological Study of the Fifth and Sixth- Generation Chinese Directors. J. Guizhou Univ. Art Ed. 2021, 2021, 72-85. (In Chinese)

[16] Li, W.; Wang, Z. Visualization Analysis on Intellectual Structures and Research Fronts of “Cinemetrics”: Quantitative Film Studies. In Proceedings of the 2023 8th International Conference on Information and Education Innovations, Manchester, UK, 13-15 April 2023; pp. 206-210.

[17] Qiao, J.Q., A glimpse into the style of early Chinese cinema (1922-1937): A quantitative film studies perspective. Movie Review 2022, 2022. https://doi.org/10.16583/j.cnki.52-1014/j.2021.22.025. (In Chinese)

[18] Tolba, A.S.; El-Baz, A.H.; El-Harby, A.A. Face recognition: A literature review. Int. J. Signal Process 2006, 2, 88-103.

[19] Hassaballah, M.; Aly, S. Face recognition: Challenges, achievements and future directions. IET Comput. Vis. 2015, 9, 614-626.

[20] Zhu, Z.; Liu, L.; Free, R.C.; et al. OPT-CO: Optimizing pre-trained transformer models for efficient COVID-19 classification with stochastic configuration networks. Inf. Sci. 2024, 680, 121141. https://doi.org/10.1016/j.ins.2024.121141.

[21] Ren, S.; He, K.; Girshick, R.; et al. Faster R-CNN: Towards real-time object detection with region proposal networks. IEEE Trans. Pattern Anal. Mach. Intell. 2016, 39, 1137-1149.

[22] Li, W. Analysis of object detection performance based on Faster R-CNN. J. Phys. Conf. Ser. 2021, 1827, 012085. https://doi.org/10.1088/1742-6596/1827/1/012085.

[23] Shuai, Q.; Wu, X. Object detection system based on SSD algorithm. In Proceedings of the 2020 International Conference on Culture-oriented Science & Technology (ICCST), Beijing, China, 28-31 October 2020; pp. 141-144. https://doi.org/10.1109/ICCST50977.2020.00033.

[24] Jeong, J.; Park, H.; Kwak, N. Enhancement of SSD by concatenating feature maps for object detection. arXiv 2017, arXiv:1705.09587.

[25] Zhu, Z.; Wang, S.; Zhang, Y. A Survey of Convolutional Neural Network in Breast Cancer. Comput. Model. Eng. Sci. 2023, 136, 2127-2172. https://doi.org/10.32604/cmes.2023.025484.

[26] Ahmad, T.; Ma, Y.; Yahya, M.; et al. Object detection through modified YOLO neural network. Sci. Program. 2020, 2020, 8403262. https://doi.org/10.1155/2020/8403262.

[27] Zhang, C.; Xu, X.; Tu, D. Face detection using improved faster rcnn. arXiv 2018, arXiv:1802.02142.

[28] Cheng, B.; Wei, Y.; Shi, H.; et al. Revisiting rcnn: On awakening the classification power of faster rcnn. In Proceedings of the 2018 European Conference on Computer Vision (ECCV), Munich, Germany, 8-14 September 2018; pp. 453-468.

[29] Zhu, Z.; Wang, S.; Zhang, Y. ROENet: A ResNet-Based Output Ensemble for Malaria Parasite Classification. Electronics 2022, 11, 2040. https://doi.org/10.3390/electronics11132040.

[30] Qian, S.; Ning, C.; Hu, Y. MobileNetV3 for Image Classification. In Proceedings of the 2021 IEEE 2nd International Conference on Big Data, Artificial Intelligence and Internet of Things Engineering (ICBAIE), Nanchang, China, 26-28 March 2021; pp. 490-497. https://doi.org/10.1109/ICBAIE52039.2021.9389905.

[31] Koonce, B.; Koonce, B. MobileNetV3. In Convolutional Neural Networks with Swift for Tensorflow: Image Recognition and Dataset Categorization; Apress: Berkeley, CA, USA, 2021; pp. 125-144.

[32] Zhao, L.; Wang, L. A new lightweight network based on MobileNetV3. KSII Trans. Internet Inf. Syst. Korean Soc. Internet Inf. (KSII) 2022, 16, 1-15.

[33] Ren; Yun; Zhu, C.; Xiao, S. Object detection based on fast/faster RCNN employing fully convolutional architectures. Math. Probl. Eng. 2018, 2018, 3598316. https://doi.org/10.1155/2018/3598316.

[34] Liu; Bin; Zhao, W.; Sun, Q. Study of object detection based on Faster R-CNN. In Proceedings of the 2017 Chinese Automation Congress (CAC), Jinan, China, 20-22 October 2017; pp. 6233-6236.

[35] Jiang, H.; Learned-Miller, E. Face Detection with the Faster R-CNN. In Proceedings of the 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017), Washington, DC, USA, 30 May-3 June 2017; pp. 650-657. https://doi.org/10.1109/FG.2017.82.

[36] Kong; Xiaohong; Li, X.; Zhu, X.; et al. Detection model based on improved faster-RCNN in apple orchard environment. Intell. Syst. Appl. 2024, 21, 200325.

[37] Cao, C.; Wang, B.; Zhang, W.; et al. An Improved Faster R-CNN for Small Object Detection. IEEE Access 2019, 7, 106838-106846. https://doi.org/10.1109/ACCESS.2019.2932731.

[38] Rehman, M.U.; Shafique, A.; Ghadi, Y.Y.; et al. A Novel Chaos-Based Privacy-Preserving Deep Learning Model for Cancer Diagnosis. IEEE Trans. Netw. Sci. Eng. 2022, 9, 4322-4337.

[39] Rehman, M.U.; Shafique, A.; Khan, I.U.; et al. An efficient deep learning model for brain tumour detection with privacy preservation. CAAI Trans. Intell. Technol. 2023, 1-16. https://doi.org/10.1049/cit2.12254.

[40] Rehman, M.U.; Shafique, A.; Khan, M.S.; et al. A novel medical image data protection scheme for smart healthcare system. CAAI Trans. Intell. Technol. 2023, 9, 821-836.