Brightness Augmentation Implementation to Evaluate Perfor-mance Classification of Face Masked Base on CNN Model

Abstract

Deep learning methods with convolutional neural network (CNN) models have increasingly been applied to facial expression recognition. However, due to the recent pandemic, many individuals wear masks for work or health reasons, obstructing the complete visibility of their faces. This can impact social interactions, particularly in areas involving facial expression cues like the mouth. This study explores the application of CNNs in identifying facial expressions obscured by masks, focusing on the VGG16 and MobileNet architectures. Additionally, the research investigates the effects of data augmentation, including geometric and brightness augmentation, on the accuracy of facial expression classification. The findings indicate that the VGG16 architecture with cross-validation (VGG16-FLCV) outperforms MobileNet-FLCV in recognizing and classifying masked facial expressions. Data augmentation, particularly brightness augmentation, significantly enhances CNN model performance. For the VGG16-FLCV architecture, the brightness range (1.00, 1.25) yields the best accuracy, with a training accuracy of 81.73% and a validation accuracy of 70.71%. The most optimal brightness ranges for VGG16-FLCV are in the dark category (0.25, 0.50), (0.50, 0.75), and (0.75, 1.00), as well asthe bright category (1.00, 1.25). Meanwhile, MobileNet-FLCV with brightness ranges (0.25, 0.50), (0.50, 0.75), (0.75, 1.00), (1.00, 1.25), and (1.25, 1.50) can be used as alternative brightness ranges without significant accuracy degradation. These findings provide valuable insights for improving the accuracy of masked facial expression recognition by applying appropriate data augmentation techniques.

Keywords

Brightness Augmentation / CNN / Cross-validation / Masked Facial Expressions

References

[1] Agrawal A. & Mittal N. (2019). Using CNN for facial expression recognition: a study of the effects of kernel size and number of filters on accuracy. The Visual Computer, 2. https://doi.org/10.1007/s00371-019-01630-9

[2] Berrar D. (2018). Cross-validation. Encyclopedia of Bioinformatics and Computational Biology: ABC of Bioinformatics, 1-3( April), 542-545. https://doi.org/10.1016/B978-0-12-809633-8.20349-X

[3] Castellano G., De Carolis B. & Macchiarulo N. (2021). Automatic emotion recognition from facial expressions when wearing a mask. ACM International Conference Proceeding Series. https://doi.org/10.1145/3464385.3464730

[4] Cheng X., Lu J., Member S. & Yuan B. (n.d.). Face Segmentor-Enhanced Deep Feature Learning for Face Recognition. 1-14.

[5] Choe J. & Shim H. (n.d.). Attention-based Dropout Layer for Weakly Supervised Object Localization.2219-2228.

[6] Cotter S.F. (n.d.).MobiExpressNet: A Deep Learning Network for Face Expression Recognition on Smart Phones. 2020 IEEE International Conference on Consumer Electronics (ICCE), 1-4.

[7] Ding H., Zhou P. & Chellappa R. (2020). Occlusion-adaptive deep network for robust facial expression recognition. IJCB 2020 -IEEE/IAPR International Joint Conference on Biometrics. https://doi.org/10.1109/IJCB48548.2020.9304923

[8] Farkhod A. & Abdusalomov A.B. (2022). Development of Real-Time Landmark-Based Emotion Recognition CNN for Masked Faces.

[9] Genç Ç., Colley A., Löchtefeld M. & Häkkilä J. (2020). Face mask design to mitigate facial expression occlusion. Proceedings -International Symposium on Wearable Computers, ISWC, 40-44. https://doi.org/10.1145/3410531.3414303.

[10] Grundmann F., Epstude K. & Id S.S. (2021). Face masks reduce emotion-recognition accuracy and perceived closeness. 1-18. https://doi.org/10.1371/journal.pone.0249792.

[11] Kandel I., Castelli M., & Manzoni L. (2022). Brightness as an Augmentation Technique for Image Classification. Emerging Science Journal, 6(4), 881-892. https://doi.org/10.28991/ESJ-2022-06-04-015.

[12] Li J., Jin K., Zhou D., Kubota N., & Ju Z. (2020). Attention Mechanism-based CNN for Facial Expression. Neurocomputing. https://doi.org/10.1016/j.neucom.2020.06.014

[13] Li Z., Kamnitsas K., & Glocker B. (2021). Analyzing Overfitting Under Class Imbalance in Neural Networks for Image Segmentation. 40(3), 1065-1077.

[14] Merghani W., & Yap M. H. (n.d.). Adaptive Mask for Region-based Facial Micro-Expression Recognition. 2-7.

[15] Pei Z., Xu H., Zhang Y., Guo M., & Yang Y. (2019). Face Recognition via Deep Learning Using Data Augmentation Based on Orthogonal Experiments. 1-16. https://doi.org/10.3390/electronics8101088.

[16] Rice L., Wong E., & Kolter J. Z. (2020). Overfitting in adversarially robust deep learning. 37th International Conference on Machine Learning, ICML 2020, PartF16814, 8049-8074.

[17] Shinde P.P. & Shah S. (2018). A Review of Machine Learning and Deep Learning Applications. 2018 Fourth International Conference on Computing Communication Control and Automation (ICCUBEA). IEEE, 1-6. https://doi.org/10.1109/ICCUBEA.2018.8697857.

[18] Waheed A., Goyal M., Gupta D., Khanna A., Al-Turjman F. & Pinheiro P.R. (2020). CovidGAN: Data Augmentation Using Auxiliary Classifier GAN for Improved Covid-19 Detection. IEEE Access, 8, 91916-91923. https://doi.org/10.1109/ACCESS.2020.2994762.

[19] Yang B. (2021). Face Mask Aware Robust Facial Expression Recognition. 240-244.

[20] Yang B., Wu J., Ikeda K., Hattori G., Sugano M., Iwasawa Y., & Matsuo Y. (2022). Face-mask-aware Facial Expression Recognition based on Face Parsing and Vision Transformer. Pattern Recognition Letters, 164, 173-182. https://doi.org/10.1016/j.patrec.2022.11.004.

[21] Zhang H., Zhang L., & Jiang Y. ( n.d.).Overfitting and Underfitting Analysis for Deep Learning Based End-to-end Communication Systems. 2019 11th International Conference on Wireless Communications and Signal Processing (WCSP), 1-6.