Analysing learners' facial expressions during learning and exploring their learning processes and emotional changes are of great significance for assisting teachers' teaching and promoting smart education. In complex learning environments, static facial expression recognition fails to capture the dynamic changes of learners' expressions losing the continuous features in the learning process, and its recognition effect is easily interfered with by factors such as occlusion and lighting variations during learning. To address the above issues, a network model based on adaptive global attention and temporal difference is proposed to recognise learners' dynamic expression sequences. Firstly, we have designed an Adaptive Global Attention (AGA) block, which adaptively models inter-channel relationships to dynamically enhance key channels that are highly correlated with learners' states while suppressing redundant information, thereby improving the model's feature representation capability under noisy environments. Secondly, we have designed a Differential Temporal Transformer (DTFormer) to extract differential information between consecutive frames, increasing the model's sensitivity to learners' facial expression dynamics and improving recognition performance. The two components complement each other in terms of spatial feature enhancement and temporal dynamic modelling effectively improving the model's overall capability for representing learners' dynamic facial expressions. Experiments were conducted on public datasets DFEW, FERV39k and the learner E-learning emotional state data set DAiSEE, and comparisons were made with classical methods using objective indicators. The results demonstrate that the proposed method outperforms the comparison methods in multiple performance indicators, thereby verifying its effectiveness.
Acknowledgements
Authors are funded by UKRI (Grant EP/W020408/1) and Grant RS718 through Swansea University. This work was supported by the Humanities and Social Science Fund of Ministry of Education of China (23YJAZH084).
Funding
The study was supported by UKRI (Grant EP/W020408/1); the Humanities and Social Science Fund of Ministry of Education of China (23YJAZH084).
Conflicts of Interest
The authors declare no conflicts of interest.
Data Availability Statement
Data available on request from the authors.
| [1] |
J. Li, E. Xue, C. Li, and Y. He, “Investigating Latent Interactions Between Students’ Affective Cognition and Learning Performance: Meta-Analysis of Affective and Cognitive Factors,” Behavioral Sciences 13, no. 7 (2023): 555, https://doi.org/10.3390/bs13070555.
|
| [2] |
Q. Yuan, “Research on Classroom Emotion Recognition Algorithm Based on Visual Emotion Classification,” Computational Intelligence and Neuroscience 2022, no. 1 (2022): 6453499, https://doi.org/10.1155/2022/6453499.
|
| [3] |
L. Peng, S. Baoye, and X. Lin, “Bearing Fault Diagnosis in Multiple Working Conditions Based on One Dimensional Convolutional Neural Network,” Journal of Shandong University of Science and Technology 42, no. 5 (2023): 88-96, https://doi.org/10.16452/j.cnki.sdkjzk.2023.05.010.
|
| [4] |
Q.Xiaohe, L.Zhiwei, L.Da, D.Xiao, H.Kaixun, and Z.Kai , “ Fault Diagnosis Method of Tansfer Learning Based Multi-Scale Graph Convolutional Networks,” Journal of Shandong University of Science and Technology 44, no. 5 (2025): 119-129, https://doi.org/10.16452/j.cnki.sdkjzk.2025.05.012.
|
| [5] |
J. Wang, “Research on Classroom Emotion Recognition Algorithm Based on Visual Emotion Classification,” (In Chinese) Educational Theory and Research 2, no. 4 (2024): 35-36, https://doi.org/10.61369/etr.6415.
|
| [6] |
B. Fang, X. Li, G. Han, and J. He, “Facial Expression Recognition in Educational Research From the Perspective of Machine Learning: A Systematic Review,” IEEE Access 11 (2023): 112060-112074, https://doi.org/10.1109/ACCESS.2023.3322454.
|
| [7] |
A. V. Savchenko, L. V. Savchenko, and I. Makarov, “Classifying Emotions and Engagement in Online Learning Based on a Single Facial Expression Recognition Neural Network,” IEEE Transactions on Affective Computing 13, no. 4 (2022): 2132-2143, https://doi.org/10.1109/TAFFC.2022.3188390.
|
| [8] |
A. Gupta, A. D’Cunha, K. Awasthi, and V. Balasubramanian, “Daisee: Towards User Engagement Recognition in the Wild,” preprint arXiv:1609.01885 (2016), https://doi.org/10.48550/arXiv.1609.01885.
|
| [9] |
J. Tang, X. Zhou, and J. Zheng, “Design of Intelligent Classroom Facial Recognition Based on Deep Learning,” Journal of Physics: Conference Series 1168, no. 2 (2019): 022043, https://doi.org/10.1088/1742-6596/1168/2/022043.
|
| [10] |
W. Yu, M. Liang, and X. Wang, “Learner Expression Recognition and Intelligent Teaching Evaluation Algorithm Based on Deep Attention Network,” (In Chinese) Journal of Computer Applications 42, no. 3 (2022): 743-749, https://doi.org/10.11772/j.issn.1001-9081.2021040846.
|
| [11] |
H. Li, L. Yang, Z. Weijia, and S. Peixuan, “Analysis of Teaching Effect Based on Facial Expression in Classroom Environment,” Modern Distance Education Research 30, no. 4 (2017), https://doi.org/10.1145/3383923.3383949.
|
| [12] |
N. Krishnnan, S. Ahmed, T. Ganta, and G. Jeyakumar, “A Video Analytics Based Solution for Detecting the Attention Level of the Students in Class Rooms,” in 2020 10th International Conference on Cloud Computing, Data Science & Engineering (Confluence), (2020), 498-501.
|
| [13] |
Z. Shou, Y. Huang, D. Li, et al., A Student Facial Expression Recognition Model Based on Multi-Scale and Deep Fine-Grained Feature Attention Enhancement,” Sensors 24, no. 20 (2024): 6748, https://doi.org/10.3390/s24206748.
|
| [14] |
Y. Chen, K. Li, F. Tian, G. Wei, and M. Seberi, “Lightweight Expression Recognition Combined Attention Fusion Network With Hybrid Knowledge Distillation for Occluded e-Learner Facial Images,” Neurocomputing 628 (2025): 129656, https://doi.org/10.1016/j.neucom.2025.129656.
|
| [15] |
Z. Chen, M. Liang, Z. Xue, and W. Yu, “STRAN: Student Expression Recognition Based on Spatio-Temporal Residual Attention Network in Classroom Teaching Videos,” Applied Intelligence 53, no. 21 (2023): 25310-25329, https://doi.org/10.1007/s10489-023-04858-0.
|
| [16] |
D. Wang, C. Yang, and G. Chen, “Using Vision Language Models to Detect Students’ Academic Emotion Through Facial Expressions,” preprint arXiv:2506.10334 (2025), https://doi.org/10.48550/arXiv.2506.10334.
|
| [17] |
D. H. Lee and J. H. Yoo, “CNN Learning Strategy for Recognizing Facial Expressions,” IEEE Access 11 (2023): 70865-70872, https://doi.org/10.1109/ACCESS.2023.3294099.
|
| [18] |
S. Luo, M. Li, and M. Chen, “Multi-Scale Integrated Attention Mechanism for Facial Expression Recognition Network,” Computer Engineering and Applications 59, no. 1 (2023): 199-206, https://doi.org/10.3778/j.issn.1002-8331.2203-0170.
|
| [19] |
D. Song and C. Liu, “A Facial Expression Recognition Network Using Hybrid Feature Extraction,” PLoS One 20, no. 1 (2025): e0312359, https://doi.org/10.1371/journal.pone.0312359.
|
| [20] |
J. Wei, G. Hu, X. Yang, A. T. Luu, and Y. Dong, “Learning Facial Expression and Body Gesture Visual Information for Video Emotion Recognition,” Expert Systems with Applications 237 (2024): 121419, https://doi.org/10.1016/j.eswa.2023.121419.
|
| [21] |
L. Wang, “Research on Facial Expression Recognition Based on Deep Learning,” in 2019 3rd International Conference on Electronic Information Technology and Computer Engineering (EITCE) (IEEE, 2019), 1144-1147, https://doi.org/10.1109/EITCE47263.2019.9095140.
|
| [22] |
X. Liu, X. Gong, and H. Zhao, “Dynamic Face Expression Recognition Based on Mixed Attention Mechanism (In Chinese),” supplement, Journal of Computer Applications 43, no. S1 (2023): 1-7, https://doi.org/10.11772/j.issn.1001-9081.2022101472.
|
| [23] |
F. An and Z. Liu, “Facial Expression Recognition Algorithm Based on Parameter Adaptive Initialization of CNN and LSTM,” Visual Computer 36, no. 3 (2020): 483-498, https://doi.org/10.1007/s00371-019-01635-4.
|
| [24] |
X. Yan, L. Xianglan, P. Xuguang, L. Fang, and Z. Haiyan, “Research on Person Re-Identification Method Based on CNN-Transformer and Attention Pyramid,” Journal of Shandong University of Science and Technology 44, no. 1 (2025): 110-118, https://doi.org/10.16452/j.cnki.sdkjzk.2025.01.010.
|
| [25] |
S. Yan, Y. Wang, X. Mai, et al., Empower Smart Cities With Sampling-Wise Dynamic Facial Expression Recognition via Frame-Sequence Contrastive Learning,” Computer Communications 216 (2024): 130-139, https://doi.org/10.1016/j.comcom.2023.12.032.
|
| [26] |
Z. Zhao and Q. Liu, “Former-DFER: Dynamic Facial Expression Recognition Transformer,” in Proceedings of the 29th ACM International Conference on Multimedia (Association for Computing Machinery, 2021), 1553-1561.
|
| [27] |
F. Ma, B. Sun, and S. Li, “Spatio-Temporal Transformer for Dynamic Facial Expression Recognition in the Wild,” preprint arXiv:2205.04749 (2022), https://doi.org/10.48550/arXiv.2205.04749.
|
| [28] |
L. Wang, X. Kang, F. Ding, S. Nakagawa, and F. Ren, “A Joint Local Spatial and Global Temporal CNN-Transformer for Dynamic Facial Expression Recognition,” Applied Soft Computing 161 (2024): 111680, https://doi.org/10.1016/j.asoc.2024.111680.
|
| [29] |
H. Li, M. Sui, and Z. Zhu, “Nr-dfernet: Noise-Robust Network for Dynamic Facial Expression Recognition,” preprint arXiv:2206.04975 (2022), https://doi.org/10.48550/arXiv.2206.04975.
|
| [30] |
T. Han, S. Dou, W. Zhang, and R. Liu, “Enhanced Dynamic Temporal Feature Extraction With Static Expression Insights for Dynamic Facial Expression Recognition,” Digital Signal Processing 168 (2025): 105470, https://doi.org/10.1016/j.dsp.2025.105470.
|
| [31] |
Y. Lü, F. Zhang, Z. Ma, B. Zheng, and Z. Nan, “Dynamic Facial Expression Recognition in the Wild via Multi-Snippet Spatiotemporal Learning,” Neurocomputing 636 (2025): 130020, https://doi.org/10.1016/j.neucom.2025.130020.
|
| [32] |
M. Stettler, A. Lappe, and M. A. Giese, “Facial Expression Recognition Based on Multi-Domain Norm-Referenced Encoding,” Neural Networks 197 (2025): 108384, https://doi.org/10.1016/j.neunet.2025.108384.
|
| [33] |
K. Jiang, Z. Wang, P. Yi, T. Lu, J. Jiang, and Z. Xiong, “Dual-Path Deep Fusion Network for Face Image Hallucination,” IEEE Transactions on Neural Networks and Learning Systems 33, no. 1 (2020): 378-391, https://doi.org/10.1109/TNNLS.2020.3027849.
|
| [34] |
X. Shi, Z. Hao, and Z. Yu, “Spikingresformer: Bridging Resnet and Vision Transformer in Spiking Neural Networks,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, (2024), 5610-5619.
|
| [35] |
X. Liu, H. Xu, and M. Wang, “Sparse Spatial-Temporal Emotion Graph Convolutional Network for Video Emotion Recognition,” Computational Intelligence and Neuroscience 2022, no. 1 (2022): 3518879, https://doi.org/10.1155/2022/3518879.
|
| [36] |
R. Singh, S. Saurav, T. Kumar, R. Saini, A. Vohra, and S. Singh, “Facial Expression Recognition in Videos Using Hybrid CNN & ConvLSTM,” International Journal of Information Technology 15, no. 4 (2023): 1819-1830, https://doi.org/10.1007/s41870-023-01183-0.
|
| [37] |
L. Wang, X. Kang, F. Ding, S. Nakagawa, and F. Ren, “MSSTNet: A Multi-Scale Spatio-Temporal CNN-Transformer Network for Dynamic Facial Expression Recognition,” in ICASSP 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (IEEE, 2024), 3015-3019.
|
| [38] |
F. Ma, B. Sun, and S. Li, “Logo-Former: Local-Global Spatio-Temporal Transformer for Dynamic Facial Expression Recognition,” in ICASSP 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (IEEE, 2023), 1-5.
|
| [39] |
M. Li, X. Zhang, C. Fan, T. Liao, and G. Xiao , “Dual-STI: Dual-Path Spatial-Temporal Interaction Learning for Dynamic Facial Expression Recognition,” Information Sciences 678 (2024): 120953, https://doi.org/10.1016/j.ins.2024.120953.
|
| [40] |
S. Woo, J. Park, J. Y. Lee, and I. S. Kweon, “CBAM: Convolutional Block Attention Module,” in Proceedings of the European Conference on Computer Vision (ECCV) (Springer International Publishing, 2018), 3-19.
|
| [41] |
Y. Xiao, Q. Yuan, K. Jiang, et al., Local-Global Temporal Difference Learning for Satellite Video Super-Resolution,” IEEE Transactions on Circuits and Systems for Video Technology 34, no. 4 (2023): 2789-2802, https://doi.org/10.1109/tcsvt.2023.3312321.
|
| [42] |
C. Huang, Q. Zeng, F. Xiong, and J. Xu, “Space Dynamic Target Tracking Method Based on Five-Frame Difference and Deepsort,” Scientific Reports 14, no. 1 (2024): 6020, https://doi.org/10.1038/s41598-024-56623-z.
|
| [43] |
X. Jiang, Y. Zong, W. Zheng, et al., Dfew: A Large-Scale Database for Recognizing Dynamic Facial Expressions in the Wild,” in Proceedings of the 28th ACM International Conference on Multimedia, (2020), 2881-2889.
|
| [44] |
Y. Wang, Y. Sun, Y. Huang, et al., Ferv39k: A Large-Scale Multi-Scene Dataset for Facial Expression Recognition in Videos,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, (2022), 20922-20931.
|
| [45] |
Y. Ma, Y. Wei, Y. Shi, X. Li, Y. Tian, and Z. Zhao, “Online Learning Engagement Recognition Using Bidirectional Long-Term Recurrent Convolutional Networks,” Sustainability 15, no. 1 (2023): 198, https://doi.org/10.3390/su15010198.
|
| [46] |
M. M. Santoni, T. Basaruddin, and K. Junus, “Convolutional Neural Network Model Based Students’ Engagement Detection in Imbalanced DAiSEE Dataset,” International Journal of Advanced Computer Science and Applications 14, no. 3 (2023), https://doi.org/10.14569/IJACSA.2023.0140371.
|
| [47] |
D. Zhu , Student Online Learning Dynamic Facial Expression Recognition Based on an Improved 3D Residual Neural Network(In Chinese). Master’s thesis (Jiangxi University of Finance and Economics, 2025).
|
| [48] |
J. Carreira and A. Zisserman, “Quo Vadis, Action Recognition? a New Model and the Kinetics Dataset,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (IEEE, 2017), 6299-6308.
|
| [49] |
K. Simonyan and A. Zisserman, “Very Deep Convolutional Networks for Large-Scale Image Recognition,” preprint arXiv:1409.1556 (2014).
|
| [50] |
K. He, X. Zhang, S. Ren, and J. Sun, “Deep Residual Learning for Image Recognition,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (IEEE, 2016), 770-778.
|
| [51] |
Z. Han, X. Meichen, P. Hong, L. Zhicai, and G. Jun, “NSNP-DFER: A Nonlinear Spiking Neural P Network for Dynamic Facial Expression Recognition,” Computers & Electrical Engineering 115 (2024): 109125, https://doi.org/10.1016/j.compeleceng.2024.109125.
|
| [52] |
G. Jung, H. Kong, S. W. Lee, “Text-Guided Weakly Supervised Framework for Dynamic Facial Expression Recognition,” Pattern Recognition (2025): 112910: SSRN 5282338, https://doi.org/10.48550/arXiv.2511.10958.
|
| [53] |
C. Huang, F. Jiang, Z. Han, et al., Modeling Fine-Grained Relations in Dynamic Space-Time Graphs for Video-Based Facial Expression Recognition,” IEEE Transactions on Affective Computing 16, no. 3 (2025): 1675-1692, https://doi.org/10.1109/taffc.2025.3530973.
|
| [54] |
H. Zhang, X. Xiao, T. Huang, S. Liu, Y. Xia, and J. Li, “An Novel End-to-End Network for Automatic Student Engagement Recognition,” in 2019 IEEE 9th International Conference on Electronics Information and Emergency Communication (ICEIEC) (IEEE, 2019), 342-345.
|
| [55] |
A. Abedi and S. S. Khan, “Improving State-of-the-Art in Detecting Student Engagement With Resnet and Tcn Hybrid Network,” in 2021 18th Conference on Robots and Vision (CRV) (IEEE, 2021), 151-157.
|
| [56] |
C. Lu, Y. Jiang, K. Fu, Q. Zhao, and H. Yang, “Lstpnet: Long Short-Term Perception Network for Dynamic Facial Expression Recognition in the Wild,” Image and Vision Computing 142 (2024): 104915, https://doi.org/10.1016/j.imavis.2024.104915.
|
| [57] |
L. Sun, Z. Lian, B. Liu, and J. Tao, “Mae-Dfer: Efficient Masked Autoencoder for Self-Supervised Dynamic Facial Expression Recognition,” in Proceedings of the 31st ACM International Conference on Multimedia, (2023), 6110-6121.
|
PDF
(1380KB)
