A dual-modal approach for detecting and classifying autism spectrum disorder using behavioral and facial features

Abstract

Background: Autism spectrum disorder (ASD) is a neurodevelopmental condition that significantly affects social connection, behavior, and knowledge acquisition. Despite increasing global prevalence, timely diagnosis remains challenging due to heterogeneity in clinical presentation. Aim: The aim of this study is to develop a dual-modal framework for early detection of ASD by analyzing behavioral assessment and image data. Methods: The proposed framework consists of two independent yet complementary modules. In the behavioral module, questionnaire responses and assessment data were analyzed using an artificial neural network classifier to predict the likelihood of ASD. In the visual module, facial images were analyzed using a DenseNet121-based deep learning model with transfer learning to detect ASD-related traits. Each module independently estimates ASD probability and categorizes severity levels. Results: The DenseNet121 model achieved strong performance in image-based ASD detection, with 91.16% (95% confidence interval [CI]: 87.8–94.2) accuracy, 91.2% (95% CI: 87.8–94.2) sensitivity, and 89.8% specificity, including when trained on a relatively small dataset. Independent training of the two modules may improve robustness and reduce modality-specific bias. Conclusion: The proposed framework demonstrates potential for enhancing early ASD detection using dual modalities. The findings support the use of deep learning-based approaches to improve detection accuracy. Relevance for patients: Early screening of ASD can facilitate timely interventions and personalized care strategies. This method offers a noninvasive, data-driven approach that may support caregivers and healthcare systems in informed decision-making, ultimately benefiting individuals with ASD and their families.

Keywords

Behavioral features / Facial features / Artificial neural network / Transfer learning / Machine learning / Deep learning model / Severity prediction

Acknowledgments

None.

Funding

None.

Conflict of interest

The authors declare that they have no competing interests.

Author contributions

Conceptualization: All authors Formal analysis: Sankari M.S Investigation: All authors Methodology: Sankari M.S Writing – original draft: Sankari M.S Writing – review & editing: All authors

Ethics approval and consent to participate

This study was conducted retrospectively using publicly available human-subject data from the Kaggle open repository. Ethical approval was not required, as confirmed by the license provided with the open-access data.

Consent for publication

Not applicable.

Availability of data

The dataset used in this study is publicly available through the Kaggle open repository: (i) https://www.kaggle.com/code/mohammedabdelaleem/gathered-asd-datasets. (ii) https://data.mendeley.com/datasets/f9dycfvwbt/3.

References

[1] Lord C, Cook EH Jr. , Leventhal BL, Amaral DG. Autism spectrum disorders. Neuron. 2000; 28(2): 355-363. doi: 10.1016/S0896-6273(00)00115-X

[2] Verywell Health Staff. Do People with Autism Lack Empathy? Verywell Health; 2011. Available from: https://www.verywellhealth.com/do-people/with-autism-lack-empathy-259887 [Last accessed on 2025 Jul 10]. https://www.verywellhealth.com/do-people/with-autism-lack-empathy-259887

[3] Baio J, Wiggins L, Christensen DL, et al. Prevalence of autism spectrum disorder among children aged 8 years - autism and developmental disabilities monitoring network, 11 sites, United States, 2014. MMWR Surveill Summ. 2018; 67(SS-6): 1-23. doi: 10.15585/mmwr.ss6706a1

[4] Sujatha R, Aarthy SL, Chatterjee J, Alaboudi A, Jhanjhi NZ. A machine learning way to classify autism spectrum disorder. Int J Emerg Technol Learn. 2021; 16(6): 182-200. doi: 10.3991/ijet.v16i06.21575

[5] Randall M, Egberts KJ, Samtani A, et al. Diagnostic tests for autism spectrum disorder (ASD) in preschool children. Cochrane Database Syst Rev. 2018; 7(7): CD009044. doi: 10.1002/14651858.CD009044.pub2

[6] Mashudi NA, Ahmad N, Noor NM. Classification of adult autism spectrum disorder using a machine learning approach. IAES Int J Artif Intell. 2021; 10(3): 743-750. doi: 10.11591/ijai.v10.i3.743

[7] Bala M, Ali MH, Satu MS, Hasan KF, Moni MA. Efficient machine learning models for early stage detection of autism spectrum disorder. Algorithms. 2022; 15(5): 166. doi: 10.3390/a15050166

[8] Wiggins LD, Rice CE, Barger B, et al. DSM-5 criteria for autism spectrum disorder maximizes diagnostic sensitivity and specificity in preschool children. Soc Psychiatry Psychiatr Epidemiol. 2019; 54(6): 693-701. doi: 10.1007/s00127-019-01674-9

[9] Ramana KS. Machine learning-based novel autism spectrum disorder screening. Turk J Comput Math Educ. 2021; 12(3): 4866-4879. doi: 10.17762/turcomat.v12i3.4132

[10] Is it Autism? Facial Features that Show Disorder. CBS News; 2011. Available from: https://www.cbsnews.com/pictures/is/it/autism/facial/features/that/show/disorder/9 [Last accessed on 2024 Jul 11]. https://www.cbsnews.com/pictures/is/it/autism/facial/features/that/show/disorder/9

[11] Bhat S, Acharya UR, Adeli H, Bairy GM, Adeli A. Autism: Causative factors, early diagnosis and therapies. Rev Neurosci. 2014; 25(6): 841-850. doi: 10.1515/revneuro-2014-0033

[12] Ardakani HA, Taghizadeh M, Shayegh F. Diagnosis of autism disorder based on deep network trained by augmented EEG signals. Int J Neural Syst. 2022; 32(11): 2250046. doi: 10.1142/S0129065722500461

[13] Khodatars M, Shoeibi A, Sadeghi D, et al. Deep learning for neuroimaging-based diagnosis and rehabilitation of autism spectrum disorder: A review. Comput Biol Med. 2021; 139: 104949. doi: 10.1016/j.compbiomed.2021.104949

[14] Tamilarasi FC, Shanmugam J. Convolutional neural network-based autism classification. In: Proceedings of the 5th International Conference on Communication and Electronics Systems (ICCES). Coimbatore, India: IEEE; 2020. p. 1208-1212. doi: 10.1109/ICCES48766.2020.9137926

[15] Mohanta A, Mittal VK. Analysis and classification of speech sounds of children with autism spectrum disorder using acoustic features. Comput Speech Lang. 2022; 72: 101287. doi: 10.1016/j.csl.2021.101287

[16] Ahmed M, Hussain S, Ali F, et al. Summarizing recent developments in autism spectrum disorder detection using machine learning and deep learning techniques. Appl Sci. 2025; 15(14): 8056. doi: 10.3390/app15148056

[17] Poonia A, Kumar S, Raghuwanshi G. Machine learning and deep learning for autism spectrum disorder discovery: A descriptive review. J Comput Cogn Eng. 2025; 4(3): 267-283. doi: 10.47852/bonviewJCCE52024757

[18] Ding Y, Zhang H, Qiu T. Deep learning approach to predict autism spectrum disorder: A systematic review and meta-analysis. BMC Psychiatry. 2024; 24(1): 739. doi: 10.1186/s12888-024-05388-7

[19] Singh A, Farooqui Z, Sattler B, Usua U, Helde M. Using Machine Learning Optimization to Predict Autism in Toddlers. In: Conference: 11th Annual International Conference on Industrial Engineering and Operations Management. 2021. p. 69206931. doi: 10.46254/an11.20211201

[20] Hassan MM, Taher SA. Analysis and classification of autism data using machine learning algorithms. Sci J Univ Zakho. 2022; 10(4): 206-212. doi: 10.25271/sjuoz.2022.10.4.1036

[21] Uddin KMM. A machine learning approach to predict autism spectrum disorder for children and adults using feature optimization. Netw Biol. 2023; 13(2): 37-46. doi: 10.17323/1996-7845.2023.2.37

[22] Talukdar J, Gogoi D, Singh TP. Comparative assessment of machine learning classifiers for autism spectrum disorder detection. Healthc Anal. 2023; 3: 100178. doi: 10.1016/j.health.2023.100178

[23] Bawa P, Kadyan V, Mantri A, Vardhan H. Investigating multiclass autism spectrum disorder classification using machine learning techniques. e-Prime Adv Electr Eng Electron Energy. 2024; 8: 100602. doi: 10.1016/j.eprime.2024.100602

[24] Haque N, Islam T, Erfan M. An exploration of machine learning approaches for early autism spectrum disorder detection. Healthc Anal. 2025; 7: 100379. doi: 10.1016/j.health.2025.100379

[25] Jahanara S, Padmanabhan S. Detecting autism from facial images. Int J Adv Res Ideas Innov Technol. 2021; 7(2): 219-225. doi: 10.13140/RG.2.2.35268.35202

[26] Akter T, Ali MH, Khan MI, et al. Improved transfer learning-based facial recognition framework for early autism detection. Brain Sci. 2021; 11(6): 734. doi: 10.3390/brainsci11060734

[27] Alam MS, Rashid MM, Roy R, Faizabadi AR, Gupta KD, Ahsan MM. Empirical study of autism spectrum disorder diagnosis using facial images and transfer learning. Bioengineering (Basel). 2022; 9(11): 710. doi: 10.3390/bioengineering9110710

[28] Hriti NS, Shaer K, Momin FMN, Mahmud H, Hasan MK. Autism Classification using Visual and Behavioral Data. medRxiv. [Preprint]; 2022. doi: 10.1101/2022.10.12.22281002

[29] Rahman KK, Subashini MM. Identification of autism in children using static facial features and deep neural networks. Brain Sci. 2022; 12(1): 94. doi: 10.3390/brainsci12010094

[30] Anjum J, Hia NA, Waziha A, Kalpoma KA. Deep Learning-Based Feature Extraction from Children’s Facial Images for Autism Detection. In: Proceedings of the Cognitive Models and Artificial Intelligence Conference; 2024. p. 155-159. doi: 10.1145/3660853.3660888

[31] Bargrizan A. ASD Final [Dataset]. Kaggle; 2025. Available from: https://www.kaggle.com/datasets/afarinbargrizan/asd-final [Last accessed on 2025 Jul 16]. https://www.kaggle.com/datasets/afarinbargrizan/asd-final

[32] Allison C, Matthews FE, Ruta L, et al. Quantitative checklist for autism in toddlers (Q-CHAT). A population screening study with follow-up: the case for multiple-point screening for autism. BMJ Paediatr Open. 2021; 5: e000700. doi: 10.1136/bmjpo-2020-000700

[33] Irmak E. COVID-19 disease severity assessment using CNN model. IET Image Process. 2021; 15(8): 1814-1824. doi: 10.1049/ipr2.12175

[34] Madhiarasan M, Louzazni M. Artificial neural networks: Architecture, types, and forecasting applications. J Electr Comput Eng. 2022; 2022: 5416722. doi: 10.1155/2022/5416722

[35] Thillaikarasi R, Kumaresan P. Autism spectrum disorder detection using facial traits: DFDK framework. Mendeley Data. 2025; 2: V3. doi: 10.17632/f9dycfvwbt.3

[36] Zhou Q, Zhu W, Li F, Yuan M, Zheng L, Liu X. Transfer learning of the resnet-18 and densenet-121 model used to diagnose intracranial hemorrhage in CT scanning. Curr Pharm Des. 2022; 28(4): 287-295. doi: 10.2174/1381612828666220208165622

[37] Levels of Autism: ASD Levels of Severity. Autism Speaks. Available from: https://www.autismspeaks.org/levels-of-autism [Last accessed on 2025 Sep 12]. https://www.autismspeaks.org/levels-of-autism

[38] Jacob SG, Sulaiman MMBA, Bennet B. Feature signature discovery for autism detection: An automated machine learning based feature ranking framework. Comput Intell Neurosci. 2023; 2023: 6330002. doi: 10.1155/2023/6330002

[39] Raj S, Masood S. Analysis and detection of autism spectrum disorder using machine learning techniques. Procedia Comput Sci. 2020; 167: 994-1004. doi: 10.1016/j.procs.2020.03.402

[40] Vakadkar K, Purkayastha D, Krishnan D. Detection of autism spectrum disorder in children using machine learning techniques. SN Comput Sci. 2021; 2(5): 386. doi: 10.1007/s42979-021-00773-8

[41] Kojovic N, Natraj S, Mohanty SP, Maillart T, Schaer M. Using 2D video-based pose estimation for automated prediction of autism spectrum disorders in young children. Sci Rep. 2021; 11(1): 15069. doi: 10.1038/s41598-021-94555-0

[42] Mohanty AS, Parida P, Patra KC. Identification of autism spectrum disorder using deep neural network. J Phys Conf Ser. 2021; 1921(1): 012006. doi: 10.1088/1742-6596/1921/1/012006

[43] Awaji B, Senan EM, Olayah F, et al. Hybrid techniques of facial feature image analysis for early detection of autism spectrum disorder based on combined CNN features. Diagnostics (Basel). 2023; 13(18): 2948. doi: 10.3390/diagnostics13182948

[44] Khan B, Bhatti SM, Akram A. Autism spectrum disorder detection in children using deep learning and facial images. Bull Bus Econ (BBE). 2024; 13(1): 557-572. doi: 10.61506/01.00241

[45] Nawghare P, Prasad J. Hybrid CNN and random forest model with late fusion for detection of autism spectrum disorder in toddlers. MethodsX. 2025; 14: 103278. doi: 10.1016/j.mex.2025.103278