Machine learning-enhanced behavioral profiling improves phenotypic screening in a genetic mouse model of autism spectrum disorder

Abstract

Aim: Genetic factors are major contributors to neurodevelopmental disorders such as autism spectrum disorder. Genetically modified animal models are widely used, yet behavioral phenotyping often relies on coarse metrics that may overlook subtle but meaningful abnormalities. Here, using Shank3B knockout (KO) mice as an example, we developed a machine learning-based behavioral analysis pipeline to enhance sensitivity and precision in genotype-related behavioral screening.

Methods: Adult male knockout and wild-type littermates were tested in standard behavioral paradigms, including the three-chamber social test, grooming assay, open field, and elevated plus maze. Videos were processed using markerless pose estimation to extract high-resolution behavioral features. Multidimensional features were analyzed with dimensionality reduction and unsupervised clustering, and genotype discrimination was evaluated using supervised classifiers.

Results:Shank3B KO mice showed reduced sociability and social novelty preference, increased repetitive grooming, reduced exploration, and elevated anxiety-like behavior. Fine-grained behavioral features revealed altered behavioral structure and transition patterns across tasks. Unsupervised clustering consistently separated genotypes into distinct behavioral states, and machine learning classifiers accurately predicted genotype based on behavioral features.

Conclusion: This study demonstrates that fine-scale, machine learning-assisted analysis applied to conventional behavioral tests facilitates the detection of genotype-specific phenotypes. The proposed pipeline provides a scalable framework for more precise behavioral screening of genetically modified models and supports translational studies of neurodevelopmental disorders.

Keywords

Genetic mouse model / behavioral phenotyping / machine learning / neurodevelopmental disorders / autism spectrum disorder

References

[1] Genetics of neurological and psychiatric disorders.Res Publ Assoc Res Nerv Ment Dis1983;60:1-300

[2] Striano P.Unraveling the genetic tapestry of neurodevelopmental disorders: a new horizon.J Transl Genet Genom2024;8:278-80

[3] Wang S,Zhang P.Genome-wide by lifetime environment interaction studies of brain imaging phenotypes.Adv Sci2025;13:e00852

[4] Smeland OB,Bahrami S.A genome-wide analysis of the shared genetic risk architecture of complex neurological and psychiatric disorders.Nat Neurosci2025;28:2439-50

[5] Zhang C,Sun P.Trans-ancestry genome-wide analyses of bipolar disorder in East Asian and European populations improve genetic discovery.Nat Neurosci2025;29:293-305

[6] Yang Y,Sun Y.Nkapl deletion drives cognitive deficits through mPFC interneuron dysfunction in a mouse model of schizophrenia.Sci Adv2025;11:eadv4779 PMCID:PMC12487883

[7] Durand CM,Boeckers TM.Mutations in the gene encoding the synaptic scaffolding protein SHANK3 are associated with autism spectrum disorders.Nat Genet2006;39:25-7 PMCID:PMC2082049

[8] Chan JKY.Common outcome, different pathways: social information-processing deficits in autism spectrum disorder and attention-deficit/hyperactivity disorder.World J Psychiatry2022;12:286-97 PMCID:PMC8900584

[9] Peça J,Ting JT.Shank3 mutant mice display autistic-like behaviours and striatal dysfunction.Nature2011;472:437-42 PMCID:PMC3090611

[10] Guo B,Chen Q.Anterior cingulate cortex dysfunction underlies social deficits in Shank3 mutant mice.Nat Neurosci2019;22:1223-34

[11] Chen Q,Gao X.Dysfunction of cortical GABAergic neurons leads to sensory hyper-reactivity in a Shank3 mouse model of ASD.Nat Neurosci2020;23:520-32 PMCID:PMC7131894

[12] Orefice LL,Chirila AM,Head JP.Peripheral mechanosensory neuron dysfunction underlies tactile and behavioral deficits in mouse models of ASDs.Cell2016;166:299-313 PMCID:PMC5567792

[13] Orefice LL,Morency DT.Targeting peripheral somatosensory neurons to improve tactile-related phenotypes in ASD models.Cell2019;178:867-86.e24 PMCID:PMC6704376

[14] Guo B,Choi S.Restoring thalamocortical circuit dysfunction by correcting HCN channelopathy in Shank3 mutant mice.Cell Rep Med2024;5:101534 PMCID:PMC11149412

[15] Balaan C,Eulalio T.Juvenile Shank3b deficient mice present with behavioral phenotype relevant to autism spectrum disorder.Behav Brain Res2019;356:137-47 PMCID:PMC6247805

[16] Mathis MW.Joint modelling of brain and behaviour dynamics with artificial intelligence.Nat Rev Neurosci2025;27:87-100

[17] Liu H,Xu J.Dissection of the relationship between anxiety and stereotyped self-grooming using the Shank3B mutant autistic model, acute stress model and chronic pain model.Neurobiol Stress2021;15:100417 PMCID:PMC8591549

[18] Mathis A,Cury KM.DeepLabCut: markerless pose estimation of user-defined body parts with deep learning.Nat Neurosci2018;21:1281-9

[19] Madhusoodanan J.DeepLabCut: the motion-tracking tool that went viral.Nature2024;629:960-1

[20] Lauer J,Ye S.Multi-animal pose estimation, identification and tracking with DeepLabCut.Nat Methods2022;19:496-504 PMCID:PMC9007739

[21] Weinreb C,Lin S.Keypoint-MoSeq: parsing behavior by linking point tracking to pose dynamics.Nat Methods2024;21:1329-39 PMCID:PMC11245396

[22] Canela-Grimau M,Busquets-Garcia A.Profiling mouse behavior with computational tools to assess age-dependent differences in associative learning.Cell Rep Methods2025;5:101144 PMCID:PMC12539235

[23] Hsu AI.B-SOiD, an open-source unsupervised algorithm for identification and fast prediction of behaviors.Nat Commun2021;12:5188 PMCID:PMC8408193

[24] Shen Y,Chen X.Behavior decoding delineates seizure microfeatures and associated sudden death risks in mouse models of epilepsy.Ann Neurol2025;99:231-47 PMCID:PMC12812430

[25] Tillmann JF,Schwarz MK.A-SOiD, an active-learning platform for expert-guided, data-efficient discovery of behavior.Nat Methods2024;21:703-11

[26] Moy SS,Perez A.Sociability and preference for social novelty in five inbred strains: an approach to assess autistic‐like behavior in mice.Genes Brain Behavior2004;3:287-302

[27] Mei Y,Zhou Y.Adult restoration of Shank3 expression rescues selective autistic-like phenotypes.Nature2016;530:481-4 PMCID:PMC4898763

[28] Tabaka O,Del Rio Triana R.Prefrontal oxytocin receptor positive cells mediate stress-induced anxiety in tuberous sclerosis complex.Commun Biol2025;8:1789 PMCID:PMC12714756

[29] Muir J,Aarrestad IK.Isolation of psychedelic-responsive neurons underlying anxiolytic behavioral states.Science2024;386:802-10 PMCID:PMC11588385

[30] Ciancone-Chama AG,Biasini E,Balasco L.Gene expression profiling in trigeminal ganglia from Cntnap2^-/- and Shank3b^-/- mouse models of autism spectrum disorder.Neuroscience2023;531:75-85

[31] Burke FF,Sparkes KM.Perinatal hyperandrogenization and immune activation in rodents model subtypes of autism.Transl Psychiatry2026;16:97 PMCID:PMC12923541

[32] Lipina TV,Jaaro‐Peled H.Enhanced dopamine function in DISC1‐L100P mutant mice: implications for schizophrenia.Genes Brain Behavior2010;9:777-89

[33] Bouargane Z,Bennis M,Lamghari-Moubarrad F.Comparative analysis of BORIS, Ethovision, DeepLabCut, and SimBA for quantifying autism spectrum disorder-like behaviors in the valproic acid mouse model.Neurosci Lett2026;875:138542

[34] Khan T,Francis NA.Quantifying social distance using deep learning-based video analysis: results from the BTBR mouse model of autism.Front Behav Neurosci2025;19:1602205 PMCID:PMC12226490

[35] Maisterrena A,Longueville J,Ey E.Female mice prenatally exposed to valproic acid exhibit complex and prolonged social behavior deficits.Prog Neuropsychopharmacol Biol Psychiatry2024;131:110948

[36] Sheppard K,Sabnis GS.Stride-level analysis of mouse open field behavior using deep-learning-based pose estimation.Cell Rep2022;38:110231 PMCID:PMC8796662

[37] Klibaite U,Aldarondo D,Ölveczky BP.Mapping the landscape of social behavior.Cell2025;188:2249-66.e23 PMCID:PMC12010356

[38] Chari T,Couto J.A reduced ability to discriminate social from non-social touch at the circuit level may underlie social avoidance in autism.Nat Commun2025;16:4600 PMCID:PMC12085590

[39] Huang K,Chen K.A hierarchical 3D-motion learning framework for animal spontaneous behavior mapping.Nat Commun2021;12:2784 PMCID:PMC8119960

[40] Han Y,Wang Y.Multi-animal 3D social pose estimation, identification and behaviour embedding with a few-shot learning framework.Nat Mach Intell2024;6:48-61

[41] Nakai N,Yamashita O.Virtual reality-based real-time imaging reveals abnormal cortical dynamics during behavioral transitions in a mouse model of autism.Cell Rep2023;42:112258