PesTest: a comprehensive benchmark for psychological emotional support capability of large language models

Abstract

Large language models (LLMs) demonstrating competent psychological emotional support capabilities have significant potential to deliver effective mental health assistance. The development of comprehensive evaluation methodologies for assessing these LLMs’ capabilities represents a critical research challenge. Current evaluation paradigms remain constrained by several limitations: they typically examine single task modalities, utilize monolingual approaches, and focus on restricted topical domains. These methodological constraints may yield misleading assessments where models exhibit strong benchmark performance yet generate potentially harmful outputs in practical applications. To address these limitations, we introduce PesTest, a novel multidimensional evaluation benchmark designed to assess psychological emotional support capabilities through multiple task modalities (judgment-based and dialogue-based evaluations), multilingual support (English and Chinese), and extensive topical coverage comprising 7 categories and 40 sub-category topics. Our comprehensive evaluation using the PesTest reveals two critical findings. First, current LLMs demonstrate particularly limited capabilities in dialogue-based support tasks. Second, we observe substantial performance variance across different psychological domains, highlighting the need for domain-specific evaluation. Furthermore, by fine-tuning the model and performing few-shot experiments, we achieve improved performance over the original model on the test set. These results validate the effectiveness of PesTest in enhancing the psychological emotional support capabilities of large language models. We hope the proposed benchmark will serve as a valuable resource for future research in this field.

Graphical abstract

Keywords

psychological emotional support / large language models / benchmark of LLMs

References

[1] Bowen P, Govender R, Edwards P, Cattell K . Work-related contact, work–family conflict, psychological distress and sleep problems experienced by construction professionals: an integrated explanatory model. Construction Management and Economics, 2018, 36( 3): 153–174

[2] Evans-Lacko S, Aguilar-Gaxiola S, Al-Hamzawi A, Alonso J, Benjet C, Bruffaerts R, Chiu W T, Florescu S, De Girolamo G, Gureje O, Haro J M, He Y, Hu C, Karam E G, Kawakami N, Lee S, Lund C, Kovess-Masfety V, Levinson D, Navarro-Mateu F, Pennell B E, Sampson N A, Scott K M, Tachimori H, Ten Have M, Viana M C, Williams D R, Wojtyniak B J, Zarkov Z, Kessler R C, Chatterji S, Thornicroft G . Socio-economic variations in the mental health treatment gap for people with anxiety, mood, and substance use disorders: results from the WHO World Mental Health (WMH) surveys. Psychological Medicine, 2018, 48( 9): 1560–1571

[3] COVID-19 Mental Disorders Collaborators . Global prevalence and burden of depressive and anxiety disorders in 204 countries and territories in 2020 due to the COVID-19 pandemic. The Lancet, 2021, 398( 10312): 1700–1712

[4] Yates B T, Taub J . Assessing the costs, benefits, cost-effectiveness, and cost-benefit of psychological assessment: we should, we can, and here’s how. Psychological Assessment, 2003, 15( 4): 478–495

[5] Liu S, Zheng C, Demasi O, Sabour S, Li Y, Yu Z, Jiang Y, Huang M. Towards emotional support dialog systems. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021, 3469−3483

[6] Sun H, Lin Z, Zheng C, Liu S, Huang M. PsyQA: a Chinese dataset for generating long counseling text for mental health support. In: Proceedings of Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021. 2021, 1489−1503

[7] Liu J M, Li D, Cao H, Ren T, Liao Z, Wu J. ChatCounselor: a large language models for mental health support. 2023, arXiv preprint arXiv: 2309.15461

[8] Turcan E, McKeown K. Dreaddit: a Reddit dataset for stress analysis in social media. In: Proceedings of the 10th International Workshop on Health Text Mining and Information Analysis (LOUHI 2019). 2019, 97−107

[9] Haque A, Reddi V, Giallanza T. Deep learning for suicide and depression identification with unsupervised label correction. In: Proceedings of the 30th International Conference on Artificial Neural Networks. 2021, 436−447

[10] Posner K, Brown G K, Stanley B, Brent D A, Yershova K V, Oquendo M A, Currier G W, Melvin G A, Greenhill L, Shen S, Mann J J . The Columbia–Suicide Severity Rating Scale: initial validity and internal consistency findings from three multisite studies with adolescents and adults. American Journal of Psychiatry, 2011, 168( 12): 1266–1277

[11] Xu X, Yao B, Dong Y, Gabriel S, Yu H, Hendler J, Ghassemi M, Dey A K, Wang D . Mental-LLM: leveraging large language models for mental health prediction via online text data. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 2024, 8( 1): 31

[12] Yang K, Zhang T, Kuang Z, Xie Q, Huang J, Ananiadou S. MentaLLaMA: interpretable mental health analysis on social media with large language models. In: Proceedings of the ACM Web Conference 2024. 2024, 4489−4500

[13] Garg M, Shahbandegan A, Chadha A, Mago V. An annotated dataset for explainable interpersonal risk factors of mental disturbance in social media posts. In: Proceedings of Findings of the Association for Computational Linguistics: ACL 2023. 2023, 11960−11969

[14] Naseem U, Dunn A G, Kim J, Khushi M. Early identification of depression severity levels on Reddit using ordinal classification. In: Proceedings of the ACM Web Conference 2022. 2022, 2563−2572

[15] Mauriello M L, Lincoln T, Hon G, Simon D, Jurafsky D, Paredes P. SAD: a stress annotated dataset for recognizing everyday stressors in SMS-like conversational systems. In: Proceedings of Extended Abstracts of 2021 CHI Conference on Human Factors in Computing Systems. 2021, 399

[16] Acosta J. Using emotion to gain rapport in a spoken dialog system. In: Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, Companion Volume: Student Research Workshop and Doctoral Consortium. 2009, 49−54

[17] Cao H, Zhang Y, Feng S, Yang X, Wang D, Zhang Y. TOOL-ED: enhancing empathetic response generation with the tool calling capability of LLM. In: Proceedings of the 31st International Conference on Computational Linguistics. 2025, 5305−5320

[18] Ong K, Dai W, Li C, Feng D, Li H, Wu J, Cheong J, Mao R, Mengaldo G, Cambria E, Liang P P. Human behavior atlas: benchmarking unified psychological and social behavior understanding. 2025, arXiv preprint arXiv: 2510.04899

[19] Poria S, Hazarika D, Majumder N, Naik G, Cambria E, Mihalcea R. MELD: a multimodal multi-party dataset for emotion recognition in conversations. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2019, 527−536

[20] Lamichhane B. Evaluation of ChatGPT for NLP-based mental health applications. 2023, arXiv preprint arXiv: 2303.15727

[21] Zung W W . A self-rating depression scale. Archives of General Psychiatry, 1965, 12( 1): 63–70

[22] Marteau T M, Bekker H . The development of a six-item short-form of the state scale of the Spielberger State—Trait Anxiety Inventory (STAI). British Journal of Clinical Psychology, 1992, 31( 3): 301–306

[23] OpenAI . GPT-4 technical report. 2023, arXiv preprint arXiv: 2303.08774

[24] Liu Z, Nie P, Cai J, Wang H, Niu Z Y, Zhang P, Sachan M, Peng K. XDailyDialog: a multilingual parallel dialogue corpus. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023, 12240−12253

[25] Xu X, Yao B, Dong Y, Yu H, Hendler J, Dey A K, Wang D. Leveraging large language models for mental health prediction via online text data. 2023, arXiv preprint arXiv: 2307.14385v1

[26] Sun Q, Wang Z, Li S, Zhu Q, Zhou G . Stance detection via sentiment information and neural network model. Frontiers of Computer Science, 2019, 13( 1): 127–138

[27] Papineni K, Roukos S, Ward T, Zhu W J. Bleu: a method for automatic evaluation of machine translation. In: Proceedings of the 40th annual meeting of the Association for Computational Linguistics. 2002, 311−318

[28] Li J, Galley M, Brockett C, Gao J, Dolan B. A diversity-promoting objective function for neural conversation models. In: Proceedings of 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2016, 110−119

[29] Ravenda F, Bahrainian S A, Raballo A, Mira A, Kando N. Are LLMs effective psychological assessors? Leveraging adaptive RAG for interpretable mental health screening through psychometric practice. In: Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025, 8975−8991

[30] Kumar A, Yunusov S, Emami A. Subtle biases need subtler measures: dual metrics for evaluating representative and affinity bias in large language models. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024, 375−392

[31] Zhang Y, Li M, Long D, Zhang X, Lin H, Yang B, Xie P, Yang A, Liu D, Lin J, Huang F, Zhou J. Qwen3 embedding: advancing text embedding and reranking through foundation models. 2025, arXiv preprint arXiv: 2506.05176

[32] Ye J, Chen X, Xu N, Zu C, Shao Z, Liu S, Cui Y, Zhou Z, Gong C, Shen Y, Zhou J, Chen S, Gui T, Zhang Q, Huang X. A comprehensive capability analysis of GPT-3 and GPT-3.5 series models. 2023, arXiv preprint arXiv: 2303.10420

[33] OpenAI . GPT-4o system card. 2024, arXiv preprint arXiv: 2410.21276

[34] Yang A, Xiao B, Wang B, Zhang B, Bian C, , et al. Baichuan 2: open large-scale language models. 2025, arXiv preprint arXiv: 2309.10305

[35] Zeng A, Liu X, Du Z, Wang Z, Lai H, et al. GLM-130B: an open bilingual pre-trained model. In: Proceedings of the 11th International Conference on Learning Representations. 2023, 1−56

[36] Du Z, Qian Y, Liu X, Ding M, Qiu J, Yang Z, Tang J. GLM: general language model pretraining with autoregressive blank infilling. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2022, 320−335

[37] Bai J, Bai S, Chu Y, Cui Z, Dang K, , et al. Qwen technical report. 2023, arXiv preprint arXiv: 2309.16609

[38] Yang A, Yang B, Hui B, Zheng B, Yu B, , et al. Qwen2 technical report. 2024, arXiv preprint arXiv: 2407.10671

[39] Yang A, Li A, Yang B, Zhang B, Hui B, , et al. Qwen3 technical report. 2025, arXiv preprint arXiv: 2505.09388

[40] Touvron H, Lavril T, Izacard G, Martinet X, Lachaux M A, Lacroix T, Rozière B, Goyal N, Hambro E, Azhar F, Rodriguez A, Joulin A, Grave E, Lample G. LLaMA: open and efficient foundation language models. 2023, arXiv preprint arXiv: 2302.13971

[41] Grattafiori A, Dubey A, Jauhri A, Pandey A, Kadian A, , et al. The Llama 3 herd of models. 2024, arXiv preprint arXiv: 2407.21783

[42] DeepSeek-AI . DeepSeek-V3 technical report. 2025, arXiv preprint arXiv: 2412.19437

[43] Hu E J, Shen Y, Wallis P, Allen-Zhu Z, Li Y, Wang S, Wang L, Chen W. LoRA: low-rank adaptation of large language models. In: Proceedings of the 10th International Conference on Learning Representations. 2022, 1−13

Rights & permissions

Higher Education Press