G-STEP: A human-centered framework for the design and deployment of responsible LLM-based mental health chatbots

Abstract

Large language model (LLM)–based chatbots are rapidly emerging as a promising vehicle for delivering companionship, psychoeducation, on-demand mental health support, and clinical workflow automation or augmentation. However, inconsistent or inadequate crisis handling, along with well-documented hallucinations, bias, and opaque accountability of LLMs, pose significant challenges for responsible deployment and clinical use of those chatbots. This paper introduces key streams of research on LLM-based Mental Health Chatbots (LLM-MHCs) and critically examines their benefits and risks. Then, we propose a five-dimensional human-centered framework for responsible design of LLM-MHCs that consists of governance, safety, transparency and explainability, empathy, and personalization (G-STEP). Finally, we outline several directions for future research, including conducting longitudinal randomized clinical trials with patients with mental health conditions to rigorously assess the effectiveness and user experience of LLM-MHCs; developing open benchmarks and comprehensive metrics for evaluating their efficacy, reliability, and equity; and supporting continuous human-in-the-loop and responsible deployment of LLM-MHCs.

Graphical abstract

Keywords

mental health / large language model (LLM) / chatbots / user safety / risk / governance

References

[1] Ayers J W, Poliak A, Dredze M, Leas E C, Zhu Z, Kelley J B, Faix D J, Goodman A M, Longhurst C A, Hogarth M, Smith D M, (2023). Comparing physician and artificial intelligence chatbot responses to patient questions posted to a public social media forum. JAMA Internal Medicine, 183( 6): 589–596

[2] Blease C, Torous J, (2023). ChatGPT and mental healthcare: balancing benefits with risks of harms. BMJ Mental Health, 26( 1): e300884

[3] Blease C, Worthen A, Torous J, (2024). Psychiatrists’ experiences and opinions of generative artificial intelligence in mental healthcare: an online mixed methods survey. Psychiatry Research, 333: 115724

[4] Campbell L O, Babb K, Lambie G, Hayes G, (2025). An examination of generative AI response to suicide inquires: content analysis. JMIR Mental Health, 12: v12i3e73623

[5] Elwahsh S, Stern N, Singh A, Ayobi A (2025). Linguistic diversity and mental well-being: co-designing custom AI chatbots with multilingual mothers. In Proceedings of the 7th ACM Conference on Conversational User Interfaces. New York, USA: ACM, 1–17

[6] Fang C MLiu A RDanry VLee EChan S W TPataranutaporn PMaes PPhang JLampe MAhmad LAgarwal S (2025). How AI and human behaviors shape psychosocial effects of extended chatbot use: A longitudinal randomized controlled study. arXiv.2503.17473

[7] Grabb DLamparth MVasan N (2024). Risks from language models for automated mental healthcare: ethics and structure for implementation. arXiv preprint arXiv:2406.11852

[8] Guo Z, Lai A, Thygesen J H, Farrington J, Keen T, Li K, (2024). Large language models for mental health applications: systematic review. JMIR Mental Health, 11: e57400

[9] Hager P, Jungmann F, Holland R, Bhagat K, Hubrecht I, Knauer M, Vielhauer J, Makowski M, Braren R, Kaissis G, Rueckert D, (2024). Evaluation and mitigation of the limitations of large language models in clinical decision-making. Nature Medicine, 30( 9): 2613–2622

[10] Hancock BBordes AWeston JLevy K (2019). Learning from dialogue after deployment: Feed yourself, chatbot! Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy. July 28–August 2, 2019. Stroudsburg, PA: Association for Computational Linguistics, 3667–3684

[11] Heinz M V, Mackin D M, Trudeau B M, Bhattacharya S, Wang Y, Banta H A, Jewett A. D, Salzhauer A J, Griffin T Z, Jacobson N C, (2025). Randomized trial of a Generative AI chatbot for mental health treatment. NEJM AI, 2( 4): AIoa2400802

[12] Hipgrave L, Goldie J, Dennis S, Coleman A, (2025). Balancing risks and benefits: Clinicians’ perspectives on the use of generative AI chatbots in mental healthcare. Frontiers in Digital Health, 7: 1606291

[13] Hua Y, Na H, Li Z, Liu F, Fang X, Clifton D, Torous J, (2025a). A scoping review of large language models for generative tasks in mental health care. NPJ Digital Medicine, 8( 1): 230

[14] Hua Y, Siddals S, Ma Z, Galatzer-Levy I, Xia W, Hau C, Na H, Flathers M, Linardon J, Ayubcha C, Torous J, (2025b). Charting the evolution of artificial intelligence mental health chatbots from rule-based systems to large language models: a systematic review. World Psychiatry; Official Journal of the World Psychiatric Association (WPA), 24( 3): 383–394

[15] Huo B, Collins G S, Chartash D, Thirunavukarasu A J, Flanagin A, Iorio A, Cacciamani G, Chen X, Liu N, Mathur P, Chan A W, Laine C, Pacella D, Berkwits M, Antoniou S A, Camaradou J C, Canfield C, Mittelman M, Feeney T, Loder E W, Agha R, Saha A, Mayol J, Sunjaya A, Harvey H, Ng J Y, McKechnie T, Lee Y, Verma N, Stiglic G, McCradden M, Ramji K, Boudreau V, Ortenzi M, Meerpohl J J, Vandvik P O, Agoritsas T, Samuel D, Frankish H, Anderson M, Yao X, Loeb S, Lokker C, Liu X, Guallar E, Guyatt G H, the The CHART Collaborative, (2025). Reporting guideline for chatbot health advice studies. JAMA Network Open, 8( 8): e2530220

[16] Kalam K, Rahman J, Islam M, Dewan S, (2024). ChatGPT and mental health: Friends or foes?. Health Science Reports, 7( 2): e1912

[17] Kian M JZong MFischer KSingh AVelentza A MSang PUpadhyay SGupta AFaruki M ABrowning W (2024). Can an LLM-Powered socially assistive robot effectively and safely deliver cognitive behavioral therapy? A study with university students. arXiv:2402.17937

[18] Kim J, Leonte K G, Chen M L, Torous J B, Linos E, Pinto A, Rodriguez C I, (2024). Large language models outperform mental and medical health care professionals in identifying obsessive-compulsive disorder. npj. Digital Medicine, 7( 1): 193

[19] Kolding S, Lundin R M, Hansen L, Østergaard S D, (2025). Use of generative artificial intelligence (AI) in psychiatry and mental health care: A systematic review. Acta Neuropsychiatrica, 37: e37

[20] Kouros T, Papa V, (2024). Digital mirrors: AI companions and the self. Societies, 14( 10): 200

[21] Lan XHan ZCheng YSheng LFeng JGao CLi Y (2025). Depression detection on social media with large language models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. Suzhou, China. 11, 2155–2171

[22] Lawrence H R, Schneider R A, Rubin S B, Matarić M J, McDuff D J, Jones B M, (2024). The opportunities and risks of large language models in mental health. JMIR Mental Health, 11: v11i9e59479

[23] Lee J, Lee D, (2023). User perception and self-disclosure towards an AI psychotherapy chatbot according to the anthropomorphism of its profile picture. Telematics and Informatics, 85: 102052

[24] Leow JChua H NJasser M BIssa BWong R T K (2025). Comparison of depression detection between LLMs and zero-shot learning using DAD dataset. The 21st IEEE International Colloquium on Signal Processing & Its Applications (CSPA), Pulau Pinang, Malaysia. New York: IEEE, 295–300

[25] Linardon J, Torous J, Firth J, Cuijpers P, Messer M, Fuller-Tyszkiewicz M, (2024). Current evidence on the efficacy of mental health smartphone apps for symptoms of depression and anxiety. A meta-analysis of 176 randomized controlled trials. World Psychiatry; Official Journal of the World Psychiatric Association (WPA), 23( 1): 139–149

[26] Liu Z, Bao Y, Zeng S, Qian R, Deng M, Gu A, Li J, Wang W, Cai W, Li W, Wang H, Xu D, Lin G N, (2024). Large language models in psychiatry: Current applications, limitations, and future scope. Big Data Mining and Analytics, 7( 4): 1148–1168

[27] Madrid-Cagigal A, Kealy C, Potts C, Mulvenna M D, Byrne M, Barry M M, Donohoe G, (2025). Digital mental health interventions for university students with mental health difficulties: A systematic review and meta-analysis. Early Intervention in Psychiatry, 19( 3): e70017

[28] McBain R K, Cantor J H, Zhang L A, Baker O, Zhang F, Halbisen A, Kofner A, Breslau J, Stein B, Mehrotra A, Yu H, (2025). Competency of large language models in evaluating appropriate responses to suicidal ideation: Comparative study. Journal of Medical Internet Research, 27: e67891

[29] Merrill K Jr, Kim J, Collins C, (2022). AI companions for lonely individuals and the role of social presence. Communication Research Reports, 39( 2): 93–103

[30] Na HHua YWang ZShen TYu BWang LWang WTorous JChen L (2025). A survey of large language models in psychotherapy: current landscape and future directions. Findings of the Association for Computational Linguistics: ACL, July. Stroudsburg: Association for Computational Linguistics, 7362–7376

[31] Ni Y, Jia F, (2025). A scoping review of AI-Driven digital interventions in mental health care: Mapping applications across screening, support, monitoring, prevention, and clinical education. Health Care, 13( 10): 1205

[32] Obradovich N, Khalsa S S, Khan W U, Suh J, Perlis R H, Ajilore O, Paulus M P, (2024). Opportunities and risks of large language models in psychiatry. NPP—Digital Psychiatry and Neuroscience, 2( 1): 8

[33] Omar M, Levkovich I, (2025). Exploring the efficacy and potential of large language models for depression: A systematic review. Journal of Affective Disorders, 371: 234–244

[34] Ophir Y, Tikochinski R, Elyoseph Z, Efrati Y, Rosenberg H, (2025). Balancing promise and concern in AI therapy: A critical perspective on early evidence from the MIT–OpenAI RCT. Frontiers in Medicine, 12: 1612838

[35] Panteli D, Buttigieg S, Keyrellous A, Ladewig K, Azzopardi Muscat N, McKee M, (2024). Artificial intelligence in public health: lessons from the European public health conference. Eurohealth, 30( 3): 13–18

[36] Park J IAbbasian MAzimi IBounds D TJun AHan JMcCarron R MBorelli JSafavi PMirbaha S (2024). Building trust in mental health chatbots: Safety metrics and LLM-based evaluation tools. arXiv preprint arXiv:2408.04650

[37] Peipert A, Lorenzo-Luaces L, (2025). Is there a treatment for the Turker blues? A fully remote nationwide randomized controlled trial of a digital intervention for depression in adult online workers. Journal of Consulting and Clinical Psychology, 93( 11): 719–734

[38] Pichowicz W, Kotas M, Piotrowski P, (2025). Performance of mental health chatbot agents in detecting and managing suicidal ideation. Scientific Reports, 15( 1): 31652

[39] Potts C, Lindström F, Bond R, Mulvenna M, Booth F, Ennis E, Parding K, Kostenius C, Broderick T, Boyd K, Vartiainen A K, Nieminen H, Burns C, Bickerdike A, Kuosmanen L, Dhanapala I, Vakaloudis A, Cahill B, MacInnes M, Malcolm M, O’Neill S, (2023). A multilingual digital mental health and well-being chatbot (ChatPal): Pre-Post multicenter intervention study. Journal of Medical Internet Research, 25: e43051

[40] Raile P, (2024). The usefulness of ChatGPT for psychotherapists and patients. Humanities & Social Sciences Communications, 11( 1): 47

[41] Robberegt S J, Brouwer M E, Kooiman B E A M, Stikkelbroek Y A J, Nauta M H, Bockting C L H, (2023). Meta-analysis: Relapse prevention strategies for depression and anxiety in remitted adolescents and young adults. Journal of the American Academy of Child and Adolescent Psychiatry, 62( 3): 306–317

[42] Shang L, Zhang L, Zhao Y, Cong A, Xu P, Lv P, Ding H, Wang H, Huang Q, Li J, Wang G, (2025). Prevalence, correlates, and treatment: epidemiological survey of common mental disorders based on DSM-5 in Beijing, China. Psychological Medicine, 55: e373

[43] Stade E C, Stirman S W, Ungar L H, Boland C L, Schwartz H A, Yaden D B, Sedoc J, DeRubeis R J, Willer R, Eichstaedt J C, (2024). Large language models could change the future of behavioral healthcare: a proposal for responsible development and evaluation. npj Mental Health Research, 3: 12

[44] Wang Y, Wang Y, Xiao Y, Escamilla L, Augustine B, Crace K, Zhou G, Zhang Y, (2025). Evaluating an LLM-Powered chatbot for cognitive restructuring: Insights from mental health professionals. arXiv, 2501.15599

[45] Wei Y, Guo L, Lian C, Chen J, (2023). ChatGPT: opportunities, risks and priorities for psychiatry. Asian Journal of Psychiatry, 90: 103808

[46] Yuan A, Garcia Colato E, Pescosolido B, Song H, Samtani S, (2025). Improving workplace well-being in modern organizations: A review of large language model-based mental health chatbots. ACM Transactions on Management Information Systems, 16( 1): 1–26

[47] Zhang D, Zhou L, Tao J, Zhu T, Gao G, (2025). KETCH: A knowledge-enhanced transformer-based approach to suicidal ideation detection from social media content. Information Systems Research, 36( 1): 572–599

Rights & permissions

Higher Education Press