Large language model enabled synthetic dataset generation for human-AI teaming in mental health assessment

i

This article has been updated. A Correction to this article was published on .

i

The Original Article was published on

i

The Original Article was published on

i

This article has been updated

i

This article was retracted on

i

This article has been updated

Abstract

Mental health assessment presents unique challenges in healthcare due to its inherently subjective nature and the scarcity of high-quality training data. This research explores the use of large language models (LLMs) to generate synthetic datasets for Human-AI teaming algorithms, focusing on mental health assessments. We created a diverse dataset that simulates human-AI collaboration scenarios in diagnostic processes. Synthetic data are labeled through an innovative approach that involves two human annotators and three LLMs, using majority voting for consensus-based annotations. The dataset initially achieves a similarity score of 83.1%, which is further improved by considering human factors that influence decision-making, with varying performance across different mental health categories, highlighting the need for targeted improvements in data collection and model architecture. Our approach addresses several key challenges in the field, including the lack of real-world training data, privacy concerns, and the need for diverse training datasets. This study serves as a foundation for future work in this critical area, which could lead to more effective and ethically sound AI-assisted mental health assessment tools.

Keywords

artificial intelligence / machine learning / large-language models / datasets / synthetic data

CHANGE HISTORY

A Correction to this paper has been published:
https://doi.org

The Retraction Note was updated to reflect reception of statements of disagreement from all authors.

This article has been retracted.Please see the Retraction Notice for more detail:
https://doi.org

References

[1] R. AlMakinah, A. Norcini-Pala, L. Disney, M. Abdullah Canbaz, Enhancing mental health support through human-ai collaboration: Toward secure and empathetic ai-enabled chatbots, Proceedings of IEEE Conference on Artificial Intelligence (CAI), 2025, 196-202. https://doi.org/10.1109/CAI64502.2025.00038

[2] A. Esteva, K. Chou, S. Yeung, N. Naik, A. Madani, A. Mottaghi, et al., Deep learning-enabled medical computer vision, NPJ Digit. Med., 4 (2021), 5. https://doi.org/10.1038/s41746-020-00376-2

[3] H. Ghanadian, I. Nejadgholi, H. Al Osman, Socially aware synthetic data generation for suicidal ideation detection using large language models, IEEE Access, 12 (2024), 14350-14363. https://doi.org/10.1109/ACCESS.2024.3358206

[4] A. Kang, J. Chen, Z. Lee-Youngzie, S. Fu, Synthetic data generation with llm for improved depression prediction, arXiv: 2411.17672. https://doi.org/10.48550/arXiv.2411.17672

[5] A. Khetan, Z. Lipton, A. Anandkumar, Learning from noisy singly-labeled data, arXiv: 1712.04577. https://doi.org/10.48550/arXiv.1712.04575

[6] N. Martinez-Martin, T. Insel, P. Dagum, H. Greely, M. Cho, Data mining for health: staking out the ethical territory of digital phenotyping, NPJ Digit. Med., 1 (2018), 68. https://doi.org/10.1038/s41746-018-0075-8

[7] T. Pitkämäki, T. Pahikkala, I. Perez, P. Movahedi, V. Nieminen, T. Southerington, et al., Finnish perspective on using synthetic health data to protect privacy: the PRIVASA project, Applied Computing and Intelligence, 4 (2024), 138-163. https://doi.org/10.3934/aci.2024009

[8] Prime Psychiatry, The great debate: is ai in mental health better at diagnosing mental illness than humans? Prime Psychiatry Office, 2024. Available from: https://primepsychiatrymd.com/blog/the-great-debate-is-ai-in-mental-health-better-at-diagnosing-mental-illness-than-humans/.

[9] LLM Radar, Exploring azure AI’s Phi-3.5-Mini-Instruct: the compact yet powerful LLM, Tal Peretz, 2024. Available from: https://blog.llmradar.ai/azure-ai-phi-3-5-mini-instruct/.

[10] M. Rollwage, J. Habicht, K. Juechems, B. Carrington, S. Viswanathan, M. Stylianou, et al., Using conversational AI to facilitate mental health assessments and improve clinical efficiency within psychotherapy services: real-world observational study, JMIR AI, 2 (2023), e44358. https://doi.org/10.2196/44358

[11] A. Sharma, I. Lin, A. Miner, D. Atkin, T. Althoff, Human-AI collaboration enables more empathic conversations in text-based peer-to-peer mental health support, Nat. Mach. Intell., 5 (2023), 46-57. https://doi.org/10.1038/s42256-022-00593-2

[12] J. Torous, S. Bucci, I. Bell, L. Kessing, M. Faurholt-Jepsen, P. Whelan, et al., The growing field of digital psychiatry: current evidence and the future of apps, social media, chatbots, and virtual reality, World Psychiatry, 20 (2021), 318-335. https://doi.org/10.1002/wps.20883