Hierarchy-aware strategy recommendation for emotional support dialogues via hyperbolic representation and conservative offline learning

Abstract

Aim: Effective emotional support requires selecting support strategies appropriate to the seeker’s state and grounded in counselling theory. We present a computational proof-of-concept that combines hyperbolic representation learning with conservative offline reinforcement learning for support-strategy recommendation.

Methods: Support strategies from Hill’s Helping Skills framework are embedded in the Poincaré ball to encode their three-level hierarchy. On this representation, we train Conservative Hyperbolic Implicit Q-Learning (CH-IQL) on the Emotional Support Conversation dataset (ESConv; 1,300 dialogues; 910/195/195 train/validation/test split). All results are reported as mean ± standard deviation over five training seeds. Policy value is measured by independent Fitted Q-Evaluation (FQE) with twin Q-networks, calibrated against the behavior policy’s Monte-Carlo ground truth and an analytic value ceiling, with dialogue-level bootstrap confidence intervals. Strategy appropriateness is additionally assessed by four blinded human raters over 120 stratified test states.

Results: Under the stochastic (sampling) deployment, CH-IQL attains an FQE-estimated value of 4.79 ± 0.07 versus 3.66 for the estimated behavior policy—positive under every seed and under two alternative, feedback-anchored reward designs—while its strategy distribution matches the entropy of human supporters (1.98 vs. 1.99) and uses all eight strategies. In blinded pairwise comparison, raters preferred the model’s strategy in 43.9% of disputed cases (95% CI 32.6–55.9%), statistically indistinguishable from the observed supporter behavior. The heuristic reward shows no dialogue-level correlation with seeker-reported outcomes (Spearman ρ = 0.009), which we disclose and address through reward-robustness experiments.

Conclusions: The framework improves the specified reward while remaining comparable to observed supporter behavior under expert judgment. These results support offline reinforcement learning with hierarchical representations as a viable strategy-recommendation mechanism, and identify outcome-correlated reward design as the key requirement for progress toward real-world utility. The system is a computational proof-of-concept and is not suitable for clinical deployment.

Keywords

emotional support dialogue systems / hyperbolic embeddings / offline reinforcement learning / strategy recommendation / Helping Skills Theory / Poincaré ball / off-policy evaluation / human assessment

References

[1] Mental health [Internet]. WHO; c2026 [cited 2026 Jul 27]. Available from:https://www.who.int/news—room/fact—sheets/detail/mental—health—strengthening—our—response

[2] State of Mental Health in America 2023 [Internet]. Mental Health America; c2026 [cited 2026 Jul 27]. Available from:https://mhanational.org/research/state—of—mental—health—in—america—2023/

[3] Patel V, Saxena S, Lund C, Thornicroft G, Baingana F, Bolton P, et al. The Lancet Commission on global mental health and sustainable development. Lancet. 2018; 392:1553-98.

[4] Abd—Alrazaq AA, Alajlani M, Ali N, Denecke K, Bewick BM, Househ M. Perceptions and Opinions of Patients About Mental Health Chatbots: Scoping Review. J Med Internet Res. 2021; 23:e17828.

[5] Vaidyam AN, Wisniewski H, Halamka JD, Keshavan MS, Torous JB. Chatbots and Conversational Agents in Mental Health: A Review of the Psychiatric Landscape. Can J Psychiatry. 2019;64:456-64.

[6] Inkster B, Sarda S, Subramanian V. An Empathy—Driven, Conversational Artificial Intelligence Agent (Wysa) for Digital Mental Well—Being: Real—World Data Evaluation Mixed—Methods Study. JMIR mHealth uHealth. 2018; 6:e12106.

[7] Fitzpatrick KK, Darcy A, Vierhile M. Delivering Cognitive Behavior Therapy to Young Adults With Symptoms of Depression and Anxiety Using a Fully Automated Conversational Agent (Woebot): A Randomized Controlled Trial. JMIR Ment Health. 2017; 4:e19.

[8] Grodniewicz JP, Hohol M. Waiting for a digital therapist: three challenges on the path to psychotherapy delivered by artificial intelligence. Front Psychiatry. 2023; 14:1190084.

[9] Sedlakova J, Trachsel M. Conversational Artificial Intelligence in Psychotherapy: A New Therapeutic Tool or Agent? Am J Bioeth. 2023; 23:4-13.

[10] Hill CE. Helping Skills: Facilitating Exploration, Insight, and Action. 4th ed. Washington, DC: American Psychological Association; 2014.

[11] Liu S, Zheng C, Demasi O, Sabour S, Li Y, Yu Z, et al. Towards emotional support dialog systems. In: Proceedings of ACL. 2021. pp. 3469-83.

[12] Nickel M, Kiela D. Poincaré embeddings for learning hierarchical representations. In: NIPS'17: Proceedings of the 31st International Conference on Neural Information Processing Systems. 2017. pp. 6341—50.

[13] Levine S, Kumar A, Tucker G, Fu J. Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems. arXiv:2005.01643 [Preprint].2020 [cited 2026 Jul 27]. Available from:https://arxiv.org/abs/2005.01643

[14] Kostrikov I, Nair A, Levine S. Offline Reinforcement Learning with Implicit Q—Learning. arXiv:2110.06169 [Preprint]. 2021 [cited 2026 Jul 27]. Available from:https://arxiv.org/abs/2110.06169

[15] Kumar A, Zhou A, Tucker G, Levine S. Conservative Q—Learning for Offline Reinforcement Learning. arXiv:2006.04779 [Preprint]. 2020 [cited 2026 Jul 27]. Available from:https://arxiv.org/abs/2006.04779

[16] Rashkin H, Smith EM, Li M, Boureau YL. Towards Empathetic Open—domain Conversation Models: A New Benchmark and Dataset. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics; 2019. pp. 5370—81.

[17] Lin Z, Madotto A, Shin J, Xu P, Fung P. MoEL: Mixture of Empathetic Listeners. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP—IJCNLP). Association for Computational Linguistics; 2019. pp. 121-32.

[18] Cheng Y, Liu W, Li W, Wang J, Zhao R, Liu B, et al. Improving Multi—turn Emotional Support Dialogue Generation with Lookahead Strategy Planning. In: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics; 2022. pp. 3014—26.

[19] Tu Q, Li Y, Cui J, Wang B, Wen JR, Yan R. MISC: A Mixed Strategy—Aware Model integrating COMET for Emotional Support Conversation. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics; 2022. pp. 308-19.

[20] Peng W, Hu Y, Xing L, Xie Y, Sun Y, Li Y. Control Globally, Understand Locally: A Global—to—Local Hierarchical Graph Network for Emotional Support Conversation. Proc Thirty—First Int Jt Conf Artif Intell. 2022:4324—30.

[21] Zheng C, Sabour S, Wen J, Zhang Z, Huang M. AugESC: Dialogue Augmentation with Large Language Models for Emotional Support Conversation. arXiv:2202.13047 [Preprint].2023 [cited 2026 Jul 27]. Available from:https://arxiv.org/abs/2202.13047

[22] Kang D, Kim S, Kwon T, Moon S, Cho H, Yu Y, et al. Can Large Language Models be Good Emotional Supporter? Mitigating Preference Bias on Emotional Support Conversation. In: Proceedings of ACL. 2024. pp. 15232-61.

[23] Nickel M, Kiela D. Learning continuous hierarchies in the Lorentz model of hyperbolic geometry. arXiv:1806.03417 [Preprint]. 2018 [cited 2026 Jul 27]. Available from:https://arxiv.org/abs/1806.03417

[24] Chami I, Ying Z, Ré C, Leskovec J. Hyperbolic graph convolutional neural networks. arXiv:1910.12933 [Preprint]. 2019 [cited 2026 Jul 27]. Available from:https://arxiv.org/abs/1910.12933

[25] Tifrea A, Bécigneul G, Ganea OE. Poincaré GloVe: Hyperbolic word embeddings. arXiv:1810.06546 [Preprint].2019 [cited 2026 Jul 27]. Available from:https://arxiv.org/abs/1810.06546

[26] Khrulkov V, Mirvakhabova L, Ustinova E, Oseledets I, Lempitsky V. Hyperbolic Image Embeddings. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2020. pp. 6417—27.

[27] Fujimoto S, Meger D, Precup D. Off—policy deep reinforcement learning without exploration. In: International Conference on Machine Learning. 2019. pp. 2052-62.

[28] Wu Y, Tucker G, Nachum O. Behavior regularized offline reinforcement learning. arXiv:1911.11361 [Preprint]. 2019 [cited 2026 Jul 27]. Available from:https://arxiv.org/abs/1911.11361

[29] Raghu A, Komorowski M, Ahmed I, Celi L, Szolovits P, Ghassemi M. Deep Reinforcement Learning for Sepsis Treatment. arXiv:1711.09602 [Preprint]. 2017 [cited 2026 Jul 27]. Available from:https://arxiv.org/abs/1711.09602

[30] Komorowski M, Celi LA, Badawi O, Gordon AC, Faisal AA. The Artificial Intelligence Clinician learns optimal treatment strategies for sepsis in intensive care. Nat Med. 2018; 24:1716—20.

[31] Fox I, Lee J, Pop—Busui R, Wiens J. Deep reinforcement learning for closed—loop blood glucose control. In: Machine Learning for Healthcare Conference. 2020. pp. 508-36.

[32] Snell C, Kostrikov I, Su Y, Yang M, Levine S. Offline RL for Natural Language Generation with Implicit Language Q Learning. arXiv:2206.11871 [Preprint]. 2023 [cited 2026 Jul 27]. Available from:https://arxiv.org/abs/2206.11871

[33] Le HM, Voloshin C, Yue Y. Batch policy learning under constraints. In: Proceedings of the 36th International Conference on Machine Learning (ICML). 2019. pp. 3703—12.

[34] Precup D, Sutton RS, Singh SP. Eligibility traces for off—policy policy evaluation. In: Proceedings of the 17th International Conference on Machine Learning (ICML). 2000. pp. 759-66.

[35] GLM Team. ChatGLM: A Family of Large Language Models from GLM—130B to GLM—4 All Tools. arXiv:2406.12793 [Preprint]. 2024 [cited 2026 Jul 27]. Available from:https://arxiv.org/abs/2406.12793

[36] Fleiss JL. Measuring nominal scale agreement among many raters. Psychol Bull. 1971; 76:378-82.

[37] Krippendorff K. Content Analysis: An Introduction to Its Methodology. 2nd ed. Thousand Oaks, CA: Sage; 2004.