Multitype Game Optimisation: A Two-Stage Fine-Tuning Framework for Multi‑Game Optimisation With Large Language Models

Abstract

Large language models (LLMs) have made remarkable advances in natural language processing, demonstrating great potential in modelling structured sequences. However, adapting these capabilities to machine gaming tasks such as Go remains challenging due to limitations in strategy generalisation and optimisation efficiency. This paper presents multitype game optimisation (MyGO), a two-stage fine-tuning framework tailored for two-player perfect information board games, exploring the applicability of LLMs to nonlinguistic decision-making domains. In the supervised fine-tuning stage, we propose a unified structural encoding method, action semantic unit (ASU), which efficiently converts heterogeneous game records into discrete token sequences compatible with LLMs. In the reinforcement learning stage, we design TA-PPO (token-level adaptive proximal policy optimisation), an enhanced PPO-based algorithm to address the issue of sparse feedback commonly encountered in game reinforcement learning. Experimental results demonstrate that the fine-tuned models achieve superior or comparable performance to traditional game-playing algorithms in terms of strategy quality, rule generalisation and inference efficiency. This work provides a scalable paradigm for fine-tuning LLMs in complex decision-making tasks and lays a foundation for future research in game AI and generalisable strategy optimisation.

Keywords

computer game / fine-tuning / large language models / reinforcement learning / reward function

Funding

This work was supported in part by the National Natural Science Foundation of China under Grants 62276285 and 62236011.

Conflicts of Interest

The authors declare no conflicts of interest.

Data Availability Statement

The SGF data for the Go task used in this study are publicly available on GitHub at https://github.com/yenw/computer-go-dataset. The SGF data for the Tibetan Jiu chess task were obtained from internal resources of the research group and are not publicly available due to institutional restrictions.

References

[1] J. Achiam, S. Adler, S. Agarwal, et al., Gpt-4 Technical Report,” arXiv preprint arXiv:2303.08774 (2023).

[2] A. Liu, B. Feng, B. Xue, et al., Deepseek-v3 Technical Report,” arXiv preprint arXiv:2412.19437 (2024).

[3] K.-H. Lee, O. Nachum, M. S. Yang, et al., Multi-Game Decision Transformers,” Advances in Neural Information Processing Systems 35 (2022): 27921-27936, https://doi.org/10.48550/ARXIV.2205.15241.

[4] D. Noever, M. Ciolino, and J. Kalin, “The Chess Transformer: Mastering Play Using Generative Language Models,” arXiv preprint arXiv:2008.04057 (2020).

[5] M. Ciolino, J. Kalin, and D. Noever, “The Go Transformer: Natural Language Modeling for Game Play,” in 2020 Third International Conference on Artificial Intelligence for Industries (AI4I) (IEEE, 2020), 23-26.

[6] D. Silver, T. Hubert, J. Schrittwieser, et al., A General Reinforcement Learning Algorithm That Masters Chess, Shogi, and Go Through Self-Play,” Science 362, no. 6419 (2018): 1140-1144, https://doi.org/10.1126/science.aar6404.

[7] J. Schrittwieser, I. Antonoglou, T. Hubert, et al., Mastering Atari, Go, Chess and Shogi by Planning With a Learned Model,” Nature 588, no. 7839 (2020): 604-609, https://doi.org/10.1038/s41586-020-03051-4.

[8] R. Zheng, S. Dou, S. Gao, et al., Secrets of Rlhf in Large Language Models Part I: Ppo,” arXiv preprint arXiv:2307.04964 (2023).

[9] H. Zhong, Z. Shan, G. Feng, et al., Dpo Meets Ppo: Reinforced Token Optimization for Rlhf,” arXiv preprint arXiv:2404.18922 (2024).

[10] A. de Wynter, “Will gpt-4 Run Doom?,” IEEE Transactions on Games (2024).

[11] X. Li, B. Liu, Z. Wei, Z. Wang, and L. Wu, “Tjong: A Transformer-Based Mahjong Ai via Hierarchical Decision-Making and Fan Backward,” CAAI Transactions on Intelligence Technology 9, no. 4 (2024): 982-995, https://doi.org/10.1049/cit2.12298.

[12] S. E. M. Noever and D. Noever, “Word Play for Playing Othello (Reverses),” arXiv preprint arXiv:2207.08766 (2022).

[13] B. Albert Gramaje, “Exploring Gpt’s Capabilities in Chess-Puzzles,” 2023.

[14] X. Feng, Y. Luo, Z. Wang, et al., Chessgpt: Bridging Policy Learning and Language Modeling,” Advances in Neural Information Processing Systems 36 (2023): 7216-7262, https://doi.org/10.48550/ARXIV.2306.09200.

[15] L. Schmid, J. Maag, S. P. D. M. Cieliebak, and P. von Däniken, ““Optimizing Language Models for Chess: The Impact of Custom Notation and Elo-Based Fine-Tuning,” ZHAW Zürcher Hochschule Für Angewandte Wissenschaften,” Technical Reports Series (2024).

[16] A. L. Davis and G. Sukthankar, “Decoding Chess Mastery: A Mechanistic Analysis of a Chess Language Transformer Model,” in International Conference on Artificial General Intelligence (Springer, 2024), 63-72.

[17] Y. Zhang, X. Han, H. Li, K. Chen, and S. Lin, “Complete Chess Games Enable Llm Become a Chess Master,” arXiv preprint arXiv:2501.17186 (2025).

[18] H. Guo, Z. Liu, Y. Zhang, and Z. Wang, “Can Large Language Models Play Games? a Case Study of a Self-Play Approach,” arXiv preprint arXiv:2403.05632 (2024).

[19] B. Sun, R. Al-Bayaty, Q. Huang, and D. Wu, “Game Theoretical Approach for Non-Overlapping Community Detection,” Tsinghua Science and Technology 26, no. 5 (2021): 706-723, https://doi.org/10.26599/tst.2020.9010017.

[20] V. Mnih, K. Kavukcuoglu, D. Silver, et al., Playing Atari With Deep Reinforcement Learning,” arXiv preprint arXiv:1312.5602 (2013).

[21] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning With a Stochastic Actor,” in International Conference on Machine Learning (Pmlr, 2018), 1861-1870.

[22] Z. Zhang, S. Xu, Y. Xia, J. Wang, S. Y. Dring, and X. Cen, “Two Improved Algorithms Based on Dqn,” in 2023 IEEE International Conference on Signal Processing, Communications and Computing (ICSPCC) (IEEE, 2023), 1-4.

[23] Z. Liang, G. Li, R. Gu, Y. Wang, and G. Shan, “Sampleviz: Concept Based Sampling for Policy Refinement in Deep Reinforcement Learning,” in 2024 IEEE 17th Pacific Visualization Conference (PacificVis) (IEEE, 2024), 359-368.

[24] S. Chu, “Apsn: Adaptive Prediction Sample Network in Deep Q Learning,” in Third International Conference on Algorithms, Microchips, and Network Applications (AMNA 2024) (SPIE, 2024), 461-465.

[25] H. Itaya, T. Hirakawa, T. Yamashita, H. Fujiyoshi, and K. Sugiura, “Mask-Attention a3c: Visual Explanation of Action-State Value in Deep Reinforcement Learning,” IEEE Access 12 (2024): 86553-86571, https://doi.org/10.1109/access.2024.3416179.

[26] M. Hessel, J. Modayil, H. Van Hasselt, et al., Rainbow: Combining Improvements in Deep Reinforcement Learning,” in Proceedings of the AAAI Conference on Artificial Intelligence, (2018), https://doi.org/10.1609/aaai.v32i1.11796.

[27] V. Zambaldi, D. Raposo, A. Santoro, et al., Relational Deep Reinforcement Learning,” arXiv preprint arXiv:1806.01830 (2018).

[28] N. Brown and T. Sandholm, “Superhuman Ai for Heads-Up No-Limit Poker: Libratus Beats Top Professionals,” Science 359, no. 6374 (2018): 418-424, https://doi.org/10.1126/science.aao1733.

[29] N. Brown and T. Sandholm, “Safe and Nested Subgame Solving for Imperfect-Information Games,” Advances in Neural Information Processing Systems 30 (2017), https://doi.org/10.48550/arXiv.1705.02955.

[30] S. Gao and S. Li, “Bloody Mahjong Playing Strategy Based on the Integration of Deep Learning and Xgboost,” CAAI Transactions on Intelligence Technology 7, no. 1 (2022): 95-106, https://doi.org/10.1049/cit2.12031.

[31] O. Vinyals, I. Babuschkin, W. M. Czarnecki, et al., Grandmaster Level in Starcraft Ii Using Multi-Agent Reinforcement Learning,” Nature 575, no. 7782 (2019): 350-354, https://doi.org/10.1038/s41586-019-1724-z.

[32] D. Silver, A. Huang, C. J. Maddison, et al., Mastering the Game of Go With Deep Neural Networks and Tree Search,” Nature 529, no. 7587 (2016): 484-489, https://doi.org/10.1038/nature16961.

[33] D. Silver, J. Schrittwieser, K. Simonyan, et al., Mastering the Game of Go Without Human Knowledge,” Nature 550, no. 7676 (2017): 354-359, https://doi.org/10.1038/nature24270.

[34] Y. Tian, Q. Gong, W. Shang, Y. Wu, and C. L. Zitnick, “Elf: An Extensive, Lightweight and Flexible Research Platform for Real-Time Strategy Games,” Advances in Neural Information Processing Systems 30 (2017), https://doi.org/10.48550/arXiv.1707.01067.

[35] X. Li, Z. Lv, S. Wang, Z. Wei, and L. Wu, “A Reinforcement Learning Model Based on Temporal Difference Algorithm,” IEEE Access 7 (2019): 121922-121930, https://doi.org/10.1109/access.2019.2938240.

[36] X. Li, Z. Lv, L. Wu, Y. Zhao, and X. Xu, “Hybrid Online and Offline Reinforcement Learning for Tibetan Jiu Chess,” Complexity 2020, no. 1 (2020): 4708075, https://doi.org/10.1155/2020/4708075.

[37] S. Wang , Research and Implementation of Computer Game Algorithms for “Jiuqi”. Master’s thesis (Minzu University of China, 2023).

[38] X. Li, Y. Chen, Y. Zhang, B. Liu, and L. Wu, “A Phased Game Algorithm Combining Deep Reinforcement Learning and Uct for Tibetan Jiu Chess,” in 2023 IEEE 47th Annual Computers, Software, and Applications Conference (COMPSAC) (IEEE, 2023), 390-395.

[39] R. S. Sutton, D. McAllester, S. Singh, and Y. Mansour, “Policy Gradient Methods for Reinforcement Learning With Function Approximation,” Advances in Neural Information Processing Systems 12 (1999).

[40] L. Ouyang, J. Wu, X. Jiang, et al., Training Language Models to Follow Instructions With Human Feedback,” Advances in Neural Information Processing Systems 35 (2022): 27730-27744, https://doi.org/10.48550/arXiv.2203.02155.

[41] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal Policy Optimization Algorithms,” arXiv preprint arXiv:1707.06347 (2017).

[42] J. Liu, A. Cohen, R. Pasunuru, Y. Choi, H. Hajishirzi, and A. Celikyilmaz, “Making Ppo Even Better: Value-Guided Monte-Carlo Tree Search Decoding,” 2023.

[43] Z. Li, T. Xu, Y. Zhang, et al., Remax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models,” arXiv preprint arXiv:2310.10505 (2023).

[44] Y. Yuan, Y. Yue, R. Zhu, T. Fan, and L. Yan, “What’s Behind Ppo’s Collapse in Long-Cot? Value Optimization Holds the Secret,” arXiv preprint arXiv:2503.01491 (2025).

[45] G. Sheng, C. Zhang, Z. Ye, et al., Hybridflow: A Flexible and Efficient Rlhf Framework,” in Proceedings of the Twentieth European Conference on Computer Systems, 2025), 1279-1297.

[46] X. Hu, S. Zhang, T. Zhang, Y. Han, L. Wang, and Z. Geng, “Multiple Space Transfer Learning Based on Maximizing Mean Variance Differences for Soft Sensor Modeling,” Expert Systems with Applications 306 (2026): 130975, https://doi.org/10.1016/j.eswa.2025.130975.

[47] Z. Zhang, Y. Han, and Z. Geng, “Learning to Detect Industrial Time-Series Anomalies From Imputation Consistency With Sparse Observations,” IEEE Transactions on Neural Networks and Learning Systems 36, no. 9 (2025): 16648-16660, https://doi.org/10.1109/tnnls.2025.3568019.

[48] X. Zhang, X. Yang, F. Tu, X. Wang, M. Yan, and Y. Liang, “Optimization of Alpha-Beta Algorithm of Jiu Chess by Combining Empirical Knowledge and Deep Reinforcement Learning,” Journal of Chongqing University of Technology (Natural Science Edition) 38, no. 5 (2024): 115-120, https://doi.org/10.3969/j.issn.1674-8425(z).2024.05.015.

[49] G. Xu, X. Li, and Y. Zhang, “Online Gaming Platform for Tibetan Jiu Chess: Ai-Vs-Ai and Human-vs-Human,” in 2024 IEEE 48th Annual Computers, Software, and Applications Conference (COMPSAC) (IEEE, 2024), 1506-1507.