Credit-Driven Adaptive Grouping for Refined Cooperative Multi-Agent Reinforcement Learning

Yirui Liu , Xueqiang Gu , Caiming Zheng , Xin Xu , Zhangling Wang , Wanpeng Zhang , Yixing Lan

CAAI Transactions on Intelligence Technology ›› 2026, Vol. 11 ›› Issue (5) : 1496 -1510.

PDF (3124KB)
CAAI Transactions on Intelligence Technology ›› 2026, Vol. 11 ›› Issue (5) :1496 -1510. DOI: 10.1049/cit2.70127
ORIGINAL RESEARCH
research-article
Credit-Driven Adaptive Grouping for Refined Cooperative Multi-Agent Reinforcement Learning
Author information +
History +
PDF (3124KB)

Abstract

Policy heterogeneity is crucial for achieving sophisticated coordination in complex collaborative tasks, which has emerged as one of the key challenges in multi-agent reinforcement learning (MARL) in recent years. Notably, the grouping paradigm has made remarkable progress in addressing policy heterogeneity. However, most existing grouping methods require predefining the number of groups or the composition and quantity of members within each group, which need to be individually configured for each scenario and are difficult to set without sufficient expert knowledge. By contrast, we propose a novel MARL grouping algorithm named Credit-driven adaptive Grouping (CreateG) which divides the entire training process into multiple phases and reallocates poorly adapted (low-credit) individuals at each training stage. With the help of the mechanism we designed, an environment-adaptive grouping is ultimately formed. Furthermore, we design a hierarchical hypernetwork architecture to accommodate this adaptive grouping mechanism. Experiments conducted on StarCraft II micromanagement hard and super hard tasks, Google Research Football and TAG scenarios show CreateG achives state-of-the-art MARL performance. Moreover, extensive ablation studies elucidate the operational mechanism of the grouping strategy and other components demonstrate how they enhance overall performance.

Keywords

grouping-based MARL / multi-agent cooperation / multi-agent reinforcement learning / policy diversity

Cite this article

Download citation ▾
Yirui Liu, Xueqiang Gu, Caiming Zheng, Xin Xu, Zhangling Wang, Wanpeng Zhang, Yixing Lan. Credit-Driven Adaptive Grouping for Refined Cooperative Multi-Agent Reinforcement Learning. CAAI Transactions on Intelligence Technology, 2026, 11 (5) : 1496-1510 DOI:10.1049/cit2.70127

登录浏览全文

4963

注册一个新账户 忘记密码

References

[1]

Y. Ren, H. Zhang, L. Du, Z. Zhang, J. Zhang, and H. Li, “ Stealthy Black-Box Attack With Dynamic Threshold Against MARL-Based Traffic Signal Control System,” IEEE Transactions on Industrial Informatics 20, no. 10 (2024): 12021-12031, https://doi.org/10.1109/tii.2024.3413356.

[2]

T. Wu, P. Zhou, K. Liu, et al., “ Multi-Agent Deep Reinforcement Learning for Urban Traffic Light Control in Vehicular Networks,” IEEE Transactions on Vehicular Technology 69, no. 8 (2020): 8243-8256, https://doi.org/10.1109/TVT.2020.2997896.

[3]

S. Morad, A. Shankar, J. Blumenkamp, and A. Prorok, “ Language-Conditioned Offline RL for Multi-Robot Navigation,” in 2025 IEEE International Conference on Robotics and Automation (ICRA) (2025), 14984-14991, https://doi.org/10.1109/ICRA55743.2025.11127288.

[4]

Y. Yang, Li Juntao, and P. Lingling, “ Multi-Robot Path Planning Based on a Deep Reinforcement Learning DQN Algorithm,” CAAI Transactions on Intelligence Technology 5, no. 3 (2020): 177-183, https://doi.org/10.1049/trit.2020.0024.

[5]

Y. Li, C. Yu, M. Shahidehpour, T. Yang, Z. Zeng, and T. Chai, “ Deep Reinforcement Learning for Smart Grid Operations: Algorithms, Applications, and Prospects,” Proceedings of the IEEE 111, no. 9 (2023): 1055-1096, https://doi.org/10.1109/JPROC.2023.3303358.

[6]

J. Wu and X. Xu, “ Decentralised Grid Scheduling Approach Based on Multi-Agent Reinforcement Learning and Gossip Mechanism,” CAAI Transactions on Intelligence Technology 3, no. 1 (2018): 8-17, https://doi.org/10.1049/trit.2018.0001.

[7]

J. Foerster, I. A. Assael, N. de Freitas, and S. Whiteson, “ Learning to Communicate With Deep Multi-Agent Reinforcement Learning,” in Advances in Neural Information Processing Systems, ed. D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, Vol. 29 (Curran Associates, Inc, 2016), https://doi.org/10.48550/arXiv.1605.06676.

[8]

S. Xie, Y. Li, X. Wang, et al., “ Hierarchical Relationship Modeling in Multi-Agent Reinforcement Learning for Mixed Cooperative-Competitive Environments,” Information Fusion 108 (2024): 102318, https://doi.org/10.1016/j.inffus.2024.102318.

[9]

P. Sunehag, G. Lever, A. Gruslys, et al., “ Value-Decomposition Networks for Cooperative Multi-Agent Learning Based on Team Reward,” in Proceedings of the 17th International Conference on Autonomous Agents and Multiagent Systems (AAMAS, 2018), 2085-2087, https://doi.org/10.65109/jsrc7365.

[10]

T. Rashid, M. Samvelyan, C. Schroeder, G. Farquhar, J. Foerster, and S. Whiteson, “ QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning,” in Proceedings of the 35th International Conference on Machine Learning. Proceedings of Machine Learning Research, eds. J. Dy and A. Krause, Vol. 80 (PMLR, 2018), 4295-4304, https://doi.org/10.48550/arXiv.1803.11485.

[11]

K. Son, D. Kim, W. J. Kang, D. E. Hostallero, and Y. Yi, “ QTRAN: Learning to Factorize With Transformation for Cooperative Multi-Agent Reinforcement Learning,” in Proceedings of the 36th International Conference on Machine Learning. Proceedings of Machine Learning Research, eds. K. Chaudhuri and R. Salakhutdinov, Vol. 97 (PMLR, 2019), 5887-5896, https://doi.org/10.48550/arXiv.1905.05408.

[12]

C. Li, T. Wang, C. Wu, Q. Zhao, J. Yang, and C. Zhang, “ Celebrating Diversity in Shared Multi-Agent Reinforcement Learning,” in Advances in Neural Information Processing Systems, ed. M. Ranzato, A. Beygelzimer, Y. Dauphin, P. S. Liang, and J. Wortman Vaughan, Vol. 34 (Curran Associates, Inc., 2021), 3991-4002, https://doi.org/10.48550/arXiv.2106.02195.

[13]

G. Papoudakis, F. Christianos, L. Schäfer, and S. V. Albrecht, “ Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Tasks,” in Thirty-Fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1) (2021), https://doi.org/10.48550/arXiv.2006.07869.

[14]

Y. Yu, Q. Yin, J. Zhang, P. Xu, and K. Huang, “ ADMN: Agent-Driven Modular Network for Dynamic Parameter Sharing in Cooperative Multi-Agent Reinforcement Learning,” in Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI-24. International Joint Conferences on Artificial Intelligence Organization, ed. K. L. Main Track (2024), 302-310, https://doi.org/10.24963/ijcai.2024/34.

[15]

J. Zhao, Y. Zhang, X. Hu, et al., “ Revisiting Qmix: Discriminative Credit Assignment by Gradient Entropy Regularization,” preprint, arXiv:2202.04427, February 09, 2022, https://doi.org/10.48550/arXiv.2202.04427.

[16]

Jiang, J. and Z. Lu, “ The Emergence of Individuality,” in Proceedings of the 38th International Conference on Machine Learning. Proceedings of Machine Learning Research. PMLR, eds. M. Meila and T. Zhang. Vol. 139 (2021), 4992-5001, https://doi.org/10.48550/arXiv.2006.05842.

[17]

A. Mahajan, T. Rashid, M. Samvelyan, and S. Whiteson, “ MAVEN: Multi-Agent Variational Exploration,” in Advances in Neural Information Processing Systems, ed. H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, Vol. 32 (Curran Associates, Inc, 2019), https://doi.org/10.48550/arXiv.1910.07483.

[18]

S. Liu, Y. Zhou, J. Song, et al., “ Contrastive Identity-Aware Learning for Multi-Agent Value Decomposition,” in Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37, no. 10 (2023), 11595-11603, https://doi.org/10.1609/aaai.v37i10.26370.

[19]

S. Li, J. K. Gupta, P. Morales, R. Allen, and M. J. Kochenderfer, “ Deep Implicit Coordination Graphs for Multi-Agent Reinforcement Learning,” in Proceedings of the 20th International Conference on Autonomous Agents and Multiagent Systems. AAMAS ’21. Virtual Event, United Kingdom (International Foundation for Autonomous Agents and Multiagent Systems, 2021), 764-772: ISBN: 9781450383073, https://doi.org/10.65109/czoy2835.

[20]

W. Duan, J. Lu, Yu G. Wang, et al., “ Layer-Diverse Negative Sampling for Graph Neural Networks,” in Transactions on Machine Learning Research (2024): ISSN: 2835-8856, https://doi.org/10.48550/arXiv.2403.11408.

[21]

Y. Zang, J. He, K. Li, et al., “ Automatic Grouping for Efficient Cooperative Multi-Agent Reinforcement Learning,” Advances in Neural Information Processing Systems 36 (2023): 46105-46121, https://doi.org/10.52202/075280-1999.

[22]

H. Na, K. Lee, S. Lee, and I.-c. Moon, “ Trajectory-Class-Aware Multi-Agent Reinforcement Learning,” in The Thirteenth International Conference on Learning Representations (2025), https://doi.org/10.48550/arXiv.2503.01440.

[23]

T. Wang, T. Gupta, A. Mahajan, B. Peng, S. Whiteson, and C. Zhang, “ {RODE}: Learning Roles to Decompose Multi-Agent Tasks,” in International Conference on Learning Representations (2021), https://doi.org/10.48550/arXiv.2010.01523.

[24]

Z. Hu, Z. Zhang, H. Li, C. Chen, H. Ding, and Z. Wang, “ Attention-Guided Contrastive Role Representations for Multi-Agent Reinforcement Learning,” in The Twelfth International Conference on Learning Representations (2024), https://doi.org/10.48550/arXiv.2312.04819.

[25]

T. Phan, F. Ritz, L. Belzner, P. Altmann, T. Gabor, and C. Linnhoff-Popien, “ VAST: Value Function Factorization With Variable Agent Sub-Teams,” in Advances in Neural Information Processing Systems, ed. A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (2021), https://dl.acm.org/doi/10.5555/3540261.3542100.

[26]

J. Shao, Z. Lou, H. Zhang, Y. Jiang, S. He, and X. Ji, “ Self-Organized Group for Cooperative Multi-Agent Reinforcement Learning,” in Advances in Neural Information Processing Systems, ed. S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, Vol. 35 (Curran Associates, Inc., 2022), 5711-5723, https://doi.org/10.52202/068431-0413.

[27]

K. Kurach, A. Raichuk, P. Stańczyk, et al., “ Google Research Football: A Novel Reinforcement Learning Environment,” in Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34, no. 4 (2020), 4501-4510, https://doi.org/10.1609/aaai.v34i04.5878.

[28]

I. Mordatch and P. Abbeel, “ Emergence of Grounded Compositional Language in Multi-Agent Populations,” in Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 32, no. 1 (2018), https://doi.org/10.1609/aaai.v32i1.11492.

[29]

B. Liao, C. Hua, Q. Xu, X. Cao, and S. Li, “ Inter-Robot Management via Neighboring Robot Sensing and Measurement Using a Zeroing Neural Dynamics Approach,” Expert Systems With Applications 244 (2024): 122938, https://doi.org/10.1016/j.eswa.2023.122938.

[30]

M. Liu, Y. Li, Y. Chen, Y. Qi, and L. Jin, “ A Distributed Competitive and Collaborative Coordination for Multirobot Systems,” IEEE Transactions on Mobile Computing 23, no. 12 (2024): 11436-11448, https://doi.org/10.1109/tmc.2024.3397242.

[31]

C. Hua, J. Xu, Z. Huang, B. Liao, and S. Li, “ Optimization-Based Finite-Time Multi-Robot Formation: A Zeroing Neurodynamics Method,” Tsinghua Science and Technology 31, no. 1 (2026): 162-179, https://doi.org/10.26599/tst.2024.9010180.

[32]

T. Wang, C. Hua, Y. Wang, X. Cao, B. Liao, and S. Li, “ Real-Time Formation Planning for Multirobot Cooperation: A Neural Informatics Perspective,” IEEE Transactions on Industrial Electronics 72, no. 12 (2025): 13703-13715, https://doi.org/10.1109/TIE.2025.3579075.

[33]

T. Wang, H. Dong, V. Lesser, and C. Zhang, “ ROMA: Multi-Agent Reinforcement Learning With Emergent Roles,” in Proceedings of the 37th International Conference on Machine Learning. Proceedings of Machine Learning Research, Vol. 119 (PMLR, 2020), 9876-9886: Hal Daumé III and Aarti Singh, https://doi.org/10.48550/arXiv.2003.08039.

[34]

S. Fu, S. Zhao, T. Li, and Y. Yan, “ QTypeMix: Enhancing Multi-Agent Cooperative Strategies Through Heterogeneous and Homogeneous Value Decomposition,” Neural Networks 184 (2025): 107093, https://doi.org/10.1016/j.neunet.2024.107093.

[35]

Liu, C. and D. Li (2025), “HYGMA: Hypergraph Coordination Networks With Dynamic Grouping for Multi-Agent Reinforcement Learning,” preprint, arXiv:2505.07207, May 12, 2025, https://doi.org/10.48550/arXiv.2505.07207.

[36]

F. A. Oliehoek and C. Amato, A Concise Introduction to Decentralized POMDPs . Springer Briefs in Intelligent Systems (Springer, 2016), https://doi.org/10.1007/978-3-319-28929-8.

[37]

H. van Hasselt, A. Guez, and D. Silver, “ Deep Reinforcement Learning With Double Q-Learning,” in Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 30, 2016), 1, https://doi.org/10.48550/arXiv.1509.06461.

[38]

M. Samvelyan, T. Rashid, C. S. de Witt, et al., “ The StarCraft Multi-Agent Challenge,” in Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems (2019), https://doi.org/10.65109/lvzz5205.

[39]

J. Hu, S. Wang, S. Jiang, and M. Wang, “ Rethinking the Implementation Tricks and Monotonicity Constraint in Cooperative Multi-agent Reinforcement Learning,” in The Second Blogpost Track at ICLR 2023 (2023), https://doi.org/10.48550/arXiv.2102.03479.

[40]

J. Wang, Z. Ren, T. Liu, Y. Yu, and C. Zhang, “ {QPLEX}: Duplex Dueling Multi-Agent Q-Learning,” in International Conference on Learning Representations (2021), https://doi.org/10.48550/arXiv.2008.01062.

[41]

B. Ellis, J. Cook, S. Moalla, et al., “ SMACv2: An Improved Benchmark for Cooperative Multi-Agent Reinforcement Learning,” in Advances in Neural Information Processing Systems, ed. A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, Vol. 36 (Curran Associates, Inc., 2023), 37567-37593, https://doi.org/10.52202/075280-1634.

[42]

H. Zhang, H. Yu, X. Wang, et al., “ Knowledge-Guided Communication Preference Learning Model for multi-agent Cooperation,” Information Sciences 667 (2024): 120395, https://doi.org/10.1016/j.ins.2024.120395.

[43]

Z. Xie, C. Ji, C. Qiao, et al., “ Mutual Information Oriented Deep Skill Chaining for multi-agent Reinforcement Learning,” CAAI Transactions on Intelligence Technology 9, no. 4 (2024): 1014-1030, https://doi.org/10.1049/cit2.12322.

[44]

W. Duan, J. Lu, and J. Xuan, “ Group-Aware Coordination Graph for Multi-Agent Reinforcement Learning,” in Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI-24, ed. K. L. Main Track (International Joint Conferences on Artificial Intelligence Organization, 2024), 3926-3934, https://doi.org/10.24963/ijcai.2024/434.

PDF (3124KB)

0

Accesses

0

Citation

Detail

Sections
Recommended

/

〈 〉