A Formal Framework for Opponent Shaping in Two-player Mixed-Motive Markov Games
Chunjiang Mu , Hao Li , Juan Shi , Shuyue Hu , Chen Chu , Zhen Wang
Recent studies have shown that in mixed-motive Markov games, one player can unilaterally influence another player’s policy through opponent shaping. However, existing opponent shaping methods are limited in shaping objectives, typically focusing on maximizing exploitation of naive learners or sustaining cooperation in self-play between two opponent-shaping players. Moreover, existing methods necessitate training a dedicated shaper for each specific shaping objective. In this work, we propose a formal framework for opponent shaping in two-player mixed-motive Markov games that enables the flexible shaping of the opponent’s behavior. This is achieved by introducing a controller that unilaterally influences the opponent’s long-run payoff through its own meta-action choices, inspired by zero-determinant (ZD) strategies in game theory. Our framework is built upon the concept of a repeated meta-game, constructed via empirical game-theoretic analysis (EGTA). In a repeated meta-game, the controller employs ZD strategies by adaptively switching between pre-trained policies, which determine the controller’s actions in the original Markov game for the subsequent steps. We establish a theoretical foundation for this construction by proving that the meta-payoff matrix is well-defined under an ergodicity condition on the induced Markov chain, and provide supporting evidence for this condition in our experimental environments. Through experiments on two mixed-motive Markov games, we demonstrate that our framework is capable of unilaterally determining the controlled player’s long-run average per-step reward, or enforcing a linear reward relationship between the controller and the controlled player. By specifying the control objectives of the ZD strategy, our framework can shape the controlled player into cooperation by punishing non-cooperative behavior, a capability that prior opponent shaping methods struggle to deliver. We further conduct robustness analyses showing that the probabilistic opponent estimation is essential for effective control, and identify the framework’s boundary: payoff control fails when the opponent’s objective transforms the interaction into a competitive game.
Multi-agent reinforcement learning / opponent shaping / game theory; payoff control
Higher Education Press 2026
/
| 〈 |
|
〉 |
PDF
(2745KB)
