AGT: Efficient Offline Reinforcement Learning With Advantage-Guided Transformer

Jiaye Wei , Xin Xu , Yixing Lan , Tenglong Liu , Yueying Wang

CAAI Transactions on Intelligence Technology ›› 2026, Vol. 11 ›› Issue (4) : 1127 -1143.

PDF (1946KB)
CAAI Transactions on Intelligence Technology ›› 2026, Vol. 11 ›› Issue (4) :1127 -1143. DOI: 10.1049/cit2.70094
ORIGINAL RESEARCH
research-article
AGT: Efficient Offline Reinforcement Learning With Advantage-Guided Transformer
Author information +
History +
PDF (1946KB)

Abstract

Offline reinforcement learning (RL) is a paradigm that seeks to train policies directly based on fixed datasets derived from previous interactions with the environment. However, offline RL faces critical challenges in environments characterised by sparse rewards and datasets dominated by suboptimal trajectories. In this article, we propose a simple and efficient offline RL framework via advantage-guided transformer (AGT). Different from previous works, AGT promotes an advantage-guided policy constraint into the loss function during policy training that adaptively learns the features of trajectories with high advantage values. Furthermore, AGT samples actions with high advantage values during action generation, replacing manual designs with advantage values to update the return-to-go (RTG) as conditional inputs, to ensure action consistency with expected returns. Theoretical analysis shows that AGT's policy improvement guarantees convergence to near-optimal policies in environments with low stochasticity. Extensive experiments on D4RL benchmarks, including robotic locomotion control, dexterous manipulation and maze navigation tasks, show AGT outperforms state-of-the-art (SOTA) offline RL methods.

Keywords

artificial intelligence / artificial neural network / learning (artificial intelligence) / machine learning

Cite this article

Download citation ▾
Jiaye Wei, Xin Xu, Yixing Lan, Tenglong Liu, Yueying Wang. AGT: Efficient Offline Reinforcement Learning With Advantage-Guided Transformer. CAAI Transactions on Intelligence Technology, 2026, 11 (4) : 1127-1143 DOI:10.1049/cit2.70094

登录浏览全文

4963

注册一个新账户 忘记密码

References

[1]

F. Hêche, O. Barakat, T. Desmettre, T. Marx, and S. Robert-Nicoud, “ Offline Reinforcement Learning in High-Dimensional Stochastic Environments,” Neural Computing & Applications 36, no. 2 (2024): 585-598, https://doi.org/10.1007/s00521-023-09029-3.

[2]

R. F. Prudencio, M. R. O. A. Máximo, and E. L. Colombini, “ A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems,” IEEE Transactions on Neural Networks and Learning Systems 35, no. 8 (2024): 10237-10257, https://doi.org/10.1109/TNNLS.2023.3250269.

[3]

S. Lamsiyah, A. E. Mahdaouy, A. Nourbakhsh, and C. Schommer, “ Fine-Tuning a Large Language Model With Reinforcement Learning for Educational Question Generation,” in Artificial Intelligence in Education-25th International Conference, AIED 2024, Recife, Brazil, July 8-12, 2024, Proceedings, Part I, Vol. 14829 of Lecture Notes in Computer Science (Springer, 2024), 424-438.

[4]

M. M. Islam, M. K. Hasan, S. Islam, et al., “ Enabling Pandemic-Resilient Healthcare: Narrowband Internet of Things and Edge Intelligence for Real-Time Monitoring,” CAAI Transactions on Intelligence Technology (2024): cit2.12314, https://doi.org/10.1049/cit2.12314.

[5]

H. Lin, W. Ding, Z. Liu, et al., “ Safety-Aware Causal Representation for Trustworthy Offline Reinforcement Learning in Autonomous Driving,” IEEE Robotics and Automation Letters 9, no. 5 (2024): 4639-4646, https://doi.org/10.1109/lra.2024.3379805.

[6]

J. Zhang, K. Wang, and C. Mu, “ Multi-Station multi-robot Task Assignment Method Based on Deep Reinforcement Learning,” CAAI Transactions on Intelligence Technology 10, no. 1 (2025): 134-146, https://doi.org/10.1049/cit2.12394.

[7]

Y. Wang, C. Yang, Y. Wen, Y. Liu, and Y. Qiao, “ Critic-Guided Decision Transformer for Offline Reinforcement Learning,” in Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2014, February 20-27, 2024, Vancouver, Canada (AAAI Press, 2024), 15706-15714.

[8]

L. Chen, K. Lu, A. Rajeswaran, et al., “ Decision Transformer: Reinforcement Learning via Sequence Modeling,” in Advances in Neural Information Processing Systems, Vol. 34 (Curran Associates Inc., 2021), 15084-15097.

[9]

Z.-M. M. Y. L. Chongchong Li, Y. Wang, Z. Ma, and Y. Liu, “ A Criterion for Selecting the Appropriate One From the Trained Models for Model-Based Offline Policy Evaluation,” CAAI Transactions on Intelligence Technology 10, no. 1 (2025): 223-234, https://doi.org/10.1049/cit2.12376.

[10]

A. Kumar, A. Zhou, G. Tucker, and S. Levine, “ Conservative q-Learning for Offline Reinforcement Learning,” in Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, Virtual (2020), 1179-1191.

[11]

I. Kostrikov, A. Nair, and S. Levine, “ Offline Reinforcement Learning With Implicit q-Learning,” in The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 (OpenReview.net, 2022).

[12]

S. Fujimoto and S. S. Gu, “ A Minimalist Approach to Offline Reinforcement Learning,” in Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, Virtual (2021), 20132-20145.

[13]

A. Kumar, J. Fu, M. Soh, G. Tucker, and S. Levine, “ Stabilizing Off-Policy q-Learning via Bootstrapping Error Reduction,” in Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada (2019), 11761-11771.

[14]

S. Fujimoto, D. Meger, and D. Precup, “ Off-Policy Deep Reinforcement Learning Without Exploration,” in Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, eds. K. Chaudhuri and R. Salakhutdinov, Vol. 97 Proceedings of Machine Learning Research, (PMLR, 2019), 2052-2062.

[15]

X. Chen, A. Ghadirzadeh, T. Yu, et al., “ Latent-Variable Advantage-Weighted Policy Optimization for Offline RL,” CoRR abs/2203 (2022): 08949, https://doi/org/10.48550/arXiv.2203.08949.

[16]

K. Sohn, H. Lee, and X. Yan, “ Learning Structured Output Representation Using Deep Conditional Generative Models,” in Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada (2015), 3483-3491.

[17]

A. Vaswani, N. Shazeer, N. Parmar, et al., “ Attention Is all You Need,” in Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA (2017), 5998-6008.

[18]

L. Wei, H. Huang, and X. Yu, “ Intersection Over Union Similarity Based Non-Maximum Suppression for Human Pose Estimation in Crowded Scenes,” IEEE Transactions on Cognitive and Developmental Systems (2023): 1, https://doi/org/10.1109/TCDS.2023.3276372.

[19]

K. Paster, S. A. McIlraith, and J. Ba, “ You Can’t Count on Luck: Why Decision Transformers and Rvs Fail in Stochastic Environments,” in Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28-December 9, 2022 (2022), 38966-38979.

[20]

S. Yang, D. Schuurmans, P. Abbeel, and O. Nachum, “ Dichotomy of Control: Separating What You Can Control From What You Cannot,” in The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 (OpenReview.net, 2023).

[21]

Y. Wu, X. Wang, and M. Hamaya, “ Elastic Decision Transformer,” in Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10-16, 2023 (2023), 18532-18550.

[22]

H. Jung, Y. Kim, and Y. Kim, “ Maneuver-Conditioned Decision Transformer for Tactical in-Flight Decision-Making,” IEEE Robotics and Automation Letters 9, no. 6 (2024): 5322-5329, https://doi.org/10.1109/lra.2024.3391691.

[23]

S. Orfanoudakis, N. K. Panda, P. Palensky, and P. P. Vergara, “ GNN-DT: Graph Neural Network Enhanced Decision Transformer for Efficient Optimization in Dynamic Environments,” CoRR abs/2502.01778 (2025), https://doi/org/10.48550/arXiv.2502.01778.

[24]

Z. Wang, X. Qu, J. Xiao, B. Chen, and J. Wang, “ P2DT: Mitigating Forgetting in Task-Incremental Learning With Progressive Prompt Decision Transformer,” in IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2024, Seoul, Republic of Korea, April 14-19, 2024 (IEEE, 2024), 7265-7269.

[25]

M. H. Modirrousta, P. F. Arani, R. Kazemi, and M. A. Shoorehdeli, “ Analysis of Anomalous Behaviour in Network Systems Using Deep Reinforcement Learning With Convolutional Neural Network Architecture,” CAAI Transactions on Intelligence Technology 9, no. 6 (2024): 1467-1484, https://doi/org/10.1049/CIT2.12359.

[26]

T. Yamagata, A. Khalil, and R. Santos-Rodríguez, “ Q-Learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RL,” in International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, Vol. 202 of Proceedings of Machine Learning Research (PMLR, 2023), 38989-39007.

[27]

S. Hu, Z. Fan, C. Huang, et al., “ Q-Value Regularized Transformer for Offline Reinforcement Learning,” in Forty-First International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 (OpenReview.net, 2024).

[28]

R. J. A. A. L. Atkinson, A. H. Waterman, and R. J. Allen, “ Does Value-Based Prioritization at Working Memory Enhance Long-Term Memory?,” Memory & Cognition 52, no. 8 (2024): 1-16, https://doi.org/10.3758/s13421-024-01532-9.

[29]

J. Schulman, P. Moritz, S. Levine, M. I. Jordan, and P. Abbeel, “ High-Dimensional Continuous Control Using Generalized Advantage Estimation,” in 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings (2016).

[30]

S. Wei, X. Wang, X. Feng, and H. Yu, “ Mast: Multi-Agent Safe Transformer for Reinforcement Learning,” IEEE Transactions on Cognitive and Developmental Systems 17, no. 4 (2025): 1-10, https://doi.org/10.1109/tcds.2025.3533744.

[31]

C. Robertson, N. A. Tong, T. T. Nguyen, Q. V. Hung Nguyen, and J. Jo, “ Resource-Adaptive and OOD-Robust Inference of Deep Neural Networks on Iot Devices,” CAAI Transactions on Intelligence Technology 10, no. 1 (2025): 115-133, https://doi.org/10.1049/cit2.12384.

[32]

I. Kostrikov, R. Fergus, J. Tompson, and O. Nachum, “ Offline Reinforcement Learning With Fisher Divergence Critic Regularization,” in Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, Vol. 139 Proceedings of Machine Learning Research (PMLR, 2021). 5774-5783.

[33]

Y. Wu, G. Tucker, and O. Nachum, “ Behavior Regularized Offline Reinforcement Learning,” CoRR abs/1911.11361 (2019), https://doi.org/10.48550/arXiv.1911.11361.

[34]

A. Singh, A. Kumar, Q. Vuong, Y. Chebotar, and S. Levine, “ Reds: Offline RL With Heteroskedastic Datasets via Support Constraints,” in Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10-16, 2023 (2023), 23921-23933.

[35]

Z. Hong, P. Agrawal, R. T. des Combes, and R. Laroche, “ Harnessing Mixed Offline Reinforcement Learning Datasets via Trajectory Weighting,” in The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 (OpenReview.net, 2023).

[36]

Y. Yue, B. Kang, X. Ma, G. Huang, S. Song, and S. Yan, “ Offline Prioritized Experience Replay,” CoRR abs/2306.05412 (2023), https://doi.org/10.48550/arXiv.2306.05412.

[37]

T. Liu, Y. Li, Y. Lan, H. Gao, W. Pan, and X. Xu, “ Adaptive Advantage-Guided Policy Regularization for Offline Reinforcement Learning,” in Forty-First International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 (OpenReview.net, 2024).

[38]

J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine, “ D4RL: Datasets for Deep Data-Driven Reinforcement Learning,” CoRR abs/2004.07219 (2020), https://doi.org/10.48550/arXiv.2004.07219.

[39]

Y. Ma, J. Hao, H. Liang, and C. Xiao, “ Rethinking Decision Transformer via Hierarchical Reinforcement Learning,” in Forty-First International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 (OpenReview.net, 2024).

[40]

O. Nachum, S. Gu, H. Lee, and S. Levine, “ Data-Efficient Hierarchical Reinforcement Learning,” in Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada (2018), 3307-3317.

[41]

L. Guo, Y. Ren, R. Li, and B. Jiang, “ Prior Knowledge-Augmented Broad Reinforcement Learning Framework for Fault Diagnosis of Heterogeneous Multiagent Systems,” IEEE Transactions on Cognitive and Developmental Systems 16, no. 1 (2024): 115-123, https://doi.org/10.1109/tcds.2023.3266791.

[42]

S. Levine, A. Kumar, G. Tucker, and J. Fu, “ Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems,” CoRR abs/2005 (2020): 01643, https://doi.org/10.48550/arXiv.2005.01643.

[43]

D. Brandfonbrener, A. Bietti, J. Buckman, R. Laroche, and J. Bruna, “ When Does Return-Conditioned Supervised Learning Work for Offline Reinforcement Learning?,” in Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28-December 9, 2022 (2022), 1542-1553.

[44]

H. van Hasselt, “ Double q-Learning,” in Advances in Neural Information Processing Systems 23: 24th Annual Conference on Neural Information Processing Systems 2010. Proceedings of a Meeting Held 6-9 December 2010, Vancouver, British Columbia, Canada (Curran Associates Inc., 2010), 2613-2621.

[45]

R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction (MIT press, 2018).

[46]

D. P. Kingma and J. Ba, “ Adam: A Method for Stochastic Optimization,” in 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings (2015).

[47]

R. Kidambi, A. Rajeswaran, P. Netrapalli, and T. Joachims, “ Morel: Model-Based Offline Reinforcement Learning,” in Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, Virtual (2020), 21810-21823.

[48]

Z. Peng, Y. Liu, C. Han, and Z. Zhou, “ Implicit Policy Constraint for Offline Reinforcement Learning,” CAAI Transactions on Intelligence Technology 9, no. 4 (2024): 973-981, https://doi.org/10.1049/cit2.12304.

[49]

T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “ Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning With a Stochastic Actor,” in ICML (2018), 1861-1870.

[50]

Z. Wang, J. J. Hunt, and M. Zhou, “ Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning,” in The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 (OpenReview.net, 2023).

[51]

A. Gupta, V. Kumar, C. Lynch, S. Levine, and K. Hausman, “ Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning,” in 3rd Annual Conference on Robot Learning, CoRL 2019, Osaka, Japan, October 30-November 1, 2019, Proceedings, eds L. P. Kaelbling, D. Kragic, and K. Sugiura, vol. 100 Proceedings of Machine Learning Research (PMLR, 2019), 1025-1037.

[52]

P. Bhargava, R. Chitnis, A. Geramifard, S. Sodhani, and A. Zhang, “ Sequence Modeling Is a Robust Contender for Offline Reinforcement Learning,” CoRR abs/2305.14550 (2023), https://doi.org/10.48550/arXiv.2305.14550.

[53]

J. Schulman, S. Levine, P. Abbeel, M. I. Jordan, and P. Moritz, “ Trust Region Policy Optimization,” in Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, Vol. 37 of JMLR Workshop and Conference Proceedings (JMLR.org, 2015), 1889-1897.

[54]

L. Sun, L. Wang, W. Ding, Y. Qian, and J. Xu, “ Neighborhood Multi-Granulation Rough Sets-Based Attribute Reduction Using Lebesgue and Entropy Measures in Incomplete Neighborhood Decision Systems,” Knowledge-Based Systems 192 (2020): 105373, https://doi.org/10.1016/j.knosys.2019.105373.

PDF (1946KB)

0

Accesses

0

Citation

Detail

Sections
Recommended

/

〈 〉