Temporal Dependency-Aware Trajectory-Level Behavioural Metric for Exploration in Reinforcement Learning

Abstract

Intrinsic motivation serves as the predominant paradigm of exploration in reinforcement learning. In pursuit of an informative and robust state representation, the behavioural metric groups behaviourally equivalent states together, which share the same single-step reward and transition distribution. However, due to the presence of uninformative rewards and the dynamic nature of procedurally generated environments, these behavioural metric-based approaches could limit the effectiveness of the learnt state representations, potentially leading to a representation collapse and an ineffective exploration. Therefore, a more comprehensive and generalisable behavioural metric is needed to overcome the above issues. In this work, we approach the exploration problem from a novel perspective, extending beyond the conventional single-step assessments to encompass a long-term consideration of the whole trajectory. Specifically, we propose a novel trajectory-level behavioural metric (TBM) that exploits temporal dependencies of the trajectory and captures the underlying sequential information of behaviour patterns. To achieve an effective trajectory representation for exploration, we develop apivotal state identifier (PSI) and a trajectory return estimator (TRE) to distinguish the diverse contributions of individual states in the trajectory. Moreover, an auxiliary representation regulariser is developed to promote the diversity and informativeness of the trajectory representation, mitigating the risk of representation mode collapse. Extensive experiments and empirical analysis conducted on procedurally generated environments showcase the superior performance of our proposed framework.

Keywords

learning (artificial intelligence) / machine learning / reinforcement learning

Acknowledgements

This work was supported by the National Natural Science Foundation of China (Grant 62276047) and Sichuan Science and Technology Programme (Grant 2025HJRC0021).

Conflicts of Interest

The authors declare no conflicts of interest.

Data Availability Statement

Code and data are available at https://github.com/AnneZhu1020/TBM.

References

[1] X. Wang, S. Wang, X. Liang, et al., Deep Reinforcement Learning: A Survey,” IEEE Transactions on Neural Networks and Learning Systems 35, no. 4 (2024): 5064-5078, https://doi.org/10.1109/tnnls.2022.3207346.

[2] V. Mnih, K. Kavukcuoglu, D. Silver, et al., Human-Level Control Through Deep Reinforcement Learning,” Nature 518, no. 7540 (2015): 529-533, https://doi.org/10.1038/nature14236.

[3] Z. Yang, K. E. Merrick, L. Jin, and H. A. Abbass, “Hierarchical Deep Reinforcement Learning for Continuous Action Control,” IEEE Transactions on Neural Networks and Learning Systems 29, no. 11 (2018): 5174-5184, https://doi.org/10.1109/tnnls.2018.2805379.

[4] Y. Yang, L. Juntao, and P. Lingling, “Multi-Robot Path Planning Based on a Deep Reinforcement Learning DQN Algorithm,” CAAI Transactions on Intelligence Technology 5, no. 3 (2020): 177-183, https://doi.org/10.1049/trit.2020.0024.

[5] J. Zhang, K. Wang, and C. Mu, “Multi-Station Multi-Robot Task Assignment Method Based on Deep Reinforcement Learning,” CAAI Transactions on Intelligence Technology 10, no. 1 (2025): 134-146, https://doi.org/10.1049/cit2.12394.

[6] M. M. Afsar, T. Crump, and B. H. Far, “Reinforcement Learning Based Recommender Systems: A Survey,” ACM Computing Surveys 55, no. 7 (2023): 145:1-145:38, https://doi.org/10.1145/3543846.

[7] P. Ladosz, L. Weng, M. Kim, and H. Oh, “Exploration in Deep Reinforcement Learning: A Survey,” Information Fusion 85 (2022): 1-22, https://doi.org/10.1016/j.inffus.2022.03.003.

[8] W. Dabney, G. Ostrovski, and A. Barreto, “Temporally-Extended ε-Greedy Exploration,” in 9th International Conference on Learning Representations, (ICLR, 2021).

[9] I. Osband, B. V. Roy, D. J. Russo, and Z. Wen, “Deep Exploration via Randomized Value Functions,” Journal of Machine Learning Research 20 (2019): 124:1-124:62, https://jmlr.org/papers/v20/18-339.html.

[10] A. Aubret, L. Matignon, and S. Hassas, “A Survey on Intrinsic Motivation in Reinforcement Learning,” CoRR (2019): 06976: abs/1908.

[11] Z. Chen, B. Luo, T. Hu, and X. Xu, “LJIR: Learning Joint-Action Intrinsic Reward in Cooperative Multi-Agent Reinforcement Learning,” Neural Networks 167 (2023): 450-459, https://doi.org/10.1016/j.neunet.2023.08.016.

[12] N. Dilokthanakul, C. Kaplanis, N. Pawlowski, and M. Shanahan, “Feature Control as Intrinsic Motivation for Hierarchical Reinforcement Learning,” IEEE Transactions on Neural Networks and Learning Systems 30, no. 11 (2019): 3409-3418, https://doi.org/10.1109/tnnls.2019.2891792.

[13] H. Tang, R. Houthooft, D. Foote, et al., #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning,” in Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems, (2017), 2753-2762.

[14] Y. Burda, H. Edwards, A. J. Storkey, and O. Klimov, “Exploration by Random Network Distillation,” in 7th International Conference on Learning Representations, Vol. 2019 (ICLR, 2019).

[15] R. Raileanu and T. Rocktäschel , “RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environments,” in 8th International Conference on Learning Representations (ICLR, 2020).

[16] B. C. Stadie, S. Levine, and P. Abbeel, “Incentivizing Exploration in Reinforcement Learning With Deep Predictive Models,” CoRR (2015): 00814: abs/1507.

[17] D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell, “Curiosity-Driven Exploration by Self-supervised Prediction,” in Proceedings of the 34th International Conference on Machine Learning (ICML, 2017), 2778-2787.

[18] C. Bai, P. Liu, K. Liu, et al., Variational Dynamic for Self-Supervised Exploration in Deep Reinforcement Learning,” IEEE Transactions on Neural Networks and Learning Systems 34, no. 8 (2023): 4776-4790, https://doi.org/10.1109/tnnls.2021.3129160.

[19] N. Ferns and D. Precup, “Bisimulation Metrics Are Optimal Value Functions,” Proceedings of the Thirtieth Conference on Uncertainty in Artificial Intelligence, UAI 2014 (2014): 210-219, https://auai.org/uai2014/proceedings/individuals/67.pdf.

[20] M. Kemertas and T. Aumentado-Armstrong, “Towards Robust Bisimulation Metric Learning,” in Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems (NeurIPS, 2021), 4764-4777.

[21] M. Chevalier-Boisvert, B. Dai, M. Towers, et al., Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Tasks,” in Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023 (NeurIPS, 2023).

[22] T. Zhang, H. Xu, X. Wang, et al., NovelD: A Simple yet Effective Exploration Criterion,” in Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems (NeurIPS, 2021), 25217-25230.

[23] A. P. Badia, P. Sprechmann, A. Vitvitskyi, et al., Never Give Up: Learning Directed Exploration Strategies,” in 8th International Conference on Learning Representations (ICLR, 2020).

[24] M. Henaff, R. Raileanu, M. Jiang, and T. Rocktäschel, “Exploration via Elliptical Episodic Bonuses,” in Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022 (NeurIPS, 2022).

[25] D. Yarats, A. Zhang, I. Kostrikov, B. Amos, J. Pineau, and R. Fergus, “Improving Sample Efficiency in Model-Free Reinforcement Learning From Images,” in Thirty-Fifth AAAI Conference on Artificial Intelligence (AAAI, 2021), 10674-10681.

[26] D. Hafner, T. P. Lillicrap, I. Fischer, et al., Learning Latent Dynamics for Planning From Pixels,” in Proceedings of the 36th International Conference on Machine Learning (ICML, 2019), 2555-2565.

[27] Z. D. Guo, B. Á Pires, B. Piot, et al., Bootstrap Latent-Predictive Representations for Multitask Reinforcement Learning,” in Proceedings of the 37th International Conference on Machine Learning (ICML, 2020), 3875-3886.

[28] M. Schwarzer, A. Anand, R. Goel, R. D. Hjelm, A. C. Courville, and P. Bachman, “Data-Efficient Reinforcement Learning With Self-Predictive Representations,” in 9th International Conference on Learning Representations (ICLR, 2021).

[29] M. Okada and T. Taniguchi, “Dreaming: Model-Based Reinforcement Learning by Latent Imagination Without Reconstruction,” in IEEE International Conference on Robotics and Automation (ICRA, 2021), 4209-4215.

[30] Y. J. Lee, J. Kim, M. Kwak, Y. J. Park, and S. B. Kim, “STACoRe: Spatio-Temporal and Action-Based Contrastive Representations for Reinforcement Learning in Atari,” Neural Networks 160 (2023): 1-11, https://doi.org/10.1016/j.neunet.2022.12.018.

[31] R. Givan, T. L. Dean, and M. Greig, “Equivalence Notions and Model Minimization in Markov Decision Processes,” Artificial Intelligence 147, no. 1-2 (2003): 163-223, https://doi.org/10.1016/s0004-3702(02)00376-4.

[32] A. Zhang, R. T. McAllister, R. Calandra, Y. Gal, and S. Levine, “Learning Invariant Representations for Reinforcement Learning Without Reconstruction,” in 9th International Conference on Learning Representations (ICLR, 2021).

[33] L. Li, T. J. Walsh, and M. L. Littman, “Towards a Unified Theory of State Abstraction for MDPs,” in International Symposium on Artificial Intelligence and Mathematics (AI&Math, 2006).

[34] K. Wang, K. Zhou, B. Kang, J. Feng, and S. Yan, “Revisiting Intrinsic Reward for Exploration in Procedurally Generated Environments,” in The Eleventh International Conference on Learning Representations (ICLR, 2023).

[35] P. Gu, M. Zhao, C. Chen, D. Li, J. Hao, and B. An, “Learning Pseudometric-based Action Representations for Offline Reinforcement Learning,” in International Conference on Machine Learning (ICML, 2022), 7902-7918.

[36] N. Ferns, P. Panangaden, and D. Precup, “Metrics for Finite Markov Decision Processes,” CoRR (2012): abs/1207.4114.

[37] P. S. Castro, “Scalable Methods for Computing State Similarity in Deterministic Markov Decision Processes,” in The Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI, 2020), 10069-10076.

[38] C. Villani , Topics in Optimal Transportation, of Graduate Studies in Mathematics Vol. 58 (American Mathematical Soc, 2003).

[39] H. Liu, M. Zhuge, B. Li, et al., Learning to Identify Critical States for Reinforcement Learning From Videos,” in 2023 IEEE/CVF International Conference on Computer Vision, (2023), 1955-1965: ICCV 2023.

[40] A. Graves, A. Mohamed, and G. E. Hinton, “Speech Recognition With Deep Recurrent Neural Networks,” in IEEE International Conference on Acoustics, Speech and Signal Processing, (ICASSP, 2013), 6645-6649.

[41] A. Vaswani, N. Shazeer, N. Parmar, et al., Attention Is all You Need,” in Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems, (2017), 5998-6008.

[42] X. Liu, Z. Wang, Y. Li, and S. Wang, “Self-Supervised Learning via Maximum Entropy Coding,” in Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems (NeurIPS, 2022).

[43] C. Kauten, “Super Mario Bros for OpenAI Gym,” GitHub (2018), https://github.com/Kautenja/gym-super-mario-bros.

[44] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal Policy Optimization Algorithms,” CoRR (2017): 06347: abs/1707.

[45] S. Wan, Y. Tang, Y. Tian, and T. Kaneko, “DEIR: Efficient and Robust Exploration Through Discriminative-Model-Based Episodic Intrinsic Rewards,” in Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence (IJCAI, 2023), 4289-4298.

[46] M. Yuan, R. C. Castanyer, B. Li, X. Jin, W. Zeng, and G. Berseth, “RLeXplore: Accelerating Research in Intrinsically-Motivated Reinforcement Learning,” Transactions on Machine Learning Research 2025 (2025), https://openreview.net/forum?id=B9BHjTN4z6.

[47] D. P. Kingma and J. Ba, “Adam: A Method for Stochastic Optimization,” in 3rd International Conference on Learning Representations, (ICLR, 2015).

[48] A. Anand, E. Racah, S. Ozair, Y. Bengio, M. Côté, and R. D. Hjelm, “Unsupervised State Representation Learning in Atari,” in Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019 (NeurIPS, 2019), 8766-8779.

[49] Y. Wu, G. Tucker, and O. Nachum, “The Laplacian in RL: Learning Representations With Efficient Approximations,” in 7th International Conference on Learning Representations, (ICLR, 2019).

[50] Y. Tassa, Y. Doron, A. Muldal, et al., Deepmind Control Suite,” CoRR (2018): 00690: abs/1801.

[51] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning With a Stochastic Actor,” in Proceedings of the 35th International Conference on Machine Learning (ICML, 2018), 1856-1865.

[52] D. Yarats, I. Kostrikov, and R. Fergus, “Image Augmentation Is all You Need: Regularizing Deep Reinforcement Learning From Pixels,” in 9th International Conference on Learning Representations, (ICLR, 2021).

[53] D. Liang, Q. Chen, and Y. Liu, “Sequential Action-Induced Invariant Representation for Reinforcement Learning,” Neural Networks 179 (2024): 106579, https://doi.org/10.1016/j.neunet.2024.106579.

[54] J. Chen, W. Z. T. Ng, Z. Chen, S. J. Pan, and T. Zhang, “State Chrono Representation for Enhancing Generalization in Reinforcement Learning,” in Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024 (NeurIPS, 2024).