Exploration and Exploitation: A Study on Sample Efficiency in Reinforcement Learning With Multifaceted Curiosity Rewards and Adaptive Experience Replay Utilisation in Sparse Reward Environments

Jingyi Huang , Guipeng Xi , Meixiu Lin , Haohui Zhang , Bo Li , Evgeny Neretin

CAAI Transactions on Intelligence Technology ›› 2026, Vol. 11 ›› Issue (4) : 1028 -1046.

PDF (4335KB)
CAAI Transactions on Intelligence Technology ›› 2026, Vol. 11 ›› Issue (4) :1028 -1046. DOI: 10.1049/cit2.70153
ORIGINAL RESEARCH OPEN ACCESS
research-article
Exploration and Exploitation: A Study on Sample Efficiency in Reinforcement Learning With Multifaceted Curiosity Rewards and Adaptive Experience Replay Utilisation in Sparse Reward Environments
Author information +
History +
PDF (4335KB)

Abstract

In recent years, the widespread application of deep reinforcement learning (DRL) in autonomous systems has highlighted the importance of achieving high sample efficiency under sparse reward conditions. To improve sample efficiency in sparse reward environments, this paper proposes a reinforcement learning framework built upon the Soft Actor Critic architecture, which integrates multifaceted curiosity rewards (MCR) and adaptive experience replay utilisation (AERU) (MCR-AERU SAC). MCR combines multi-level intrinsic motivational signals, such as state prediction error and model uncertainty, to provide rich exploration incentives, encouraging the controlled entity to deviate from existing trajectories and discover new high-reward behaviours. AERU dynamically adjusts the experience replay priorities based on posterior temporal difference error (TD error), focusing on utilising ‘partially successful’ transitions that are easily overlooked. The synergy between MCR and AERU enables the proposed framework to achieve an optimal balance between exploration and exploitation, significantly accelerating policy convergence and improving sample utilisation. Extensive experiments in complex dynamic training environments demonstrate that the proposed MCR-AERU SAC algorithm achieves up to 1.41 times the early-stage gain rate and a 55.56% improvement in task success rate compared to the HER-SAC baseline, demonstrating superior sample efficiency and excellent robustness in large-scale sparse reward environments.

Keywords

adaptive experience replay / curiosity reward / exploration-exploitation trade-off / mixed exploration strategy / reinforcement learning sample efficiency

Cite this article

Download citation ▾
Jingyi Huang, Guipeng Xi, Meixiu Lin, Haohui Zhang, Bo Li, Evgeny Neretin. Exploration and Exploitation: A Study on Sample Efficiency in Reinforcement Learning With Multifaceted Curiosity Rewards and Adaptive Experience Replay Utilisation in Sparse Reward Environments. CAAI Transactions on Intelligence Technology, 2026, 11 (4) : 1028-1046 DOI:10.1049/cit2.70153

登录浏览全文

4963

注册一个新账户 忘记密码

References

[1]

O. K. Pal, M. S. H. Shovon, M. F. Mridha, and J. Shin, “ In-Depth Review of AI-Enabled Unmanned Aerial Vehicles: Trends, Vision, and Challenges,” Discover Artificial Intelligence 4, no. 1 (December 2024): 97, https://doi.org/10.1007/s44163-024-00209-1.

[2]

A. A. Laghari, A. K. Jumani, R. A. Laghari, H. Li, S. Karim, and A. A. Khan, “ Unmanned Aerial Vehicles Advances in Object Detection and Communication Security Review,” Cognitive Robotics 4 (2024): 128-141, https://doi.org/10.1016/j.cogr.2024.07.002.

[3]

J. Y. Huang, Y. J. Cui, G. P. Xi, et al., “ GTrXL-SAC-Based Path Planning and Obstacle-Aware Control Decision-Making for UAV Autonomous Control,” Drones 9, no. 4 (April 2025): 275, https://doi.org/10.3390/drones9040275.

[4]

Z. Hu, M. Z. Butt, N. Nasir, and M. B. Salleh, “ Vision-Based Autonomous Navigation for Quadrotor UAVs in Unknown and Constrained Environments,” International Journal of Intelligent Robotics and Applications 9, no. 4 (May 2025): 1743-1754, https://doi.org/10.1007/s41315-025-00462-2.

[5]

B. Li, J. M. Wang, C. Song, Z. Yang, K. Wan, and Q. Zhang, “ Multi-UAV Roundup Strategy Method Based on Deep Reinforcement Learning CEL-MADDPG Algorithm,” Expert Systems with Applications 245 (July 2024), https://doi.org/10.1016/j.eswa.2023.123018.

[6]

X. Wang, S. Wang, X. X. Liang, et al., “ Deep Reinforcement Learning: A Survey,” IEEE Transactions on Neural Networks and Learning Systems 35, no. 4 (April 2024): 5064-5078, https://doi.org/10.1109/TNNLS.2022.3207346.

[7]

R. S. Sutton and A. G. Barto, “ Reinforcement Learning: An Introduction,” IEEE Transactions on Neural Networks 9, no. 5 (September 1998): 1054, https://doi.org/10.1109/TNN.1998.712192.

[8]

B. Li, J. Huang, S. Bai, et al., “ Autonomous Air Combat Decision-Making of UAV Based on Parallel Self-Play Reinforcement Learning,” CAAI Transactions on Intelligence Technology 8 (2022): 1-18, https://doi.org/10.1049/cit2.12109.

[9]

X. W. Fu and Y. Sun, “ A Combined Intrusion Strategy Based on Apollonius Circle for Multiple Mobile Robots in Attack-Defense Scenario,” IEEE Robotics and Automation Letters 10, no. 1 (January 2025): 676-683, https://doi.org/10.1109/LRA.2024.3512361.

[10]

M. Cao, L. Shu, L. Yu, et al., “ Beyond Sparse Rewards: Enhancing Reinforcement Learning With Language Model Critique in Text Generation,” 19 (February 2024), https://doi.org/10.48550/arXiv.2401.07382.

[11]

X. Y. Liu, H. Yang, J. Gao, et al., “ FinRL: Deep Reinforcement Learning Framework to Automate Trading in Quantitative Finance,” in Proceedings of the Second ACM International Conference on AI in Finance, Virtual Event (ACM, 2021), 1-9, https://doi.org/10.1145/3490354.3494366.

[12]

P. Ladosz, L. Weng, M. Kim, and H. Oh, “ Exploration in Deep Reinforcement Learning: A Survey,” Information Fusion 85 (September 2022): 1-22, https://doi.org/10.1016/j.inffus.2022.03.003.

[13]

M. Yuan, R. Castanyer, B. Li, et al., “ RLeXplore: Accelerating Research in Intrinsically-Motivated Reinforcement Learning,” (April 2025), https://doi.org/10.48550/arXiv.2405.19548.

[14]

D. Pathak, P. Agrawal, A. A. Efros, et al., “ Curiosity-Driven Exploration by Self-Supervised Prediction,” (May 2017), https://doi.org/10.48550/arXiv.1705.05363.

[15]

X. Fu, J. Zhu, Z. Wei, H. Wang, and S. Li, “A UAV Pursuit-Evasion Strategy Based on DDPG and Imitation Learning,” International Journal of Aerospace Engineering 2022 (May 2022): 1-14, https://doi.org/10.1155/2022/3139610.

[16]

L. Wu and K. Chen, “ Goal Exploration Augmentation via Pre-Trained Skills for Sparse-Reward Long-Horizon Goal-Conditioned Reinforcement Learning,” (December 2023), https://doi.org/10.48550/arXiv.2210.16058.

[17]

J. Zhang, W. Zhang, D. Zhou, et al., “ Uncertainty-Aware Reward-Free Exploration With General Function Approximation,” (June 2024), https://doi.org/10.48550/arXiv.2406.16255.

[18]

M. Albaba, S. Christen, T. Langarek, et al., “ RILe: Reinforced Imitation Learning,” (April 2025), https://doi.org/10.48550/arXiv.2406.08472.

[19]

B. Li, J. Y. Huang, H. H. Zhang, L. Huai, and E. Neretin, “ Enhanced Exploration for Multi-UAV Cooperative Roundup: An I2C-MATD3 Reinforcement Learning Framework,” Defence Technology 58 (October 2025): 374-389, https://doi.org/10.1016/j.dt.2025.10.032.

[20]

Y. Burda, H. Edwards, D. Pathak, et al., “ Large-Scale Study of Curiosity-Driven Learning,” (August 2018), https://doi.org/10.48550/arXiv.1808.04355.

[21]

A. L. Strehl and M. L. Littman, “ An Analysis of Model-Based Interval Estimation for Markov Decision Processes,” Journal of Computer and System Sciences 74, no. 8 (December 2008): 1309-1331, https://doi.org/10.1016/j.jcss.2007.08.009.

[22]

S. Mohamed and D. J. Rezende, “ Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning,” (September 2015), https://doi.org/10.48550/arXiv.1509.08731.

[23]

R. Raileanu and T. Rocktäschel, “ RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environments,” (February 2020), https://doi.org/10.48550/arXiv.2002.12292.

[24]

J. Li and P. Gajane, “ Curiosity-Driven Exploration in Sparse-Reward Multi-Agent Reinforcement Learning,” (February 2023), https://doi.org/10.48550/arXiv.2302.10825.

[25]

M. Andrychowicz, F. Wolski, A. Ray, et al., “ Hindsight Experience Replay,” (February 2018), https://doi.org/10.48550/arXiv.1707.01495.

[26]

H. Oh, M. Murooka, T. Motoda, et al., “ Self-Augmented Robot Trajectory: Efficient Imitation Learning via Safe Self-Augmentation With Demonstrator-Annotated Precision,” 11 (September 2025), https://doi.org/10.48550/arXiv.2509.09893.

[27]

C. Finn, S. Levine, and P. Abbeel, “ Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization,” (March 2016), https://doi.org/10.48550/arXiv.1603.00448.

[28]

J. Lin, Y. Q. Du, O. Watkins, et al., “ Learning to Model the World With Language,” (May 2024), https://doi.org/10.48550/arXiv.2308.01399.

[29]

A. Nair, V. Pong, M. Dalal, et al., “ Visual Reinforcement Learning With Imagined Goals,” (December 2018), https://doi.org/10.48550/arXiv.1807.04742.

[30]

M. Fang, T. Zhou, Y. Du, et al., “ Curriculum-Guided Hindsight Experience Replay,” in Proceedings of the 33rd International Conference on Neural Information Processing Systems (Curran Associates Inc., 2019) 12623-12634, Article 1131.

[31]

T. Haarnoja, A. Zhou, K. Hartikainen, et al., “ Soft Actor-Critic Algorithms and Applications,” (January 2019), https://doi.org/10.48550/arXiv.1812.05905.

[32]

Z. J. Hu, X. G. Gao, K. F. Wan, Q. Wang, and Y. Zhai, “ Asynchronous Curriculum Experience Replay: A Deep Reinforcement Learning Approach for UAV Autonomous Motion Control in Unknown Dynamic Environments,” IEEE Transactions on Vehicular Technology (2023): 1-16, https://doi.org/10.1109/TVT.2023.3285595.

[33]

B. Norman and J. Clune, “ First-Explore, Then Exploit: Meta-Learning to Solve Hard Exploration-Exploitation Trade-Offs,” (November 2024), https://doi.org/10.48550/arXiv.2307.02276.

[34]

R. Strudel, R. Garcia, J. Carpentier, et al., “ Learning Obstacle Representations for Neural Motion Planning,” (November 2020), https://doi.org/10.48550/arXiv.2008.11174.

[35]

Z. J. Hu, X. G. Gao, K. F. Wan, Y. Zhai, and Q. Wang, “ Relevant Experience Learning: A Deep Reinforcement Learning Method for UAV Autonomous Motion Planning in Complex Unknown Environments,” Chinese Journal of Aeronautics 34, no. 12 (2021): 187-204, https://doi.org/10.1016/j.cja.2020.12.027.

[36]

R. Devidze, P. Kamalaruban, and A. Singla, “ Exploration-Guided Reward Shaping for Reinforcement Learning Under Sparse Rewards,” in 36th Conference on Neural Information Processing Systems (NeurIPS 2022), (Curran Associates, Inc., 2022).

[37]

H. Lin, C. Huang, and Z. Chen, “ An Advantage-Based Optimization Method for Reinforcement Learning in Large Action Space,” (December 2024), https://doi.org/10.48550/arXiv.2412.12605.

[38]

J. Yan, B. Luo, and X. Xu, “ Hierarchical Reinforcement Learning for Handling Sparse Rewards in Multi-Goal Navigation,” Artificial Intelligence Review 57, no. 6 (May 2024): 156, https://doi.org/10.1007/s10462-024-10794-3.

[39]

D. Yunis, J. Jung, F. Dai, et al., “ Subwords as Skills: Tokenization for Sparse-Reward Reinforcement Learning,” (October 2024), https://doi.org/10.48550/arXiv.2309.04459.

[40]

T. Schaul, J. Quan, I. Antonoglou, et al., “ Prioritized Experience Replay,” (February 2016), https://doi.org/10.48550/arXiv.1511.05952.

[41]

Y. C. Pan, J. C. Mei, A. M. Farahmand, et al., “ Understanding and Mitigating the Limitations of Prioritized Experience Replay,” (June 2022), https://doi.org/10.48550/arXiv.2007.09569.

[42]

A. Kayal, E. Pignatelli, and L. Toni, “ The Impact of Intrinsic Rewards on Exploration in Reinforcement Learning,” Neural Computing & Applications 37, no. 21 (July 2025): 16269-16303, https://doi.org/10.1007/s00521-025-11340-0.

[43]

H. R. Tang, R. Houthooft, D. Foote, et al., “ #Exploration: A Study of Count-Based Explorationfor Deep Reinforcement Learning,” (November 2016), https://doi.org/10.48550/arXiv.1611.04717.

[44]

M. G. Bellemare, S. Srinivasan, G. Ostrovski, et al., “ Unifying Count-Based Exploration and Intrinsic Motivation,” (November 2016), https://doi.org/10.48550/arXiv.1606.01868.

[45]

A. S. Klyubin, D. Polani, and C. L. Nehaniv, “ Empowerment: A Universal Agent-Centric Measure of Control,” in 2005 IEEE Congress on Evolutionary Computation (IEEE, 2005), 128-135, https://doi.org/10.1109/CEC.2005.1554676.

[46]

K. Gregor, D. Rezende, and D. Wierstra, “ Variational Intrinsic Control,” (November 2016), https://doi.org/10.48550/arXiv.1611.07507.

[47]

A. V. D. Oord, Y. Z. Li, and O. Vinyals, “ Representation Learning With Contrastive Predictive Coding,” (January 2019), https://doi.org/10.48550/arXiv.1807.03748.

[48]

X. Wang, Y. Chen, and W. Zhu, “ A survey on Curriculum Learning,” IEEE Transactions on Pattern Analysis and Machine Intelligence (2021): 1, https://doi.org/10.1109/TPAMI.2021.3069908.

[49]

Y. Kim, “ Convolutional Neural Networks for Sentence Classification,” (September 2014), https://doi.org/10.48550/arXiv.1408.5882.

PDF (4335KB)

0

Accesses

0

Citation

Detail

Sections
Recommended

/

〈 〉