A Novel State Space Exploration Method for the Sparse-Reward Reinforcement Learning Environment

Liu, Xi; Ma, Long; Chen, Zhen; Zheng, Changgang; Chen, Ren; Liao, Yong; Yang, Shufan

Abstract

Sparse-reward reinforcement learning environments pose a particular challenge because the agent receives infrequent rewards, making it difficult to learn an optimal policy. In this paper, we propose NSSE, a novel approach that combines that stratified state space exploration with prioritised sweeping to enhance the informativeness of learning, thus enabling fast learning convergence. We evaluate NSSE on three typical Atari sparse reward environments. The results demonstrate that our state space exploration method exhibits strong performance compared to two baseline algorithms: Deep Q-Network (DQN) and noisy Deep Q-Network (Noisy DQN).

• Presentation Conference Type: Conference Paper (Published)
• Conference Name: 43rd SGAI International Conference on Artificial Intelligence
• Start Date: Dec 12, 2023
• End Date: Dec 14, 2023
• Online Publication Date: Nov 8, 2023
• Publication Date: 2023
• Deposit Date: Jun 6, 2024
• Publisher: Springer
• Peer Reviewed: Peer Reviewed
• Pages: 216-221
• Series Title: Lecture Notes in Computer Science
• Series Number: 14381
• Series ISSN: 0302-9743
• Book Title: Artificial Intelligence XL: 43rd SGAI International Conference on Artificial Intelligence, AI 2023, Cambridge, UK, December 12–14, 2023, Proceedings
• ISBN: 978-3-031-47993-9
• DOI: https://doi.org/10.1007/978-3-031-47994-6_18
• Keywords: Sparse-reward, Replay Sub-buffers, DQN, Exploration, Reinforcement Learning