Towards Interpretable Reinforcement Learning with State Abstraction Driven by External Knowledge

• BOUGIE Nicolas

  Sokendai, The Graduate University for Advanced Studies National Institute of Informatics

• ICHISE Ryutaro

  Sokendai, The Graduate University for Advanced Studies National Institute of Informatics

抄録

Advances in deep reinforcement learning have demonstrated its effectiveness in a wide variety of domains. Deep neural networks are capable of approximating value functions and policies in complex environments. However, deep neural networks inherit a number of drawbacks. Lack of interpretability limits their usability in many safety-critical real-world scenarios. Moreover, they rely on huge amounts of data to learn efficiently. This may be suitable in simulated tasks, but restricts their use to many real-world applications. Finally, their generalization capability is low, the ability to determine that a situation is similar to one encountered previously. We present a method to combine external knowledge and interpretable reinforcement learning. We derive a rule-based variant version of the Sarsa(λ) algorithm, which we call Sarsa-rb(λ), that augments data with prior knowledge and exploits similarities among states. We demonstrate that our approach leverages small amounts of prior knowledge to significantly accelerate the learning in multiple domains such as trading or visual navigation. The resulting agent provides substantial gains in training speed and performance over deep q-learning (DQN), deep deterministic policy gradients (DDPG), and improves stability over proximal policy optimization (PPO).

収録刊行物

• IEICE Transactions on Information and Systems

  IEICE Transactions on Information and Systems E103.D (10), 2143-2153, 2020-10-01

  一般社団法人 電子情報通信学会

キーワード

• reinforcement learning
• symbolic reinforcement learning
• reasoning about knowledge
• interpretable reinforcement learning

詳細情報

• CRID: 1390567172583610496
• NII論文ID: 130007920639
• DOI: 10.1587/transinf.2019edp7170
• ISSN: 17451361
• ISSN: 09168532
• Web Site: https://www.jstage.jst.go.jp/article/transinf/E103.D/10/E103.D_2019EDP7170/_pdf
• 本文言語コード: en
• データソース種別:
  • JaLC
  • Crossref
  • CiNii Articles
  • KAKEN
• 抄録ライセンスフラグ: 使用不可