Event-driven Temporal Models for Explanations - ETeMoX: Explaining Reinforcement Learning

Parra-Ullauri, Juan Marcelo; Garcoa-Dominguez, Antonio; Bencomo, Nelly; Zheng, Changgang; Zhen, Chen; Boubeta-Puig, Juan; Ortiz, Guadalupe; Yang, Shufan

Abstract

Modern software systems are increasingly expected to show higher degrees of autonomy and self-management to cope with uncertain and diverse situations. As a consequence, autonomous systems can exhibit unexpected and surprising behaviours. This is exacerbated due to the ubiquity and complexity of Artificial Intelligence (AI)-based systems. This is the case of Reinforcement Learning (RL), where autonomous agents learn through trial-and-error how to find good solutions to a problem. Thus, the underlying decision-making criteria may become opaque to users that interact with the system and who may require explanations about the system’s reasoning. Available work for eXplainable Reinforcement Learning (XRL) offers different trade-offs: e.g. for runtime explanations, the approaches are model-specific or can only analyse results after-the-fact. Different from these approaches, this paper aims to provide an online model-agnostic approach for XRL towards trustworthy and understandable AI. We present ETeMoX, an architecture based on temporal models to keep track of the decision-making processes of RL systems. Runtime models are stored on a temporal graph database and queried during system execution on demand to extract history-aware explanations. In cases where the resources are limited (e.g. storage capacity or time to response), the architecture also integrates complex event processing, an event-driven approach, for detecting matches to event patterns (complex events) that need to be stored, instead of keeping the entire history. The approach is applied to a mobile communications case study using autonomous airborne base stations, which are positioned using RL algorithms to maximise user coverage. In order to test the generalizability of our approach, three variants of the underlying RL algorithms are used: Q-Learning, State-Action-Reward-State-Action (SARSA) and Deep Q-Network (DQN). The experiments are performed during training to support developers in gaining insights about the learning process in reinforcement learning. The encouraging results show that using the proposed configurable architecture, RL developers are able to obtain explanations about the evolution of a metric, relationships between metrics, and were able to track situations of interest happening over time windows.

Citation

Parra-Ullauri, J. M., Garcoa-Dominguez, A., Bencomo, N., Zheng, C., Zhen, C., Boubeta-Puig, J., …Yang, S. (2022). Event-driven Temporal Models for Explanations - ETeMoX: Explaining Reinforcement Learning. Software and Systems Modeling, 21, 1091-1113. https://doi.org/10.1007/s10270-021-00952-4

• Journal Article Type: Article
• Acceptance Date: Nov 4, 2021
• Online Publication Date: Dec 18, 2021
• Publication Date: 2022-06
• Journal: Software and Systems Modeling
• Print ISSN: 1619-1366
• Publisher: Springer
• Peer Reviewed: Peer Reviewed
• Volume: 21
• Pages: 1091-1113
• DOI: https://doi.org/10.1007/s10270-021-00952-4
• Keywords: Temporal Models, Complex Event Processing, Artificial Intelligence, Explainable Reinforcement Learning, Event-driven Monitoring
• Public URL: http://researchrepository.napier.ac.uk/Output/2782463