Explaining Agent Interactions through their Causal Behavior and Counterfactuals

Reinforcement learning (RL) agents often operate as black boxes, making it difficult to understand their decision-making in dy- namic environments. We propose a novel framework for explainable RL (XRL) based on structural causal models (SCMs). Our approach learns an SCM of the environment dynamics and reward process in a mobile network simulator (mobile-env ), and uses this causal model to gener- ate counterfactual explanations and perform interventions to understand agent behavior. We demonstrate that the learned SCM can closely ap- proximate the environment’s transition dynamics while remaining in- terpretable. By leveraging do-calculus and counterfactual reasoning, our framework explains the long-term effects of actions through causal chains and highlights key influential factors. Experiments on a wireless network control task show that our method provides meaningful explanations of agent decisions (e.g., why a certain action leads to higher reward), with minimal loss in policy performance. We present comparative evaluations against baseline explanation approaches and discuss how our SCM-based explanations improve transparency and trust in RL policies