On Temporal Credit Assignment and Data-Efficient Reinforcement Learning

This paper introduces a novel performance measure for evaluating Reinforcement Learning (RL) algorithms, specifically addressing the temporal credit assignment problem. The authors argue that existing measures for generalization and exploration do not adequately capture an algorithm's ability to attribute outcomes to past actions and states. They propose "misallocation" (MALLOC), an information-theoretic metric that quantifies the difference between an algorithm's credit attribution and that of an optimal policy. To define MALLOC, the paper utilizes Partial Information Decomposition (PID), a concept from information theory, and employs Shapley values from game theory to assign credit to individual steps in a trajectory, offering a more nuanced understanding of how RL agents learn from delayed rewards.