Discrete Representations for Continual Reinforcement Learning

This episode explores whether learned discrete representations actually improve reinforcement learning, world modeling, and continual adaptation compared with standard continuous latent spaces. It explains how vector-quantized codebook latents, sparse binary-style features, and older ideas like tile coding relate to modern world models, and why the real advantage may come from reduced interference rather than discreteness alone. The discussion centers on three evaluation settings: predicting future dynamics in latent space, improving downstream control in model-free RL, and helping agents adapt to shifting tasks without forgetting earlier behavior. Listeners would find it interesting because it cuts through the “discrete vs. continuous” hype and turns the paper into a sharper engineering question about which representation bottlenecks produce more stable, reusable abstractions under changing conditions. Sources: 1. Harnessing Discrete Representations For Continual Reinforcement Learning — Edan Meyer, Adam White, Marlos C. Machado, 2023 http://arxiv.org/abs/2312.01203 2. Neural Discrete Representation Learning — Aaron van den Oord, Oriol Vinyals, Koray Kavukcuoglu, 2017 https://papers.nips.cc/paper/2017/hash/7a98af17e63a0ac09ce2e96d03992fbc-Abstract.html 3. Mastering Atari with Discrete World Models — Danijar Hafner, Timothy P. Lillicrap, Mohammad Norouzi, Jimmy Ba, 2021 https://openreview.net/forum?id=0oabwyZbOu 4. Transformers are Sample-Efficient World Models — Vincent Micheli, Eloi Alonso, Francois Fleuret, 2023 https://openreview.net/forum?id=vhFu1Acb0xb 5. Harnessing Discrete Representations for Continual Reinforcement Learning — Edan Jacob Meyer, Adam White, Marlos C. Machado, 2024 https://openreview.net/forum?id=tCXURNlAZ3 6. Mastering Diverse Domains through World Models — Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, Timothy P. Lillicrap, 2023 https://scholar.google.com/scholar?q=Mastering+Diverse+Domains+through+World+Models 7. Fuzzy Tiling Activations: A Simple Approach to Learning Sparse Representations Online — Yangchen Pan, Kirby Banman, Martha White, 2021 https://scholar.google.com/scholar?q=Fuzzy+Tiling+Activations%3A+A+Simple+Approach+to+Learning+Sparse+Representations+Online 8. Investigating the Properties of Neural Network Representations in Reinforcement Learning — Han Wang, Erfan Miahi, Martha White, Marlos C. Machado, Zaheer Abbas, Raksha Kumaraswamy, Vincent Liu, Adam White, 2022 https://scholar.google.com/scholar?q=Investigating+the+Properties+of+Neural+Network+Representations+in+Reinforcement+Learning 9. Smaller World Models for Reinforcement Learning — Jan Robine, Tobias Uelwer, Stefan Harmeling, 2021 https://scholar.google.com/scholar?q=Smaller+World+Models+for+Reinforcement+Learning 10. Continual Learning as Computationally Constrained Reinforcement Learning — Saurabh Kumar, Henrik Marklund, Ashish Rao, Yifan Zhu, Hong Jun Jeon, Yueyang Liu, Benjamin Van Roy, 2023 https://scholar.google.com/scholar?q=Continual+Learning+as+Computationally+Constrained+Reinforcement+Learning 11. Efficient World Models with Context-Aware Tokenization — Vincent Micheli, Eloi Alonso, Francois Fleuret, 2024 https://scholar.google.com/scholar?q=Efficient+World+Models+with+Context-Aware+Tokenization 12. AdaWorld: Learning Adaptable World Models with Latent Actions — authors not identifiable from the snippet, recent https://scholar.google.com/scholar?q=AdaWorld%3A+Learning+Adaptable+World+Models+with+Latent+Actions 13. A Survey of Continual Reinforcement Learning — authors not identifiable from the snippet, recent https://scholar.google.com/scholar?q=A+Survey+of+Continual+Reinforcement+Learning 14. Stable Continual Reinforcement Learning via Diffusion-Based Trajectory Replay — authors not identifiable from the snippet, recent https://scholar.google.com/scholar?q=Stable+Continual+Reinforcement+Learning+via+Diffusion-Based+Trajectory+Replay 15. AI Post Transformers: DreamerV3 World Models Across 150 Tasks — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-04-20-dreamerv3-world-models-across-150-tasks-af5edb.mp3 16. AI Post Transformers: TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-03-25-turboquant-online-vector-quantiz-1967b7.mp3 17. AI Post Transformers: Latent Space as a New Computational Paradigm — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-04-05-latent-space-as-a-new-computational-para-810f39.mp3