From Entropy to Epiplexity- Rethinking Information for Computationally Bounded Intelligence
from Mechanical Dreams Hosted by Mechanical Dirk
In this episode: • Introduction: Is Shannon Information Theory Broken?: Professor Norris and Linda introduce the episode, with Norris expressing skepticism about challenging the foundations of information theory. Linda introduces the paper 'From Entropy to Epiplexity' and the premise that traditional theory fails to account for computational bounds. • The Paradox of Deterministic Creation: The hosts discuss the first major paradox: how deterministic processes like AlphaZero or synthetic data generation seem to create new knowledge, despite the Data Processing Inequality suggesting otherwise. Linda explains why cryptographic randomness proves that 'computational difficulty' looks like entropy. • Defining Epiplexity and Time-Bounded Entropy: Linda breaks down the core definitions of the paper, explaining Epiplexity as the structural information a specific model can actually learn, versus Time-Bounded Entropy, which is the residual unpredictability relative to that model's resources. • Emergence, Induction, and the Chess Experiment: A deep dive into the paper's experiments with Cellular Automata and Chess. The hosts discuss how the order of data (Forward vs. Reverse) impacts what a model learns and how limited compute forces models to learn emergent rules rather than brute-force simulation. • Practical Implications for LLMs and Conclusion: The discussion moves to real-world application, specifically how Epiplexity explains why pre-training on text transfers better than images. Norris admits the utility of the theory for data selection in Large Language Models.
Embed this episode
No transcript yet.