EPISODE · Oct 12, 2025 · 14 MIN
The Markovian Thinker
from Best AI papers explained · host Enoch H. Kang
Reinforcement learning (RL) methods for training Large Language Models (LLMs) to produce long chains of thought (LongCoT) are constrained by the standard thinking environment, where the state is unbounded, leading to quadratic computational costs as reasoning length increases. This paper propose Markovian Thinking, a paradigm where the reasoning policy conditions only on a constant-size state, effectively decoupling thinking length from context size and yielding linear compute and constant memory benefits. This concept is instantiated with Delethink, an RL environment that organizes reasoning into fixed-size chunks. At the chunk boundaries, the context resets, and the policy must learn to write a concise textual carryover sufficient for seamless continuation of the reasoning process in the next chunk. Models trained with Delethink, such as R1-Distill 1.5B, match or exceed the performance of LongCoT-RL while significantly reducing computational overhead, demonstrating superior test-time scaling capability far beyond their training budget. They emphasize that redesigning the thinking environment is a powerful lever for achieving efficient and scalable reasoning in LLMs.
What this episode covers
Reinforcement learning (RL) methods for training Large Language Models (LLMs) to produce long chains of thought (LongCoT) are constrained by the standard thinking environment, where the state is unbounded, leading to quadratic computational costs as reasoning length increases. This paper propose Markovian Thinking, a paradigm where the reasoning policy conditions only on a constant-size state, effectively decoupling thinking length from context size and yielding linear compute and constant memory benefits. This concept is instantiated with Delethink, an RL environment that organizes reasoning into fixed-size chunks. At the chunk boundaries, the context resets, and the policy must learn to write a concise textual carryover sufficient for seamless continuation of the reasoning process in the next chunk. Models trained with Delethink, such as R1-Distill 1.5B, match or exceed the performance of LongCoT-RL while significantly reducing computational overhead, demonstrating superior test-time scaling capability far beyond their training budget. They emphasize that redesigning the thinking environment is a powerful lever for achieving efficient and scalable reasoning in LLMs.
NOW PLAYING
The Markovian Thinker
No transcript for this episode yet
Similar Episodes
Mar 31, 2026 ·54m
Mar 27, 2026 ·14m
Mar 24, 2026 ·42m
Mar 20, 2026 ·42m
Mar 17, 2026 ·41m
Mar 13, 2026 ·44m