Skip to content
“How can LLM RL Work Despite Information-Theoretic Inefficiency” by beren episode artwork

Oct 16, 2026 19 min

“How can LLM RL Work Despite Information-Theoretic Inefficiency” by beren

from LessWrong (30+ Karma)

Cross-posted from my personal blog The success of RL in LLMs has been puzzling me for a while. People have developed various information-theoretic style arguments by which they argue that RL is extremely informationally inefficient compared to pretraining, that it can only impart a tiny amount of bits, and that it can only bring out behaviours that are already in the base models etc. The basic intuition here is extremely obvious and essentially falls immediately out of the formulation of the two methods. Pretraining (and SFT etc) compute a loss on every token so every token gives them information. RL, of the policy gradient style that works for LLMs, computes a single scalar reward, often binary, denoting success or failure at the end of a long rollout which can often be hundreds of thousands of tokens long. In a naive sense, this means that RL accumulates 1/rollout_length bits per sample compared to pretraining. What is worse is that computing these rollouts requires decoding the actual model which is much more expensive than prefill due to the memory-bandwidth-limited nature of transformer inference at long sequence lengths. This makes RL doubly inefficient -- first we waste substantial compute doing rollouts vs [...] The original text contained 13 footnotes which were omitted from this narration. --- First published: October 16th, 2026 Source: https://www.lesswrong.com/posts/TbrmHshmcQMobveD3/how-can-llm-rl-work-despite-information-theoretic --- Narrated by TYPE III AUDIO.

Episode details come from the podcast's RSS feed.

Embed this episode

Ready to play

“How can LLM RL Work Despite Information-Theoretic Inefficiency” by beren

0:00 19:47

No transcript yet.

Requests help us decide what to transcribe next.

Frequently asked questions

How long is this episode of LessWrong (30+ Karma)?

This episode is 19 minutes long.

When was this LessWrong (30+ Karma) episode published?

This episode was published on October 16, 2026.

Can I download this LessWrong (30+ Karma) episode?

Yes. Use the download control on the episode player to save the publisher-provided media file.