Provable Long-Range Benefits of Next-Token Prediction

This academic paper rigorously investigates the power of next-token prediction for training large language models (LLMs), specifically focusing on Recurrent Neural Networks (RNNs). The core finding is that simply minimizing the next-token log loss during training is sufficient to yield an LLM whose output is computationally indistinguishable from the true training distribution over long sequences of up to $k$ tokens, provided the model size is sufficiently large. The authors establish this through a complexity-theoretic approach involving "distinguishers"—bounded algorithms attempting to tell the generated text from real data. Crucially, the paper introduces a self-boosting" mechanism, proving that loss minimization itself drives the model away from being distinguishable, without needing explicit knowledge or training of a distinguisher. Furthermore, the analysis provides **polynomial bounds on the required model size and bit size** needed to achieve this long-range coherence.