Predicting Neural Scaling Laws without Training: A Data Manifold Oracle

This paper introduces the Data Manifold Oracle (DMO), a training-free framework designed to predict neural scaling laws by analyzing raw text through compression statistics. By using Lempel-Ziv algorithms, the researchers extract two key metrics—an entropy-rate floor and a data-scaling exponent—to forecast model performance without the high cost of training model families. The authors prove an exact symbolic obstruction, demonstrating that raw text alone cannot reveal a dataset's geometric dimension without an external scale. Empirically, the DMO effectively ranks the scaling behavior and loss saturation of various corpora, including web, code, and math data. The research further extends this to DMO-Doc, a selector that identifies high-quality documents to improve pretraining and post-training outcomes. Ultimately, the work establishes that fundamental properties of machine learning performance are visible in the statistical structure of data before a single gradient step is taken.