Reusing pre-training data at test time is a compute multiplier

The academic paper investigates the efficiency of Large Language Model (LLM) pre-training by quantifying the amount of knowledge left unextracted from training datasets. The authors demonstrate that employing retrieval-augmented generation (RAG) at test time, which involves reusing the pre-training data, leads to significant accuracy improvements across benchmarks like MMLU, Math-500, and SimpleQA, even after decontamination efforts. The study establishes that retrieval acts as a compute multiplier, with performance gains for MMLU sometimes equivalent to about a 5x increase in pre-training compute alone. Furthermore, the researchers show that combining RAG with additional test-time compute techniques, such as self-consistency and reranking, yields even greater gains, suggesting substantial room for improvement in both dataset quality and current pre-training methodologies. Overall, the findings indicate that LLMs are not fully utilizing the information present in existing datasets and that retrieval offers a powerful, additive way to enhance performance.