EPISODE · May 14, 2026 · 24 MIN
Efficient Pre-Training with Token Superposition
from Daily Paper Cast · host Jingwen Liang, Gengyu Wang
🤗 Upvotes: 30 | cs.CL Authors: Bowen Peng, Théo Gigant, Jeffrey Quesnelle Title: Efficient Pre-Training with Token Superposition Arxiv: http://arxiv.org/abs/2605.06546v1 Abstract: Pre-training of Large Language Models is often prohibitively expensive and inefficient at scale, requiring complex and invasive modifications in order to achieve high data throughput. In this work, we present Token-Superposition Training (TST), a simple drop-in method that significantly improves the data throughput per FLOPs during pre-training without modifying the parallelism, optimizer, tokenizer, data, or model architecture. TST is done in two phases: (i) A highly efficient superposition phase where we combine many contiguous tokens into one bag and train using a multi-hot cross-entropy (MCE) objective, and (ii) a recovery phase where we revert back to standard training. We extensively evaluate TST on the scale of 270M and 600M parameters and validate on 3B and a 10B A1B mixture of experts model, demonstrating that it is highly robust in different settings. Ultimately, TST consistently outperforms baseline loss and downstream evaluations, and under equal-loss settings, TST yields up to a 2.5x reduction in total pre-training time at the 10B A1B scale.
What this episode covers
🤗 Upvotes: 30 | cs.CL Authors: Bowen Peng, Théo Gigant, Jeffrey Quesnelle Title: Efficient Pre-Training with Token Superposition Arxiv: http://arxiv.org/abs/2605.06546v1 Abstract: Pre-training of Large Language Models is often prohibitively expensive and inefficient at scale, requiring complex and invasive modifications in order to achieve high data throughput. In this work, we present Token-Superposition Training (TST), a simple drop-in method that significantly improves the data throughput per FLOPs during pre-training without modifying the parallelism, optimizer, tokenizer, data, or model architecture. TST is done in two phases: (i) A highly efficient superposition phase where we combine many contiguous tokens into one bag and train using a multi-hot cross-entropy (MCE) objective, and (ii) a recovery phase where we revert back to standard training. We extensively evaluate TST on the scale of 270M and 600M parameters and validate on 3B and a 10B A1B mixture of experts model, demonstrating that it is highly robust in different settings. Ultimately, TST consistently outperforms baseline loss and downstream evaluations, and under equal-loss settings, TST yields up to a 2.5x reduction in total pre-training time at the 10B A1B scale.
NOW PLAYING
Efficient Pre-Training with Token Superposition
No transcript for this episode yet
Similar Episodes
No similar episodes found.