EP008: RoBERTa Proves BERT Was Just Undertrained

The paper "RoBERTa: A Robustly Optimized BERT Pretraining Approach" presents a replication study of BERT which finds that the original model was significantly undertrained.To address this, the authors introduce RoBERTa, an improved training recipe that modifies BERT in the following ways:• Training Methodology: It utilizes dynamic masking rather than static masking, removes the next sentence prediction (NSP) objective, and trains on longer sequences.• Scale: The model is trained for longer periods with larger mini-batches and learning rates.• Data: It uses a significantly larger dataset totaling over 160GB of uncompressed text, including a new dataset collected by the authors called CC-NEWS.By implementing these design choices, RoBERTa matches or exceeds the performance of all post-BERT methods published at the time, achieving state-of-the-art results on the GLUE, RACE, and SQuAD benchmarks. The authors conclude that with these optimizations, BERT's original masked language modeling objective is competitive with newer alternatives.