EP012: Google T5 Turns Every Task Into Text

The paper "Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer" presents a comprehensive empirical survey of transfer learning techniques in Natural Language Processing (NLP). The authors introduce a unified framework that casts every text processing problem—including translation, question answering, and classification—as a "text-to-text" task, where the model is fed input text and trained to generate target text.Key contributions and findings include:• Unified Framework: By treating all tasks as text-to-text, the authors could apply the same model, objective, and training procedure across diverse benchmarks. They introduce the "Text-to-Text Transfer Transformer" (T5) for this purpose.• Systematic Study: The paper conducts extensive experiments comparing different model architectures, pre-training objectives, unlabeled datasets, and training strategies. The study found that a standard encoder-decoder architecture using a "denoising" objective (reconstructing corrupted text) performed best.• C4 Dataset: The authors released the "Colossal Clean Crawled Corpus" (C4), a massive dataset of clean English text scraped from the web, to facilitate pre-training at scale.• State-of-the-Art Results: By combining the insights from their study with massive scale—training models with up to 11 billion parameters on over 1 trillion tokens—the authors achieved state-of-the-art performance on benchmarks such as GLUE, SuperGLUE, SQuAD, and CNN/Daily Mail.Overall, the paper demonstrates that a simple text-to-text approach, when scaled up with large models and datasets, can yield effective general language understanding.