EP003: How ELMo Made Word Vectors Dynamic

The paper "Deep contextualized word representations" introduces a novel type of word representation called ELMo (Embeddings from Language Models). Unlike traditional word embeddings that provide a single, context-independent vector for each word, ELMo representations are deep contextualized vectors derived from all internal layers of a deep bidirectional language model (biLM) pre-trained on a large text corpus.Key aspects of the paper include:• Modeling Syntax and Semantics: ELMo effectively captures both complex characteristics of word use (syntax and semantics) and how these uses vary across linguistic contexts, such as modeling polysemy.• Deep Internal Representations: The researchers show that exposing the deep internals of the biLM is crucial; ELMo is a linear combination of all internal biLM states, which allows downstream models to select signals most useful for specific tasks. Analysis reveals that lower-level states better capture syntactic information (like part-of-speech tagging), while higher-level states capture more semantic, context-dependent information (like word sense disambiguation).• State-of-the-Art Performance: Simply adding ELMo to existing architectures significantly improved the state of the art across six challenging NLP tasks: question answering (SQuAD), textual entailment (SNLI), semantic role labeling (SRL), coreference resolution, named entity recognition (NER), and sentiment analysis (SST-5).• Efficiency: Using ELMo greatly increases sample efficiency, allowing models to reach state-of-the-art performance with significantly fewer parameter updates and smaller training sets.The authors conclude that these rich, universal representations are easily integrated into various neural NLP architectures and provide substantial gains across a broad range of language understanding problems.