EP025: RoPE Solves Sequence by Rotating Vectors

"RoFormer: Enhanced Transformer with Rotary Position Embedding"Core Innovation: Rotary Position Embedding (RoPE) The paper proposes RoPE to address the limitation that standard Transformer models are position-agnostic. Unlike previous methods that add positional embeddings to word vectors, RoPE encodes absolute positions by multiplying context representations (queries and keys) with a rotation matrix. This mathematical formulation ensures that the self-attention mechanism naturally captures relative position dependencies based on the difference in rotations between tokens.Key Advantages• Decaying Dependency: RoPE models the intuition that the connection strength between tokens should decrease as their relative distance increases.• Flexibility & Compatibility: The method accommodates varying sequence lengths and, unlike many relative position encoding schemes, is compatible with linear self-attention architectures like Performer.Performance The enhanced model, RoFormer, demonstrated consistent improvements over baselines such as BERT and the standard Transformer:• Faster Convergence: It achieved lower loss and faster convergence during pre-training.• Better Translation: It surpassed the baseline Transformer in English-to-German machine translation tasks.• Long Text Handling: RoFormer significantly outperformed BERT and WoBERT on long text classification tasks (e.g., Chinese legal documents), especially as sequence lengths increased to 1024 tokens.