Decoupling the "What" and "Where" With Polar Coordinate Positional Embeddings
from Mechanical Dreams Hosted by Mechanical Dirk
In this episode: • Introduction: Linda and Professor Norris introduce the podcast and the focus of the episode: the PoPE paper. • The Problem with RoPE: A discussion on Rotary Position Embedding and how it entangles content and positional information. • Introducing PoPE: Linda explains the mathematical shift to polar coordinates to decouple the what and the where. • Empirical Triumphs: Reviewing the massive performance jump on the Indirect Indexing task, plus music, genomics, and language modeling. • Length Extrapolation and Conclusion: Analyzing PoPE's zero-shot length extrapolation capabilities compared to YaRN, followed by episode wrap-up.
Embed this episode
No transcript yet.