EP010: ALBERT Outperforms BERT With Parameter Sharing

The paper, "ALBERT: A Lite BERT for Self-supervised Learning of Language Representations," introduces a modified BERT architecture designed to address the memory limitations and training speed issues encountered when scaling up natural language processing models.The authors achieve this through two primary parameter-reduction techniques:• Factorized embedding parameterization: This separates the size of the hidden layers from the vocabulary embedding size by decomposing the large embedding matrix into two smaller matrices. This allows the hidden layer size to be increased without significantly inflating the parameter count.• Cross-layer parameter sharing: This technique shares parameters across all layers of the network, preventing the model size from growing with network depth.Additionally, ALBERT replaces the original BERT's "next sentence prediction" (NSP) loss with a sentence-order prediction (SOP) loss. The authors argue that SOP is a more challenging task that better models inter-sentence coherence, leading to consistent improvements in downstream tasks.