Low-Rank Adaptation (LoRA)

This technical paper proposes a novel technique called Low-Rank Adaptation (LoRA) for adapting large language models (LLMs) to specific downstream tasks. LoRA addresses the challenge of fine-tuning LLMs, which requires updating all model parameters, by injecting low-rank decomposition matrices into each layer of the Transformer architecture. This significantly reduces the number of trainable parameters, resulting in a substantial decrease in storage requirements, memory usage, and training time. The paper shows that LoRA performs comparably or even better than fine-tuning on various tasks, including natural language understanding (NLU) and generation (NLG), while providing additional benefits such as efficient task switching and lower hardware barrier to entry. The paper concludes by investigating the low-rank structure of model updates, providing insights into the effectiveness of LoRA and the underlying mechanisms of model adaptation.