EPISODE · Apr 16, 2025 · 17 MIN
Adaptive SVD for Continual Learning in Large Language Models
from Neural intel Pod · host Neuralintel.org
This research addresses the challenge of catastrophic forgetting in large language models during continual learning, where adapting to new tasks degrades performance on old ones. To overcome this, the authors introduce a novel approach that utilizes adaptive singular value decomposition (SVD) to identify and preserve important knowledge while allowing flexible learning of new information. Their method dynamically determines task-specific low-rank parameter subspaces for updates, ensuring these updates remain orthogonal to critical directions learned from prior tasks. This constrained full fine-tuning technique achieves state-of-the-art results on various benchmarks, demonstrating effective knowledge retention and adaptation without increasing the model's parameter count or requiring storage of past gradients.
What this episode covers
This research addresses the challenge of catastrophic forgetting in large language models during continual learning, where adapting to new tasks degrades performance on old ones. To overcome this, the authors introduce a novel approach that utilizes adaptive singular value decomposition (SVD) to identify and preserve important knowledge while allowing flexible learning of new information. Their method dynamically determines task-specific low-rank parameter subspaces for updates, ensuring these updates remain orthogonal to critical directions learned from prior tasks. This constrained full fine-tuning technique achieves state-of-the-art results on various benchmarks, demonstrating effective knowledge retention and adaptation without increasing the model's parameter count or requiring storage of past gradients.
NOW PLAYING
Adaptive SVD for Continual Learning in Large Language Models
No transcript for this episode yet
Similar Episodes
Mar 14, 2026 ·23m
Mar 11, 2026 ·16m
Feb 28, 2026 ·14m