Continual Learning of Large Language Models: A Comprehensive Survey

Topic 4 closes by stretching specialization across time. Maya opens with an interpreter in Lisbon losing words in her own first language — first-language attrition as the human face of catastrophic forgetting — and the survey behind the episode maps the machine version: erosion with good manners, where the model stays fluent while old domains, formats, and safety behaviors quietly slip. The hosts walk the survey's two axes (vertical apprenticeship versus horizontal keeping-up), its three training stages, and four named defense families — the Songbook, the Tether, the New Wing, and the Diet — then stage the field's real argument: Leo's case that weights are a terrible database versus Maya's case that retrieval hands the model a note card without changing what the model is. The resolution is a rule of half-life, and the hospital discharge-note summarizer returns to show what a trustworthy update lifecycle actually looks like. Sources: • Continual Learning of Large Language Models: A Comprehensive Survey: https://arxiv.org/pdf/2404.16789 • LoRA: Low-Rank Adaptation of Large Language Models: https://arxiv.org/pdf/2106.09685 • QLoRA: Efficient Finetuning of Quantized LLMs: https://arxiv.org/pdf/2305.14314 • LowRA: Accurate and Efficient LoRA Fine-Tuning of LLMs under 2 Bits: https://arxiv.org/pdf/2502.08141