Fine-Tuning and Specialization: LoRA and Beyond

Topic 4 opens with the move that defines modern model specialization: freeze the giant base model and train a tiny low-rank update beside it. Maya and Leo map the topic's landmarks — the Rank Knob, the Precision Floor, the What-Got-Worse Test, and the Long Haul — and stage the field's real arguments: PEFT-everywhere versus full fine-tuning's capacity case, and how many bits the frozen base can lose before quality quietly collapses. A hospital discharge-note summarizer anchors the whole topic, setting up deep dives on LoRA, QLoRA, LowRA, and continual learning. Sources: • LoRA: Low-Rank Adaptation of Large Language Models: https://arxiv.org/pdf/2106.09685 • QLoRA: Efficient Finetuning of Quantized LLMs: https://arxiv.org/pdf/2305.14314 • LowRA: Accurate and Efficient LoRA Fine-Tuning of LLMs under 2 Bits: https://arxiv.org/pdf/2502.08141 • Continual Learning of Large Language Models: A Comprehensive Survey: https://arxiv.org/pdf/2404.16789