LowRA: Accurate and Efficient LoRA Fine-Tuning of LLMs under 2 Bits

The paper that asks how few bits an adapter can survive on. Maya opens with a mosaic master firing a portrait in four tile shades, and the analogy turns out to be the method: LowRA pushes LoRA fine-tuning below two bits per parameter through three deliberate decisions — the Palette (which values the codes stand for), the Cut Lines (where bucket boundaries sit), and the Bit Budget (where bits get spent) — plus the CUDA kernels that keep the memory win from leaking back out as runtime. The reported floor: accuracy down to about 1.15 bits, with up to fifty percent memory reduction. Then the staged argument: Leo refuses to bet compliance rules on four levels per number, Maya argues that on constrained hardware the alternative to a 1.15-bit adapter is no adaptation at all, and the resolution lands on a rule — the precision budget and the testing budget move together. The hospital discharge-note summarizer returns with a whole shelf of department adapters to manage. Sources: • LowRA: Accurate and Efficient LoRA Fine-Tuning of LLMs under 2 Bits: https://arxiv.org/pdf/2502.08141 • QLoRA: Efficient Finetuning of Quantized LLMs: https://arxiv.org/pdf/2305.14314 • LoRA: Low-Rank Adaptation of Large Language Models: https://arxiv.org/pdf/2106.09685