LoRA Without Regret

The paper from Thinking Machines Lab provides a comprehensive analysis of Low-Rank Adaptation (LoRA), a parameter-efficient fine-tuning (PEFT) method for large language models. The research demonstrates that when applied correctly — specifically across all layers, including MLP/MoE layers — LoRA can achieve the same performance and sample efficiency as full fine-tuning (FullFT), particularly in typical post-training scenarios that are not capacity constrained. Key practical findings include that LoRA is significantly more memory-efficient and that its optimal learning rate is consistently about ten times higher than that for FullFT. The authors establish a "low-regret regime" where LoRA's performance matches FullFT, especially noting its equivalence in reinforcement learning (RL) tasks, which require comparatively little capacity. The text also investigates technical details such as batch size effects and the invariance of certain LoRA hyperparameters.