LoRA

Also known as: Low-Rank Adaptation

LoRA — Low-Rank Adaptation — is a parameter-efficient fine-tuning technique that injects small trainable low-rank matrices into a frozen base model's weight matrices instead of updating all parameters. Because only the adapter matrices are trained, LoRA reduces GPU memory requirements and training time dramatically compared to full fine-tuning, while achieving comparable task performance on many benchmarks. Multiple LoRA adapters can be maintained in parallel and swapped onto the same base model at serving time.