Explainer
LoRA (low-rank adaptation)
Train a small correction beside unchanged base weights to save training memory.
Parameter-efficient fine-tuning (PEFT) saves training memory by updating a small share of the parameters and leaving the rest frozen, meaning unchanged (Hugging Face PEFT). LoRA, low-rank adaptation, trains a small set of extra numbers called an adapter. The adapter supplies a correction to the original model. LoRA represents that correction using two small matrices, tables of numbers, beside each chosen original matrix. Training fewer numbers reduces the memory needed for gradients and optimizer records. The original weights still need to be stored.
Going deeper
The rank is the shared width of the two thin matrices: how many independent directions the correction can use. It does not set how large the change is. A rank of 1 can still be large. A higher rank trains more numbers and uses more memory. It does not by itself mean better answers, and it is not a count of layers or of examples. As of 28 October 2026, the PEFT library (version 0.21.2) defaults to 8 and Unsloth's guide suggests 16 or 32 (huggingface.co, unsloth.ai). Alpha scales the correction by alpha divided by rank, and the QLoRA authors found that adapting all linear layers was needed to match full fine-tuning (arxiv.org). Contested: a later study found LoRA behind full fine-tuning on code and maths while forgetting less, "LoRA learns less and forgets less" (Biderman et al. 2024, arxiv.org).
In the long read: Full fine-tuning, LoRA and QLoRA. Every related term is in Fine-tuning, every concept.
Read next: QLoRA