Explainer
Quantization
Storing a model's numbers with fewer bits, so it needs less memory.
Bit width is the space used to represent each number. Quantization represents weights with fewer bits, such as reducing 16-bit weights to 8 or 4 bits, to save memory. The exact memory saving and quality depend on the method and model. In training, QLoRA keeps the frozen base at 4 bits while small adapters train (arxiv.org). In serving, methods such as GPTQ and AWQ compress a finished model (arxiv.org, arxiv.org), and GGUF files let tools such as llama.cpp run models on ordinary hardware (huggingface.co). Quantize the exact model you will serve, then re-run your tests, safety included. If a LoRA adapter is being merged in, merge first and then quantize, because quantization can shift quality.
Going deeper
Contested: GPTQ reports negligible loss at 3 to 4 bits on very large models, while a study of LLaMA3 found non-negligible degradation, especially at very low bit widths (Huang et al. 2024, arxiv.org). The answer depends on the model, the method, the bit width and the task.
In the long read: Shipping it and keeping it working. Every related term is in Fine-tuning, every concept.
Read next: QLoRA