Quantization

Also known as: model quantization, weight quantization

Quantization reduces the numerical precision of model weights and activations from the training-time standard of 32-bit or 16-bit floating point to lower bit-width formats such as 8-bit integer or 4-bit NF4. This shrinks the model's memory footprint and accelerates inference on hardware with limited bandwidth, often with small accuracy degradation for well-calibrated quantization schemes. In fine-tuning, quantization is used in methods like QLoRA to keep the base model in GPU memory while the adapter trains in higher precision.