Gradient Accumulation

Gradient accumulation simulates training with a larger batch size than fits in GPU memory by accumulating gradients across multiple forward-backward passes before performing a single optimizer update. This allows fine-tuning large models on limited hardware while preserving the training dynamics that come with larger effective batch sizes, such as more stable gradient estimates. Accumulated gradient steps trade throughput for memory efficiency and are commonly combined with mixed-precision training.