Learning Rate

Also known as: LR, step size

The learning rate controls how large an update step the optimizer takes in the direction of the gradient during each training iteration. In fine-tuning, choosing the right learning rate is critical: too high and the model quickly overwrites pretraining knowledge, causing catastrophic forgetting; too low and convergence is impractically slow. Fine-tuning typically uses much smaller learning rates than pretraining, often combined with warmup schedules and cosine decay to stabilize training on small datasets.