Overfitting

Overfitting occurs when a model's training loss continues to decrease while its performance on held-out validation data stagnates or degrades, indicating the model has memorized training examples rather than learning generalizable patterns. In LLM fine-tuning, overfitting often manifests as verbatim repetition of training phrases, inflexible outputs, or a model that handles only seen formats. It is mitigated by early stopping, data augmentation, regularization techniques, and keeping fine-tuning datasets diverse.