← All chapters
Fine-Tuning & Alignment
Adapting models with fine-tuning, preference optimization, and alignment.
23 terms
AdapterAn adapter is a small trainable module inserted between or alongside frozen layers of a pretrained model to enable task-specific adaptation without modifying the original weights. Classic adapters add two linear projections with a bottleneck dimension inside each transformer layer. During fine-tuning only adapter weights are updated; at inference, the base model and adapter run together. LoRA is a modern variant that applies a rank-decomposed adapter to the weight matrices directly, achieving similar efficiency.AlignmentAlignment in the context of language model training refers to the process of ensuring a model's behavior matches intended human values, goals, and policies — covering dimensions such as helpfulness, honesty, and harmlessness. Alignment techniques include instruction tuning, RLHF, DPO, and constitutional AI approaches. Alignment is distinct from capability: a highly capable model that acts deceptively or pursues unintended goals is misaligned even if it performs well on benchmarks.Catastrophic ForgettingCatastrophic forgetting occurs when a neural network significantly degrades its performance on previously learned tasks after being fine-tuned on new data. In LLM fine-tuning, a model trained too aggressively on a narrow domain can lose general language capabilities and reasoning skills present in the base model. Mitigation strategies include using lower learning rates, mixing original pretraining data into the fine-tuning corpus, and using PEFT methods that preserve most base model weights unchanged.Checkpoint MergingCheckpoint merging combines the weights of two or more independently fine-tuned model checkpoints into a single model, often by taking a linear interpolation of their parameters. Techniques like SLERP and TIES merging can blend capabilities from different fine-tuning runs — for example, merging a code-focused checkpoint with an instruction-following checkpoint — without rerunning expensive training. The resulting merged model can exhibit complementary strengths, though careful validation is needed to confirm merged capabilities are retained.Dataset FormattingDataset formatting refers to the structure imposed on training examples before they are fed into a fine-tuning run — including how system messages, user turns, and assistant responses are arranged using the model family's chat template, how special tokens mark role boundaries, and how output fields are masked so the loss is computed only on tokens the model should learn to generate. Incorrect formatting is one of the most common causes of poor fine-tuning results because the model trains on the wrong supervision signal.DistillationKnowledge distillation trains a smaller student model to mimic the output distribution of a larger teacher model, transferring capability while dramatically reducing inference cost. Instead of training the student only on hard labels, distillation uses the teacher's soft probability outputs — or intermediate representations — as a rich training signal. Distillation is widely used to produce efficient deployment models and can be combined with fine-tuning to adapt a student to a specific domain while retaining quality from a frontier teacher.DPODirect Preference Optimization is an alignment technique that trains a language model directly on preference pairs — chosen and rejected responses — without requiring a separate reward model or reinforcement learning. DPO reformulates the RLHF objective into a classification loss on the model's own log-probabilities, making training simpler, more stable, and computationally cheaper than standard RLHF. It has become a popular alternative for preference alignment, especially when reward model training is impractical.EpochAn epoch is one complete pass through the full fine-tuning dataset during training. Because fine-tuning datasets are typically small, models are often trained for multiple epochs — but too many epochs on a small, homogeneous dataset leads to overfitting. Common fine-tuning practice uses 1–5 epochs with careful monitoring of validation loss; instruction-tuning runs frequently converge in 2–3 epochs and plateau or degrade thereafter without additional data diversity.Fine-TuningFine-tuning is the process of continuing to train a pretrained language model on a smaller, task-specific dataset to adapt its behavior for a target application. Unlike prompting, fine-tuning modifies the model's weights, allowing it to internalize patterns, formats, and domain knowledge that would be impractical to convey in a prompt alone. The result is a specialized model variant that often outperforms the base model on the target task while requiring fewer runtime tokens for instruction.Fine-Tuning vs RAGThe fine-tuning versus RAG decision determines whether to adapt a model by baking knowledge into its weights through training or by fetching knowledge at inference time. Fine-tuning is preferred for internalizing stable task formats, styles, and procedural behaviors that do not change frequently. RAG is preferred for accessing current, verifiable, or large-scale factual knowledge that would require retraining to update. Many production systems combine both: fine-tuning for behavioral shaping and RAG for factual grounding.Full Fine-TuningFull fine-tuning updates every parameter in the base model during training, as opposed to PEFT methods that freeze most weights. It offers the greatest flexibility and can achieve the deepest behavioral adaptation, but demands significant GPU memory — often requiring multiple high-end accelerators for models over a few billion parameters. Full fine-tuning also carries a higher risk of catastrophic forgetting and typically requires careful learning rate scheduling to avoid destroying pretraining knowledge.Gradient AccumulationGradient accumulation simulates training with a larger batch size than fits in GPU memory by accumulating gradients across multiple forward-backward passes before performing a single optimizer update. This allows fine-tuning large models on limited hardware while preserving the training dynamics that come with larger effective batch sizes, such as more stable gradient estimates. Accumulated gradient steps trade throughput for memory efficiency and are commonly combined with mixed-precision training.Instruction TuningInstruction tuning fine-tunes a pretrained language model on a dataset of instruction-response pairs so it learns to follow natural-language directions reliably. Without instruction tuning, a raw pretrained model predicts the next token but does not understand that it should answer questions, complete tasks, or follow constraints. Instruction tuning transforms the base model into an assistant that responds helpfully to diverse directives, and is the first stage of most production LLM alignment pipelines.Learning RateThe learning rate controls how large an update step the optimizer takes in the direction of the gradient during each training iteration. In fine-tuning, choosing the right learning rate is critical: too high and the model quickly overwrites pretraining knowledge, causing catastrophic forgetting; too low and convergence is impractically slow. Fine-tuning typically uses much smaller learning rates than pretraining, often combined with warmup schedules and cosine decay to stabilize training on small datasets.LoRALoRA — Low-Rank Adaptation — is a parameter-efficient fine-tuning technique that injects small trainable low-rank matrices into a frozen base model's weight matrices instead of updating all parameters. Because only the adapter matrices are trained, LoRA reduces GPU memory requirements and training time dramatically compared to full fine-tuning, while achieving comparable task performance on many benchmarks. Multiple LoRA adapters can be maintained in parallel and swapped onto the same base model at serving time.OverfittingOverfitting occurs when a model's training loss continues to decrease while its performance on held-out validation data stagnates or degrades, indicating the model has memorized training examples rather than learning generalizable patterns. In LLM fine-tuning, overfitting often manifests as verbatim repetition of training phrases, inflexible outputs, or a model that handles only seen formats. It is mitigated by early stopping, data augmentation, regularization techniques, and keeping fine-tuning datasets diverse.PEFTParameter-efficient fine-tuning is a family of methods that adapt a pretrained model by training only a small number of additional or selected parameters rather than the entire weight set. PEFT approaches include LoRA, prefix tuning, prompt tuning, and adapter insertion. By reducing the number of trainable parameters by orders of magnitude, PEFT lowers hardware requirements, reduces catastrophic forgetting risk, and makes it practical to maintain many task-specific model variants without storing full copies.Preference DataPreference data consists of records that pair the same prompt with two or more model responses labeled by human annotators according to which response is better — or ranked across quality dimensions such as helpfulness, safety, and honesty. It is the foundational input for both RLHF reward model training and DPO alignment fine-tuning. The quality and diversity of preference data strongly determines the final model's alignment with the intended values and desired behaviors.QLoRAQLoRA combines 4-bit quantization of the frozen base model with LoRA adapter training, making it possible to fine-tune large language models on a single consumer GPU. The base model's weights are stored in NF4 format to minimize memory, while the trainable LoRA matrices are kept in higher precision. QLoRA has dramatically democratized fine-tuning by bringing it within reach of researchers and practitioners without access to large-scale GPU clusters.QuantizationQuantization reduces the numerical precision of model weights and activations from the training-time standard of 32-bit or 16-bit floating point to lower bit-width formats such as 8-bit integer or 4-bit NF4. This shrinks the model's memory footprint and accelerates inference on hardware with limited bandwidth, often with small accuracy degradation for well-calibrated quantization schemes. In fine-tuning, quantization is used in methods like QLoRA to keep the base model in GPU memory while the adapter trains in higher precision.Reward ModelA reward model is a language model trained to predict a scalar quality score for a given prompt-response pair, representing how much a human would prefer that response. It is trained on human-labeled preference data and then used as a proxy evaluator during RL fine-tuning to provide a reward signal without requiring constant human feedback. Reward model quality is a bottleneck in the RLHF pipeline: a poorly calibrated reward model leads to reward hacking, where the policy exploits scoring loopholes rather than genuinely improving.RLHFRLHF trains a language model to produce outputs that humans prefer by using human preference data to train a reward model, then optimizing the LLM's policy against that reward with reinforcement learning — typically Proximal Policy Optimization. The human feedback phase involves annotators comparing pairs of model responses; the reward model learns to predict those preferences; and RL fine-tuning updates the LLM to maximize predicted reward. RLHF is responsible for much of the helpfulness and safety improvement seen in modern assistant models.SFTSupervised fine-tuning trains a language model on a curated dataset of prompt-completion pairs using standard cross-entropy loss, where the model learns to predict the correct completion token by token. SFT is typically the first adaptation stage after pretraining and is used for instruction following, task specialization, and style transfer. The quality and format of the SFT dataset directly shape the model's output style and capability ceiling before any preference alignment stage.