DPO

Also known as: Direct Preference Optimization

Direct Preference Optimization is an alignment technique that trains a language model directly on preference pairs — chosen and rejected responses — without requiring a separate reward model or reinforcement learning. DPO reformulates the RLHF objective into a classification loss on the model's own log-probabilities, making training simpler, more stable, and computationally cheaper than standard RLHF. It has become a popular alternative for preference alignment, especially when reward model training is impractical.