RLHF

Also known as: Reinforcement Learning from Human Feedback

RLHF trains a language model to produce outputs that humans prefer by using human preference data to train a reward model, then optimizing the LLM's policy against that reward with reinforcement learning — typically Proximal Policy Optimization. The human feedback phase involves annotators comparing pairs of model responses; the reward model learns to predict those preferences; and RL fine-tuning updates the LLM to maximize predicted reward. RLHF is responsible for much of the helpfulness and safety improvement seen in modern assistant models.