Preference Data

Also known as: human preference dataset, comparison data

Preference data consists of records that pair the same prompt with two or more model responses labeled by human annotators according to which response is better — or ranked across quality dimensions such as helpfulness, safety, and honesty. It is the foundational input for both RLHF reward model training and DPO alignment fine-tuning. The quality and diversity of preference data strongly determines the final model's alignment with the intended values and desired behaviors.