Pairwise Comparison

Also known as: A/B eval, side-by-side evaluation

Pairwise comparison presents two model outputs for the same input and asks an evaluator — human or LLM — to pick the better response or declare a tie. It sidesteps the difficulty of absolute scoring by grounding judgment in relative preference, which is often easier and more consistent. Win rates across many pairs form a Bradley-Terry or Elo-style ranking, making pairwise evaluation the standard method for comparing model versions or providers.