Position Bias

Position bias is the tendency of an LLM judge to favor a response based on where it appears rather than its quality — often preferring whichever option is presented first. It threatens the validity of pairwise comparisons. Mitigations include randomizing the order of candidates and running each comparison in both orderings, then only trusting judgments that remain consistent when positions are swapped.