LLM Judge Calibration

LLM judge calibration is the process of aligning an automated judge's scores with trusted human ratings before relying on it at scale. It involves scoring a labeled sample, measuring agreement, and tuning the rubric or prompt until the judge tracks human preferences closely. Calibration catches systematic quirks like leniency or position bias so downstream evaluation numbers can be interpreted with confidence.