Evaluation Harness

An evaluation harness is the automated framework that runs a suite of test cases against a model or prompt configuration and computes quality metrics. It ingests a dataset of inputs with expected outputs or rubrics, calls the model under test, scores each response, and aggregates results into a report. Harnesses are run both during CI to catch regressions before deployment and periodically in production to detect model drift as the underlying model or data distribution shifts.