Offline Evaluation

Also known as: batch evaluation

Offline evaluation measures model quality against a static dataset before deployment, without exposure to real users. It enables fast iteration by running a full benchmark suite against every candidate model or prompt change in CI/CD, catching regressions early. Offline eval is complementary to online evaluation: offline evals are cheap and reproducible, while online evals capture real distribution shift and user behavior that static datasets cannot represent.