Online Evaluation

Also known as: live evaluation, production evaluation

Online evaluation measures model quality on real traffic after deployment, using signals such as user thumbs-up or thumbs-down ratings, session completion rates, implicit engagement metrics, or shadow scoring by an LLM judge. It captures distribution shift — the real inputs users send often differ substantially from curated eval sets. Online evaluation requires logging infrastructure, sampling strategies, and careful metric design to distinguish quality changes from traffic composition changes.