Evaluation · 104 pages
Evaluation-Driven Development for LLMs
Build LLM features the way you build tested software — with evals that gate every prompt and model change.
Contents
- Why evals come first
- Building a golden dataset
- LLM-as-judge without fooling yourself
- Regression gates in CI
- Sampling live traffic into evals
Free preview
Most teams add evaluation after something breaks. Evaluation-driven development flips that order: you decide how you will measure quality before you write the prompt, so every change can be scored instead of argued about. It turns a subjective 'this feels better' into a number you can defend.
The core loop
- Curate a small golden dataset that represents real usage
- Define a scorer — exact match, rubric, or a calibrated LLM judge
- Run the eval on every candidate change
- Gate deployment on the score, not on vibes
A flawed judge is worse than no judge. Always validate your LLM-as-judge against human labels on a sample before trusting its scores.
Judges you can trust
def judge(question, answer, reference):
prompt = f"Score 1-5 how well the answer matches the reference.\nQ: {question}\nAnswer: {answer}\nReference: {reference}"
return int(model(prompt).strip())The book walks through calibrating this judge, wiring the eval into CI as a regression gate, and continuously refreshing the dataset from sampled production traffic so your benchmark never drifts away from what users actually ask.
Want the full guide?
Join the newsletter and we'll send you the complete guide and new releases.