← All chapters
Evaluation, Guardrails & Safety
Measuring quality, adding guardrails, and testing for regressions and failures.
30 terms
Answer AbstentionAnswer abstention evaluates whether a system correctly declines to answer questions it cannot support, rather than fabricating a response. A good benchmark includes unanswerable or out-of-scope questions and rewards the model for saying it does not know. Measuring abstention alongside accuracy captures a safety property that pure correctness metrics miss, since confidently wrong answers can be more harmful than none.Answer CompletenessAnswer completeness measures whether a response covers all parts of a multi-part question rather than addressing only the first or easiest part. It complements faithfulness and relevance: an answer can be accurate and on-topic yet still incomplete. Completeness is often scored with a rubric or an LLM judge that checks each required sub-point against the generated answer.Answer RelevanceAnswer relevance measures how directly and completely a model's response addresses the user's question. A response can be factually accurate and faithful to its source context yet still score low on relevance if it answers a related but different question, buries the key point, or omits important aspects of what was asked. Answer relevance is typically assessed by an LLM judge or by comparing the response embedding to the question embedding.BenchmarkA benchmark is a standardized evaluation suite used to compare model capabilities across the research community. Examples include MMLU for knowledge breadth, HumanEval for code generation, and TruthfulQA for factuality. Public benchmarks enable reproducible comparisons but also suffer from contamination when training data overlaps with benchmark test sets. Production teams typically supplement public benchmarks with internal task-specific evals that better reflect real user needs.CalibrationCalibration refers to whether a model's expressed confidence or probability scores accurately reflect empirical accuracy. A well-calibrated model that claims 80% confidence should be right about 80% of the time. Poor calibration — overconfidence or underconfidence — undermines reliability in high-stakes decisions. Calibration is measured with expected calibration error, reliability diagrams, and related tools; it can be improved through temperature scaling or other post-hoc methods.Evaluation DriftEvaluation drift occurs when a fixed benchmark gradually stops reflecting real production behavior, either because user queries have shifted or because the system has been tuned to overfit the static test set. It produces reassuring scores that mask real regressions. Countering drift means periodically refreshing the evaluation set from live traffic so it continues to represent the actual usage distribution.Evaluation RubricAn evaluation rubric defines the criteria and scoring scale used to judge model outputs, whether by a human annotator or an LLM judge. A rubric might specify dimensions like accuracy, completeness, tone, and groundedness, each rated on a scale such as 1–5 with anchoring examples. Explicit rubrics reduce subjectivity, improve inter-annotator agreement, and make LLM-as-judge scoring more consistent and reproducible across evaluation runs.Evaluation SamplingEvaluation sampling selects a representative subset of live traffic to score, rather than evaluating every request. Stratifying the sample across topics, difficulty, and user segments keeps the estimate cheap while staying faithful to real usage. The scored sample doubles as a growing regression set, so issues that appear in production get caught offline on the next change.Exact MatchExact match is the fraction of model outputs that match the reference answer character-for-character after normalization steps such as lowercasing and punctuation removal. It is the strictest automatic metric: a response that is correct in meaning but paraphrased the reference differently scores zero. Exact match is most appropriate for closed-form tasks such as short extractive QA, multiple-choice benchmarks, and structured data extraction where there is a single canonical correct form.Faithfulness ScoreA faithfulness score quantifies how well a generated answer is supported by the evidence provided in the input context. Each claim in the response is verified against the source passages, and the score reflects the fraction that can be attributed. High faithfulness indicates the model is grounding its answer in supplied evidence rather than hallucinating; it is a primary evaluation metric in retrieval-augmented generation systems and multi-hop QA benchmarks.Golden DatasetA golden dataset is a curated collection of inputs paired with high-quality ground-truth outputs, used as the authoritative test set for evaluating a model or system. Each example is carefully reviewed — often by subject-matter experts — to ensure correctness and representativeness. Golden datasets underpin regression testing because any performance change against them signals a meaningful shift in model behavior rather than noise.Ground TruthGround truth is the authoritative, human-verified correct answer or label used as the reference point in an evaluation. A model's output is compared against ground truth to compute objective metrics such as exact match, F1, or semantic similarity. The quality of ground truth directly limits evaluation reliability: noisy or inconsistently labeled references produce misleading metrics even when the underlying model is performing well.GuardrailA guardrail is a component that intercepts, scores, or blocks model inputs or outputs that violate defined policies — such as requests for harmful content, personally identifiable information leakage, or off-topic responses. Guardrails may be implemented as classifiers, rule-based filters, or a secondary LLM call that reviews the primary model's output before it reaches the user. Evaluating guardrail effectiveness requires both pass-rate testing on legitimate inputs and adversarial testing to measure bypass resistance.Hallucination DetectionHallucination detection is the process of identifying claims in a model's output that are factually incorrect, unsupported by source documents, or internally contradictory. Detection methods range from automated NLI-based entailment checks and LLM judges to human expert review. In RAG systems, hallucination detection overlaps with faithfulness measurement, but it also targets errors introduced from parametric knowledge rather than retrieval gaps.Human EvaluationHuman evaluation collects judgments from annotators on model output quality along dimensions like accuracy, helpfulness, clarity, and safety. It remains the gold standard because humans can catch subtle reasoning errors, cultural nuances, and novel failure modes that automated metrics miss. Human evaluation is expensive and slow, so it is typically reserved for model release decisions, calibrating automated judges, and diagnosing failure categories surfaced by cheaper evals.Inter-Annotator AgreementInter-annotator agreement quantifies how consistently different human annotators apply the same labels or scores to the same model outputs. High agreement — measured with Cohen's kappa, Krippendorff's alpha, or percent agreement — validates that the evaluation task and rubric are clear and that the resulting labels are reliable. Low agreement signals ambiguous instructions or inherently subjective evaluation criteria, and any metrics built on such labels carry significant noise.Jailbreak TestingJailbreak testing specifically probes whether adversarial prompt manipulations can bypass a model's safety guidelines and elicit content it was trained to refuse. Techniques include role-play framing, instruction injection, token-smuggling, and multi-turn manipulation. Systematic jailbreak testing generates a library of attack patterns that can be run as a regression suite whenever the model or its prompt guard is updated, tracking whether known bypasses have been closed.LLM Judge CalibrationLLM judge calibration is the process of aligning an automated judge's scores with trusted human ratings before relying on it at scale. It involves scoring a labeled sample, measuring agreement, and tuning the rubric or prompt until the judge tracks human preferences closely. Calibration catches systematic quirks like leniency or position bias so downstream evaluation numbers can be interpreted with confidence.LLM-as-JudgeLLM-as-judge uses a capable language model — typically a frontier model — to score or rank the outputs of another model being evaluated. The judge reads the model's response alongside a rubric and optionally the source context, then produces a score, verdict, or comparison decision. It scales cheaply to large eval sets and can capture nuanced quality dimensions that rule-based metrics miss, but it inherits the judge model's own biases and blind spots.Offline EvaluationOffline evaluation measures model quality against a static dataset before deployment, without exposure to real users. It enables fast iteration by running a full benchmark suite against every candidate model or prompt change in CI/CD, catching regressions early. Offline eval is complementary to online evaluation: offline evals are cheap and reproducible, while online evals capture real distribution shift and user behavior that static datasets cannot represent.Online EvaluationOnline evaluation measures model quality on real traffic after deployment, using signals such as user thumbs-up or thumbs-down ratings, session completion rates, implicit engagement metrics, or shadow scoring by an LLM judge. It captures distribution shift — the real inputs users send often differ substantially from curated eval sets. Online evaluation requires logging infrastructure, sampling strategies, and careful metric design to distinguish quality changes from traffic composition changes.Pairwise ComparisonPairwise comparison presents two model outputs for the same input and asks an evaluator — human or LLM — to pick the better response or declare a tie. It sidesteps the difficulty of absolute scoring by grounding judgment in relative preference, which is often easier and more consistent. Win rates across many pairs form a Bradley-Terry or Elo-style ranking, making pairwise evaluation the standard method for comparing model versions or providers.Position BiasPosition bias is the tendency of an LLM judge to favor a response based on where it appears rather than its quality — often preferring whichever option is presented first. It threatens the validity of pairwise comparisons. Mitigations include randomizing the order of candidates and running each comparison in both orderings, then only trusting judgments that remain consistent when positions are swapped.Precision, Recall, and F1Precision measures the fraction of a model's positive predictions that are correct; recall measures the fraction of actual positives the model successfully identifies. F1 is the harmonic mean of the two, balancing both concerns. In NLP evaluation, token-level F1 is a standard metric for extractive QA and named entity recognition tasks where partial matches matter, while precision and recall individually matter when the cost of false positives and false negatives differ greatly.Red TeamingRed teaming is a proactive adversarial evaluation practice in which testers deliberately try to elicit unsafe, incorrect, or policy-violating behavior from a model. Human red-teamers craft creative edge-case prompts, while automated red-teaming uses a separate model to generate adversarial inputs at scale. The findings feed into safety fine-tuning, guardrail design, and model policy updates before or after public deployment.Reference-Free EvaluationReference-free evaluation scores a model's output based solely on the input and the response itself, without requiring a gold-standard reference answer. This is valuable when ground truth is costly to produce or ambiguous — for example, evaluating an open-ended conversational response. LLM-as-judge approaches are inherently reference-free. The trade-off is that without a verified correct answer to anchor judgment, scores reflect perceived quality rather than factual correctness.Regression GateA regression gate is an automated check in the deployment pipeline that blocks a prompt or model change when its evaluation score drops below a threshold or falls behind the current version. By turning evaluation into a pass/fail gate, teams prevent silent quality regressions from reaching users and make every change prove it is at least as good as what it replaces.Regression TestingRegression testing re-runs a fixed eval suite against a new model version, prompt change, or system update to detect any degradation in previously passing cases. In LLM workflows, regressions are especially insidious because a change that improves one behavior can silently harm another. A well-maintained regression suite — anchored to a golden dataset — is the primary safeguard against shipping model updates that break production quality without anyone noticing.Semantic Similarity ScoreA semantic similarity score measures how close two texts are in meaning by comparing their embedding vectors — typically with cosine similarity — rather than requiring exact string overlap. It captures paraphrases and synonym variations that exact match misses. In NLG evaluation, semantic similarity to a reference is used as a proxy for content quality, though it does not verify factual accuracy and can score fluent but incorrect paraphrases as high-quality responses.Toxicity EvaluationToxicity evaluation detects harmful, offensive, or policy-violating content in model outputs across dimensions such as hate speech, explicit material, threats, and self-harm promotion. Evaluators combine classifier-based tools like Perspective API with human review and red-teaming. Continuous toxicity monitoring is essential in production because harmful outputs can emerge from novel input combinations that were not present during initial safety testing.