Regression Testing for Prompts

A prompt change that improves performance on the cases you were looking at can silently degrade performance on cases you weren't. This is a regression, and it happens constantly in AI development because prompts affect the entire output distribution, not just the specific examples you were tuning on. Regression testing for prompts means running a consistent eval suite before and after any change to catch these shifts automatically.

What counts as a prompt change

• Editing the system prompt or any instruction in the user template
• Changing the model version (even minor version bumps can shift behavior)
• Modifying the retrieval strategy, reranking logic, or context window size in a RAG pipeline
• Adding or removing few-shot examples from the prompt
• Changing generation parameters: temperature, top-p, max tokens

A prompt version control workflow

Treat every prompt as a versioned artifact. Store prompts in plain text or YAML files in your repository alongside the eval results they produce. When you propose a change, run the eval suite against both the current prompt and the candidate, compare the results, and require the candidate to meet or beat the current score on all major categories before merging.

def compare_prompts(
    baseline_prompt: str,
    candidate_prompt: str,
    dataset: list[dict],
) -> dict:
    baseline_scores = [score(run(baseline_prompt, ex["input"]), ex) for ex in dataset]
    candidate_scores = [score(run(candidate_prompt, ex["input"]), ex) for ex in dataset]

    delta = {
        "mean_delta": mean(candidate_scores) - mean(baseline_scores),
        "regressions": [
            ex["id"]
            for ex, b, c in zip(dataset, baseline_scores, candidate_scores)
            if c < b - 0.1  # more than 10% worse
        ],
    }
    return delta

Gating deployments on eval results

The most mature teams gate prompt deployments the same way they gate code deployments: by running the eval suite in CI and blocking merge if scores fall below defined thresholds. This requires fast evals (under five minutes ideally) and clear pass/fail criteria. For slower LLM-judge evals, run them asynchronously and report results as a required check on the pull request.

Handling the long tail

Aggregate scores can mask category-level regressions. A new prompt might improve summarization quality by 10% on average while halving accuracy on medical terminology inputs. Segment your eval results by task type, input length, topic domain, and any other relevant dimension. Require improvements to hold across all segments before shipping.