Evaluating Agents

Evaluating a single model call is hard enough; evaluating an agent is harder. The output of an agent is not just a text response but a trajectory — a sequence of decisions, tool calls, and observations. A good evaluation framework captures both the final outcome and the quality of the path taken to get there.

What to measure

• Task completion rate: did the agent accomplish the stated goal? (binary or graded)
• Step efficiency: how many tool calls did it take vs. the minimum needed?
• Trajectory quality: did the agent choose the right tools in a sensible order?
• Error recovery: when a tool failed, did the agent recover gracefully or give up?
• Safety compliance: did it avoid prohibited actions and escalate appropriately?

Building a task evaluation harness

An agent eval harness runs the agent against a set of tasks with known expected outcomes, captures the full trajectory, and scores each run. The key components are a task dataset, a sandboxed tool environment with mock or real tools, a scorer that grades outcomes, and a trajectory logger for inspection.

async def evaluate_agent(tasks: list[dict]) -> dict:
    results = []
    for task in tasks:
        trajectory = []
        outcome = await run_agent_traced(
            task=task["prompt"],
            on_step=lambda step: trajectory.append(step),
        )
        score = grade_outcome(
            expected=task["expected"],
            actual=outcome,
            trajectory=trajectory,
        )
        results.append({"task": task["id"], "score": score, "steps": len(trajectory)})
    return {
        "completion_rate": sum(r["score"] > 0.5 for r in results) / len(results),
        "mean_steps": sum(r["steps"] for r in results) / len(results),
        "results": results,
    }

LLM-as-judge for trajectory evaluation

A human reviewing every trajectory is too slow for CI. Use a stronger model as a judge to score trajectories automatically. Give the judge the task description, the expected outcome, and the full trajectory, and ask it to rate efficiency, correctness, and safety on a rubric. Validate the judge against a held-out human-labeled set to confirm correlation.

Regression testing and CI integration

Run your eval suite whenever the system prompt, tool definitions, or model version changes. A drop in completion rate or a rise in average step count is a regression signal. Treat the eval suite as a test suite: it should be fast enough to run in CI and have clear pass/fail thresholds that gate deployments.