Idempotent, Replayable Runs

Also known as: durable execution, checkpointing, exactly-once side effects

Checkpoint each step and key every side effect, so a crashed or retried run resumes instead of restarting or double-charging.

9 min read

The problem

Agent runs are long, expensive, and non-deterministic, and processes die in the middle of them. Without durable state, a crash at step nine of twelve throws away everything and restarts from zero — re-paying for every model call and, far worse, re-executing every side effect. The customer gets refunded twice, the email goes out twice, the record is created twice. Retries make it worse, not better.

The pattern

Persist run state after every step so a resumed run continues from the last checkpoint rather than the beginning. Give every side-effecting operation a deterministic idempotency key derived from the run and step, and have the executing service deduplicate on it. Replay then becomes safe: repeated steps return their recorded result instead of executing again.

Use when

Runs are long enough that losing one is materially expensive
Tools have irreversible side effects — payments, messages, provisioning
Runs suspend for human approval and must survive a deployment
You need to debug production runs by replaying them exactly

Avoid when

Runs are short, cheap, and free of side effects — restarting is simpler
All operations are naturally idempotent reads
The infrastructure cost of durable state exceeds the cost of occasional re-runs

Tradeoffs

GainCrashes and deploys stop destroying in-flight work, and retries stop duplicating effects
CostA persistence layer on the hot path, with its own latency and failure modes
GainRuns become replayable, which makes production debugging dramatically easier
CostCheckpoints hold potentially sensitive data and need retention and access policy
GainSuspension for human approval becomes natural rather than exceptional
CostState schema changes must be versioned or in-flight runs break on deploy

This is standard durable-execution practice applied to agents. What makes it more pressing here is that the steps are unusually expensive and the side effects are unusually easy to trigger by accident.

Checkpoint after every step

A checkpoint is the run's full resumable state: the message history, the step index, accumulated results, and a status. Write it after each step completes, before the next begins. Version the schema — long-running suspended runs will otherwise break the first time you deploy a change to the state shape.

class Checkpoint(BaseModel):
    schema_version: int = 3
    run_id: str
    step: int
    status: Literal["running", "awaiting_approval", "done", "failed"]
    messages: list[Message]
    completed_effects: dict[str, str]   # idempotency_key -> recorded result
    updated_at: datetime


async def run(run_id: str, task: str, max_steps: int = 20) -> Result:
    cp = await store.load(run_id) or Checkpoint(
        run_id=run_id, step=0, status="running",
        messages=[Message(role="user", content=task)], completed_effects={},
        updated_at=now(),
    )
    if cp.schema_version < 3:
        cp = migrate(cp)

    while cp.step < max_steps and cp.status == "running":
        cp = await execute_step(cp)
        await store.save(cp)          # durable before the next step starts

    return Result(status=cp.status, messages=cp.messages)

Deterministic idempotency keys

The key must be identical across replays of the same logical operation, which means it must derive from the run identity and the step — never from a timestamp or a random value. Include a hash of the arguments so that a genuinely different call at the same step gets a different key.

def idempotency_key(run_id: str, step: int, tool: str, args: dict) -> str:
    payload = json.dumps(args, sort_keys=True, separators=(",", ":"))
    digest = hashlib.sha256(payload.encode()).hexdigest()[:16]
    return f"{run_id}:{step}:{tool}:{digest}"   # stable across replays


async def execute_effect(cp: Checkpoint, tool: str, args: dict) -> str:
    key = idempotency_key(cp.run_id, cp.step, tool, args)

    if key in cp.completed_effects:
        # Already done in a previous attempt. Return the recorded result.
        metrics.increment("effect.replayed", tags={"tool": tool})
        return cp.completed_effects[key]

    result = await TOOLS[tool](**args, idempotency_key=key)   # provider dedupes too
    cp.completed_effects[key] = result
    await store.save(cp)                                       # record before returning
    return result

warning

Save the effect record before returning it to the loop. A crash in that window — effect executed, record not written — is precisely the case that causes a duplicate on replay, and it is the one teams most often leave open.

Two layers of protection

Local deduplication via `completed_effects` handles resumption within your system. Passing the same key to the downstream provider handles the case your checkpoint cannot see: the request reached the provider, the response was lost, and your local record was never written. Most payment and messaging APIs accept an idempotency key precisely for this; use it, and pick a key whose lifetime exceeds your maximum run duration.

Checkpoint after every step, and version the state schema
Derive idempotency keys from run id and step — never from time or randomness
Record the effect result before returning it to the loop
Pass the same key downstream so the provider deduplicates too
Set a retention policy: checkpoints hold conversation content and tool arguments

tip

Replayability is worth it for debugging alone. Loading a production run's checkpoint into a local process and stepping through it beats any amount of log archaeology when an agent has done something inexplicable.

Related patterns

Human-in-the-Loop Approval Gate Budget and Loop Ceilings Trace-Driven Failure Triage