Durable Execution for AI Agents: Survive a Crash at Step 9

Durable execution for AI agents means a long run can crash and pick up where it stopped, without repeating work or repeating actions.

Without it, a crash at step 9 of 12 gives you two bad choices. Start over and pay again. Or resume by hand and hope nothing ran twice.

Here is how I’d build it: save state after every step, give every side effect an idempotency key, cap your retries, and decide early whether a workflow engine earns its keep.

The Problem With Long Runs

Say an agent processes a vendor onboarding packet. It runs for 40 minutes. It reads six documents, checks two systems, drafts a record, creates the vendor in your ERP at step 8, and emails the vendor at step 9. The process dies at step 9. This example is hypothetical, but the shape is common.

What do you do now?

If you restart from step 1, you repeat 40 minutes of model calls. Worse, you might create the vendor twice. If you resume at step 9, you need to know exactly what step 8 did and whether the email already went out.

Anthropic’s engineers made the same point when they described building their research system. Agents are stateful and errors compound. Agents run for long periods and keep state across many tool calls, so they need durable execution. You cannot just restart from the beginning, because restarts are expensive. Anthropic built systems that resume from where the agent was, and combined the model’s adaptability with deterministic safeguards such as retry logic and regular checkpoints.

That last sentence is the whole design. Let the model do the thinking. Let boring, deterministic code do the bookkeeping.

What Durable Execution for AI Agents Means

Durable execution is a pattern: record each completed step so a crashed process can resume from the last record instead of from the start. Our durable execution glossary entry has the short definition.

Three parts matter.

1. A state store. A database or engine that survives your process dying.
2. Checkpoints. After each step finishes, write its result to the store before moving on.
3. Replay or resume. On restart, load the record. Skip steps that are done. Continue at the first unfinished step.

AWS describes the same mechanism for Lambda durable functions, launched in December 2025. Steps add automatic checkpointing and retries. When execution ends unexpectedly, Lambda resumes from the last checkpoint and replays the handler from the beginning while skipping completed operations. After a step completes, it is skipped on replay. Waits suspend the function without compute charges, for up to a year. Temporal’s AI documentation describes the same idea: a workflow resumes automatically after a crash, a network timeout, or a multi-day wait for a human to approve a step.

What to checkpoint

Save what you need to resume and to explain what happened.

• Run ID, step index, and status per step. Where are we?
• Each step’s input and output. The exact tool arguments and the result.
• Model outputs that drove decisions. Models are not deterministic. If a replay re-asks the model, it may choose a different action. Store the answer and reuse it.
• Side-effect records. Which external actions ran, with their idempotency keys and returned IDs.
• Version info. Which prompt and code version started the run.

Waiting for a human

Long runs often pause for approval. A good durable design makes this free: save state, stop the process, resume on the human’s answer. See human-in-the-loop AI agents for where those pauses belong.

Idempotency Keys: How to Avoid Double Actions

Checkpoints solve “where was I.” Idempotency solves “did that already happen.”

There is a gap between them. Say the process dies after the step 8 ERP call but before the checkpoint write. On resume, the record says step 8 never finished. A naive retry creates the vendor twice.

The AWS Builders’ Library states the underlying problem plainly. A timeout or failure does not mean the side effects did not happen. Its view: APIs with side effects are not safe to retry unless they provide idempotency, which guarantees the side effects happen once no matter how often you retry.

Temporal says the same about its retried Activities. Their idempotency post notes that Activity execution is not atomic, so Temporal recommends Activities be idempotent, or at least that running one twice does not cause unwanted side effects. Its example builds the key from the workflow run ID and the activity ID.

The fix is a key that stays the same across retries of the same step. Copy that pattern:

• Key = run ID + step ID. Generate it when the step is first planned. Reuse it on every retry. Never generate a new one per attempt.
• Send it to the system you call, if it supports keys. Stripe’s API is the classic example: for API v1 it saves the status code and body of the first request made with a key, and later requests with the same key return the saved result. Keys are up to 255 characters, and Stripe notes they can be removed after at least 24 hours. Know the retention window of any system you depend on.
• If the target has no key support, a lookup before the write is not enough on its own. The first attempt may still land after your search, or the search may read stale data. You need a uniqueness guard that the target enforces: a unique external reference or tax ID field, so a second create fails instead of duplicating.
• If the outcome is ambiguous and there is no guard, do not retry automatically. Mark the step as unknown, reconcile it against the target later, and escalate to a person. A human review beats a duplicate payment.

Sort your tools into three groups. This is also a tool design task.

Tool type	Example	Retry rule
Read-only	Fetch a record, search a document	Retry freely, within limits
Idempotent write	Create with key, set field to value	Retry with the same key
Non-idempotent write	Send email, post payment with no key	No automatic retry. Use a uniqueness guard, or reconcile and ask a human

The non-idempotent row is where agents hurt people. Put the extra effort there.

Retry and Backoff Rules

Retries fix transient failures and create new ones if you are careless. The model may also retry on its own by calling the same tool again. Set rules at three levels.

Level 1: transport retries. Network errors, rate limits, and 5xx responses. The AWS guidance applies: use exponential backoff with jitter, so retries spread out in time instead of arriving in bursts. Retry at a single layer of the stack. In their example, independent retries at each layer multiplied the load on the database 243 times. Cap the count, and consider limiting retries with a token bucket.

Level 2: step retries. The orchestrator retries a failed step. Distinguish transient errors from permanent ones. A timeout is transient. A validation error or a “permission denied” will not fix itself. Mark permanent errors as non-retryable and fail fast.

Be careful with engine defaults. Temporal’s documented default retry policy has a 1 second initial interval, a backoff coefficient of 2.0, a maximum interval of 100 times the initial interval, and unlimited attempts. Unlimited retries are fine for a payment API that is briefly down. They are not fine for a step that costs model tokens each time. Set a maximum.

Level 3: agent-level retries. The model tries a different approach. Anthropic’s team found that telling the agent a tool is failing, and letting it adapt, works surprisingly well. I agree, with a limit. Count attempts per goal and escalate after a fixed number. A loop with no ceiling burns budget while looking busy.

A starting rule set, which you should tune against real failures:

• Transient errors: up to 5 attempts with exponential backoff and jitter.
• Permanent errors: zero retries, log and escalate.
• Agent-level retries: 3 attempts per goal, then hand off.
• Total run budget: a hard cap on steps and spend.

Those numbers are my guesses, not benchmarks.

Workflow Engine or Hand-Built Loop

You have two options. Build the checkpoint loop yourself, or adopt an engine such as Temporal, AWS Lambda durable functions, or a similar tool.

A hand-built loop can be small. A table holds run_id, step_index, status, input, output, and idempotency_key. A worker picks up unfinished runs on start, skips succeeded steps, and continues. You understand all of it.

An engine gives you retries, timers, waits for human signals, history, and replay. You pay in new concepts and another system to run.

My decision rule:

Situation	My pick
Runs under a few minutes, one or two side effects	Hand-built loop, or rerun the whole thing if every write is idempotent
Runs of 10+ minutes, 5+ steps with side effects	Seriously consider an engine
Waits of hours or days for humans	Engine, or at least a proper state store and scheduler
Many concurrent runs, strict audit needs	Engine
Small team, no one to operate it	Managed engine, or hand-built if scope is small
Multi-agent flows with handoffs	Engine, since coordination state multiplies (multi-agent systems)

Either way, the questions stay the same: where does state live, and what happens on failure? The AI agent architecture guide covers the surrounding design.

One more factor: deploys. Anthropic notes that agent systems are highly stateful and run almost continuously, so when you ship an update, agents may be anywhere in their process. They use rainbow deployments, shifting traffic gradually while old and new versions run side by side. At minimum, store the code and prompt version in each run, and let in-flight runs finish on the version they started with. See AI agent maintenance after deploy.

Failure Drills

Do not trust recovery you have not broken on purpose. Run these before launch and after major changes.

1. Kill at every step. Run the workflow once per step, killing the process at that step. Resume each time. Check for duplicates and gaps.
2. Kill between the action and the checkpoint. This is the nasty window. Crash right after the external call, before you write the result. Resume. The idempotency key or a uniqueness guard in the target must stop the second write. If neither exists, the step must go to reconciliation, not an automatic retry.
3. Timeout after success. Make the downstream system do the work but return a timeout. Does the key or uniqueness guard stop a duplicate? If the step has neither, does it stop and wait for reconciliation instead of retrying?
4. Duplicate delivery. Send the same trigger twice. Only one run should act.
5. Poison step. Make one step fail permanently. Does the run stop, record the reason, and escalate, or does it loop?
6. Deploy mid-run. Ship a new version while a run is in flight. Does the run finish cleanly?
7. Outage and silence. Block the model API for several minutes, and leave an approval unanswered. Do runs wait, resume, or time out safely?

For each drill, write the expected result before you run it. Log the actual result. Any mismatch is a bug. Keep the drills in your regression suite.

The Cost of Durability

Durability is not free. Know what you pay.

• Latency. Every checkpoint is a write. Next to a model call, that is small. In tight loops of fast tool calls, it adds up. Measure it in your setup.
• Storage. Step inputs and outputs can be large. Decide retention, and redact sensitive data in stored state.
• Complexity. Engines replay your orchestration code, so it must be deterministic. Model calls belong inside recorded steps.
• Operations. An engine is another service to run and monitor.
• Discipline. Idempotency is design work on every write tool. No engine does it for you.

Against that, count what a crash costs without it: repeated model spend, duplicate actions, and manual cleanup. For a 40-second run that is cheap to repeat, skip the engine. For a 40-minute run that touches money, durability pays off the first time something dies.

Bottom Line

A long agent run will fail. Plan the resume before you plan the happy path.

Checkpoint after every step. Give every side effect a stable idempotency key. Cap retries at every level. Pick an engine when runs are long, waits are long, or audits are strict.

Next step: list your agent’s write tools, mark which are idempotent, and run the kill-between-action-and-checkpoint drill.

FAQ

What is durable execution? Each completed step is saved, so a crashed process resumes from the last saved step instead of starting over. Temporal and AWS Lambda durable functions provide it, and a database table can do a simpler version.

How do agents resume after a crash? On restart, the system loads the run record, skips steps marked successful, and continues at the first unfinished step. It reuses stored model outputs, so the replay follows the original path.

How do I avoid double actions? Give each side effect an idempotency key built from the run ID and step ID, and reuse it on every retry. If the target system does not support keys, use a uniqueness guard it enforces, such as a unique reference field, so a second create fails. A lookup before retrying is not enough, because the first attempt can still land after the lookup. If there is no key and no guard, do not retry automatically. Reconcile the step and send it to a person.

When do I need a workflow engine? Consider one when runs last over about ten minutes, include several side effects, wait hours or days for people, run at high volume, or need strict audit trails. For short runs with one or two writes, a simple table and a safe retry rule often do the job.

How do I test failure recovery? Kill the process at every step, and specifically between an external call and its checkpoint write. Also test timeouts after success, duplicate triggers, a permanently failing step, a deploy mid-run, and an unanswered approval. Write the expected result first, then compare.

Three Ways to Work With TMA

Need an agent built? We deploy production AI agents in your infrastructure. Working pilot. Real data. Measurable ROI. → Schedule Demo

Want to co-build a product? We’re not a dev agency. We’re co-builders. Shared cost. Shared upside. → Partner with Us

Want to join the Guild? Ship pilots, earn bounties, share profit. Community + equity + path to exit. → Become an AI Architect