AI Agent Incident Response: A Runbook for When It Goes Wrong

Your agent will make a mistake. The question is whether you have a plan before it does.

AI agent incident response is the same loop as any incident: detect, contain, assess, fix, tell people, learn. Agents add three twists. They act across several systems. They fail quietly. And their own account of what they did cannot be trusted as evidence.

This is the short runbook I’d put on one page and print.

Why agents need their own runbook

You may already have an incident process for outages. Keep it. But an agent incident differs in ways that break a standard playbook.

The failure is often silent. A server outage throws errors. An agent that sends a wrong refund amount throws nothing. Everything looks healthy while it does damage.

Instructions are not controls. A prompt that says “do not change production” is a request. Only permissions are controls. Fortune reported on a case where a Replit coding agent reportedly deleted a live database during a declared code freeze. The agent told the user a rollback would not work. The user recovered the data by hand. Replit’s CEO called the deletion “unacceptable and should never be possible.” This is a user’s account relayed by press, so treat the details as reported, not proven. The lesson holds either way: a freeze that lives in the prompt is not a freeze, and the agent’s description of what it did is not a log.

You own what the agent says. In Moffatt v. Air Canada, a British Columbia tribunal held the airline responsible for wrong information its chatbot gave a customer. Treat agent output as a company statement.

Most teams are not ready. In a 2025 Pacific AI survey of 351 practitioners, 54% had an AI incident response playbook. It’s a small, vendor-run survey. Still, it matches what I see: policies exist, playbooks lag.

NIST’s April 2025 revision of SP 800-61 folds incident response into overall risk management rather than treating it as a separate IT chore. That is the right frame. The runbook below fits inside it.

Severity levels and who decides

Decide the levels before an incident, not during. Atlassian’s guidance says teams without defined severity levels waste the first minutes debating how bad things are. NIST’s guidance on triage says the same in other words: do a preliminary review, estimate severity and urgency, and prioritize by scope, likely impact, and time sensitivity.

Here are three levels written for agents.

Level	What happened	Examples	Response
Sev 1	Harm reached money, people, or data, or the agent acted outside its permissions	Wrong payments sent. Customer data exposed. Messages sent to customers in error. A destructive action in production	Freeze now. Wake the incident lead. Executive and legal informed
Sev 2	Wrong output or action reached records or users, reversible, limited scope	Bad updates to internal records. A batch of wrong drafts approved and sent internally	Freeze the affected action. Fix within the day
Sev 3	A human caught it before effect, or quality drifted	Reviewer rejected a wrong draft. Accuracy dips on a test set	Log, fix in normal sprint

Who decides severity: the incident lead, a named on-call person, makes the call in the first 15 minutes. The agent’s business owner can raise it. Nobody lowers a Sev 1 or 2 without the incident lead and the security or risk owner agreeing in writing.

If you are unsure, rate it higher. Google’s SRE book gives the same advice in other words: declare early. It is better to declare and close it fast than to spin up the process hours into a growing problem. Its simple triggers also work for agents: you need a second team to fix it, customers can see it, or it is still unsolved after an hour.

Roles

Borrow the roles from the same Google chapter and keep them small:

• Incident lead. Holds the whole picture and assigns work.
• Operator. The only person who changes the system during the incident.
• Communicator. Sends updates and keeps the incident document current.
• Scribe or planner. Tracks follow-ups and handoffs.

On a small team one person can hold two roles. Never let one person both lead and operate a Sev 1.

Detect

You cannot respond to what you cannot see. Ways agent incidents surface:

• A user or reviewer reports something odd
• A spike in human overrides or rejections
• A spike in tool errors, retries, or cost
• An output check or guardrail triggers
• A sample audit finds a pattern
• A vendor or third party tells you

Wire the first three into alerts. If you have no monitoring yet, start with agent observability tools and the checks in AI agent maintenance after deploy.

Write down who gets told and how. A report that sits in a shared inbox is not detection.

Contain: freeze actions

Containment comes before understanding. Stop the harm first, then ask why.

Do these in order. Stop at the earliest step that works.

1. Preserve evidence. Snapshot the logs, prompts, configuration, model and tool versions, and the current run state. Do this in a minute, before you change anything. Incident records can hold sensitive data, so NIST notes the incident lead should safeguard them.
2. Drop to draft-only. Remove or disable every write tool. The agent can still propose. It cannot act.
3. Pause triggers. Stop schedulers, queues, and inbound webhooks so no new work arrives.
4. Cut credentials. If you suspect misuse or an attack, revoke or rotate the agent’s tokens and keys.
5. Flip the kill switch. If steps 2 to 4 are not enough, shut the agent off entirely. See AI agent kill switch for how to build one before you need it.
6. Quarantine bad output. Hold unsent messages, unposted entries, and queued actions.

Two rules matter more than the rest:

• Freeze with permissions, not with instructions. Do not tell the agent to stop. Take away what it can do. This is the root of excessive agency, which OWASP traces to too much functionality, too many permissions, and too much autonomy.
• Do not ask the agent what happened and believe it. Read the logs. If you do not have logs, that is the first finding of the postmortem.

Assess impact

Once the bleeding has stopped, scope it. You are answering five questions:

1. When did it start? Find the first bad action, not the first report.
2. What did it do? List every action between the start and the freeze.
3. Who and what was affected? Records, customers, vendors, amounts, external messages.
4. Is it still wrong somewhere? Bad data may have spread to other systems.
5. Can it be undone? For each action, mark reversible, reversible with effort, or permanent.

This only works if you kept an audit trail of agent actions. Pull the log, filter by agent and time window, and build a table: action, target, time, amount, status, reversible. Hand the table to the incident lead.

Update severity as you learn more. Raise it as soon as the evidence calls for it.

Fix and verify

Fix in this order:

1. Undo the damage. Reverse the actions you can. Prepare corrections for the rest. Have a person sign off on each correction batch.
2. Fix the cause. The cause is usually one of: an overly broad permission, an ambiguous instruction, bad or stale data, a tool that returned a confusing result, or a model or vendor change you did not test.
3. Add a regression test. Put the failing case in your test set so it fails loudly next time.
4. Restore in steps. Bring the agent back in the order it was frozen, reversed: draft-only first, then with human approval on each action, then normal. Watch it at each step.
5. Verify. Re-run the failing case. Check the logs for the first 24 hours. Have the incident lead and the agent’s owner both sign the all-clear.

Do not skip step 4. Going from frozen to fully live in one move is how you get a second incident.

Communicate

Silence makes incidents worse. Decide who is told and when, as part of the runbook.

Audience	Sev 1	Sev 2	Sev 3
Incident team	Immediately	Immediately	Ticket
Agent owner and business owner	Immediately	Within the hour	Same day
Security, risk, legal	Immediately	Same day	As needed
Executives	Within the hour	Summary after	None
Affected customers or partners	As soon as facts are confirmed, reviewed by legal	If they were affected	None
Regulators	Check the legal duty with counsel	Check with counsel	None

I’m not a lawyer. Some industries and laws set deadlines for reporting certain incidents. Have counsel confirm yours before you need it.

Send updates on a fixed rhythm, such as every 30 minutes for a Sev 1. Use four lines:

What we know: <one sentence>
What we do not know yet: <one sentence>
What we have done: <one sentence>
Next update at: <time>

Do not guess in writing. Do not blame the model, the vendor, or a person in an update. State facts.

Postmortem template

Write the postmortem within a few days, while memory is fresh. Google’s SRE book defines it as a written record of the incident, its impact, the actions taken, the root causes, and the follow-up actions. It also says to set the criteria for writing one before an incident. My criteria: every Sev 1 and Sev 2, and any Sev 3 that repeats.

Keep it blameless. That does not mean no accountability. It means you assume people acted on good intent with the information they had, and you fix the system that gave them bad information or too much power. The same Google chapter says you can’t fix people, but you can fix systems and processes. Atlassian’s handbook puts the goal as never having the same incident twice, with specific changes committed by specific dates.

Copy this template:

Title:
Date / severity / duration:
Incident lead / author:

1. Summary (3 sentences: what happened, impact, current status)
2. Timeline (UTC): first bad action, detection, freeze, fix, all-clear
3. Impact: records, people, money, external messages, reversibility
4. Detection: how we found out. How long after the first bad action?
5. Root cause(s): what allowed it? Permissions, instructions, data, tools, testing, change control
6. What went well
7. What went poorly
8. Where we got lucky
9. Action items (each with an owner and a due date):
   - Prevent: remove the cause
   - Detect: find it faster next time
   - Mitigate: shrink the blast radius
   - Test: add the regression case
10. Open questions

How to prevent repeats

Action items die when they have no owner. So:

• Every item gets one named owner and a date.
• Track them on the same board as feature work.
• Review open items in the monthly agent review.
• Close the loop: when an item ships, add the regression case or alert it was meant to create.

For a pattern that keeps showing up, change the design. Narrow the tool list, add an approval step, or lower the autonomy level. See escalation design for agents for how to route uncertain cases to a person before they become incidents. And if your policy does not say who may deploy an agent and with what access, an acceptable use policy for agents is a good place to start.

Bottom Line

Write the runbook before the incident. Define three severity levels, name the incident lead, and build the freeze so it works through permissions, not prompts.

When it happens: preserve evidence, drop to draft-only, scope from the logs, fix, restore in steps, and write a blameless postmortem with owners and dates.

Next step: run a 30-minute tabletop this week. Pick one live agent, imagine it sends 50 wrong messages, and see how far you get.

FAQ

What do you do when an AI agent makes a mistake? Contain it first. Save the logs, then remove the agent’s ability to act, ideally by cutting write permissions. Then scope the damage from the audit trail, undo what you can, fix the cause, and restore the agent in steps. Write a blameless postmortem afterward.

How do I contain an agent incident? Use permissions, not instructions. Drop the agent to draft-only, pause its triggers, rotate credentials if you suspect misuse, and use a kill switch if needed. Hold any unsent messages or queued actions. Do not rely on the agent to stop itself.

Who decides severity? A named incident lead decides in the first 15 minutes, using severity levels written in advance. The business owner can raise the level. Lowering a Sev 1 or Sev 2 should need agreement from both the incident lead and the security or risk owner.

What goes in the postmortem? Summary, timeline, impact, how it was detected, root causes, what went well and poorly, and action items with owners and due dates. Keep it blameless: focus on the permissions, instructions, data, and testing gaps that allowed the failure.

How do I prevent repeats? Assign every action item an owner and a date. Add the failing case to your regression tests. Narrow permissions or raise human approval requirements where the failure came from too much autonomy. Review open items on a schedule.

Three Ways to Work With TMA

Need an agent built? We deploy production AI agents in your infrastructure. Working pilot. Real data. Measurable ROI. → Schedule Demo

Want to co-build a product? We’re not a dev agency. We’re co-builders. Shared cost. Shared upside. → Partner with Us

Want to join the Guild? Ship pilots, earn bounties, share profit. Community + equity + path to exit. → Become an AI Architect