The AI incident response playbook.

What is AI Incident Response?

AI incident response is the structured process of detecting, containing, investigating, and recovering from failures or security incidents in AI and machine learning systems. Unlike traditional IT incident response, which focuses on infrastructure availability and data breaches, AI incident response addresses the unique failure modes of probabilistic systems.

How AI Incident Response Differs from Traditional IR

The NIST Computer Security Incident Handling Guide (SP 800-61) provides the foundational framework for incident response, but requires AI-specific adaptations. MITRE’s ATLAS framework (Adversarial Threat Landscape for Artificial-Intelligence Systems) extends traditional ATT&CK with ML-specific tactics and techniques.^[6][7]

Types of AI Incidents

AI incidents fall into several distinct categories, each requiring different detection and response procedures:

The AI Incident Response Lifecycle

Based on NIST SP 800-61, the AI incident response lifecycle consists of six phases. Unlike traditional IR, AI incidents often require iteration between investigation and containment as root causes emerge through model analysis.

Building an AI Incident Response Team

AI incident response requires a cross-functional team combining traditional security skills with AI/ML expertise. Larger organizations may maintain dedicated AI security teams; smaller organizations can augment existing IR teams with ML specialists.

Core Roles and Responsibilities

Detection and Monitoring

Effective AI incident response begins with robust detection capabilities. The average AI incident takes 4.5 days to detect—compared to 2.3 days for traditional security incidents—because organizations lack AI-specific monitoring.^[2]

Detection Methods

• Performance Monitoring: Track accuracy, precision, recall, F1, AUC-ROC across demographic groups. Alert on degradation beyond thresholds (e.g., 5% accuracy drop, 10% disparity increase).
• Drift Detection: Monitor input distribution shift (data drift) and prediction distribution shift (concept drift) using statistical tests (KS test, PSI, JS divergence).
• Anomaly Detection: Identify unusual prediction patterns, confidence distributions, or feature values that may indicate adversarial inputs or data quality issues.
• Output Validation: Check for hallucinations using retrieval-augmented generation, fact-checking pipelines, or human-in-the-loop review for high-stakes decisions.
• User Reports: Establish clear channels for users to report unexpected behavior, bias, or errors. Many incidents (Amazon hiring, SafeRent) were detected through user complaints.

Monitoring Infrastructure

Production AI systems should implement comprehensive observability:

Containment Strategies

Containment stops ongoing harm while preserving forensic evidence. AI incidents require model-specific containment tactics beyond traditional infrastructure isolation.

Containment Options (Ordered by Invasiveness)

Containment Decision Matrix

Investigation and Root Cause Analysis

AI incident investigation requires both traditional forensics and ML-specific analysis techniques. The goal is to determine what happened, why it happened, and what data/models were affected.

Model Forensics Techniques

• Model Interrogation: Analyze decision boundaries, feature importance, and activation patterns to understand model behavior. Use SHAP values, LIME, or integrated gradients to explain individual predictions.
• Training Data Analysis: Inspect training data for quality issues, bias, or poisoning. Check data lineage to identify when/where corruption occurred. Compare training distribution to production inputs.
• Prediction Analysis: Review logged predictions during incident window. Identify patterns in misclassifications, confidence scores, or demographic disparities. Look for adversarial input signatures.
• Version Comparison: Compare incident model version to previous stable version. Use model diff tools to identify changed weights, architecture, or preprocessing. Check deployment logs for configuration changes.
• Supply Chain Review: Audit third-party models, datasets, libraries, and APIs. Check for known vulnerabilities in ML frameworks (CVEs in TensorFlow, PyTorch, etc.). Validate model provenance and checksums.

MITRE ATLAS Framework

For adversarial incidents, map attacker techniques to MITRE ATLAS (Adversarial Threat Landscape for AI Systems). ATLAS extends ATT&CK with ML-specific tactics:^[7]

Communication and Disclosure

AI incident communication requires balancing transparency, legal obligations, and reputation management. Regulatory requirements increasingly mandate disclosure of AI failures.

EU AI Act Article 73: serious incident reporting

Article 73 of Regulation (EU) 2024/1689 requires providers of high-risk AI systems to report serious incidents to the market surveillance authority of the member state where the incident occurred. Fifteen days is the outer limit, not the deadline — two shorter clocks override it, and the obligation starts as soon as a causal link to the AI system is established or reasonably suspected rather than when the investigation finishes:^[14]

An initial report may be incomplete. Where the full picture is not yet available, the expectation is that the provider files on time and follows up with a complete report, not that it waits for certainty and misses the window.

The other two clocks: GDPR Article 33 and SOC 2

Most teams plan for one deadline and are then caught by a shorter one. A single AI incident routinely engages three regimes at once, and they start at different moments, run for different periods, and are owed to different people. The EU AI Act clock runs on a serious incident involving a high-risk system. The GDPR clock runs on personal data, which is the one most AI incidents actually touch. SOC 2 sets no external deadline at all but decides, after the fact, whether the response was auditable.

Three consequences worth designing for. GDPR is usually the binding constraint at 72 hours, and it runs on awareness rather than on confirmation, so a partial notification followed by updates is the expected pattern and not an admission of disorder. The GDPR and EU AI Act clocks can both be running on the same incident with different content owed to different authorities. And SOC 2 is the one that judges you retrospectively: CC7.3 asks whether the event was evaluated, CC7.4 whether it was responded to, and CC7.5 whether recovery happened, with an auditor sampling incidents from the period and testing whether the documented process was the process actually followed.

That last point is where AI incidents tend to fail an audit rather than fail a deadline. The response usually happened; what is missing is the record that it happened in the order claimed. If containment was a model rollback at 02:14 and a guardrail change at 02:40, the evidence needs to show both, with times and an approver, and it needs to have been written when the events occurred rather than reconstructed for the auditor afterwards.

Stakeholder Communication Matrix

Communication Best Practices

• Be Specific About Impact: Don’t say "some users may have been affected." Quantify: "Approximately 1,200 loan applications processed between March 1-5 may have been subject to biased scoring."
• Explain in Plain Language: Avoid jargon like "model drift" or "concept shift." Say: "Our fraud detection system became less accurate because customer behavior changed during the pandemic."
• Provide Recourse: Tell affected users what they can do. "If your application was denied between these dates, you can request manual review at [email/link]."
• Don’t Disclaim Responsibility: Air Canada tried to argue its chatbot was a "separate legal entity"—and lost. You’re responsible for your AI’s outputs.^[12]

Recovery and Remediation

Recovery restores normal operations with validated fixes. Unlike traditional IR where recovery means "restore from backup," AI recovery often requires retraining, revalidation, and gradual rollout.

Recovery Steps

Post-Incident Review and Lessons Learned

Post-incident reviews convert incidents into organizational learning. Conduct within 1-2 weeks while details are fresh, with blame-free focus on process improvement.

Post-Incident Review Agenda

Documentation Requirements

Comprehensive incident documentation serves multiple purposes: organizational learning, regulatory compliance, legal protection, and customer transparency.

• Incident Report: Formal write-up including timeline, root cause, impact, response actions, and lessons learned. Share with leadership and retain for compliance.
• Technical Analysis: Detailed forensic findings, model analysis results, data quality assessment, and remediation validation. Archive for future reference.
• Communications Log: Record of all stakeholder notifications, regulatory filings, customer communications. Demonstrates compliance with disclosure obligations.
• Evidence Preservation: Retain logs, model snapshots, code versions, and data samples. May be required for regulatory investigation or litigation.

AI Incident Response Playbook Template

Every organization should maintain incident-specific playbooks. This template provides a starting structure:

Playbook: AI agent incidents

Everything above assumes the failure is a prediction. When the system is an agent, the failure is an action that has already happened somewhere else — a record written, a message sent, a payment moved, a ticket closed. Three things change, and they change the order of the response.

Containment comes before diagnosis. A degraded model keeps producing bad answers; an agent left running keeps taking bad actions, including during the investigation. Revoke first, understand second. Blast radius is a function of permissions rather than traffic, so the question is not how many requests were served but which tools the agent could reach and what credentials it held. And the forensic unit is a sequence rather than a single inference: you have to reconstruct an ordered chain of tool calls and the reasoning between them, which is only possible if a per-action record exists.

The recurring failure in agent post-mortems is not a missing control. It is that the team cannot say with confidence which controls ran on the action under dispute, because the only record is an application log the agent’s own process wrote. A log a compromised component can edit answers a debugging question but not an evidentiary one, and the difference surfaces exactly when a regulator or a counterparty is asking.

GLACIS Solution

AI Incident Response with Verifiable Evidence

Traditional incident response documentation is backward-looking—assembled after the fact to explain what happened. GLACIS provides forward-looking evidence: cryptographic proof that your AI controls executed correctly in production, with tamper-evident audit trails for every inference.

Faster Detection

Real-time attestation of model behavior, drift metrics, and fairness indicators. Alert when controls fail—not days later when users complain. Average detection time drops from 4.5 days to hours.

Comprehensive Forensics

Every inference logged with model version, input features, output, confidence scores, and policy evaluations. Reconstruct exactly what happened during incident window without guessing from incomplete logs.

Regulatory Compliance

Pre-built templates for EU AI Act Article 73 serious incident reports. Evidence that demonstrates what controls were in place, when they failed, and what corrective action was taken—mapped to regulatory requirements.

Stakeholder Confidence

Share verifiable evidence with customers, regulators, and boards. Not "trust us, we investigated"—actual cryptographic proof that third parties can independently validate.

The challenge: AI incidents are inevitable. The question is whether you can prove your controls worked when they should have, and detect when they don’t. Evidence beats documentation.

Frequently Asked Questions

References