AI Failure Contingency Runbook

How to use this document

• What it is for. This is a base operational runbook for the first minutes and hours when an AI service that the organization's processes depend on fails: vendor outage, model degradation, harmful outputs, data exposure or loss, attacks, unannounced changes and runaway consumption. It states what to do, in what order, with which manual fallback and how quickly to restore service.
• Who should own it. Usually operations or the site reliability engineering (SRE) team, with the owners of the processes that depend on AI. Enter the function in [RESPONSIBLE FUNCTION] and the person in [OWNER].
• Decisions to take first. Decide the recovery objectives for each system (aid B), the manual fallbacks for each process (Section 7), who activates the runbook (Section 4) and how it relates to your business continuity plan and your incident response plan (Section 1.2).
• Fields to complete. Replace the bracketed fields and complete the recovery objectives table and the contact directory. Keep [OPTIONAL] clauses only if they apply.
• Country variants. This runbook has no country variants. Deadlines for reporting cybersecurity incidents or data breaches to authorities vary by country and industry: do not set them here; they are handled in the incident response plan, using the laws listed in "Related regulatory references."
• Review. Have the technical teams validate the steps in each scenario and have system owners confirm that the manual fallbacks are feasible with the staff available.
• Approval and publication. Have it approved by the operations or technology leadership, record the approval in "Version history," keep it accessible outside the usual systems as well (printed copy or another repository) and test it with an exercise.

AI Failure Contingency Runbook

Field	Value
Organization	[LEGAL ENTITY NAME]
Owning function	[RESPONSIBLE FUNCTION]
Runbook owner	[OWNER]
Approved by	[APPROVED BY]
Effective date	[DATE]
Review frequency	[REVIEW FREQUENCY]
Activation channel	[COMPLETE: on-call channel, phone] and [AI GOVERNANCE EMAIL]
Location of the backup copy	[COMPLETE]

1. Purpose and relationship to other plans

1.1 This runbook sets out the operational sequence that [ORGANIZATION NAME] follows when its AI systems in production fail and time matters more than thorough analysis: contain, keep operating with fallbacks, stabilize and coordinate.

1.2 It applies together with:

• the "AI Incident Response Plan," which governs classification, investigation, notifications to authorities and affected people, and lessons learned, and whose record is the single incident record;
• the "AI Rollback and Safe-State Protocol," which defines how to degrade, roll back or disable each system;
• the business continuity plan, of which this runbook is an annex for services that depend on AI;
• the "AI Post-Deployment Monitoring Plan," which provides the alerts that usually trigger this runbook.

1.3 This runbook covers only the operational sequence and the non-AI fallbacks. It does not repeat the technical rollback steps (they are in the rollback protocol) or the classification, investigation and notification rules (they are in the incident response plan).

2. Scope and scenarios

2.1 It applies to AI systems in production that support business processes or customer services, whether in-house or from vendors.

2.2 Scenarios covered:

No.	Scenario	Examples
E1	Vendor or API outage or degradation	The service does not respond, responds with errors or with unacceptable latency; quota exhausted
E2	Model degradation	Drop in accuracy, drift, more errors or fabricated information
E3	Harmful or inappropriate outputs	Offensive, dangerous, discriminatory or false responses reaching people
E4	Data exposure or loss	The system reveals personal or confidential data; loss of an index, logs or reference data
E5	Attack or misuse	Prompt injection, agent taking unauthorized actions, abuse of the service
E6	Unannounced vendor change	New model version that alters results; change in terms or data location
E7	Runaway consumption or cost	Agent loop, usage spikes, anomalous billing
E8	Failure or compromise of a self-hosted model	Inference server down or saturated; weights hash differs from the approved one; security advisory for the inference server or the upstream model; endpoint exposed without authentication