Runbook for an LLM outage: detection, escalation, postmortem

By Ivo Donker — compiled with AI assistance · August 7, 2026

When an external large language model (LLM) goes down, it is not just a single microservice that crashes — the functional intelligence of an entire application degrades on the spot. Robust software design provides architectural pillars such as circuit breakers and fallbacks, yet in practice the operational manual that sets out how engineers act during an active outage is often missing. Without a structured process, an incident leads to chaotic communication, delayed mitigation and missed service level agreement claims. See the anchor article on building robust integrations to understand how these operational runbook procedures dovetail with the broader system design.

This article serves as the operational runbook for LLM outages. It connects observability signals directly to incident response actions, escalation paths and postmortem analysis. It does not cover the technical fallback layer itself; see the guide to graceful degradation during LLM outages for implementing backup models and degradation strategies. Nor does it cover which SLA figures to negotiate in a contract; see the overview of SLAs and uptime among LLM providers for drafting contractual guarantees.

1. The runbook as an operational artifact

A runbook is not static documentation written once at project handover. It is a living, version-controlled artifact that must be immediately available to the on-duty site reliability engineer (SRE) or developer. Ownership of the runbook sits primarily with the team responsible for managing the LLM integration infrastructure or API gateway.

The runbook comes into force the moment automated monitoring crosses controlled thresholds, or when end users report a significant loss of functionality that regular application errors do not explain. To keep the instructions accurate, every change to the API architecture, routing logic or fallback providers must be reflected in the runbook procedures within 24 hours. Every incident is direct cause to revise the playbook.

2. Phase 1: detection and signal analysis

Failure mode

The failure mode in this phase centers on being unable to establish in time whether an elevated error rate is caused by an external provider outage (infrastructure failure, model degradation or capacity problems at the vendor) or by an internal fault in your own application code (invalid JSON payloads, corrupted API keys or exceeded prompt token limits).

Detection

Automatic detection rests on continuously analyzing observability signals at the API endpoint. Distinguish between hard provider errors (HTTP status codes 500, 502, 503 and 504), rate limit errors (HTTP 429) and application errors (HTTP 400 or 422). For setting up telemetry pipelines and log structures correctly, see the guide to observability and logging, which explains how to collect detailed metrics. When HTTP 5xx errors exceed a threshold of 5% over a rolling 60-second window, you are looking at a potential provider outage.

Alongside error rates, latency percentiles are critical indicators. A sudden rise in the P99 or P95 latency level often signals congestion at the provider before any HTTP error codes are actually returned. See the benchmark guide to latency percentiles to determine which measurements are representative of your specific workloads.

Distinguish transient failure (network spikes or brief restarts) from structural outages by analyzing the error frequency and the provider's response headers (such as Retry-After). If 80% of requests fail for more than three consecutive minutes, you are dealing with a structural outage.

// Provider-onafhankelijke detectielus voor LLM-endpoint gezondheid
type HealthStatus struct {
    IsHealthy     bool
    ErrorRate     float64
    P95LatencyMs  int64
    FailureReason string
}

func EvaluateLLMEndpoint(metrics MetricsWindow, timeoutBudgetMs int64) HealthStatus {
    // Definieer de kritieke alarmdrempels
    const maxErrorThreshold = 0.05 // 5% fouten toegestaan
    const maxP95LatencyMs = 4500   // 4.5 seconden P95 limiet

    if metrics.TotalRequests == 0 {
        return HealthStatus{IsHealthy: true, ErrorRate: 0, P95LatencyMs: 0}
    }

    errorRate := float64(metrics.FailedRequests) / float64(metrics.TotalRequests)
    
    // Foutpad 1: HTTP 5xx of netwerkfouten overschrijden drempel
    if errorRate > maxErrorThreshold {
        return HealthStatus{
            IsHealthy:     false,
            ErrorRate:     errorRate,
            P95LatencyMs:  metrics.P95LatencyMs,
            FailureReason: "PROVIDER_ERROR_RATE_EXCEEDED",
        }
    }

    // Foutpad 2: Latency overschrijdt het gestelde deadline-budget
    if metrics.P95LatencyMs > maxP95LatencyMs || metrics.P95LatencyMs > timeoutBudgetMs {
        return HealthStatus{
            IsHealthy:     false,
            ErrorRate:     errorRate,
            P95LatencyMs:  metrics.P95LatencyMs,
            FailureReason: "LATENCY_BUDGET_EXHAUSTED",
        }
    }

    return HealthStatus{
        IsHealthy:    true,
        ErrorRate:    errorRate,
        P95LatencyMs: metrics.P95LatencyMs,
    }
}

Mitigation

As soon as the detection loop identifies a deviation, the application immediately isolates outbound traffic to the failing provider. Automated monitoring records the exact time of the incident, collects the most recent 100 error trace IDs and marks the provider as unavailable on the internal status dashboard. This prevents follow-up requests from stalling in long waits.

The cost of this mitigation

• Latency: The detection loop adds a minimal processing time of roughly 1 to 5 milliseconds per aggregated metric evaluation.
• Money: Additional storage and processing costs for detailed distributed tracing and high-resolution metric collection.
• Complexity: Requires robust telemetry infrastructure (such as Prometheus/Grafana or OpenTelemetry) and carefully tuned alerting rules to avoid false positives.

3. Phase 2: escalation and communication

Failure mode

The failure mode in this phase covers the absence of timely action, misjudging the business impact, or unclear and contradictory communication to internal stakeholders and external users during an active disruption.

Detection

Escalation is triggered as soon as automatic mitigation (a circuit breaker, for instance) stays active for more than 5 minutes, or when the loss of functionality directly affects critical business processes such as payment processing or customer service automation.

Mitigation

The escalation process follows a strictly defined ladder:

1. Level 1: automated alert (T0). A pager alert to the on-call engineer, triggered automatically from the detection metrics.
2. Level 2: incident commander (T+10 min). If the outage has not resolved itself or been absorbed by transparent fallbacks after 10 minutes, the incident commander takes over operational coordination.
3. Level 3: stakeholder notification (T+15 min). Inform the product and service teams. Update the internal status page.
4. Level 4: external communication (T+30 min). Publish an incident notice on the public status page for external users if service delivery is visibly affected.

During an active outage, accurate record-keeping is essential for pursuing the vendor's contractual obligations. SLA expectations are set out in contracts, but they require specific actions during downtime: record the timestamp of the first failed call, retain the HTTP response codes and the provider's unique request IDs, and generate an aggregated report of total downtime. For the legal and administrative detail of claims, read the guidance on AI contracts and SLA claims which sets out step by step how to gather evidence for credit claims and how the duty to notify vendors works.

The cost of this mitigation

• Latency: No direct effect on runtime application latency.
• Money: Operating costs for on-call rotations (standby compensation) and communication platforms such as PagerDuty or Statuspage.
• Complexity: Organizational overhead; requires regular training of the incident response team and clear communication playbooks.

4. Phase 3: temporary mitigation and operationalization

Failure mode

In this phase, the failure mode is either blocking application workflows entirely because the system stays dependent on an unavailable model, or overloading the backup infrastructure through an uncontrolled, abrupt switchover.

Detection

The system detects that the primary provider is not responding within the configured timeout or is repeatedly returning errors. Retries and exponential backoff attempt to absorb brief network flickers. To keep retries from overloading the failing provider further, consult the guidelines on retries and backoff for the right configuration of jitter and maximum retry attempts.

When the circuit breaker trips, the temporary mitigation protocol kicks in. That protocol switches to a secondary route or activates a functional degradation mode.

Mitigation

Depending on the severity and type of the outage, the following mitigating measures are activated step by step:

• Enable a fallback provider: Redirect API traffic to an alternative LLM vendor or a locally hosted model.
• Model degradation: Switch from a heavy, slower model to a lighter, faster one from the same or another provider in order to preserve basic functionality. (If the question of "which model" arises in the application architecture, see the model selection criteria on model choice per task on hub.llmnet.nl).
• Functional degradation: Offer a simplified experience, such as showing cached answers, disabling complex tool use, or displaying a static notice to the user.

Running retries and switching to backup providers creates a risk that a request is processed twice at the provider if the original call was received but the response was delayed. An idempotency key prevents repeated API calls from causing unintended side effects. See the overview of idempotency in LLM calls for the exact implementation of unique keys at the API level.

Every request must also be tightly bounded by a deadline budget, so that threads in the application do not wait indefinitely on a slow provider. See the guide to timeouts and cancellation to learn how to cascade timeouts through your entire chain.

The cost of this mitigation

• Latency: Switching to a secondary provider brings network overhead with it. Caching, by contrast, cuts latency considerably.
• Money: Duplicate infrastructure costs for keeping standby capacity at alternative providers or running your own gateway. A gateway requires dedicated resources; see the handbook on self-hosting an LLM gateway for more on managing your own routing layer.
• Complexity: Prompts must be standardized so that they behave identically across different model architectures.

5. Phase 4: postmortem and incident review

Failure mode

The failure mode in the postmortem phase is a repeat of the same outage because action items are not assigned clearly, or the emergence of a blame culture in which engineers hide mistakes instead of addressing the underlying systemic weaknesses.

Detection

The postmortem phase starts automatically within 24 to 48 hours of formally downgrading the incident status to "resolved".

Mitigation

An effective postmortem follows the principle of a blameless culture. The goal is not to assign blame but to identify system faults, missing monitoring and shortcomings in automation. The postmortem analysis runs through the following steps:

1. Timeline reconstruction: Map out in detail when the first deviation occurred (T0), when the alert fired (T1), when mitigation was deployed (T2) and when service was fully restored (T3).
2. Analysis of runbook performance: Explicitly assess whether the steps in this runbook were carried out correctly. Where did delays occur? Were the alert thresholds set too tightly or too loosely?
3. Assigning action items: Every improvement gets one specific owner and a firm delivery date. Action items range from adjusting alerting to restructuring the fallback logic.
4. Validation through testing: Verify that the improvements actually work by simulating a provider outage in a test environment. Read the handbook on testing LLM integrations to learn how to apply layered failure simulations and chaos engineering to API integrations.

If the incident revealed that a change in prompt structure caused the errors — because the provider rolled out a new model version that responded differently to existing prompts, for example — a quick rollback to an earlier prompt version must be possible. See the article on version control for prompts in code to make sure prompt changes are recorded atomically and traceably in your CI/CD pipeline.

The cost of this mitigation

• Latency: No effect on runtime latency.
• Money: The engineering team's time investment in analyzing logs, writing the report and carrying out remedial work.
• Complexity: Managing action lists and keeping test and failover documentation continuously up to date.

6. Checklist for the on-duty engineer

Use the short checklist below when an LLM outage actually occurs:

• [ ] Verify the source of the outage: Check telemetry for 5xx errors, timeouts and P99 latency. Rule out internal faults (4xx, invalid keys).
• [ ] Check the automated mitigation: Determine whether the circuit breaker tripped correctly and whether traffic was automatically routed to the fallback provider or degradation mode.
• [ ] Inform stakeholders: Escalate along the ladder and update the status page if the disruption lasts longer than 5 minutes.
• [ ] Record the evidence: Save log files, request IDs, error codes and exact time intervals to support later SLA credit claims.
• [ ] Monitor recovery: Once the provider outage is resolved, use the detection loop to confirm that the error rate drops below 1% before you gradually shift traffic back (canary release) to the primary provider.
• [ ] Schedule the postmortem: Schedule the review session within 48 hours of the incident and prepare the timeline reconstruction.