# Everything Is Green and the CEO Is Furious: Incident Response When the Model Is the Bug

> 🗓️ **Last updated: July 2026**

The screenshot is on a leadership Slack channel by 9:14 in the morning. A customer asked the assistant a routine policy question. The assistant answered confidently and in detail. The answer was wrong in a way that, if acted on, would have cost the customer real money. The customer screenshotted it, posted it, and tagged the company account. The post is gaining traction. The CEO has asked, in writing, what is being done.

The on call engineer pulls up the dashboards. The system is healthy. Latency normal, error rate zero, no recent deploys. The git log shows nothing in the last week that would touch this surface. The prompt template has not changed. The model version pinned in the manifest is the same revision that was in production yesterday and the week before. The retrieval index is the same. By every metric the platform tracks, nothing is wrong.

The model said the thing. The model is the bug. There is no commit to revert.

This is the failure shape that breaks classical incident response. The on call playbook assumes that an incident has a cause, the cause is in a system you control, and the fix is a code change or a config change that reverses the cause. When the model is the bug, all three assumptions wobble. The cause may not be in any system you control. The fix may not be a code change. And the "revert" may not exist.

This article is about how to run an incident when the failing component is the model itself.

* * *

## What this article is not

It is not a general incident response tutorial. The classical roles, severity levels, war room structure, and postmortem discipline all still apply. What changes are the triage steps, the mitigation surface, and the shape of the postmortem when the bug lives inside a probability distribution rather than inside your repository.

* * *

## The thesis

Classical incident response assumes you can find the bad commit and revert it. Model incidents have no commit. The playbook needs three things classical incidents do not: a replay capture that lets you reproduce the failure deterministically from logged inputs, a triage taxonomy that distinguishes between the five places the bug actually lives, and a runtime mitigation layer that lets you change behaviour without a deploy. Get those three right and a model incident becomes a tractable engineering problem. Skip any of them and you end up in the "wait for the vendor" state, with an angry CEO and no plan.

* * *

## What classical incident response looked like

The classical playbook is well documented and largely universal. The shape, condensed:

1. **Detection.** An alert fires, a customer reports, a metric breaches.
2. **Triage.** Severity is assigned, typically SEV1 for a customer facing outage, SEV2 for major degradation, SEV3 and below for partial issues, following the framework PagerDuty publishes openly as part of their incident response guidance.
3. **Incident command.** A single incident commander coordinates. A communications lead handles internal and external updates. Responders investigate.
4. **Diagnosis.** Logs, traces, dashboards. A recent change is identified: a deploy, a config push, a dependency upgrade.
5. **Mitigation.** Revert the change, fail over, scale out, drain traffic. The faster path is almost always reversal of the most recent change.
6. **Resolution.** The metric returns to baseline. The incident is closed.
7. **Postmortem.** Following the model formalised in Chapter 15 of the Google SRE book, *Postmortem Culture: Learning from Failure*, the team writes a blameless retrospective focused on systemic causes rather than individual fault.

This works because classical incidents have causal arrows that point at code. The bug is in a function, a query, a config, a state. The team finds the arrow and breaks it.

* * *

## What changes when the model is the bug

Three things change at once when the bug lives inside the model.

**Detection moves outside the system.** The customer facing screenshot in the opener was not detected by any internal metric. The model did not error. The latency was fine. The deterministic plumbing all worked. The bug surfaced because a human read the output and recognised it as wrong. The first signal of a model incident is very often external, a support ticket, a screenshot, a Trust and Safety report, a regulator's email, and the detection latency is the time it takes a customer to notice, not the time it takes an alert to fire.

**The "recent change" hunt fails.** The triage instinct in a classical incident is to look for the last thing that moved. In a model incident, often nothing moved in any system you control. The model's behaviour shifted because retrieved context shifted, because a downstream prompt exploited an interaction that had always been possible but never triggered, because the hosted version silently updated on a moving alias, or because today is simply the day the dice rolled badly on a failure that was always possible at low probability. Article 2 covers why "what moved" is harder to answer for LLM systems. In an incident, that difficulty becomes time to mitigate.

**Mitigation does not look like a revert.** Classical mitigation reverses the most recent change. Model incident mitigation is constructive: you ship something new, a tighter system prompt, a content filter, a fallback model, a kill switch on a specific tool call, because there was nothing to revert. Every patch survives into the next incident.

* * *

## The replay imperative

The single most important capability in model incident response is replay. Given a failure case, the exact prompt the customer sent, the exact retrieval context the system assembled, the exact tool calls and intermediate responses, the exact model version pinned at the time, can you rerun that case offline and reproduce the failure deterministically?

If yes, the incident becomes a reproducible bug, and a reproducible bug is a tractable engineering problem. The team can iterate on prompt changes, fallback configurations, and filter rules against the replay until the bad output goes away, and then test the mitigation against the wider eval suite to confirm no regression elsewhere.

If no, if the system cannot reconstruct what happened well enough to reproduce it, then the team is reduced to guessing. Every proposed fix is a hypothesis with no way to verify before deploy. The incident enters the "wait and see" state, where mitigations are shipped speculatively and the only way to know if they worked is to wait for the next failure. This is a miserable place to run an incident from.

The observability discipline from Article 3 of this series is the prerequisite. Replay requires that every model call captured its full input, full output, model version, sampling parameters, retrieval context, and tool call trace, at the time it happened, retained long enough to still exist when the incident is opened. Sampling based observability is fine for capacity planning. It is fatal for incident response, because the one call you need is invariably the one that was sampled out.

The two practical questions to answer in advance, before the incident:

*   **What is the maximum gap between failure and reproduction?** If a customer reports a bad output from three days ago and the logs only retain twenty four hours of full request bodies, the replay never happens.
*   **What is the privacy posture?** Full fidelity replay implies storing user prompts and model outputs, which has data protection implications. The retention window and access controls need to be defined before the SEV1 ticket opens, not negotiated during it.

* * *

## The triage taxonomy

When the model is the bug, "the model" is a useful shorthand but a terrible diagnostic category. The actual bug usually lives in one of five distinct places, and the mitigation differs for each. Naming them up front is the difference between a triage that converges and a triage that loops.

**1. Hallucination on the model itself.** The prompt and the context were reasonable. The model produced a confident, fabricated answer. The bug lives inside the model's training distribution. Mitigation surface: a tighter system prompt with explicit grounding instructions, retrieval enforcement ("only answer from the provided context"), a stricter judge model on the output, or in severe cases a switch to a different model with better grounding behaviour on this class of prompt.

**2. Retrieval poisoning or retrieval miss.** The model received misleading or insufficient context and answered the question as if the context were correct. The bug lives in the retrieval layer or the source data. Mitigation: rerank, source filtering, source authority weighting, a blacklist on specific documents, a refresh of the index. Multi tenant RAG systems, covered in Series 2 Article 3, have an extra dimension here: the bad context may belong to one tenant and the failure may surface for that tenant only.

**3. Prompt regression.** A recent prompt change, a tweak to the system instruction, a new tool description, a refactored output schema, interacts badly with a class of input nobody tested. The bug is in the prompt commit. Mitigation: revert the prompt to its previous version, which is the closest model incidents get to a classical revert, then roll the fix out again with the failing case added to the regression suite. Article 2's version discipline patterns are the prerequisite for a clean revert.

**4. Vendor side degradation.** The hosted model started behaving differently. Either the alias pin moved to a new revision (avoidable, covered in Article 2), or the dated revision the team pinned itself drifted, through capacity issues, regional routing, or a partial outage. Mitigation: the vendor status page is the first stop. If the incident is on the vendor side, the response is communication and partial degradation modes, not engineering. Article 4 covers the deprecation discipline that prevents this from becoming an emergency in the first place.

**5. Downstream contract break.** The model output is correct, but a system downstream of the model is mishandling it: a parser, a renderer, a tool call adapter, a workflow step. The user sees a broken response, but the model is innocent. This was the opening scene of Article 4. Mitigation lives in the downstream system, not in the model layer. The triage error is to spend an hour tweaking prompts when the actual bug is in a regex three services away.

The triage flow, in practice: capture the replay, walk the five categories in order, eliminate each on evidence. Most incidents in production fall into categories 2, 3, or 5, not category 1. Declaring "the model hallucinated" without ruling out the other four is one of the most common errors in model incident response, and one of the most expensive.

* * *

## Runtime mitigations

A model incident usually cannot be mitigated by deploy. Or, more precisely, the deploy cycle for a model system, change the prompt, run the eval suite, push to staging, canary, full rollout, is far too slow for an active customer facing incident. The runtime mitigation layer is what bridges the gap between "the incident is open" and "the proper fix is in production."

Three patterns, ordered by how surgical they are.

**Filter and reject.** A content filter sits between the model and the user. For the duration of the incident, it rejects outputs matching a specific pattern, the offending phrase, a class of factual claim, a tool call to a specific function. Rejected outputs return a graceful fallback. The cost is precision loss and the risk of over blocking. The benefit is that the bad output stops reaching customers immediately.

**Fallback to a different model.** If the failure is localised to a specific model on a specific class of prompts, route that class to a different model. The fallback may be slower or more expensive, but on this class of prompts it is correct, and during an incident "correct and degraded" beats "fast and wrong" every time. The fallback model has to already be wired in and eval tested for this class. Standing up a new vendor integration during a SEV1 is not feasible, no matter how calmly leadership asks for it.

**Kill switch a capability.** Disable the tool, the retrieval source, the workflow step, or the model entirely for the affected surface. The kill switch is the bluntest instrument and the most reliable. It assumes that "off" is a defined state the rest of the system handles gracefully, where Article 4's rollback semantics apply here too.

Each of these mitigations is a patch that survives into the next incident unless someone deliberately removes it. Without discipline, the system slowly becomes a thicket of incident era filters that nobody dares touch. Every runtime mitigation needs an owner, an expiry date, and a review cadence.

* * *

## Postmortems on probabilistic failures

The blameless postmortem template, what happened, when it was detected, when it was mitigated, contributing factors, action items, applies to model incidents largely unchanged. Two adaptations matter.

**"Root cause" is sometimes the wrong framing.** Classical postmortems converge on a primary contributing factor: a bad commit, a missed alert, a configuration drift. Model incidents often have no single cause. The system was always capable of producing this output, and the conditions to elicit it happened to align today. The honest postmortem says the failure surface was visible in retrospect and the system should have had a filter, a stricter retrieval rule, or a different prompt, not that there was a bug to fix. Forcing a single root cause onto a distributional failure produces postmortems that read as deflection.

**Action items need a recurrence probability framing.** Classical action items reduce a probability that was effectively one back to effectively zero. Model incident action items reduce a probability that was always non zero. "Add this case to the regression eval suite" is honest. "Fix the model so this can never happen" is not. The postmortem distribution list also widens, since contributing factors typically span the model, retrieval, prompt, downstream consumer, and vendor layers.

* * *

## Failure modes, severity ordered

The patterns that turn model incidents from manageable into ruinous, roughly worst first:

**1. No replay capability.** The system cannot reproduce the failure offline. Every mitigation is a guess. The incident closes when the customer stops complaining, not when the cause is understood. Mitigation: full fidelity observability per Article 3, with a retention window matched to incident detection latency.

**2. "Blame the user."** The triage concludes that the failure was the user's fault: a malformed prompt, an unusual phrasing, an edge case. The action item is to do nothing. The same incident recurs with a different user a week later. The OWASP LLM Top 10 explicitly classes prompt injection as a system risk, not a user error, and the same framing applies to most edge case prompts. The user surface is the production surface.

**3. Infinite mitigation.** Each incident adds a filter or a prompt patch. None are ever removed. After a year the system is a layer cake of incident era mitigations, the regression suite takes hours, and a new prompt change can no longer be reasoned about. Mitigation: every runtime mitigation has an owner and an expiry date, and the postmortem reviews previous mitigations for retirement.

**4. Single source triage.** One engineer can read the replay. They become the bottleneck on every model incident, and then they go on holiday. Mitigation: replay tooling exposed to the whole team, with a runbook walking the five triage categories.

**5. Model vendor lock in revealed at the worst moment.** The incident requires falling back to a different model. There is no other model integrated. The team discovers during the SEV1 that switching vendors is a two week project. Mitigation: at least one alternate model is integrated, eval tested on production traffic patterns, and wired into the routing layer before the incident.

**6. "Wait for the vendor" as a strategy.** The triage concludes the bug is on the vendor side. The team's response is to wait. No degraded service mode is engaged. Customers continue hitting the failure. Mitigation: every "wait for the vendor" decision is paired with a degraded service mode that is actually shipped during the wait, a filter, a fallback, a feature disablement.

**7. Postmortem theatre.** The postmortem is filed, the action items are assigned, and none of the action items ship. The next similar incident finds the same gaps. Mitigation: action item completion is tracked at the same cadence as the original incident severity. SEV1 action items have a one week SLA for closure, not a "when we get to it."

* * *

## The Architect's Checklist

1. Every model call captures full input, full output, model version, sampling parameters, retrieval context, and tool call trace, retained long enough to outlast incident detection latency.
2. A replay harness exists. Any engineer on call can rerun a logged failure case offline against the production model version.
3. The triage runbook walks the five categories explicitly: hallucination, retrieval, prompt regression, vendor degradation, downstream contract break.
4. At least one alternate model is integrated, eval tested, and wired into a routing layer that can fail over during an incident.
5. A runtime filter layer exists between the model and the user, with a way to push a new pattern based block without a deploy.
6. Kill switches exist for individual tools, retrieval sources, workflow steps, and the model itself. Each "off" state is a defined operation the rest of the system handles gracefully.
7. Every runtime mitigation has an owner, a documented incident reference, and an expiry or review date.
8. The incident severity framework explicitly includes model output quality as a SEV class, not just availability and latency.
9. The first external detection channel, support, Trust and Safety, social, has a defined escalation path into on call.
10. Vendor status pages are integrated into the on call dashboard.
11. Postmortem templates for model incidents replace "root cause" framing with "failure surface and recurrence probability."
12. SEV1 and SEV2 action items have closure SLAs, tracked at the same cadence as the original incident.
13. The on call rotation includes someone who can read the replay. The replay tooling is not a single engineer skill.

* * *

## The Architect's Mental Model

A model incident is a deterministic system incident with the causal arrow pointing into a probability distribution. The classical disciplines, incident command, blameless postmortems, severity classification, action item closure, survive almost unchanged. The deterministic toolchain, find the commit, revert it, monitor, does not.

What replaces it is replay, a triage taxonomy, and a runtime mitigation layer. Replay turns the incident from speculation into engineering. The taxonomy stops the team from looping on "the model is broken." The runtime layer stops the customer from being the next person to discover what the team already knows.

> *Classical incidents have a commit. Model incidents have a customer.*

* * *