# The Better Model Broke Production: Rolling Out a New Model Safely

> 🗓️ **Last updated: July 2026**

The new model revision cleared the eval suite on a Monday. The numbers were unambiguous: faithfulness up three points, hallucination rate down, latency within budget. Shadow traffic ran for two weeks against a five per cent mirror of production. The diff harness flagged some longer responses but no factual regressions. The canary cohort, another five per cent and this time live, ran for four clean days. The on call dashboards were green. Promotion to one hundred per cent went out at eleven on Wednesday morning. Everyone moved on to the next thing, which is exactly when the trouble started.

By Friday afternoon, an invoice service three teams away was paging on a one per cent rate of empty invoices going to customers. The first hour of triage assumed a database issue. The second hour found the parser. The parser had a regex that expected the model's response to start with a numeric line, and the old model had reliably done so for eighteen months. The new model, the better one, prepended a polite framing sentence. The regex returned nothing. The exception was swallowed. The invoice rendered blank. The downstream team had not been told about the model upgrade, because nobody on the LLM team knew the parser existed.

The model did not get worse. The system around the model got worse, because the system was holding an unwritten contract that the model had quietly stopped honouring.

This is the failure shape that separates a model rollout from a code rollout. The artefact behind a model rollout is not a binary that fails closed when its contract changes. It is a probabilistic component whose contract was never signed, only inferred. Every downstream system that consumed its output for long enough built an implicit contract on top of patterns the model happened to follow. The upgrade did not break anything the eval suite measured. It broke contracts the eval suite did not know existed.

This article is about how to roll out a model upgrade so that conversation does not happen on a Friday afternoon.

* * *

## What this article is not

It is not a tour of every progressive delivery tool. It is not an argument that one canary configuration is correct. It is the argument that model rollouts require a longer schedule, a wider observation surface, and more deliberate downstream coordination than code rollouts of comparable risk, and that the team rolling out the model is almost never the team that will first see the breakage.

* * *

## The thesis

A model upgrade is a distributed system migration wearing a config change as a disguise. The new dependency has a different behavioural distribution, a different cost profile, and a different set of failure modes than the old one. The same discipline that catches breakage in classical migrations, which is to say shadow traffic, canary cohorts, eval gated promotion, rehearsed rollback, and downstream coordination, applies here too, with adaptations for the probabilistic nature of the artefact. Skip any of those steps and you ship the rollout that ended the opening scene.

* * *

## What the classical rollout discipline looked like

Pre LLM, the standard playbook for shipping a high risk change was well documented. The Google SRE book formalises it. Spinnaker, Argo Rollouts, and equivalents implement it. The Netflix originated Kayenta automates the analytical step. The shape is familiar:

1. Build the new artefact. Run unit and integration tests.
2. Deploy to a staging environment that approximates production traffic.
3. Use a blue green or feature flag mechanism so the new artefact runs alongside the old.
4. Route a small fraction of real traffic to the new artefact.
5. Compare latency, error rate, and business KPIs across cohorts.
6. Expand the cohort on evidence, not on the calendar.
7. Roll back instantly if a metric crosses a threshold.

This works for classical services because the contract between consumer and provider is enforced by a schema, errors fail closed, behavioural diffs are deducible from the code diff, and rollback restores the old behaviour exactly. LLM systems break each of those assumptions. The contract is the model's behavioural distribution, not the response schema. Errors fail open, because a wrong answer returns a clean two hundred. Behavioural diffs are empirical, not deducible from any diff you can read. And rollback may restore the old model, but it does not restore the database rows the new model already populated. Every step of the classical playbook still applies. None of them apply unchanged.

* * *

## What changes when the model changes

It is tempting to think of a model upgrade as a config change, a single string in a deployment manifest going from one revision to another. That framing is the source of most production incidents. The blast radius of changing that one string is larger than the blast radius of almost any code change you could name.

When the model changes, six distinct things change at once:

*   **Output format.** Response length, sentence structure, preamble patterns, refusal phrasing, the use of bullets versus prose. Anything downstream that pattern matched on the old format is now exposed.
*   **Cost profile.** A more capable model is often more verbose. Tokens out per response can rise twenty to forty per cent without any deliberate decision being made.
*   **Latency distribution.** Time to first token and total generation time shift. Streaming consumers feel this immediately.
*   **Refusal and safety distribution.** The set of prompts the model now declines, and the wording of those refusals, both move.
*   **Tool use behaviour.** When the model emits a function call, which tool it picks, and how it fills the arguments, all of these drift between revisions.
*   **Strengths and weaknesses on specific topics.** The new model may be better on average and worse on the three sub domains your traffic over indexes on.

Each of these has at least one downstream system that depends on it without knowing it does. The model upgrade is a contract change against a contract that was never signed. The rollout's job is to surface every one of those implicit contracts before the customer does.

* * *

## Shadow traffic for LLM systems

Shadow traffic, sometimes called dark launch or traffic mirroring, sends a copy of real production requests to the new model in parallel with the production model. Only the production model's output is served to users. The shadow output is logged and analysed offline.

For LLM systems, shadow traffic is the only honest way to measure how a model will behave on real traffic before you bet a user facing metric on it. Golden eval sets, by definition, cannot match the long tail distribution of what users actually ask. Shadow traffic does.

What shadow traffic surfaces that the eval suite does not:

*   **Real cost.** Tokens in and tokens out on real prompts, not on golden inputs. A twenty per cent verbosity increase shows up as a real dollar number.
*   **Real latency.** Including the slow tail on prompts the eval set never contained.
*   **Behavioural drift on the real distribution.** Response format changes, tool use frequency changes, refusal rate changes, all measurable on a representative sample.
*   **Failure mode discovery.** Prompts that crash the new model, prompts that produce empty responses, prompts that return malformed structured output.

What shadow traffic does not catch:

*   **Downstream contract breaks.** No downstream system sees the shadow output, so no downstream parser fails on it. The Friday afternoon failure mode in the opener is invisible to shadow.
*   **Multi turn drift.** Shadow harnesses usually replay single turns. Multi turn conversations, where the model's reply becomes context for the next turn, are harder to shadow honestly.
*   **Effects on user behaviour.** Shadow cannot tell you whether users would retry, escalate, or churn.

Practical constraints: shadow traffic doubles inference cost for its duration, so sample at a rate the budget tolerates. Five to ten per cent is usually enough to get statistically useful comparisons. It creates a second copy of every shadowed prompt, with PII and retention implications, so the shadow log lives under the same data protection regime as the production log. The diff testing pattern that Twitter open sourced under the name Diffy is the canonical reference for how to compare two services' responses to the same input at scale without reviewing every pair by hand. The same pattern adapts to LLM outputs, with the comparison function replaced by a semantic similarity or judge model check rather than a byte level diff.

* * *

## Canary rollouts and eval gated promotion

A canary rollout routes a small fraction of real production traffic to the new model and lets the new model's outputs actually reach users. The cohort grows over time, gated by evidence that the new model is behaving acceptably.

Canary differs from shadow in one decisive way: canary outputs reach users, so canary can surface the downstream contract break that shadow cannot. The cost is that the canary cohort is also the cohort of users absorbing whatever regressions the canary reveals. There is no free lunch here, only a smaller bill.

Two adaptations matter for LLM canaries.

First, **the metric set has to include behaviour, not just errors.** Classical canary analysis watches HTTP error rate, latency, and a few business KPIs. An LLM canary that watches only those will wave through a rollout where the new model is more polite, more verbose, more expensive, and silently breaking a downstream parser, none of which move HTTP level metrics. The canary needs to watch the eval suite from Article 1 of this series, run continuously on the canary cohort's traffic, alongside cost per request, refusal rate, response length distribution, and the error rate of any downstream system that touches LLM output.

Second, **the cohort should grow on evidence, not on a schedule.** Argo Rollouts and Spinnaker both support pause and analyse steps in a canary pipeline. Kayenta, the Netflix originated automated canary analysis tool now maintained as part of Spinnaker, implements the statistical comparison automatically. The pattern is: ramp to five per cent, run for long enough to clear the eval gate at that volume, ramp to twenty per cent, repeat. "Long enough" for LLM systems is usually longer than for classical services. Weekly usage patterns, time zone variation, and the slow tail of edge case prompts all need to wash through the cohort before the metric is trustworthy. Twenty four hours is rarely enough. A full week per ramp step is closer to defensible.

Two pitfalls. The first: CSAT and user feedback metrics on a small canary cohort have low statistical power and are confounded by user reaction effects. A small worsening in the model can trigger retries, escalations, or visible frustration, all of which feed back into the metric you are using to decide. Article 1 covers the confound. The practical implication for rollouts is that CSAT alone is not a canary gate. The second: the eval suite the canary clears must be a frozen suite that the model was not tuned on. If the team that prepared the new model also prepared the eval set, the gate is a rubber stamp with extra steps.

* * *

## Rollback semantics

In classical rollouts, rollback restores the old artefact and the old behaviour, atomically. With LLM systems, rollback restores the old model, but it does not restore the artefacts the new model already produced.

If the new model populated database rows with generated summaries during the canary window, those rows remain after rollback. If the new model wrote to a vector store, those embeddings remain. If the new model populated a cache that other services read from, that cache remains. The system after rollback is the system on the old model, plus a residue of state created by the new model. Whether that residue matters depends on the system. In retrieval heavy systems and any pipeline that materialises model output, it usually does.

Rollback discipline therefore has two parts. The fast part is the model swap: change the version pin, restart the inference clients, the next request hits the old model. The slow part is the state remediation: identify which downstream stores were populated by the new model during the window, and decide whether to invalidate, regenerate, or leave them. Both parts need to be rehearsed before the rollout, not improvised during the incident. The version pinning, alias usage, and snapshot discipline from Article 2 of this series are the prerequisites. Without them, "roll back" is not a defined operation, it is a hope.

* * *

## The vendor driven deprecation problem

Sometimes the upgrade is not yours. The vendor retires the dated revision you pinned, and you have to migrate by a date you did not pick.

OpenAI and Anthropic both publish deprecation timelines for their hosted models, and both have given months of notice on past deprecations. That notice window is the rollout schedule you have, whether you want it or not. The rollout discipline does not change, shadow, canary, eval gate, rollback drill, but the latest acceptable date for completing the rollout is the vendor's cut off, minus your own safety margin. Treat the deprecation feed as an inbound dependency on your roadmap. Subscribe to it. Wire it into the same intake process that picks up security advisories. Article 2 covered the version discipline side of this. The rollout side is the discipline that turns a deprecation notice into a successful migration before the cut off, rather than a scramble after it.

For self hosted models on stacks like vLLM, you own the deprecation calendar yourself, which is more freedom and more responsibility in equal measure. The temptation to defer upgrades indefinitely is real. The consequence is accumulating a model that is years behind the frontier on capability and cost efficiency, which becomes its own production problem soon enough.

* * *

## Failure modes, severity ordered

The failure modes worth designing against, in roughly the order they will hurt the most:

**1. Downstream contract break.** The opening scene failure. A downstream system pattern matched on the old model's output shape, the new model changed shape, the contract broke silently. Mitigation: publish a behavioural diff summary before the rollout, route it to every team that consumes LLM output, give them time to update their parsers.

**2. Cost shock.** The new model is more capable and more verbose. Tokens out per response rises. The latency tail extends. Inference cost climbs twenty to forty per cent overnight. Mitigation: cost per request is a first class canary metric, ramp gates include a cost ceiling, and the cost bounded patterns from the Series 1 closer apply at the system level here.

**3. Silent regression on a sub cohort.** Overall eval scores improve. A specific user segment, one language, one domain, one tenant, gets dramatically worse. Aggregated metrics hide it. Mitigation: stratified eval reporting on the canary cohort, with per segment thresholds, not just an overall mean.

**4. Insufficient observation time.** The canary ran for twenty four hours, the rollout looked fine, the weekly pattern was never tested. Monday morning, the metric that matters is off. Mitigation: minimum observation windows expressed in usage cycles, not hours.

**5. Eval gate gaming.** The new model was tuned against the same eval set used as the rollout gate. The gate passes. Real production users do not agree. Mitigation: the rollout eval set is held out from training and tuning, owned by a different team, and rotated periodically.

**6. Rollback not actually possible.** The team tried to roll back and discovered the old model revision was already deprecated, or the alias had already moved, or the inference cache referenced the new model in a way that broke under rollback. Mitigation: rehearse rollback before the rollout, in a staging environment that simulates the version state.

**7. No downstream coordination.** The rollout was a Slack message in one channel. The team that owned the invoice parser was not in that channel. Mitigation: the rollout has an explicit downstream stakeholder list, the behavioural diff summary goes to all of them, and the rollout is paused until acknowledgement comes back.

* * *

## The Architect's Checklist

1. Pin a dated model revision in production, never a moving alias. Set up in Article 2.
2. Establish a frozen rollout eval set, held out from any training or tuning by the team preparing the upgrade.
3. Run shadow traffic at five to ten per cent of production for at least one full usage cycle, typically a week, before the canary stage.
4. Compare shadow output to production using a semantic similarity or judge model harness, not a byte level diff.
5. Define the canary metric set explicitly: eval suite scores, cost per request, latency distribution, refusal rate, response length distribution, and downstream system error rate.
6. Stratify canary metrics by tenant, language, and domain, whatever segments matter to the business, and gate on per segment thresholds, not aggregates.
7. Ramp the canary on evidence, not on a schedule. Five per cent, evidence, twenty, evidence, fifty, evidence, one hundred.
8. Set minimum observation windows in usage cycles, usually one week per ramp, not hours.
9. Publish a behavioural diff summary before the rollout begins. Distribute it to every downstream system owner. Pause the rollout until each acknowledges.
10. Define rollback in two halves, model swap and state remediation, and rehearse both in staging.
11. Wire vendor deprecation feeds into the same intake process as security advisories.
12. Run cost monitoring during the rollout with a hard ceiling that pauses the ramp automatically, using the observability discipline from Article 3 of this series.
13. Record an audit entry for every rollout: which dated revision replaced which, on what date, with what eval results, signed off by whom.

* * *

## The Architect's Mental Model

Model rollouts are not code rollouts. They are vendor library upgrades with no semver, no changelog, and downstream consumers who built parsers against the library's incidental behaviour. The discipline that makes them safe is the discipline of a major distributed system migration, which is to say shadow, canary, eval, rollback, and downstream coordination, adapted for an artefact whose contract is empirical rather than typed.

The team that owns the model is rarely the team that will discover the breakage. The rollout's design has to reflect that asymmetry. Slow down the ramp. Widen the observation surface. Publish the behavioural diff before the rollout, not after the page.

> *Code rollouts fail closed. Model rollouts fail confidently.*

* * *