Prompt and Model Versioning

🗓️ Last updated: July 2026

The release diff is empty. No commits since Tuesday. No infrastructure changes. The dashboards from a week ago and the dashboards from this morning track different distributions of user behaviour. Somewhere in the system, something changed — and nothing in the git log points to it.

The team starts the usual hunt. Build pipelines: green. Container hashes: identical. Database migrations: none. Feature flags: untouched. The hunt continues for two hours before someone runs the offline eval against this morning's logs and notices that the model's responses have shifted in tone — slightly more cautious, slightly more verbose, slightly less helpful on a particular class of question.

The model is the same model. The prompt is the same prompt. The retrieval pipeline returns the same documents. And yet, the system's behaviour has moved.

What changed?

The team is using a model alias — call it vendor-flagship-latest. The vendor rolled out a minor revision overnight. The training-data cutoff shifted by a few weeks. A safety policy was tightened. None of these moves required the team to deploy anything. None of them registered in source control. And the team had no online eval running continuously that would have caught the shift the moment it happened.

This article is about the discipline that turns invisible changes into visible ones.

What this article is NOT

This is not a prompt engineering guide. It will not teach you how to write better prompts. It will not compare model providers, opine on chain-of-thought formatting, or grade the relative merits of XML tags versus markdown headings in system messages.

This is an article about versioning — about treating the prompt as code and the model as a dependency, with all the source-control discipline, dependency-pinning discipline, and rollback discipline those framings demand. The thesis is structural, not stylistic: most teams ship LLM systems where the most behaviour-defining components live outside the version-control loop, and then they wonder why behaviour shifts overnight.

The thesis

A prompt is code.

A model is a dependency.

A retrieval index is a build artefact.

A function-calling schema is an API contract.

Most teams ship LLM systems treating all four of these as configuration — strings in a database table, an alias that points to "whatever the vendor is serving today", an index that gets rebuilt on a cron, a list of tools defined inline somewhere in the orchestration layer. None of those four components has a version in the operational sense. None of them passes through a code review, an eval-gated deploy, or a rollback procedure. And yet, between them, they account for almost all the production behaviour of the system.

The fix is to treat each of them with the same operational discipline you would apply to a service in production: source-controlled, version-pinned, eval-gated, rollback-safe.

What the classical version looked like

For two decades, the industry has had a clear answer to "how do we keep production behaviour from changing under us":

• Code lives in source control with a complete history of every change
• Dependencies are pinned to specific versions in a manifest — package.json, requirements.txt, go.mod, Cargo.toml
• Lockfiles record the exact resolved version of every transitive dependency
• Versioning follows a published contract — SemVer is the dominant one, codifying the difference between a patch, a minor, and a breaking change
• Releases are tagged, immutable, and reproducible — git checkout v2.4.1 gives you the same artefact today as it did six months ago

This stack rests on a structural assumption: behaviour is a function of code, and code is what source control tracks. As long as the code is pinned, the behaviour is pinned.

LLM systems break that assumption at every layer.

The four moving parts

In a classical service, source control tracks essentially everything that determines behaviour. In an LLM service, four components move outside the source-control loop unless you specifically pull them in:

1. The prompt — often a string in a database row, edited via an admin tool, with no diff, no review, no tag
2. The model — referenced by an alias such as vendor-flagship-latest that resolves to whatever the vendor is serving today
3. The retrieval index — rebuilt on a schedule, with no record of what was indexed when, no diff against the previous build, no rollback
4. The function-calling schema — defined inline in orchestration code, edited freely, with no notion that a downstream contract depends on its shape

Each of these is a behaviour-defining surface. Each can change in production without a deploy. Each can change in production without anyone noticing.

The versioning discipline applies the same operational pattern to all four: source-controlled, version-pinned, diff-able, rollback-safe, eval-gated.

Prompts as code

Treat every prompt as a first-class source-controlled artefact. That is the floor.

In practice, this means:

• Every prompt lives in source control — either inline in the code, or in a structured prompt file alongside the code
• Every prompt has a semantic version, incremented on change
• Every prompt change goes through code review, with a visible diff
• Every prompt change runs against a regression eval suite before it can be promoted
• The promotion mechanism is the same one used for code — a release artefact, an environment tag, a deploy

Several mature patterns exist. Microsoft's Prompty format stores prompts as text files with YAML front matter, typed named inputs, and explicit model configuration — explicitly modelled after how source files store code. Tools like LangSmith, Langfuse, PromptLayer, and Helicone offer prompt registries with version history, diff views, and experiment tooling, so that a prompt edit is traceable and reversible.

The pattern matters more than the tool. The pattern is: a prompt is not a configuration value. A prompt is a behaviour-defining instruction with operational characteristics — failure modes, edge cases, latency profile, cost profile, evaluation surface — that demand the same kind of governance you give to code.

Two design choices are worth being explicit about.

Inline versus registry. Inline prompts (in the repo, in the code) are simplest, naturally source-controlled, and trivially reviewable. Registry prompts (managed in an external tool) decouple prompt updates from code deploys — useful when prompt iteration needs to happen faster than the release cycle. The trade-off is the registry now sits in your operational dependency graph, with all the availability, audit, and rollback concerns that implies. For most systems, inline is the right starting point; move to a registry only when there is a concrete reason that justifies the new dependency.

Templating discipline. Prompts that interpolate runtime variables — user names, tool outputs, retrieved documents — should make those variables explicit, named, and validated. An untyped template that silently consumes whatever upstream input arrives is the prompt equivalent of dynamic SQL: it works until it doesn't, and the failure is uninspectable. Pin the variables. Validate the shapes. Treat the template as a function signature.

Models as dependencies

Treat every model the way you would treat a third-party library upgrade.

For classical dependencies, the discipline is well understood: pin to a specific version, not a moving tag; read the changelog before upgrading; run a regression suite against the new version before promoting; roll out the upgrade gradually, with a rollback path; subscribe to the vendor's deprecation schedule.

For LLM models, the same discipline applies — with one important difference. Classical dependencies, properly pinned, do not change. LLM models, even when pinned, can be deprecated by the vendor on a published schedule, after which the pinned version stops responding. Pinning buys you predictability for a window, not forever.

Two patterns matter.

Pin to dated revisions, not aliases. Major LLM vendors publish dated model revisions — names like claude-3-5-sonnet-20240620 or gpt-4o-2024-11-20. The dated form is the equivalent of a Git commit SHA. The alias form (claude-3-5-sonnet-latest, gpt-4o) is the equivalent of a floating branch. Production systems should reference the dated form, full stop. If you cannot point at a specific revision string in your config, you are not running pinned.

Track vendor deprecation as a first-class risk. OpenAI publishes a model deprecation page; Anthropic publishes version policies; the trend across the major vendors is to give months of notice between announcement and shutoff. Treat that calendar the way an enterprise IT department treats an OS support window. Plan the upgrade. Run the eval. Schedule the cutover. Do not be in the position of discovering the deprecation when the API starts returning errors.

When the cutover happens, treat it exactly like any other dependency upgrade. The new model is a new dependency. It needs a regression suite. It needs a shadow-traffic comparison. It needs a canary. Do not let the fact that the vendor scheduled it for you bypass the discipline you would apply to a self-driven upgrade.

Eval-gated deploys

The discipline above buys you traceability — every change becomes visible, diff-able, attributable. It does not, on its own, buy you safety. Safety comes from gating every change behind an eval that has to pass before the change reaches production.

The shape of the gate matters. For a classical code change, the gate is a unit test suite that exercises every code path. For an behaviour-defining change in an LLM system — prompt, model, retrieval, schema — the gate is a regression eval suite that exercises every behaviour the system is contractually required to deliver:

• Held-out examples that the previous version handled correctly — does the new version still handle them?
• Examples that the previous version handled badly — does the new version improve, or at least not regress?
• Examples designed to surface specific failure modes — hallucination, refusal, off-topic drift, schema breakage
• Examples that probe the system at its operational edges — long inputs, ambiguous inputs, hostile inputs

The gate is automated. The gate runs on every prompt change, every model change, every retrieval-index rebuild, every schema edit. The promotion mechanism — code merge, registry promotion, model pinning — refuses to advance if the gate fails. The team does not get to override the gate manually without an audit trail.

The size of the regression suite grows with the system. A young system may start with a few dozen examples; a mature system maintains hundreds, drawn from real production traces, real complaints, real edge cases that have surfaced over time. The suite is not a static asset — it accretes. Every production incident contributes new examples. Every behavioural surprise becomes a held-out test. Every user-reported failure becomes a regression case. Over time, the suite becomes the most complete behavioural specification the system has — and the deploy gate that enforces it becomes the most important deployment-time safety mechanism in the platform.

A change that does not pass eval does not reach production. That is the rule the discipline reduces to.

Rollback semantics for non-deterministic code

Classical rollback is conceptually simple: redeploy the previous version, behaviour returns to where it was. The artefact is byte-identical; the behaviour is byte-identical.

LLM rollback is not that.

A model rollback may bring back a deprecated revision the vendor has already decommissioned. A prompt rollback restores the previous string, but if the model has changed independently underneath, the combined behaviour is new. A retrieval-index rollback restores the previous index, but the documents in the world have moved on. A schema rollback restores the previous tool definitions, but the agent has already learned, in production, to expect the new ones.

The discipline is to treat rollback as a coordinated operation across all four moving parts. Pin the prompt. Pin the model revision. Pin the index. Pin the schema. Capture all four pins together as a release tag. A rollback restores all four pins atomically.

The harder corollary is that rollback paths atrophy unless they are exercised. A rollback that has never been run in production is a rollback the team has no confidence in. Schedule rollback drills the same way mature platform teams schedule disaster-recovery drills — pick a release pin from a few weeks ago, restore all four components atomically in a non-production environment, and verify that the regression eval suite still passes. Catch the broken parts of the rollback path before they are needed at 3 AM.

The other corollary is that rollback windows are bounded by what the system can still serve. If the vendor has retired the model revision you rolled back to, rollback no longer works. Treat that horizon — the vendor's deprecation date — as the effective end of your rollback window for that release. Plan accordingly.

Failure modes

These are the failures the discipline is designed to prevent. Severity-ordered, worst first.

Silent vendor upgrade. The team uses a model alias. The vendor revises the model in place. Production behaviour shifts overnight with no signal in source control. Detection latency is bounded only by how aggressively the team is running online evals. Mitigation: pin to dated revisions; never reference aliases in production.

Prompt drift in a registry. A prompt registry sits outside source control. A team member edits the prompt through an admin UI. There is no diff, no review, no eval gate. The change reaches production immediately. Detection depends on whether someone notices the behaviour change in dashboards. Mitigation: every registry change goes through review; every registry change runs the eval gate; the registry maintains a full version history. A registry edit is not a separate workflow from a code change — it is a code change, just stored elsewhere.

Schema drift across services. The agent expects a tool to return one shape. A downstream service quietly changes the response shape. The agent now produces broken outputs on a specific class of input. The change does not register in any LLM-system component — it is a downstream contract break. Mitigation: treat function-calling schemas as inter-service contracts; version them; add consumer-driven contract tests; alert when production traces show schema-mismatch errors.

Dependency-surprise from prompt-included context. A prompt template interpolates retrieved documents. The retrieval index is rebuilt on a cron. New documents enter the index. Some of them carry instructions or content that interact with the prompt in unintended ways. Behaviour shifts. Detection requires correlating the index-build timestamp against the behaviour-shift timestamp — a join the team is rarely set up to do. Mitigation: version the retrieval index; capture the index version in every trace; treat the index as one of the pinned components in a release.

Lossy rollback. The team rolls back the prompt but not the model, or the model but not the prompt. The resulting combined behaviour is something the team has never tested. The dashboards stabilise but the system is now in a never-validated state. Mitigation: rollbacks are coordinated, all four pins together, captured as release tags.

The architect's checklist

The shipping bar for prompt and model versioning is structural. Walk this checklist before declaring a system production-ready:

🗓️ Every prompt is source-controlled — either inline in the repo or in a registry with a full version history visible to the team

🗓️ Every model reference in production is a dated revision, not an alias

🗓️ Every prompt change, model change, index rebuild, and schema change runs a regression eval suite before promotion

🗓️ The promotion mechanism refuses to advance on a failed eval, with no informal override path

🗓️ Releases are tagged across all four pinned components — prompt version, model revision, index version, schema version — so a rollback is a single atomic operation

🗓️ The team subscribes to vendor deprecation schedules and plans upgrades against them as known work, not as surprise work

🗓️ Production traces capture all four pin identifiers, so any behaviour shift can be correlated against a versioned cause

🗓️ Schema changes between services run as consumer-driven contracts, alerted on at the trace layer

🗓️ The rollback window is documented, bounded by the earliest vendor deprecation date in the current release pin

If a system meets this checklist, "the diff is empty and the behaviour isn't" stops being a real failure mode. There is no behaviour-defining surface that lives outside the versioning loop.

The mental model

Think of the LLM system as a service whose source code is distributed across four repositories — the prompt repo, the model repo, the index repo, the schema repo. None of those repositories is one the team owns end-to-end. The prompt repo lives in the registry. The model repo lives at the vendor. The index repo lives at the pipeline that builds the index. The schema repo is fragmented across the services the agent calls.

The versioning discipline is the discipline of making those four distributed repositories behave as if they were one. Pinned. Tagged. Diff-able. Rollback-safe. Eval-gated.

That is the work. It is not glamorous. It does not surface in product demos. It is, however, what stands between a system that ships reliably and a system that drifts.

A prompt is code. A model is a dependency. Treat them like the things they are, and the diff will stop lying to you.

🗓️ Coming next: Observability for Probabilistic Systems — why the classical four-signal model misses the failures that matter most in LLM systems, and what to instrument instead.

🗓️ Series progress: