🗓️ Last updated: September 2026

Three coding agents start from the same checkout. One upgrades a package. One refactors the service that uses it. One fixes a test by changing the shared fixture. Each finishes successfully in the filesystem it happened to inherit. The combined branch resembles a group project in which nobody was told there were other group members.

Parallel coding is not primarily a coordination problem. It is first an isolation problem.

The unit of work needs a boundary

Give each agent a dedicated branch and working tree created from a known commit. Record that base commit with the task. The agent may inspect the wider repository, but it should modify only its own checkout. This creates a reviewable patch, prevents uncommitted changes from bleeding between tasks, and makes cleanup unsurprising.

Git worktrees are a practical local mechanism because they allow multiple working trees attached to one repository. Separate clones or ephemeral runners may be more appropriate for remote execution. The mechanism matters less than the invariant: one task owns one isolated filesystem and one branch.

Do not share build outputs between concurrent tasks unless the toolchain explicitly supports it. Generated files, package caches, local databases, ports, and containers can create hidden coupling. A clean source branch is not isolated if every agent points at the same mutable test database.

Concurrency starts with task topology

Before dispatching work, identify dependencies between tasks. Two independent documentation changes can proceed together. A schema migration and the code consuming that schema have an ordering decision. Two changes to the same central file may be logically independent and mechanically expensive to merge.

A useful dispatcher classifies tasks as independent, ordered, or conflicting. Independent tasks may run concurrently. Ordered tasks wait for a predecessor or branch from its result. Conflicting tasks should be combined or deliberately sequenced.

This is not multi agent choreography. The agents do not need to hold a committee meeting. The delivery system needs to understand the version graph.

Preserve the handoff artifact

Each task should finish with more than changed files. Require a compact handoff containing the base commit, commits produced, files changed, tests executed, test results, unresolved concerns, and any deviation from the contract. This artifact lets a reviewer understand the work without replaying an entire conversation.

Long running agent research from Anthropic similarly emphasizes incremental progress and structured artifacts that carry state across sessions. Git history already provides part of that structure. Use it. A transcript is useful for diagnosis, but a coherent commit remains the native unit of software review.

Merge through evidence

An agent branch should pass its scoped verification before entering the merge queue. The target branch must then run the required integration checks because other changes may have landed. Never assume that two green branches produce one green branch. Version control has spent decades offering educational examples to the contrary.

For conflicts, ask for a fresh analysis against the updated target. A textual merge that removes conflict markers may still violate behavior. Rerun tests and review the combined intent. High risk conflicts should return to a human owner.

Keep commits small and meaningful. An agent that mixes a functional change, formatting sweep, package upgrade, and generated file refresh into one commit has created a review hostage situation.

Control shared resources

Allocate unique database names, container names, ports, and temporary directories per task. Use scoped credentials and disposable environments for untrusted execution. Do not let two agents deploy to the same shared development slot unless testing race conditions is the actual assignment.

Limit concurrency according to the resources that are genuinely scarce: build capacity, external API quotas, integration environments, and reviewer attention. Starting fifty tasks is easy. Reviewing fifty simultaneous architectural decisions is merely a creative way to move the queue.

Failure modes

The first failure is branch isolation without environment isolation. Tests alter shared state and create results that cannot be reproduced.

The second is stale success. A branch passed tests against an old base and enters the target without current integration checks.

The third is invisible dependency. Two tasks were described as independent but change the same contract. A task topology review would have exposed the relationship before compute and reviewer time were spent.

The fourth is transcript dependence. Nobody can understand the patch without reading the agent's entire session. The code, commits, tests, and handoff should explain the result.

The architect's checklist

1. Create one isolated branch and filesystem for every task.

2. Record the exact base commit and execution environment.

3. Classify task dependencies before parallel dispatch.

4. Isolate mutable test resources and deployment targets.

5. Require scoped tests before queueing and integration tests after merging.

6. Preserve a structured handoff beside the commits.

Parallel agents can increase throughput. Without isolation, they mostly increase the speed at which unrelated assumptions meet in one branch.

Sources

1. Git documentation for multiple worktrees

2. Anthropic on harnesses for long running agents

🗓️ Last updated: September 2026

The coding agent reports that the feature is complete. The project compiles. Four tests pass. The fifth test was deleted because it represented "legacy behavior." This is one way to achieve a green build, in the same sense that removing the battery improves a smoke alarm's noise profile.

Generated code does not need a new definition of correctness. It needs a stronger and more explicit path to the definition the system already has.

Verification is not model evaluation

An LLM evaluation asks whether a model or agent behaves acceptably across a set of tasks. Software verification asks whether this particular change preserves the required behavior of this particular system. Series 3 covered the first problem. This article is about the second.

The distinction prevents a common mistake. A strong benchmark score does not make an individual patch correct. A weak benchmark score does not make a passing, reviewed patch incorrect. The unit of evidence is the change and the repository around it.

Build a verification ladder

Every task should climb a known sequence of checks. The exact tools vary, but the layers are stable.

1. Formatting and generated file checks

2. Compilation and static analysis

3. Focused unit tests

4. Contract and integration tests

5. End to end behavior where the risk requires it

6. Security, compatibility, and performance checks for affected surfaces

Run cheap, local checks first so obvious defects return quickly. Run broad checks before merge. The task contract should identify which layers are required. The agent may discover that additional checks are prudent, but it may not decide that required checks feel unnecessary today.

Protect the oracle

A test is useful only if its expected result comes from an independent requirement. When an agent changes production code and rewrites the failing assertion to match, the build may become green without the software becoming right.

Treat changes to existing tests as a separate review concern. Require an explanation that ties the changed expectation to a contract, requirement, approved behavior change, or corrected defect in the test. A test update is not suspicious by default. An unexplained test update is.

For defects, prefer a regression test that fails before the implementation change and passes after it. For new behavior, connect acceptance criteria to observable tests. For refactoring, require existing behavior to remain stable and use characterization tests where coverage is weak.

Verify more than the happy path

Coding agents are very good at satisfying the example placed directly in front of them. Production defects often live one input beyond that example. Include boundary values, invalid state, cancellation, retries, concurrency, authorization, serialization, and partial failure according to the domain.

Use property based or generative tests when invariants matter more than individual examples. Use contract tests for service boundaries. Use snapshot tests carefully; a large snapshot approved without inspection is simply a text file that has learned to say yes.

Make hidden checks legitimate

Some acceptance checks should remain outside the agent's immediate edit surface. This prevents the implementation and its only judge from being produced by the same reasoning process. Hidden checks are common in benchmarks, but the production equivalent is broader: contract suites owned by another component, policy tests, compatibility checks, or independently maintained acceptance tests.

Hidden does not mean mysterious. Engineers should know the categories of behavior being protected. Surprise requirements are poor governance. Independent evidence is good verification.

Record the evidence

The pull request should state exactly which commands ran, their results, and which checks could not run. Continuous integration remains the authoritative execution environment. Local success is useful feedback, not a substitute for clean execution on the target platform.

Store test reports and relevant artifacts. For a UI change, retain screenshots or visual comparisons where appropriate. For an API change, retain contract results. For a migration, retain forward and rollback evidence. "Tests passed" is less useful than it sounds when nobody can identify the tests.

Failure modes

The first failure is compile driven development. Compilation proves that the compiler accepted the program. The compiler has never claimed to understand the business.

The second is test laundering. The agent modifies the test until the implementation passes, with no external evidence that behavior should change.

The third is flaky acceptance. Repeated reruns eventually create a green check. Fix or quarantine the flaky test under an explicit policy; do not convert probability into proof.

The fourth is broad change with narrow verification. A shared package changes, but only the local unit suite runs. Verification scope should follow impact, not convenience.

The architect's checklist

1. Define a standard verification ladder for the repository.

2. Tie required checks to the task's risk and affected surfaces.

3. Review changed expectations separately from implementation changes.

4. Require regression evidence for defects and contract evidence for interfaces.

5. Run authoritative checks in a clean continuous integration environment.

6. Preserve named commands, reports, and unresolved failures in the handoff.

The agent can write the patch and some of its tests. It cannot be the sole author of reality. That position remains occupied by the system's contracts, users, and occasionally production at three in the morning.

Sources

1. SWE-bench repository and benchmark documentation

2. Anthropic on evaluating AI agents

🗓️ Last updated: September 2026

A pull request arrives with thirty seven changed files, polished comments, comprehensive tests, and a description that explains the architecture with unusual confidence. The reviewer reads the summary, sees green checks, and approves. Two days later the team discovers that the patch duplicated an authorization rule and bypassed the actual policy service.

Nothing in the pull request looked careless. That was part of the problem.

Code review must change when producing plausible code becomes cheap. The reviewer's job is no longer to reward effort or infer care from presentation. It is to test whether the change belongs in the system.

Review intent before implementation

Begin with the task contract. What outcome was requested? Which boundaries were protected? Which files were expected to change? Which checks were required? If the pull request cannot answer these questions, reviewing the diff immediately is premature.

Then inspect the change map. Group files by purpose: production behavior, tests, configuration, dependencies, migrations, generated output, and documentation. Unexpected groups deserve attention. A small feature that changes deployment configuration may be correct, but it has crossed a boundary the task should explain.

Read the dangerous parts first

Review by risk, not file order. Start with public contracts, authentication, authorization, data access, migrations, concurrency, error handling, dependencies, and destructive operations. Generated formatting changes can wait. The review interface may sort alphabetically. Architecture does not.

Ask four questions of every material change:

1. Is this behavior required?

2. Is this the correct architectural location?

3. What existing invariant could it violate?

4. What evidence would expose that violation?

These questions work regardless of whether the author is a human, an agent, or the increasingly common partnership in which neither wishes to discuss individual responsibility.

Demand provenance without demanding a transcript

The reviewer needs the base commit, task contract, agent and tool version where relevant, files changed, commands run, and unresolved concerns. The reviewer usually does not need thousands of tokens of internal conversation. Preserve the full session for diagnosis according to policy, but present a concise, structured handoff for review.

Require citations from the summary to code, tests, and design decisions. A claim that compatibility was preserved should point to the contract check. A claim that a race was fixed should point to the synchronization change and concurrency test. Fluent explanation is not evidence. It is an invitation to find evidence.

Control diff shape

Large diffs reduce review quality. Ask agents to separate mechanical changes from behavioral ones. Package upgrades, formatting, generated output, and feature logic should not share a commit unless they are inseparable. Small coherent commits allow reviewers to inspect intent and revert safely.

Set escalation thresholds for dependency additions, public API changes, migrations, permission changes, and unusually broad diffs. A threshold need not reject the change. It changes the review route and required owners.

Use code ownership for domain boundaries. A coding agent can edit a payment module. It cannot manufacture the payment team's accountability by adding them to the pull request description.

Review the tests as production code

Tests written by the same agent are useful but not independent. Inspect whether they prove the requirement or simply reproduce the implementation. Look for deleted cases, weakened assertions, excessive mocking, snapshots accepted wholesale, and tests that never exercise the changed branch.

Run focused experiments when the risk warrants it. Revert the production fix and confirm the regression test fails. Change a boundary value. Simulate a timeout. Review is not confined to reading. Sometimes the quickest way to understand code is to make its assumptions uncomfortable.

Preserve human attention

Do not send every generated patch to the same approval queue. Classify work by risk. Documentation and mechanical refactors with strong checks may use lighter review. Contract, security, data, and infrastructure changes require owners with the relevant context.

Measure review load, rework, and escaped defects. If agent throughput rises while reviewer queues and defect rates rise faster, the organization has automated typing and created a new bottleneck. The dashboard may still call this productivity because dashboards enjoy optimism.

Failure modes

The first failure is summary substitution. The reviewer reads the agent's explanation instead of the code. Summaries guide attention; they do not replace inspection.

The second is green check hypnosis. Automated verification reduces risk but cannot prove that the requested behavior was correct or complete.

The third is authorship bias in either direction. Generated code is not automatically suspect, and human code is not automatically wise. Apply the same engineering standards while adjusting scrutiny for scale and provenance.

The architect's checklist

1. Review the task contract before the diff.

2. Inspect high risk boundaries before routine files.

3. Require a structured handoff with evidence links.

4. Separate mechanical and behavioral changes.

5. Route sensitive changes to accountable domain owners.

6. Track reviewer load, rework, and escaped defects together.

The purpose of review is not to determine whether code looks as though a competent person wrote it. Modern tools can arrange that appearance very efficiently. The purpose is to determine whether the change deserves to become part of the system.

Sources

1. GitHub documentation for reviewing proposed changes

2. Anthropic on harness design for long running application development

🗓️ Last updated: September 2026

The pilot begins with one developer and one coding agent. Within a month, several teams have different tools, different repository instructions, different permission models, and different definitions of completion. Management asks how much faster engineering has become. Nobody can say, but the token dashboard is exceptionally detailed.

The unit an engineering organization should optimize is not generated code. It is an accepted, verified, reviewable change. The pull request is the product of the coding agent workflow.

Assemble the delivery loop

A production workflow has seven explicit stages.

1. Intake turns an issue into a task contract.

2. Risk classification chooses permissions, environment, and review route.

3. Provisioning creates an isolated branch, worktree, and disposable runtime.

4. Execution lets the agent inspect, modify, and run scoped checks.

5. Verification runs authoritative repository checks.

6. Review evaluates intent, architecture, evidence, and risk.

7. Integration merges through current target branch checks and records the outcome.

Each stage has an owner and a failure state. The agent may participate in several stages, but it should not silently redefine them. A workflow in which the same process writes the task, implements the change, invents the acceptance test, approves the result, and merges it is efficient in the way an unsigned blank cheque is efficient.

Classify before execution

Use a small risk model based on affected components and action types. Documentation, local tests, and mechanical refactors may run with minimal privileges. Dependency changes, workflows, secrets handling, authentication, authorization, infrastructure, migrations, and public contracts require tighter controls and designated reviewers.

Risk classification controls network access, available credentials, writable paths, runtime limits, and approval requirements. It also decides whether the agent may open a pull request automatically or must return a proposal.

Keep agent environments disposable. Start from a known image and commit. Supply only task specific credentials. Capture commands and artifacts. Destroy the environment after the handoff according to retention policy. A developer laptop with years of ambient credentials is not an execution platform. It is a museum of implicit trust.

Standardize the repository interface

The platform should not hardcode every project's build system. Define a small repository contract: setup, fast verification, full verification, formatting, generated file policy, and instruction discovery. Let each repository implement the contract through checked in scripts or task definitions.

This keeps the platform generic while preserving local truth. It also makes onboarding measurable. A repository that cannot satisfy the contract is not ready for unattended agent work. That finding may be more valuable than the first generated patch.

Keep the model replaceable

Model capability changes rapidly. The durable assets are the task format, repository interface, isolation boundary, verification evidence, review policy, and outcome data. Avoid coupling the delivery workflow to one provider's conversation format or tool names.

Store model and harness versions with each run so results can be compared. Route tasks by capability and risk where useful, but do not treat model selection as the architecture. The model is an important dependency. It is not the entire software delivery system, despite marketing's best efforts.

Measure accepted outcomes

Useful measures follow work from intake to production:

1. Time from ready task to merged change

2. First pass verification rate

3. Human review time and queue delay

4. Rework requested after review

5. Revert and escaped defect rate

6. Percentage of runs that escalate correctly

7. Cost per accepted change

Segment results by task class and risk. Documentation changes and schema migrations should not share one productivity average. Avoid lines of code, prompts sent, and raw pull request count as success measures. A system can maximize all three while making the product worse with remarkable consistency.

Compare against an appropriate baseline and include human time. An agent that saves twenty minutes of implementation and consumes an hour of review has not created forty minutes of innovation. It has moved work to a more senior salary band.

Roll out by repository and task class

Begin with repositories that have reliable setup and verification. Choose bounded task classes. Establish baseline measures. Expand permissions and autonomy only when evidence supports it. Keep a disable path at the organization, repository, and task level.

Use failed runs as repository feedback. Repeated setup failures indicate environment debt. Repeated scope violations indicate weak task contracts or boundaries. Repeated review surprises indicate missing architecture documentation or tests. The agent workflow becomes a diagnostic instrument for the engineering system around it.

Failure modes

The first failure is tool first adoption. The organization buys seats before defining the work and evidence flow. Usage grows; accountable outcomes remain elusive.

The second is universal autonomy. Every repository and task receives the same permissions because policy design would delay the launch announcement.

The third is local optimization. Agent execution becomes faster while verification, review, and integration queues grow.

The fourth is vendor shaped architecture. Repository and process design become dependent on one product's current interface, making future capability improvements oddly expensive to adopt.

The architect's checklist

1. Optimize for accepted changes, not generated code.

2. Make intake, risk, isolation, verification, review, and integration explicit stages.

3. Define a portable repository contract for setup and checks.

4. Provision disposable environments with task specific authority.

5. Record model, harness, task, evidence, review, and outcome data.

6. Expand autonomy only by task class and measured results.

Coding agents do not create a software factory by themselves. They create a very fast workstation. The factory is the delivery system that decides what work enters, what evidence leaves, and what is allowed to reach production.

Sources

1. Visual Studio Code documentation for custom agent instructions

2. Anthropic on harnesses for long running agents

3. GitHub documentation for reviewing proposed changes

🗓️ Last updated: August 2026

The ticket says, "Improve the checkout flow." A coding agent changes validation, renames a response field, adds a dependency, and redesigns the error page. Every change is plausible. None can be confidently accepted because the requested outcome was never defined.

The team calls this agent unpredictability. The ticket was six words long.

Coding agents turn vague intent into code faster than most teams can discover that the intent was vague. The quality of the task contract therefore becomes a delivery constraint. A good contract does not prescribe every line. It defines the outcome, the permitted surface, the evidence of completion, and the limits within which engineering judgment may operate.

From ticket to executable contract

A useful task contains six elements.

1. The observable outcome

2. The current behavior

3. The allowed change surface

4. The constraints that must remain true

5. The verification commands

6. The conditions that require escalation

Consider a stronger version of the checkout request:

Outcome: Reject expired promotion codes before payment authorization.
Current behavior: Expired codes fail after payment authorization begins.
Scope: Checkout application service and its tests.
Constraints: Do not change the public CheckoutResponse schema.
Verification: Run the checkout unit and integration test suites.
Escalate: Stop if the payment provider contract must change.

The agent still chooses the implementation. The team has defined what success means and where independent judgment ends.

Acceptance criteria must observe behavior

Avoid criteria that merely restate implementation wishes. "Use a validator" is not an outcome. "An expired code returns the existing invalid promotion response before any payment authorization call" can be tested.

Good criteria describe externally visible behavior, important invariants, and failure cases. They can be checked by tests, static analysis, contract comparison, or a human observing a defined result. This matters because an agent needs a stopping condition. Without one, it may stop when the code compiles or continue improving the neighborhood until the task becomes a renovation project.

Control the change budget

State the expected surface of change. A small defect should not modify twelve projects without explanation. A migration may legitimately touch hundreds of files. The task should distinguish expected breadth from suspicious breadth.

Use a diff budget as a review signal, not an arbitrary line limit. Ask the agent to stop and report when it needs to change a public contract, add a production dependency, create a migration, weaken a test, or edit outside the named modules. These are architectural transitions. They deserve conscious acceptance rather than energetic autocomplete.

Separate discovery from implementation

Some tickets contain unknowns that should be investigated before code changes begin. Create a discovery phase with a concrete output: relevant files, observed behavior, likely root cause, proposed change, affected contracts, and verification plan. Review that output for high risk work before authorizing implementation.

This is not mandatory ceremony for every typo. It is a way to spend human attention before the expensive decisions become a large diff. Teams already do this with design notes and technical spikes. The agent has not abolished planning. It has merely made skipping planning impressively fast.

Make the environment part of the contract

A task is only verifiable in a stable environment. Record fixture assumptions, required services, feature flags, seed data, and commands. If a test is flaky, say so and fix it. Do not teach the agent to rerun a failing test until statistics become acceptance criteria.

Research benchmarks such as SWE-bench frame repository work as resolving real issues against tests in a defined environment. The broader lesson is more useful than any leaderboard score: task quality, environment stability, and verification quality are inseparable when judging coding work.

Define escalation as success

An agent that stops at an unexpected boundary has behaved correctly. Write explicit escalation conditions and reward them operationally. Examples include ambiguous product behavior, contradictory tests, missing credentials, a suspected security issue, or a change requiring data loss.

If every escalation is treated as failure, the system will optimize for confident improvisation. Humans do this too, although we usually promote it under a different name.

Failure modes

The first failure is solution shaped work: the ticket names a class and method but never defines the user outcome. The implementation can be completed exactly and still solve the wrong problem.

The second is hidden acceptance. A reviewer holds critical requirements in memory and reveals them after the pull request arrives. The agent cannot satisfy criteria it never received. Neither can a new engineer.

The third is self verification. The agent writes a test that matches its implementation and treats agreement between the two as independent evidence. Existing behavior, contract tests, and external acceptance criteria must anchor verification.

The architect's checklist

1. Define an observable outcome and the current behavior.

2. State the permitted modules and protected contracts.

3. Supply exact verification commands and failure cases.

4. Identify changes that require approval before implementation continues.

5. Use a discovery phase for ambiguous or high risk work.

6. Treat a justified escalation as a correct result.

A task contract does not remove judgment. It gives judgment a boundary. That is useful for agents and mildly revolutionary for several backlog grooming meetings.

Sources

1. SWE-bench repository and benchmark documentation

2. Anthropic on evaluating AI agents

🗓️ Last updated: August 2026

A coding agent opens a repository, reads three files, changes a fourth, runs the wrong test command, and announces success. The team blames the model. The repository contains four build scripts, two abandoned architecture documents, and a README last updated when the application still had one database.

The model may indeed be imperfect. It has also been asked to infer a development system that the humans stopped documenting years ago. Apparently intelligence was expected to compensate for archaeology.

Coding agents do not work from source code alone. They work from the repository as an operating environment. File layout, naming, build commands, tests, local instructions, and dependency boundaries all become part of the context from which the agent decides what to inspect and what to change. The repository is therefore not merely where the prompt lands. The repository is most of the prompt.

What this article is not

This is not a guide to writing one enormous instruction file. A repository cannot be made legible by placing a small novel beside an illegible codebase. Instructions help most when they reveal stable facts that the code and tooling cannot express clearly.

Start with the executable truth

An agent needs one reliable path to restore, build, test, lint, and run the project. Those commands should work from a clean checkout. Put them in the repository and run them in continuous integration. If the documented command differs from the CI command, the documentation is decorative.

Keep the default path unsurprising. A command that requires three private environment variables, a manually created database, and knowledge passed down through oral tradition is not a build command. It is an initiation ceremony.

The repository should answer these questions without guesswork:

1. What command establishes a working environment?

2. What is the fastest relevant verification command?

3. Which generated files must not be edited?

4. Where do architectural boundaries live?

5. Which changes require broader tests or human approval?

These answers benefit humans immediately. Agent readiness is often ordinary engineering hygiene with a more impatient customer.

Write instructions for decisions, not syntax

Repository instruction files — AGENTS.md, CLAUDE.md, .github/copilot-instructions.md, and equivalents depending on the tool — can describe project wide commands, conventions, and architectural rules. Folder specific instructions can narrow the guidance for a monorepo or a subsystem. Current Visual Studio Code documentation, for example, supports repository instructions and scoped instruction files (currently experimental) for coding agents.

Useful instructions state facts with operational consequences:

Run dotnet test tests/Orders.UnitTests for changes under src/Orders.
Do not edit files under Generated.
Database migrations require an integration test and schema review.
The Domain project must not reference Infrastructure.

Weak instructions ask for virtues:

Write clean, scalable, production ready code.
Follow best practices.
Be careful.

The second group sounds respectable and constrains almost nothing. A model can produce a confident interpretation of "best practices" just as easily as a committee can.

Keep instructions short enough to remain visible. Remove rules that tools already enforce. Formatting belongs in a formatter. Dependency boundaries belong in architecture tests. Secret scanning belongs in a scanner. Prose is appropriate where judgment is required, but prose should not impersonate enforcement.

Design the repository map

Large repositories need progressive disclosure. The root should explain the system shape and point toward deeper documentation. Each major area should identify its purpose, public boundaries, verification commands, and local hazards. The agent can then load the smallest useful context instead of pouring the entire repository into one conversation and hoping relevance emerges from volume.

A concise architecture map should name the deployable units, shared libraries, data stores, external contracts, and ownership boundaries. Link to source files and executable tests where possible. Avoid copying implementation details into documents that will drift independently.

Context engineering research from Anthropic emphasizes curating relevant information and controlling context growth during long running work. The same principle applies at repository scale. More context is not automatically better context. A hundred thousand tokens of stale detail is still stale detail, now with a larger invoice.

Make boundaries machine visible

If a layer must not reference another layer, test that rule. If an API contract must remain compatible, validate it. If a package can only be used in one module, encode the restriction in project structure or analysis. An agent is much more reliable when the repository rejects invalid changes than when a paragraph merely discourages them.

This is the key shift. The repository should be a constrained environment in which the easiest successful change is also an architecturally valid change.

Failure modes

The first failure is the repository constitution: hundreds of instructions that compete with the actual task and with one another. Stable, specific rules beat comprehensive prose.

The second is undocumented setup state. The agent succeeds in an environment that contains cached packages, old build outputs, or developer credentials, then fails from a clean runner.

The third is context by proximity. A file sits nearby, so the agent assumes it belongs to the same architectural boundary. Good naming and explicit maps reduce that ambiguity.

The fourth is instruction drift. If the command in AGENTS.md no longer works, remove or repair it immediately. Incorrect context is worse than absent context because it arrives wearing authority.

The architect's checklist

1. Make restore, build, and test commands work from a clean checkout.

2. Keep one documented default path aligned with continuous integration.

3. Use repository instructions for stable decisions and local constraints.

4. Provide a concise map of deployables, modules, contracts, and owners.

5. Encode architecture rules in tests and tooling wherever possible.

6. Test the repository setup in the same isolated environment used by agents.

A coding agent reveals the quality of the environment around the code. If it cannot discover how the system works, the uncomfortable possibility is that the repository has been relying on human memory as infrastructure.

Sources

1. Visual Studio Code documentation for custom agent instructions

2. Anthropic on effective context engineering for agents

🗓️ Last updated: August 2026

It is the architecture review the Friday before a greenfield It is the architecture review the Friday before a greenfield .NET agentic service ships. The senior architect projects Program.cs onto the meeting room wall: sixty lines of generic host wire up. A staff engineer with three months on the team asks the only question that matters. Where does the prompt template live? The architect points at one line: services.Configure<PromptCatalog>(builder.Configuration.GetSection("Prompts"));. The catalog binds from appsettings.json, overlays from the environment specific file, surfaces through IOptionsMonitor<PromptCatalog> for hot reload, and is read on each invocation by the agent host. The whole answer takes ninety seconds. The architecture is defensible because every question of "where does X live" has exactly one answer the team can point to in the wire up file.

That is the bar a reference architecture clears. The five preceding articles in this series each described a component: the Microsoft Agent Framework substrate, the IChatClient middleware pipeline, structured outputs as a typed contract, hosted service lifecycle, and the streaming surface. The closer assembles them into a single service the architect can defend before the first commit.

What This Article Is Not

This is not a repository you can copy and paste. It is not a vendor scaffold from dotnet new. It is not an Azure deployment recipe; the deployment surface here is generic .NET hosting, and the cloud specific wire up is a wrapper around the same generic host. It is not the only valid layering for a .NET agentic service; it is the layering a senior architect should be able to defend in a review and adjust deliberately when the project's constraints demand a different cut.

The Thesis

A .NET agentic service is a generic host with five seams the architect owns. The layering across Domain, Application, Infrastructure, and Hosting is the same canonical layering ASP.NET Core taught a decade ago. What changes is the lifetime of the IChatClient pipeline, the placement of the AgentSessionStore, the configuration surface for prompt versions and model selection, the OpenTelemetry pipeline that wraps the agent middleware, and the choice between a unified web host or a worker plus gateway split. Get those five seams right and the service is boring to operate. Get one wrong and it surfaces in production at the worst possible moment: a lifetime mismatch that captures stale options into a singleton, an environment variable override that silently changes the model on production traffic, a span graph that explodes when streaming turns chunks into events.

What the Classical .NET Version Looked Like

The pre agentic .NET version of "service that calls an external API and exposes results to a client" was small. A web host. A typed HTTP client through IHttpClientFactory. A Minimal API endpoint. An IOptions<T> block bound from appsettings.json. A logger at a constructor. The layering was implicit because the surface was small and the failure modes were familiar.

An agentic service breaks that pattern. The model client is the substrate of every interesting code path. The conversation has state that outlives the request. The middleware wrapping the model call carries telemetry, function invocation, structured output translation, and cost guardrails. The runtime mixes interactive HTTP and long running background work. The configuration surface needs to allow prompt version rollouts and model swaps without a redeploy. The shape that emerges is recognisably the four layer pattern ASP.NET Core has always supported, but each layer carries weight the classical small API version never had.

The Four Layers

Domain holds the records that describe the service's contract with itself: the ChatMessage and ChatRole from Microsoft.Extensions.AI, the application specific structured output records from Article 3, the session and identity types, and the tool contracts. Domain is dependency free. It compiles without a chat client implementation, without a hosting reference. The test is whether the project builds on its own.

Application holds the use cases: orchestration that takes a user request, resolves the right prompt, calls the agent, runs the tools, validates the structured output, and returns a result. Application depends on Domain and on the abstractions in Microsoft.Extensions.AI, IChatClient, IEmbeddingGenerator, ChatOptions, never on a specific provider. The test is whether the use case can be unit tested with a stub IChatClient that returns a canned ChatResponse.

Infrastructure holds the implementations: the configured OpenAIClient or AzureOpenAIClient wrapped into an IChatClient, the Redis backed AgentSessionStore from Article 4, the distributed cache, the function invocation registrations, and the OpenTelemetry exporter wiring. Infrastructure depends on Application's abstractions. The test is whether a swap from OpenAI to Azure OpenAI changes only the registration line in Program.cs.

Hosting is the composition root: Program.cs itself. It reads configuration, wires the M.E.AI middleware pipeline, registers the agent and session store, configures IHostedService workers from Article 4, maps the streaming endpoints from Article 5, and adds OpenTelemetry. Hosting depends on every other layer. The test is whether a new engineer can read it top to bottom and answer every "where does X live" question without opening another file.

The four layer cut is older than the Microsoft Agent Framework and older than Microsoft.Extensions.AI. The discipline, no chat client implementation leaking into a use case, no Program.cs line bleeding into a domain record, is what keeps the service defensible when the model surface shifts again in eighteen months.

Dependency Injection Lifetimes

The five seams that decide whether the agentic service runs cleanly all sit in the DI container. The lifetimes are not interchangeable.

IChatClient is Singleton. The AddChatClient extension registers the pipeline as a singleton by default, and the framework's own XML doc says exactly that: "The client is registered as a singleton service." Per request state lives in the ChatOptions and IEnumerable<ChatMessage> passed into GetResponseAsync or GetStreamingResponseAsync. The FunctionInvokingChatClient carries instance configuration, AllowConcurrentInvocation, MaximumIterationsPerRequest, but its per invocation context is tracked in an AsyncLocal<FunctionInvocationContext?>, so the singleton lifetime is safe under concurrent requests.

ChatClientAgent is Singleton. The Microsoft.Agents.AI ChatClientAgent from Article 1 is a thin sealed class over IChatClient. Its per conversation state lives in the AgentSession callers pass into RunAsync and RunStreamingAsync, not on the agent instance.

AgentSessionStore is Singleton. The Redis backed or in memory store from Article 4 is a stateless wrapper over a connection multiplexer. The persistence layer is the source of state; the store object itself is not.

AgentSession is per conversation, resolved at runtime, not registered. Sessions live and die with the conversation, not with the request. Application code resolves them from the store using a conversation identifier; they do not have a DI lifetime because they are not DI services. The session is an entity; the store is the service.

Tools registered as AIFunction instances are Singleton when stateless and resolved from scope when they need request scoped dependencies. The wrapper pattern is to register the stateful dependency as Scoped and to construct the function inside the request handler with the scoped service captured.

IOptions<T>, IOptionsSnapshot<T>, and IOptionsMonitor<T> are three lifetimes of the same binding. IOptions<T> resolves once at app start. IOptionsSnapshot<T> is "used to access the value of TOptions for the lifetime of a request", the framework's own XML doc, which makes it Scoped. IOptionsMonitor<T> is Singleton with a hot reload event via OnChange(Action<TOptions, string?>). Prompt catalogs and model selection are IOptionsMonitor<T> so a config change rolls out without a redeploy. Static structural settings, connection strings, maximum iteration counts, are IOptions<T>.

The lifetime mismatch the architect catches at review is a Scoped service captured inside a Singleton's constructor, the classic "captive dependency" the .NET container's scope validation mode flags in Development but not in Production. Turning on scope validation in Production is a defensible decision for an agentic service because the cost of a captured stale options instance is an entire production conversation served against the wrong prompt.

Configuration

The configuration sources the generic host reads are fixed in order, last source wins. The framework loads appsettings.json, then appsettings.{EnvironmentName}.json, then optional {ApplicationName}.settings.json and its environment specific overlay if the application name is set, then User Secrets when EnvironmentName == "Development", then environment variables, then command line arguments. The ordering is implemented in HostingHostBuilderExtensions.ApplyDefaultAppConfiguration and is the canonical pattern for Host.CreateApplicationBuilder and WebApplication.CreateBuilder. The architect's lever is which layer carries which kind of value.

Structural defaults live in appsettings.json and ship in the container image. Per environment overrides live in appsettings.Production.json and friends. Secrets do not live in any of those; they live in the cloud secret store and are pulled in by a configuration provider. builder.Configuration.AddAzureKeyVault(new Uri("https://kv.vault.azure.net/"), new DefaultAzureCredential()) from the Azure.Extensions.AspNetCore.Configuration.Secrets package overlays Key Vault secrets onto the configuration root. The same pattern applies to AWS Secrets Manager and HashiCorp Vault via their providers. The discipline is that the API key is never in source, never in the image, and never typed into a CI variable that is not itself protected.

The prompt catalog and the model selection are the two surfaces that change most often without code. IOptionsMonitor<PromptCatalog> bound to a Prompts section gives the operations team a path to roll a prompt version without a deploy: flip the version pointer, the file watcher fires, OnChange callbacks rebuild dependent caches. The same pattern handles model swaps via a ModelSelection record with Provider, ModelId, and Endpoint. The middleware is wired once at startup; the values it reads are resolved fresh on each invocation.

The trap with hot reload is that not every dependent state recomputes on OnChange. A cache key generator that captures the prompt version at registration time will never see a new version come through, because the key was closed over at startup. The fix is to make the generator a function of IOptionsMonitor<T>.CurrentValue, evaluated per request.

Observability

The OpenTelemetry pipeline for the agentic service has three layers. The M.E.AI middleware emits the GenAI semantic conventions tags through its OpenTelemetryChatClient, model id, prompt token count, completion token count, finish reason, on each call. The application code adds its own activity spans through ActivitySource for the use case layer above the chat client. The OpenTelemetry .NET SDK collects both, batches, and exports through the OTLP exporter to the collector.

The wire up is small. services.AddOpenTelemetry() from the OpenTelemetry.Extensions.Hosting package returns the OTel builder. .ConfigureResource(r => r.AddService("agentic-service")) sets the service identity. .WithTracing(t => t.AddSource("YourApp.UseCases").AddSource("Experimental.Microsoft.Extensions.AI").AddOtlpExporter()) subscribes the tracer to both the application's ActivitySource and the M.E.AI source, and ships the spans to the collector. .WithMetrics(m => m.AddMeter("Experimental.Microsoft.Extensions.AI").AddOtlpExporter()) does the same for metrics; the meter name is identical to the trace source name because the M.E.AI middleware constructs both from OpenTelemetryConsts.DefaultSourceName.

The UseOpenTelemetry() extension on ChatClientBuilder is the M.E.AI hook that turns the middleware on. It is one line in the builder pipeline. The OTel SDK in the host process picks up the activity through AddSource because the middleware writes to a known source name. The two halves of the wire up, middleware on the builder, source on the tracer provider, are deliberately separate so the architect can compose them with whatever resource attributes, samplers, and exporters the deployment demands.

The trap on the OTel pipeline is span explosion under streaming. Article 5 covered why the streaming chat span lifetime spans the entire enumeration. The corollary is that per chunk span events on a high traffic streaming endpoint can blow up the collector's queue. The defence is the sampler, ParentBased(TraceIdRatioBased(0.05)) for production traffic, full sampling in staging, and a per event budget cap on chunk level events.

Deployment

The deployment surface is one of two shapes. The first is the unified web host: a single ASP.NET Core process serving the HTTP API, running the SignalR hub from Article 5, and hosting the background workers from Article 4 in the same process. The second is the worker plus gateway split: a Minimal API gateway running the streaming endpoints and SignalR hub, plus worker processes running the agent loop as BackgroundService. The two shapes share the same composition root pattern.

The decision between them is operational, not architectural. The unified host is simpler to deploy: one container, one set of health probes, one log stream. The split is the right shape when the worker fleet scales independently of the gateway, or when a long running agent loop should not share a pod with traffic serving threads. Either shape uses Host.CreateApplicationBuilder(args) for the worker, WebApplication.CreateBuilder(args) for the web host, and the same Microsoft.Extensions.AI, Microsoft.Agents.AI, and OpenTelemetry packages in both projects.

The composition root for the unified web host fits on one screen.

var builder = WebApplication.CreateBuilder(args);

builder.Configuration.AddAzureKeyVault(
    new Uri(builder.Configuration["KeyVault:Uri"]!),
    new DefaultAzureCredential());

builder.Services.Configure<PromptCatalog>(
    builder.Configuration.GetSection("Prompts"));
builder.Services.Configure<ModelSelection>(
    builder.Configuration.GetSection("Model"));

builder.Services.AddChatClient(sp =>
{
    var model = sp.GetRequiredService<IOptionsMonitor<ModelSelection>>().CurrentValue;
    return new AzureOpenAIClient(new Uri(model.Endpoint), new DefaultAzureCredential())
        .GetChatClient(model.ModelId)
        .AsIChatClient();
})
.UseOpenTelemetry()
.UseLogging()
.UseDistributedCache()
.UseFunctionInvocation();

builder.Services.AddSingleton<AgentSessionStore, RedisAgentSessionStore>();
builder.Services.AddSingleton<ChatClientAgent>(sp =>
    new ChatClientAgent(sp.GetRequiredService<IChatClient>(),
        new ChatClientAgentOptions { Name = "PlatformCopilot" }));

builder.Services.AddHostedService<OrderInvestigationWorker>();
builder.Services.AddSignalR();

builder.Services.AddOpenTelemetry()
    .ConfigureResource(r => r.AddService("agentic-service"))
    .WithTracing(t => t
        .AddSource("PlatformCopilot.UseCases")
        .AddSource("Experimental.Microsoft.Extensions.AI")
        .AddOtlpExporter())
    .WithMetrics(m => m
        .AddMeter("Experimental.Microsoft.Extensions.AI")
        .AddOtlpExporter());

builder.Services.AddHealthChecks()
    .AddCheck<AgentLivenessCheck>("agent", tags: new[] { "live" });

var app = builder.Build();
app.MapAgentEndpoints();
app.MapHub<ChatHub>("/hub/chat");
app.MapHealthChecks("/health/live",
    new HealthCheckOptions { Predicate = r => r.Tags.Contains("live") });
app.Run();

Every line of that file answers one of the architect's reviewable questions. Where do secrets come from? Key Vault. Where is the prompt catalog? Prompts section, monitored. What is the model? Model section, hot reloadable. What lifetime is the chat client? Singleton, default. Where does session state live? RedisAgentSessionStore, singleton over a Redis multiplexer. What runs in the background? OrderInvestigationWorker from Article 4. What does streaming use? ChatHub from Article 5. The composition root reads like a series of architectural commitments, each one a single line.

The configuration shape that file binds against is equally small.

{
  "KeyVault": { "Uri": "https://kv-platform-prod.vault.azure.net/" },
  "Model": {
    "Provider": "AzureOpenAI",
    "Endpoint": "https://oai-platform.openai.azure.com/",
    "ModelId": "gpt-4o-2024-11-20"
  },
  "Prompts": {
    "Version": "2026-08-15",
    "Catalog": {
      "Triage": "You are an internal platform copilot.",
      "Summary": "Summarise the resolution in three sentences."
    }
  },
  "Otlp": { "Endpoint": "http://otel-collector.observability:4317" }
}

The PromptCatalog and ModelSelection records on the application side are plain C# records with init only setters, bound by name from the matching configuration sections. Adding a prompt is one entry under Prompts:Catalog. Rolling a prompt version is one change to Prompts:Version. Swapping models is one change to Model:ModelId. The deploy artefact does not change.

Failure Modes, Worst First

The well architected version of this surface is invisible. The failure modes are the ones that surface as production incidents.

The first is the captive dependency. A Scoped service captured into a Singleton's constructor, usually IOptionsSnapshot<T> held by a Singleton agent, silently freezes at construction time. Production traffic serves against the wrong prompt for the lifetime of the process. The defence is validateScopes: true on the service provider in Production, not only Development.

The second is the hot reload that does not propagate. IOptionsMonitor<T>.OnChange fires on the source's reload event. A computed value derived from CurrentValue and cached without an OnChange subscription will serve the stale version forever. The defence is to call CurrentValue at the point of use, never at registration time.

The third is the secret in the prompt history. A user prompt that includes an API key or a customer record ends up in the model provider's request log, in the distributed cache, and in the OpenTelemetry span as a tag value. Once the secret is in a third party log the rotation cost is real. The defence is a pre flight redaction pass on the message list before it reaches GetResponseAsync, plus an OTel tag filter to ensure the span does not carry the raw content.

The fourth is the configuration drift between staging and production. The two appsettings.{Environment}.json files diverge over time. A change made to one without the other surfaces as a "works in staging" incident. The defence is a single source of truth for environment specific values, usually a shared store like Azure App Configuration, with the JSON files reserved for genuinely environment local concerns.

The fifth is the OTel span explosion under streaming. Every chunk on a high traffic endpoint emits a chunk level metric and may emit an event on the active span. The collector queue grows, the exporter drops, the application starts blocking when its buffer fills. The defence is a sampler with a low ratio on production traffic and an explicit cap on per event budget on the streaming span.

The sixth is the silent provider swap. A ModelSelection.ModelId change rolled through hot reload swaps the model on next request without an audit trail. The user facing behaviour shifts. The cost surface shifts. The defence is to emit an ActivitySource event whenever the resolved ModelId differs from the previous value, and to gate the configuration source's write surface so only one place can flip the model.

The seventh is the scope validation bypass. The default for WebApplication.CreateBuilder enables scope validation in Development only. Production runs without it. A Scoped service mistakenly registered as a Singleton dependency does not throw; it just captures and freezes. The defence is the explicit builder.Host.UseDefaultServiceProvider(opts => { opts.ValidateScopes = true; opts.ValidateOnBuild = true; }) line in the composition root, applied to every environment.

The Architect's Checklist

🗓️ The four layer cut, Domain, Application, Infrastructure, Hosting, is enforced by project references; no Application project references Microsoft.AspNetCore.*, no Domain project references Microsoft.Extensions.AI.

🗓️ IChatClient is registered as Singleton through services.AddChatClient(...) and the entire ChatClientBuilder middleware pipeline is wired in one place.

🗓️ ChatClientAgent and AgentSessionStore are both Singleton; AgentSession is resolved per conversation from the store, not registered in DI.

🗓️ Prompt versions and model selection bind to IOptionsMonitor<T> and the dependent values are read from CurrentValue on each invocation, not captured at registration.

🗓️ Secrets come from a configuration provider that overlays at runtime, Azure Key Vault, AWS Secrets Manager, or HashiCorp Vault, never from a checked in file or a CI environment variable.

🗓️ Scope validation is enabled in every environment, not only Development, via UseDefaultServiceProvider(opts => opts.ValidateScopes = true) on the host.

🗓️ The OpenTelemetry pipeline subscribes both the application's own ActivitySource and the Microsoft.Extensions.AI source through .AddSource(...) on the tracer provider builder.

🗓️ The OTel sampler ratio is production tuned; streaming endpoints carry an explicit per event budget cap.

🗓️ User content redaction runs before the messages reach GetResponseAsync and before any span tag carries them.

🗓️ The composition root in Program.cs reads top to bottom and answers every "where does X live" question without opening another file.

🗓️ The unified web host versus worker plus gateway decision is documented as an operational choice with a written trigger for when the split becomes worth its operational cost.

🗓️ Every architectural decision in the reference architecture is defensible against the question "what does the code review say if you got this wrong": the answer is a specific failure mode from the list above.

The Architect's Mental Model

The .NET agentic service is a generic host with the four layer cut, the five DI seams, and one composition root the team can read together. The framework choices, Microsoft Agent Framework over Semantic Kernel, Microsoft.Extensions.AI as the substrate, ASP.NET Core for the HTTP surface, IHostedService for background work, OpenTelemetry for observability, are decisions the architect makes once and lives with for a release cycle. The day to day work of operating the service is configuration changes, prompt version rolls, and model swaps, none of which should require a code change.

The well architected reference is not a starter template. It is the smallest set of files a senior architect can defend in a one hour review. The discipline is the same discipline that has carried Microsoft.Extensions.* for a decade: abstract what changes, register what survives, monitor what moves.

Abstract what changes, register what survives, monitor what moves. Everything else is just Program.cs with the lights on.

It is 23:47 on a Thursday. A senior platform engineer types why is the warehouse webhook 502-ing into the internal copilot built on top of a .NET 10 + Microsoft Agent Framework service. The cursor blinks. Two hundred and eighty milliseconds later the first token lands on the page. The answer streams in, word by word, for forty one seconds. By the time the model finishes the engineer has already opened a second tab on the webhook log, read three lines, and stopped reading the assistant output entirely. The shape of that conversation has almost nothing to do with the model's intelligence and almost everything to do with the transport layer underneath it. The first token arrived early enough to hold attention. The connection stayed open without a proxy buffering the chunks. The cancellation token fired the moment the engineer's tab lost focus and the model stopped spending budget on a paragraph nobody would ever read.

The architect's job on the .NET side of that interaction is not the model. It is the discipline of the pipe. .NET has had the right streaming primitive, IAsyncEnumerable<T>, since C# 8 and .NET Core 3.0, more than three years before streaming chat became the default UX. The architect's choice in 2026 is the transport above it, and the cost of getting it wrong is paid in first token latency, in dropped cancellations, in stalled spinners behind misconfigured proxies, and in billing line items for paragraphs the user closed the tab on at the second token.

What This Article Is Not

This is not a tutorial on JavaScript EventSource, the WHATWG fetch streams API, or the Blazor server side rendering pipeline. It is not a benchmark of SSE versus WebSockets versus HTTP/2 server push. The honest answer is "depends on direction, depends on proxy, and you almost certainly want SSE for one way." It is not a SignalR primer.

It is an architect's read on IAsyncEnumerable<T> as the language level streaming primitive on .NET, on Microsoft.Extensions.AI's and the Microsoft Agent Framework's streaming surface as the LLM side adapter, and on the two transport surfaces ASP.NET Core ships out of the box, Minimal APIs with Server Sent Events and SignalR streaming hubs, together with the failure modes that make a streaming endpoint quietly cost more than the model call it wraps.

The Thesis

Token by token streaming is not a UX flourish. It is the contract between the model's slowest to emit byte and the user's willingness to wait. On .NET the streaming primitive is IAsyncEnumerable<T>. The LLM surface is IChatClient.GetStreamingResponseAsync returning IAsyncEnumerable<ChatResponseUpdate>. The agent surface is AIAgent.RunStreamingAsync returning IAsyncEnumerable<AgentResponseUpdate>. The transport above is either Server Sent Events on a Minimal API endpoint via TypedResults.ServerSentEvents, or a SignalR streaming hub method that returns the same IAsyncEnumerable<T> straight to a typed client. The architect's job is to pick the right transport for the direction of the stream, propagate the cancellation token honestly from the browser down through the model call, keep the framing readable end to end without an intermediate proxy buffering it, and put a cost cap on the producer so a reconnect storm does not become a billing event.

What the Classical .NET Version Looked Like

The pre streaming chat .NET pattern was Task<string>. You sent a request, you awaited a complete response, you rendered it. Latency was hidden behind a spinner. If the answer took twelve seconds, the user stared for twelve seconds and then saw the whole answer at once. The streaming workaround in that world was a polling loop, the client polled /api/chat/status?id=... every five hundred milliseconds and the server stored partial output in a cache.

IAsyncEnumerable<T> shipped with C# 8 in September 2019, alongside await foreach and the WithCancellation extension. The [EnumeratorCancellation] attribute that closes the cancellation contract came shortly after with .NET 5 and C# 9 in November 2020. Microsoft.Extensions.AI exposes IChatClient.GetStreamingResponseAsync as a first class member of the chat interface, alongside the non streaming GetResponseAsync. The package's 10.x line ships with the .NET 10 wave. The Microsoft Agent Framework, when it consolidated Semantic Kernel and AutoGen in April 2026, kept the same shape one altitude up: AIAgent.RunStreamingAsync returns IAsyncEnumerable<AgentResponseUpdate>. The classical Task<string> surface still exists, but treating it as the primary surface for an interactive agent in 2026 is a decision to ship a polling architecture.

The Streaming Primitive on .NET

IAsyncEnumerable<T> is the language level contract for an asynchronous sequence. Three pieces of the contract carry the whole article.

The first is [EnumeratorCancellation], the attribute from System.Runtime.CompilerServices that marks one CancellationToken parameter on an async IAsyncEnumerable<T> method as the merge point. The compiler then routes the token that the caller passes to WithCancellation(ct) into that parameter, on top of whatever the method already received. Without the attribute, a caller doing await foreach (var x in source.WithCancellation(ct)) sees no signal from ct reach the body of the iterator. The token is silently dropped, which is the single most expensive one line bug in this entire article.

The second is ConfigureAwait(false) on the awaits inside a library level streaming method. The synchronisation context capture is the wrong default for library code that does not need to marshal back to a UI thread, and on a streaming endpoint that emits hundreds of updates per response the cost is real. The Agent Framework's own AIAgent.RunStreamingAsync uses ConfigureAwait(false) on the inner await foreach. Library code should match.

The third is the shape of the unit you yield. Microsoft.Extensions.AI's ChatResponseUpdate is a delta, a single chunk that carries Role, Text, Contents, MessageId, ResponseId, ConversationId, CreatedAt, FinishReason, ModelId, and a ContinuationToken for stream resumption. Updates are not snapshots; they are slices. The extension ToChatResponseAsync collapses them into a single ChatResponse when server side code needs the assembled message. The agent altitude is the same shape one level up: AgentResponseUpdate carries tool call events, intermediate steps, and the final message.

The trap that catches juniors here is treating updates as snapshots and overwriting the rendered message on every chunk. They are deltas. The renderer concatenates. Get this wrong and the final answer is the last token, repeated.

The LLM Streaming Surface, Two Altitudes

IChatClient.GetStreamingResponseAsync(IEnumerable<ChatMessage> messages, ChatOptions? options, CancellationToken cancellationToken) returns IAsyncEnumerable<ChatResponseUpdate>. This is the M.E.AI substrate the Series 4 Article 2 covered as the unified client layer. The middleware pipeline on ChatClientBuilder, UseLogging, UseOpenTelemetry, UseDistributedCache, UseFunctionInvocation, wraps the streaming method the same way it wraps GetResponseAsync. The OpenTelemetry middleware emits a span whose lifetime spans the entire enumeration, not just the call, which is correct because the streaming call is the enumeration.

AIAgent.RunStreamingAsync(IEnumerable<ChatMessage> messages, AgentSession? session = null, AgentRunOptions? options = null, CancellationToken cancellationToken = default) returns IAsyncEnumerable<AgentResponseUpdate>. The agent altitude adds the session parameter Article 4 introduced: the same AgentSession that survives the pod restart can be the streaming context for the next response. The signature on RunStreamingAsync carries [EnumeratorCancellation] on the cancellation parameter, the way every library grade async iterator should.

The architect picks the altitude by whether the streaming endpoint needs the tool call surface and the agent session boundary, or whether it is happy with a chat level delta. An interactive copilot endpoint that exposes one model and no tools can target IChatClient directly. A multi step agent endpoint that runs tools between updates targets AIAgent.

Minimal API SSE in .NET 10

ASP.NET Core 10 shipped a typed Server Sent Events result on TypedResults and Results. The shape, straight from the public API surface, is one of two overloads, ServerSentEvents<T>(IAsyncEnumerable<SseItem<T>> values) for full control over event type and event id, or ServerSentEvents<T>(IAsyncEnumerable<T> values, string? eventType = null) for the common case of plain typed events. The result wires up the HTTP framing the way an SSE consumer expects: Content-Type: text/event-stream, Cache-Control: no-cache,no-store, Pragma: no-cache, Content-Encoding: identity, and it disables output buffering through the response body feature so chunks are flushed as they arrive. Cancellation flows through HttpContext.RequestAborted, which the framework passes to the underlying SseFormatter.WriteAsync. For non string item types the result JSON serialises with the application's configured JsonOptions; for IAsyncEnumerable<SseItem<string>> it skips JSON and writes the string straight.

The endpoint shape is small enough to read in one breath.

app.MapPost("/chat/stream", (
    [FromBody] ChatRequest request,
    IChatClient chat,
    HttpContext http,
    CancellationToken ct) =>
{
    var messages = new List<ChatMessage>
    {
        new(ChatRole.System, "You are an internal platform copilot."),
        new(ChatRole.User, request.Prompt),
    };

    async IAsyncEnumerable<SseItem<string>> Stream(
        [EnumeratorCancellation] CancellationToken token)
    {
        await foreach (var update in chat
            .GetStreamingResponseAsync(messages, options: null, token)
            .ConfigureAwait(false))
        {
            if (update.Text is { Length: > 0 } text)
            {
                yield return new SseItem<string>(text, eventType: "token");
            }

            if (update.FinishReason is { } reason)
            {
                yield return new SseItem<string>(reason.Value, eventType: "done");
            }
        }
    }

    return TypedResults.ServerSentEvents(Stream(http.RequestAborted));
});

Four things in that block carry the article's architecture. HttpContext.RequestAborted is the canonical cancellation source for an HTTP request: when the browser closes the tab, the proxy drops the connection, or the client cancels the fetch, this token fires. [EnumeratorCancellation] on the iterator's token parameter merges it with whatever the framework passes through. ConfigureAwait(false) is on the inner await foreach because the iterator is library grade code with no business marshalling back to a sync context. And the iterator emits two event types, token for the chunks and done for the final reason, which lets the browser EventSource route them through different handlers without parsing the body.

The non obvious detail is what happens after the model finishes generating but the user has already closed the tab. The token from RequestAborted fires. The await foreach throws OperationCanceledException. The streaming method on the chat client is contractually required to honour the token and stop pulling from the provider. If the underlying provider implementation does not honour cancellation, and a real number of community provider packages did not in 2024 and early 2025, the model keeps generating tokens the user will never see, and the bill keeps ticking. The architect's responsibility is to verify the provider's cancellation behaviour by integration test, not by reading the docs and hoping.

SignalR Streaming Hub Methods

Server Sent Events is one way, server to client. The moment the streaming endpoint needs the user to send messages mid stream, a stop button, a clarification, an interrupt, SSE is no longer enough. SignalR is the other transport surface ASP.NET Core ships, and a streaming hub method on SignalR is a hub method that returns IAsyncEnumerable<T>, ChannelReader<T>, Task<IAsyncEnumerable<T>>, or Task<ChannelReader<T>>. The framework recognises the return type and turns the method into a streaming invocation automatically.

public sealed class ChatHub(IChatClient chat) : Hub
{
    public async IAsyncEnumerable<string> StreamReply(
        string prompt,
        [EnumeratorCancellation] CancellationToken cancellationToken)
    {
        var messages = new List<ChatMessage>
        {
            new(ChatRole.System, "You are an internal platform copilot."),
            new(ChatRole.User, prompt),
        };

        await foreach (var update in chat
            .GetStreamingResponseAsync(messages, options: null, cancellationToken)
            .ConfigureAwait(false))
        {
            if (update.Text is { Length: > 0 } text)
            {
                yield return text;
            }
        }
    }
}

The CancellationToken parameter is not the connection's lifetime token; it is the per invocation token that SignalR fires when the client unsubscribes from this particular stream. On the .NET client, the call is hubConnection.StreamAsync<string>("StreamReply", prompt, ct) returning an IAsyncEnumerable<string> the consuming code reads with await foreach. The browser JavaScript client uses connection.stream("StreamReply", prompt).subscribe({ next, error, complete }) and disposes the subscription to send the cancellation upstream.

The architectural choice between SSE and SignalR comes down to two questions. First, does the stream need to carry messages from the client back to the server while it is open, or does it only need to deliver tokens? Second, does the deployment need to traverse a proxy or CDN that aggressively buffers text/event-stream with no way to set X-Accel-Buffering: no? SignalR's WebSocket transport sidesteps both. SSE pays the proxy tax every time.

Cancellation Propagation, End to End

The cancellation contract on a streaming chat endpoint is one token, flowed through five layers without being dropped at any of them. The browser closes the tab. The proxy notices and closes the connection. The framework fires HttpContext.RequestAborted or the SignalR per invocation token. The endpoint passes that token to [EnumeratorCancellation]. The await foreach flows it to IChatClient.GetStreamingResponseAsync, which passes it to the underlying provider HTTP call. The provider stops the model response if its API supports it.

Any layer that drops the token leaks money. Three common drops: a try/catch (Exception) that swallows OperationCanceledException and continues iterating; a hand rolled SSE endpoint with Response.WriteAsync that does not pass the request aborted token down; a SignalR hub method whose CancellationToken parameter is missing [EnumeratorCancellation]. All three look correct on review. All three quietly burn provider budget on responses nobody is watching.

Buffering, Backpressure, and the Proxy Tax

ServerSentEventsResult<T> disables output buffering on the response body. The browser flushes its receive buffer on each chunk. The middle layer, proxy, gateway, WAF, CDN, is where the streaming contract breaks in production.

Nginx with default settings buffers text/event-stream and delivers in eight kilobyte chunks. The fix is X-Accel-Buffering: no on the response header. AWS Application Load Balancer streams correctly on HTTP/1.1 with keep alive on. Cloudflare respects SSE when Cache-Control: no-cache is set, which ServerSentEventsResult<T> already does. Azure Front Door requires the origin to send Cache-Control explicitly. The architect verifies this with an end to end test from a real browser through the production edge, not from a curl on the cluster, where everything always works.

Backpressure on a streaming method is implicit in IAsyncEnumerable<T>: the producer's yield return does not return until the consumer's MoveNextAsync completes. The producer cannot outrun the consumer. The trap is the channel based variant: when the hub method writes to Channel.CreateUnbounded<T> and returns the reader, the producer outruns the consumer indefinitely and the channel grows. The fix is Channel.CreateBounded<T> with BoundedChannelFullMode.Wait, which makes WriteAsync await capacity and reintroduces backpressure.

The Cost Surface

Streaming changes the cost shape of an LLM endpoint. A non streaming call is paid for once: the full response, billed at the end. A streaming call is paid for the same number of output tokens, but the cost is committed as the response generates, and the failure mode is asymmetric. If the client closes the tab at the second token of a five hundred token reply and the cancellation is honoured at every layer, the architect pays for two tokens. If cancellation is dropped at any layer, the architect pays for five hundred. Multiply by a reconnect storm, a flaky network that drops and reopens the SSE connection every five seconds, and the bill on a copilot endpoint can run an order of magnitude over the steady state number for the same number of useful answers delivered.

This is where Series 3 Article 6, Cost Guardrails for LLM Systems, earns its keep. The streaming endpoint sits behind the same per request token budget, the same per tenant rate limit, the same per session daily cap as the non streaming endpoint. The ChatClientBuilder middleware pipeline is the right place to enforce them, because the budget enforcement wraps the streaming method exactly the way it wraps the non streaming one.

Failure Modes, Worst First

The streaming chat failure modes are not the model's. They are the pipe's.

The first and worst is dropped cancellation. Any layer between the browser and the provider that swallows the token leaks money. The damage is silent: the response generated successfully, the user closed the tab, the bill ticked up. There is no error in the log to grep for, which is exactly why it survives to the invoice.

The second is the proxy buffer. The model emitted the first token at 280 ms; nginx held it until the eight kilobyte buffer filled; the user saw nothing for thirty seconds and reloaded the page; the reload kicked off a second streaming call; both calls billed.

The third is the swallowed enumerator exception. The iterator throws on a transient provider 502; the outer try/catch (Exception) logs and continues; the next iteration calls the disposed enumerator and the connection drops; the client retries; the cycle repeats until a circuit breaker fires several minutes later.

The fourth is the reconnect storm. A flaky network on the client closes and reopens the SignalR connection every few seconds; each reconnect resubscribes and starts a new model call; cost runs ten times steady state. The fix is an idempotency key on the streaming invocation so the server can detect "this is the same stream the client was just on" and resume from a continuation token rather than start fresh.

The fifth is the unbounded channel. The hub method writes to Channel.CreateUnbounded<T>; the producer outruns the consumer; memory grows linearly with stream length; the pod OOMs on a long conversation.

The sixth is JSON serialisation on the hot path. TypedResults.ServerSentEvents JSON serialises every non string event using the configured JsonOptions. The per event cost is small individually and adds up over a hundred thousand event day. Using IAsyncEnumerable<SseItem<string>> and pre encoding at the source skips the serialiser.

The seventh is the cancellation token at the wrong altitude. A hub method that declares CancellationToken without [EnumeratorCancellation] receives a token, but the token the framework merges from WithCancellation at the call site is silently discarded. The compiler does not warn. The endpoint looks correct. The cancellation never fires.

The Architect's Checklist

🗓️ Every public IAsyncEnumerable<T> method has [EnumeratorCancellation] on its cancellation token parameter.

🗓️ Every await inside a streaming iterator uses ConfigureAwait(false) when the method is library grade code.

🗓️ The SSE endpoint reads HttpContext.RequestAborted and passes that token to the chat client streaming call.

🗓️ The SignalR streaming hub method accepts a CancellationToken parameter marked with [EnumeratorCancellation] so client side unsubscribe fires.

🗓️ The reverse proxy in front of the SSE endpoint is verified to not buffer text/event-stream; X-Accel-Buffering: no on nginx, equivalent header on the edge.

🗓️ Channel based producer paths use Channel.CreateBounded<T> with BoundedChannelFullMode.Wait, never CreateUnbounded.

🗓️ The integration test suite includes a "client cancels at the second token" scenario that verifies the provider stopped generating.

🗓️ Streaming endpoints enforce the same per request token budget, per tenant rate limit, and per session daily cap as the non streaming endpoints.

🗓️ Reconnect aware streaming uses an idempotency key plus ChatResponseUpdate.ContinuationToken for resumption rather than restarting the model call.

🗓️ OpenTelemetry middleware on ChatClientBuilder is in the pipeline so the span scope covers the entire enumeration, not just the call.

🗓️ JSON serialisation on the SSE hot path is avoided where it can be; pre encode to IAsyncEnumerable<SseItem<string>> when the event payload is small and uniform.

🗓️ The provider package's cancellation behaviour is verified end to end against the provider's billing console, not asserted from the docs.

The Architect's Mental Model

The model is not a function that returns a string. It is a sequence that yields words. .NET already had the right primitive, IAsyncEnumerable<T>, three years before streaming chat became the dominant LLM interaction model. The transport above is a choice between two ASP.NET Core surfaces, SSE on a Minimal API for the one way case, SignalR for the bidirectional and fan out cases, and the failure modes are not in the model, they are in the pipe.

The well architected version of this surface is invisible. The user sees only the word arriving. The architect's job is to keep the connection that brought it boring.

The model yields the words. The pipe decides whether anyone is still there to read them.

🗓️ Last updated: August 2026

The architect inherits a brownfield .NET 10 service that classifies incoming customer-support emails into six ticket categories, extracts the order number if there is one, and routes the result to a queue. The classifier has been working for two years. It is built on top of a chat completion call that asks the model to "respond with the category and the order number." The application code parses the model's reply with a regex. The regex was six lines when the service shipped; it is now sixty-four. The product team has just added eight new email formats and two new categories. The team's senior dev has been writing new regex branches for two days and the parse function is starting to look like a small programming language of its own.

This is the recurring shape of a .NET LLM codebase that learned to talk to models before structured outputs existed. The model is asked to "respond in JSON" and the application code parses the reply on a best-effort basis. The parse function grows quietly until the day a new format silently breaks a downstream queue and someone has to read sixty-four lines of regex to find out why.

The answer is to stop asking the model to respond in JSON and start asking it to respond as a typed record. The mechanism is structured outputs — a contract negotiated between the application and the model, expressed as a JSON schema generated from a C# type, enforced by the provider at decode time, deserialised back into the same C# type at the boundary. The application code never parses a string.

What this article is NOT

This is not a JSON tutorial. It is not a comparison of System.Text.Json and Newtonsoft.Json — the answer in 2026 is System.Text.Json with a source generator, and the discussion is closed. It is not a defence of string-parsing model responses as a stopgap; if you are still doing that in 2026, the technical debt is compounding faster than the team can pay it down.

It is an architect's read on why structured outputs are the contract surface between a .NET service and a model, and how to design that contract so it earns its place in the dependency graph.

Thesis

A model call that returns free text is a function with a stringly-typed return value. A model call that returns a structured output is a function with a strongly-typed return value. The first is the API design that 2010-era PHP shipped on. The second is the API design that ASP.NET Core's typed minimal-API endpoints ship on. The same architectural argument that won the HTTP layer applies to the model layer. The C# version of structured outputs is a record on the consumer side, a source-generated JsonSerializerContext for the serialisation pass, and the Microsoft.Extensions.AI GetResponseAsync<T> extension on top of the provider's structured-output surface. Designed well, the model becomes a pluggable backend behind a typed contract. Designed poorly, the typed contract becomes a stricter version of the stringly-typed mess it replaced.

What the classical .NET version looked like

The pre-2024 pattern was a chat completion call with a system prompt that said "respond only in JSON with the following fields," followed by a JsonSerializer.Deserialize<T> call that hoped for the best. The failure modes were familiar to anyone who shipped that code. The model wrapped the JSON in a markdown code fence and the deserialiser threw on the leading backticks. The model added an explanatory sentence before the JSON and the parse failed. The model returned null for a non-nullable property and the deserialiser threw. The model returned a number as a string. The model returned an extra field nobody asked for. Each failure mode produced a retry loop, a fallback heuristic, or a regex preprocessor that survived until the next model upgrade silently changed its output shape.

The "respond only in JSON" contract was a convention. The model was free to honour it or not. The application code carried the cost of every deviation.

OpenAI shipped strict structured outputs on August 6, 2024, with gpt-4o-2024-08-06. The contract changed from convention to constraint. The model now receives the JSON schema as part of the request and the provider's decode path is guaranteed to emit only tokens that conform to the schema. A response either matches the schema or the response is a programmatically detectable refusal. There is no third option. Other major providers shipped equivalent contracts over the following twelve months. As of mid-2026, structured outputs are the assumed default for any production .NET LLM service that ingests model output into a typed downstream system.

The four moving parts

A structured-output call on .NET has four moving parts. Getting each of them right separately is what makes the whole thing a typed contract instead of a stricter version of the old mess.

The contract type. A C# record that names every field the application needs from the model. Records win because they are immutable by default, they generate value-based equality for free, and they read at the call-site as data rather than as classes. The record is the contract — what the model returns and what the application consumes are the same shape.

The JSON schema. Generated from the contract type. On .NET 9+ this is System.Text.Json.Schema.JsonSchemaExporter for general use, and Microsoft.Extensions.AI.AIJsonUtilities.CreateJsonSchema for the schemas the model layer actually needs — the latter applies the conventions the providers expect, including required-by-default property handling and provider-aware refinements. The schema is generated, never hand-written. Hand-written schemas drift from the C# type the moment the type changes.

The response format. The wire-level contract sent to the provider. On OpenAI's Chat Completions API the shape is response_format: { type: "json_schema", json_schema: { name, schema, strict: true } } — the schema and the strict flag together turn the request from a hint into a constraint. On OpenAI's Responses API the equivalent lives under text: { format: { type: "json_schema", name, schema, strict: true } }. On Anthropic, the same effect is achieved through the tool-use surface — define a tool whose input schema is the contract, instruct the model to call it, read the tool-call arguments as the structured output. On Microsoft.Extensions.AI, every one of these provider-specific shapes hides behind a single ChatOptions.ResponseFormat setting backed by ChatResponseFormat.ForJsonSchema(schema).

The deserialisation. The decode-time round-trip from the model's JSON output back to the C# record. The choice that matters at this step is whether the deserialisation runs through reflection (the default) or through a source-generated JsonSerializerContext (AOT-safe, trim-safe, faster). For any .NET service that ships on Native AOT or runs under aggressive trimming — and every new .NET 10 service should at least consider both — source-generated serialisation is the only correct answer.

The Microsoft.Extensions.AI typed surface

The convenience method that ties the four moving parts together is GetResponseAsync<T> — an extension method on IChatClient exposed by ChatClientStructuredOutputExtensions. The signature reads as if the model layer were just another typed RPC:

public record OrderInfo(
    [property: Description("The numeric order ID extracted from the email, if present.")]
    string? OrderId,
    [property: Description("The category the email belongs to.")]
    TicketCategory Category,
    [property: Description("A short one-line summary of what the customer is asking for.")]
    string Summary);

public enum TicketCategory
{
    OrderStatus,
    Refund,
    ShippingDelay,
    ProductQuestion,
    AccountAccess,
    Other
}

ChatResponse<OrderInfo> response =
    await chatClient.GetResponseAsync<OrderInfo>(
        new ChatMessage(ChatRole.User, emailBody));

if (response.TryGetResult(out OrderInfo? info))
{
    await ticketQueue.EnqueueAsync(info);
}
else
{
    logger.LogWarning(
        "Model declined or returned unparseable output. Raw text: {Text}",
        response.Text);
}

What is happening underneath: the extension method generates a JSON schema from OrderInfo via AIJsonUtilities.CreateJsonSchema, attaches it as ChatOptions.ResponseFormat via ChatResponseFormat.ForJsonSchema, sends the request through whatever middleware the pipeline composed, receives the model's JSON output, deserialises it into OrderInfo, and returns a ChatResponse<OrderInfo> whose TryGetResult pattern hands the application either the typed value or a fallback path that exposes the raw text. The application code reads as if the model were a function. That is the entire architectural point.

The [property: Description] attributes on the record's positional parameters become part of the generated schema and are visible to the model at request time. The property: target is load-bearing — a bare [Description("...")] on a positional parameter binds to the constructor parameter, which reflection-based schema generators do not read. The property: target binds the attribute to the generated property, which they do. A well-described record produces a model response that matches the application's intent. A record with no descriptions produces a model response that matches the field names and hopes for the best. Descriptions are not decoration; they are the prompt's contribution to the contract.

Source-generated JSON and the AOT question

Reflection-based serialisation works. It has worked since .NET Core 3.0. It will continue to work. But it is incompatible with Native AOT and increasingly fragile under trimming, and both deployment modes are growing as a share of new .NET 10 services. A service that ships on Native AOT cannot afford to discover at runtime that the deserialiser cannot construct a contract type without reflection. The fix is JsonSerializerContext and the source generator.

[JsonSourceGenerationOptions(WriteIndented = false)]
[JsonSerializable(typeof(OrderInfo))]
[JsonSerializable(typeof(TicketCategory))]
internal partial class TicketJsonContext : JsonSerializerContext
{
}

The partial class is empty; the source generator emits the bodies. The application can then pass TicketJsonContext.Default into the JsonSerializerOptions it uses anywhere the typed structured-output deserialisation runs. M.E.AI's typed extensions accept a JsonSerializerOptions for exactly this reason; the AOT-safe path is one constructor call away. The contract type and the JSON context live next to each other in the codebase, evolve together, and the source generator catches the case where a contract field is added without the context being regenerated.

The architectural rule that earns this its place: any C# type that crosses the model boundary in either direction belongs in a JsonSerializerContext. The cost is one attribute. The benefit is permanent AOT-safety on that surface.

When you need finer control than the typed extension gives

GetResponseAsync<T> is the right answer for the eighty-percent case. For the remaining twenty — agent-loop call-sites where the schema needs custom names, response formats where the schema description carries domain meaning the record name cannot, or scenarios where the same schema is used across multiple call-sites and the caller wants to register it once and reuse it — the lower-level surface is ChatResponseFormat.ForJsonSchema:

JsonElement schema = AIJsonUtilities.CreateJsonSchema(typeof(OrderInfo));

var options = new ChatOptions
{
    ResponseFormat = ChatResponseFormat.ForJsonSchema(
        schema: schema,
        schemaName: "order_info",
        schemaDescription: "Structured extraction of order details from a support email.")
};

ChatResponse response = await chatClient.GetResponseAsync(
    new ChatMessage(ChatRole.User, emailBody),
    options);

The deserialisation is now the application's responsibility — JsonSerializer.Deserialize<OrderInfo>(response.Text, TicketJsonContext.Default.OrderInfo). The cost is one extra line. The benefit is full control over the schema name and description the provider sees in the request. For most call-sites this is overkill; for the ones where the schema name shows up in the provider's telemetry and the team cares about the readability of that telemetry, the explicit form earns its keep.

Refusals — when the model says no

Strict structured outputs introduced a programmatic refusal path. The model can decline to produce output when the request would violate its safety policy, when the prompt is ambiguous, or when the schema cannot be satisfied from the available context. The refusal arrives as a structured response, not as an exception, and the application is expected to detect it and handle it.

On Microsoft.Extensions.AI the refusal manifests as TryGetResult returning false while response.Text carries the model's refusal message. The application code's responsibility is to distinguish refusal from parse failure — both produce the same TryGetResult outcome — and to route each to the correct fallback. A refusal on an order-extraction call probably means the email contained content the model would not engage with, and the right answer is to escalate to a human queue. A parse failure on the same call usually means a transient provider issue, and the right answer is to retry with backoff. Conflating the two produces silent escalations on transient errors and silent retries on safety refusals — neither is acceptable in production.

The architectural pattern: every structured-output call-site has an explicit refusal path that is observable in telemetry, distinct from the success path, and routed to a queue that humans actually read. The refusal is part of the contract. Treating it as an error is a category mistake.

The Responses API note

OpenAI introduced the Responses API as the recommended surface for new code, with the Chat Completions API entering long-term support. The wire shape moves response_format under text.format, the streaming model becomes the default, and the multi-turn state is handled server-side. None of this changes how a .NET application written against IChatClient looks. M.E.AI's OpenAI adapter wraps whichever underlying surface the SDK uses, and the application code remains untouched. The architect's discipline is to verify, at adapter-pinning time, that the adapter version in use targets the surface the team has chosen — Chat Completions for stability today, Responses for forward-compatibility tomorrow — and to write down which surface the service is built against in the same place the rest of the dependency choices live.

Failure modes, severity-ordered

The patterns that turn a healthy structured-output adoption into a fragile one, worst first.

1. Treating refusals as exceptions. A try/catch wrapping the typed call that logs the refusal as an error and retries on backoff produces silent safety-policy bypass attempts, a corrupted telemetry signal, and eventually an account-level rate limit. Mitigation: refusal handling is explicit, structurally distinct from parse failure, and instrumented separately.

2. Schema drift between the record and the prompt. Adding a field to the record without updating the system prompt produces a schema-compliant response that has nothing useful in the new field. Mitigation: the system prompt references the record's purpose and lets the schema express the field-level contract; new fields land in both places in the same commit.

3. Reflection-based serialisation on an AOT service. Works in development, fails at first request in production. Mitigation: every contract type lives in a JsonSerializerContext; the project disables reflection-based serialisation in the production configuration.

4. Over-rich contract types. A record with thirty fields and three levels of nesting overwhelms the model's schema-adherence budget on smaller models and silently degrades extraction quality. Mitigation: contract types are flat where possible, narrow in scope, and the architect resists the temptation to overload the schema with everything the call-site might one day want.

5. Catch-all string properties. A field declared as string? that should be an enum, a date, or an integer pushes the parse problem from compile time to runtime. Mitigation: enums for closed sets, DateTimeOffset for timestamps, numeric types for numbers. The compile-time contract is the strongest contract.

6. Schema strict mode disabled. Disabling strict: true because "the model gets confused" is debugging in the wrong layer. Mitigation: keep strict on; the confusion is a prompt or contract issue, not a strictness issue.

7. Mixing structured outputs and free-text in the same call. A request that asks for a structured object and also a friendly summary produces neither cleanly. Mitigation: structured calls produce structured output, period. Conversational calls produce text. Separate concerns, separate call-sites.

The architect's checklist

🗓️ Every model-to-application data hand-off is expressed as a C# record contract type with [property: Description] attributes on each positional parameter.

🗓️ Schemas are generated from the contract type — never hand-written — via AIJsonUtilities.CreateJsonSchema or the M.E.AI typed extension's built-in path.

🗓️ The application uses chatClient.GetResponseAsync<T> for the typed eighty-percent case, and ChatResponseFormat.ForJsonSchema for the explicit-schema-name twenty-percent case.

🗓️ TryGetResult is the access pattern; response.Text is the fallback path; refusal and parse failure are handled distinctly.

🗓️ Every contract type appears in a JsonSerializerContext partial class, with the source generator enabled.

🗓️ Reflection-based serialisation is not used at the model boundary in production.

🗓️ The system prompt references the schema's purpose and lets the schema enforce the field-level contract; the two evolve in the same commit.

🗓️ Contract types are flat, narrow, and per-call-site; no single record carries the union of every call-site's fields.

🗓️ Enums close the set wherever the model is choosing from a finite list; string is not used as a substitute for an enum or a typed value.

🗓️ Refusals are routed to a human queue with structured telemetry, distinct from transient-failure retries.

🗓️ The provider adapter pinning record names the API surface the service targets — Chat Completions or Responses — and the adapter version is reviewed at each major release.

🗓️ Tests at the contract boundary use the generated schema as a fixture; a schema-version change is a test-detected change, not a runtime-discovered one.

Mental model

Structured outputs are the typed return statement of the model call. The C# record is the return type, the JSON schema is the type signature the provider checks, and the deserialiser is the implicit conversion from wire format back to the type. The application code reads as if the model were a method. The contract is the type. The type is the contract.

The architect who treats structured outputs as "JSON mode but stricter" misses the design centre. The architect who treats them as "the model's return type, expressed in C# and enforced at the provider" misses nothing.

String-parsing model responses was a workaround. Structured outputs are the design.

What's next

Article 4 — Hosting Long-Running Agents — IHostedService, BackgroundService, and the Worker Lifecycle — drops one altitude lower. An agent loop that takes minutes to run, a queue of pending tool calls, a server that restarts mid-loop, a checkpoint that survives the restart, an OpenTelemetry trace that spans both halves of the loop. The .NET hosting model already has the primitives. The architect's job is to wire them to the agent's lifecycle without inventing a parallel scheduler. Subscribe at pragmaticstack.hashnode.dev for Friday's drop.

Series 4 progress

🗓️ Last updated: August 2026

The architect picks up a brownfield .NET 10 service in the second week of August. It is a customer support backend that has been calling OpenAI directly through the official OpenAI .NET SDK since the project started. The ticket on the board this sprint reads: add structured logging on every model call, emit OpenTelemetry spans matching the rest of the service, support Azure OpenAI as a secondary provider for a regulated tenant, and put a cache in front of the deterministic prompts the support classifier sends a few thousand times an hour. Four tickets. One sprint. The team's senior dev has been writing the wrapper classes by hand for three days, and the abstractions are starting to look like a small framework of their own. Nobody asked for a small framework.

This is the recurring shape of a brownfield .NET LLM codebase in 2026. The application code talks directly to a vendor SDK. The cross cutting concerns, logging, telemetry, caching, function calling, provider switching, get bolted on as wrapper classes that nobody wants to own. The team is six months from where they need to be and they cannot rewrite the world.

The answer is not a new framework. The answer is Microsoft.Extensions.AI, a small, deliberately boring set of abstractions that ships in the same .NET 10 stack as Microsoft.Extensions.Logging and Microsoft.Extensions.Caching. The value of M.E.AI is not what it adds. Its value is what it asks you to standardize on, and what it lets you compose on top of that standard. Boring, in this case, is the whole pitch.

What This Article Is Not

This is not a Microsoft.Extensions.AI tutorial. The Microsoft Learn docs are excellent at that, and a blog should not try to replace them. It is not a comparison of Semantic Kernel and M.E.AI; Article 1 of this series covered that consolidation. It is not an argument that you should rewrite working OpenAI SDK code today.

It is an architect's read on why M.E.AI is the load bearing layer underneath the rest of the .NET LLM stack, and how to design the rest of your service so that the layer earns its keep.

The Thesis: The Pipeline Is the Point

Microsoft.Extensions.AI is the .NET LLM ecosystem's ASP.NET Core moment. It is a small set of interfaces, IChatClient, IEmbeddingGenerator, ChatMessage, ChatOptions, and a middleware pipeline pattern that composes cross cutting concerns the same way ASP.NET Core's request pipeline composes HTTP middleware. The interfaces are not the point. The pipeline is. An architecture that takes the pipeline seriously inherits logging, telemetry, caching, function invocation, and provider switching as configuration concerns rather than code concerns. An architecture that treats M.E.AI as yet another SDK wrapper pays the cost of the abstraction without collecting the benefit.

The Prior Art You Already Know: ASP.NET Core Middleware

ASP.NET Core's middleware pipeline is the prior art every .NET architect already understands. A request enters the pipeline, passes through ordered middleware components, hits an endpoint, and returns through the same components on the way out. Authentication, authorization, exception handling, response compression, request logging: every cross cutting concern in the HTTP stack is a middleware component composed by configuration. The application code at the bottom of the stack handles only the domain logic.

The same pattern, applied to model calls, is what Microsoft.Extensions.AI shipped. A chat call enters a pipeline, passes through ordered middleware (logging, OpenTelemetry, caching, function invocation), hits the actual provider client, and the response returns through the same components. Application code at the bottom calls IChatClient.GetResponseAsync and gets back a ChatResponse. The cross cutting concerns are configured at composition time and disappear from the application's reading view.

The architect who has internalized the ASP.NET Core middleware model has already internalized the M.E.AI middleware model. The vocabulary is the same. The abstraction is the same. Only the pipeline contents change. If you have ever written app.UseAuthentication() and not thought twice about it, you already know how this ends.

The Four Load Bearing Types

Microsoft.Extensions.AI 10.6.0, the current stable as of mid 2026, GA in the .NET 10 stack, exposes a deliberately small surface. The four types worth knowing on day one are below. Everything else in the namespace either supports these four or is a convenience over them.

IChatClient. The abstraction over any chat completion style model. The interface exposes two methods that matter:

Task<ChatResponse> GetResponseAsync(
    IEnumerable<ChatMessage> messages,
    ChatOptions? options = null,
    CancellationToken cancellationToken = default);

IAsyncEnumerable<ChatResponseUpdate> GetStreamingResponseAsync(
    IEnumerable<ChatMessage> messages,
    ChatOptions? options = null,
    CancellationToken cancellationToken = default);

A non streaming call returns the full ChatResponse when the model finishes. A streaming call returns an IAsyncEnumerable<ChatResponseUpdate> that yields incremental updates as they arrive. The streaming method is the foundation for Article 5 of this series.

IEmbeddingGenerator. The corresponding abstraction for embedding models. Generic over input type and embedding type so the same interface covers text embeddings, image embeddings, and multimodal embeddings. The pattern matches IChatClient: pipeline composable, provider pluggable, deliberately small.

ChatMessage. The unit of conversation. A role (ChatRole.System, ChatRole.User, ChatRole.Assistant, ChatRole.Tool), a content payload, optional name and author identification. The same ChatMessage shape is produced by every provider adapter: OpenAI, Azure OpenAI, Anthropic via community adapter, Ollama via OllamaSharp. The application code never sees a vendor specific message type.

ChatOptions. Per call configuration: temperature, max tokens, top p, tool list, response format, stop sequences. The architectural point is that ChatOptions is per call, not per client. A single registered IChatClient can serve many call sites with different configurations. This is the difference between a model call being a configuration concern and being a wiring concern.

The Pipeline (Where the Value Actually Lives)

The composition surface is ChatClientBuilder. The pattern mirrors ASP.NET Core's IApplicationBuilder and WebApplicationBuilder: a fluent chain of Use* extension methods that wrap the inner client in middleware components, in declared order, returning a single composed IChatClient at the end. A representative composition looks like this:

services
    .AddChatClient(provider => new OpenAIClient(apiKey)
        .GetChatClient("gpt-4o").AsIChatClient())
    .UseLogging()
    .UseOpenTelemetry(sourceName: "MyApp.Chat")
    .UseDistributedCache()
    .UseFunctionInvocation();

Each Use* extension method comes from Microsoft.Extensions.AI. The semantics:

UseLogging wraps the inner client with a logging middleware that emits the request, the response, and any exceptions through ILogger. The category and verbosity are configurable. In production, the architect typically wants the request and response payloads at Debug level only, with timing and metadata at Information.

UseOpenTelemetry wraps the inner client with an OpenTelemetry middleware that emits spans and metrics aligned with the OpenTelemetry GenAI semantic conventions Series 3 Article 3 covered in detail. The middleware participates in the same System.Diagnostics.ActivitySource infrastructure the rest of the .NET service is already using. No separate wiring.

UseDistributedCache wraps the inner client with a caching middleware that hashes the inputs into a cache key, looks the response up in an IDistributedCache implementation (Redis, SQL Server, in memory), and short circuits the model call on a hit. The cache key composition and TTL are configurable. The architect's design responsibility is to be honest about which calls are actually cacheable, which is fewer than you would like.

UseFunctionInvocation wraps the inner client with a function calling middleware that recognizes tool call responses, dispatches them to the registered AIFunction instances, and feeds the tool results back into the model loop until the conversation reaches a terminal assistant response. This is the middleware that turns the model call from a single round trip into the agent loop everyone assumes is going to be hard to implement. It is not hard to implement. It is one method call.

Order matters. Logging and OpenTelemetry wrap function invocation so that each tool dispatch is traced and logged individually. Caching wraps function invocation if you want tool aware caching; caching wraps only the outer call if you want naive caching. That is the architect's choice, and the architect's choice to write down for the next person, who will otherwise reverse engineer it from a stack trace at midnight.

Provider Adapters: Swapping Vendors Without Touching Your Code

IChatClient is the abstraction. The providers are concrete adapters that surface their vendor SDK behind the interface. The architecture is the substrate; the provider is the bottom of the pipeline. As of mid 2026, the production ready adapter set on .NET is:

• Microsoft.Extensions.AI.OpenAI, which wraps the official OpenAI .NET SDK. Adapter method: AsIChatClient on OpenAI.Chat.ChatClient.
• Microsoft.Extensions.AI.AzureAIInference, which wraps the Azure AI Inference SDK for Azure hosted models.
• Azure OpenAI, where the official Azure.AI.OpenAI SDK exposes its chat clients through the OpenAI adapter shape. In practice this is the most common configuration in Microsoft hosted production.
• OllamaSharp, the recommended Ollama path on .NET. The earlier official Microsoft.Extensions.AI.Ollama preview package is deprecated; the OllamaSharp library implements IChatClient directly. Useful for local model development.
• ONNX, Phi, and on device adapters, for inference inside the host process. Lower latency, lower throughput, no network egress, no vendor cost.

Provider switching at the M.E.AI layer is a single registration change, the services.AddChatClient(...) line, with no impact on application code. The pipeline and the consumers below it are provider agnostic. The architect's discipline is to keep them that way: zero provider specific types reach the application layer. This is the one rule that, broken once, quietly undoes the entire abstraction.

The Function Calling Surface

Function calling, the mechanism by which the model invokes typed .NET methods, is the most consequential middleware in the pipeline. M.E.AI ships the convention.

The unit is AIFunction. The factory is AIFunctionFactory.Create. Any delegate, lambda, or method handle becomes an AIFunction: the factory inspects the parameter types, generates a JSON schema, picks up [Description] attributes from the method and its parameters, and returns an AIFunction you can attach to ChatOptions.Tools. A representative tool registration:

AIFunction getOrderStatus = AIFunctionFactory.Create(
    (string orderId) => orderStatusService.LookupAsync(orderId),
    new AIFunctionFactoryOptions
    {
        Name = "get_order_status",
        Description = "Look up the current status of a customer order by ID."
    });

var options = new ChatOptions { Tools = [getOrderStatus] };
var response = await chatClient.GetResponseAsync(messages, options);

With UseFunctionInvocation in the pipeline, the middleware handles the model's tool call response, executes getOrderStatus, feeds the result back into the conversation, and returns the final assistant message. The application code never parses a tool call by hand.

The architectural point is that a tool is now a typed C# method surfaced by metadata. No string parsing, no manual schema wiring, no provider specific tool format. The architect's job is to keep the tool methods at the right granularity, small enough to be model friendly, large enough to do one useful unit of work, and to recognize that every tool method is also part of the public contract of the agent.

A second architectural point gets lost in the convenience: a tool method is a C# method, which means it is unit testable as a C# method. The model layer is not in the test path. Tool methods should be tested in the same project, with the same conventions, as any other application service, and the agent level integration test that wires the tool into a real model call should be a separate, smaller suite. Confusing the two test surfaces produces unit tests that depend on a live model and integration tests that exercise no model behavior. Both are common failures in 2026 .NET LLM codebases. Both are avoidable by treating the tool method as ordinary code, because it is.

Why M.E.AI Lives Separate from the Agent Framework

Article 1 noted that the Microsoft Agent Framework sits on top of M.E.AI's IChatClient. The natural question is why M.E.AI exists at all if the Agent Framework already wraps it.

The answer is that not every .NET LLM service is an agent. Many production services need a model call with caching, telemetry, and a tool or two, not a multi turn agent with handoffs, sessions, and orchestration. M.E.AI is the layer where those services live. The Agent Framework is the layer where agent services live. Both layers are first party. Both are part of the .NET stack. Picking the right altitude for the service is part of the architectural decision, and the altitude is reversible: a service that grows into an agent can adopt the Agent Framework without rewriting its M.E.AI substrate.

The corollary: a service whose only LLM interaction is a single classifier call per request should not be running the Agent Framework. M.E.AI plus a caching middleware is the right altitude and the simpler answer. Reaching for the agent framework here is the .NET equivalent of bringing a forklift to carry a sandwich.

How M.E.AI Relates to Semantic Kernel

Semantic Kernel, now in maintenance mode following the Microsoft Agent Framework's April 2026 GA, provides documented bridge extensions (AsChatClient, AsChatCompletionService) between its IChatCompletionService and M.E.AI's IChatClient. An existing Semantic Kernel codebase can introduce M.E.AI for new call sites without disturbing the existing Kernel composition; the two coexist, and the M.E.AI pipeline can wrap or be wrapped by Kernel mediated calls depending on which surface owns the call site.

The honest framing: M.E.AI is the substrate both Semantic Kernel and the Agent Framework now share. Code that goes through IChatClient directly survives any future framework decision. Code that goes through framework specific composition has a shorter half life. The architect's preference, for new code, should be to drop one layer below the framework whenever the framework is not doing useful work.

For an existing Semantic Kernel codebase, the practical migration pattern is not a rewrite. It is an additive one. New call sites land on IChatClient directly with the M.E.AI pipeline; existing Kernel mediated call sites stay where they are. Over time, as the Kernel composition becomes the smaller surface and the M.E.AI composition becomes the larger one, the center of gravity moves on its own. The architect's job is to make sure the new code lands in the right place, not to retroactively rewrite the old code. The two surfaces are designed to coexist precisely because Microsoft expects this transition to happen gradually across most production codebases.

Failure Modes, Worst First

The patterns that turn a healthy M.E.AI adoption into a fragile one, worst first.

1. Swallowed exceptions inside middleware. A custom middleware that wraps the inner client in try/catch and returns an empty ChatResponse on failure looks defensive and is catastrophic. The downstream caller treats the response as success, the model call silently failed, and the failure is invisible in telemetry. Mitigation: middleware lets exceptions propagate unless the architect has consciously decided on a fallback semantic that the caller can detect.
2. Caching nondeterministic prompts. UseDistributedCache caches against a hash of the inputs. A prompt that includes the current timestamp, the user's session ID, or any other per call identifier will produce a cache miss every time and a unique cache entry every time, wasting cache memory and producing zero hit rate. Mitigation: caching is opt in per call site, gated on the call site being demonstrably deterministic, and the cache key composition is reviewed.
3. Over pipelining. Twelve middlewares chained on every IChatClient because each one might be useful. Cost goes up, because every middleware allocates per call, and the failure surface multiplies. Mitigation: a minimum viable pipeline at each composition point, and a separate IChatClient registration for separate concerns rather than a maximalist one pipeline fits all composition.
4. Cross call state in middleware. A middleware that maintains a static dictionary, a per instance cache, or any other shared state across calls is a contention surface and a correctness hazard. Mitigation: middleware is request scoped or stateless; shared state lives in an injected service, not in the middleware itself.
5. Treating ChatOptions as a singleton. Building a single ChatOptions instance at startup and reusing it across calls produces behavior that is miserable to debug the day one call site mutates a field. Mitigation: ChatOptions is per call, constructed at the call site, immutable after construction.
6. Provider lock in through the side door. Application code that catches a provider specific exception type, depends on a provider specific tokenizer, or parses a provider specific error code defeats the abstraction. Mitigation: provider types stay at the adapter boundary; cross cutting exception types and tokenizer interfaces live in the application's own infrastructure layer.
7. Skipping the OpenTelemetry middleware. A service that wires logging without OpenTelemetry will be observably blind at the LLM boundary the first time a production incident requires correlating a model latency spike with a downstream timeout. Mitigation: UseOpenTelemetry ships with every IChatClient composition that runs in production, full stop. The cost is negligible. The value is total.

The Architect's Checklist

🗓️ Application code talks to IChatClient, never to a vendor SDK type.

🗓️ IChatClient is registered in DI through services.AddChatClient(...), with the provider adapter as the inner client.

🗓️ Provider specific types (OpenAIClient, ChatClient, vendor SDK message types) stop at the registration boundary.

🗓️ Every production IChatClient composition includes UseLogging and UseOpenTelemetry; OTel emits through the same ActivitySource as the rest of the service.

🗓️ UseDistributedCache is added only on call sites with reviewed cache key determinism.

🗓️ UseFunctionInvocation is added on call sites that use tools; tool functions are constructed via AIFunctionFactory.Create with explicit names and descriptions.

🗓️ ChatOptions is constructed per call, not reused across calls.

🗓️ Custom middleware lets exceptions propagate by default; any fallback semantic is written down and reviewed.

🗓️ The middleware order is documented at each composition point: what wraps function invocation, what wraps caching, what wraps the outer call.

🗓️ Streaming call sites use GetStreamingResponseAsync and treat the IAsyncEnumerable<ChatResponseUpdate> as the unit of cancellation.

🗓️ Provider adapter packages are pinned to a known stable version range; the adapter set is reviewed at each major release.

🗓️ IEmbeddingGenerator registrations follow the same pipeline discipline as IChatClient: embedding calls are equally observable, equally cacheable, equally provider agnostic.

🗓️ No application code depends on a Semantic Kernel or Agent Framework type for work that M.E.AI can do directly.

The Architect's Mental Model

M.E.AI is to the .NET LLM stack what IHttpClientFactory was to the .NET HTTP stack. The interface is small. The composition surface is the value. The pipeline is the thing that earns the abstraction its place in the dependency graph. Every cross cutting concern that used to be a hand written wrapper class becomes a one line middleware registration. Every provider switch becomes a one line composition change. Every observability requirement becomes a one line UseOpenTelemetry call that lights up the same telemetry the rest of the service already emits.

The architect who treats M.E.AI as yet another SDK abstraction misses the design center. The architect who treats it as a pipeline misses nothing.

ASP.NET Core won by being the substrate. M.E.AI is winning the same way.

What's Next

Article 3, Structured Outputs in C#: Schema, Records, and the End of String Parsing, goes one level up the stack. The ChatOptions.ResponseFormat surface, the JSON schema from C# records pattern, the source generator path for AOT safe serialization, the refusal case when the model declines to produce structured output, and the architectural pattern of mapping the model contract to a typed C# record at the boundary. Subscribe at pragmaticstack.hashnode.dev for Friday's drop.

🗓️ Last updated: July 2026

Picture the .NET architect who, sometime in 2025, did everything right. They read the docs, they avoided the hype, they picked Semantic Kernel because it was the Microsoft blessed option with a real team behind it, and they shipped an agent on top of it. Kernel, plugins, connectors, a planner doing the orchestration. Clean. Defensible in a design review.

Then April 2026 arrived, and Microsoft announced that the thing they standardized on was now, effectively, the previous generation. Semantic Kernel and AutoGen had been folded into a single unified SDK called the Microsoft Agent Framework, which reached 1.0 general availability in April 2026. The product lead's own framing was that you should think of the Agent Framework as Semantic Kernel v2.0, built by the same team. Comforting, in the way that "we're not breaking up, we're just seeing other abstractions" is comforting.

This is the first article in AI for the .NET Architect. Before we touch a single line of Microsoft.Extensions.AI or write one IHostedService, we have to deal with the fact that the foundational framework most .NET LLM code was written against just got consolidated out from under it. The good news, and it is genuinely good news, is that the consolidation is a much smaller earthquake than the renaming makes it look. Most of what you built was never really the framework's anyway.

The Two Framework Problem You Inherited in Mid 2026

If you started a .NET agent project any time in the last two years, you inherited a confusing inheritance. There was Semantic Kernel, the enterprise leaning SDK with the Kernel object, plugins, and planners. There was AutoGen, the research leaning multi agent framework that came out of Microsoft Research and was very good at agents talking to each other. And underneath both, quietly, there was Microsoft.Extensions.AI, a small set of abstractions that neither replaced nor competed with the other two but somehow sat below them.

Three things, overlapping, from the same company, with the same logo, pointed at roughly the same problem. Every architecture review that touched AI spent its first twenty minutes just establishing which of the three the team was actually using and why. That was not a great use of twenty minutes.

The April 2026 consolidation is, more than anything, Microsoft admitting that the three framework situation was untenable and collapsing it into one. Semantic Kernel and AutoGen become the Microsoft Agent Framework. Microsoft.Extensions.AI stays exactly where it was, as the substrate. If you only remember one sentence from this article, make it that one: the consolidation happened one layer up from the part you should have been building on.

What Semantic Kernel Actually Gave You

It is worth being honest about what Semantic Kernel did and did not do, because the honest version is what tells you how much of your code is at risk.

Semantic Kernel gave you four things that mattered:

• The Kernel. A dependency injection container with opinions about AI. It held your services, your plugins, and your configuration, and it was the object everything else hung off of.
• Plugins. Native functions you exposed to the model, decorated so the framework could describe them to the LLM and invoke them when the model asked. If you wrote a method and tagged it as a KernelFunction, you had built a tool the model could call.
• Connectors. Adapters to the model providers, so your code did not hardcode the OpenAI SDK or the Azure OpenAI SDK directly. In theory you could swap providers by swapping a connector.
• Planners. The orchestration layer that took a goal and decomposed it into steps. The Sequential Planner and the Stepwise Planner were the famous ones, and they were the part everyone demoed.

Here is the uncomfortable part. The planners, the most distinctive Semantic Kernel feature, the thing that made it feel like more than an SDK wrapper, were already deprecated well before the consolidation. The Sequential Planner and the Stepwise Planner were superseded by function calling planning, which is a fancy way of saying the model itself decides which tools to call, in which order, using the native tool calling the model providers built into their APIs. The framework stopped needing to plan because the models learned to plan.

So if you built your architecture on the Sequential Planner, you were already standing on a deprecated abstraction before April 2026 said a word. The consolidation did not break your planner. Your planner was scheduled for demolition the day the models got good at tool calling.

What the Microsoft Agent Framework Actually Is

Strip away the launch blog enthusiasm and the Agent Framework is a focused thing: a unified SDK for building agents, distributed under the Microsoft.Agents.AI package family, that takes the agent and multi agent ideas from Semantic Kernel and AutoGen and rebuilds them on a common foundation.

The core abstraction is AIAgent. Where Semantic Kernel made you assemble a Kernel and hang services and plugins off it, the Agent Framework gives you an agent as a first class concept, with ChatClientAgent as the canonical concrete implementation you will actually instantiate. An agent has instructions, tools, and the ability to carry state across turns, and it is built directly on top of the IChatClient abstraction from Microsoft.Extensions.AI.

That last detail is the whole game, and it is why this consolidation is survivable. The Agent Framework is not a parallel universe with its own model client and its own plumbing. It sits on the exact same IChatClient substrate that a plain Microsoft.Extensions.AI application uses, the same one Article 2 of this series is entirely about. The agent is a layer of orchestration on top of a model client you already understand, not a replacement for it.

Multi agent coordination, the AutoGen contribution, comes along too: the framework supports agents that hand work to other agents, which is the pattern AutoGen pioneered and Semantic Kernel never did cleanly. If you were running AutoGen for the agent to agent conversation and Semantic Kernel for the enterprise plumbing, the consolidation is unambiguously good for you. You get one SDK instead of two that pretended not to know each other.

What You Keep, What You Replace, What Was Never the Framework's Job

This is the section to bring to your own migration review, because it sorts your codebase into three honest piles.

What you keep. The Microsoft.Extensions.AI substrate survives untouched. The function calling pattern survives, because it was always the model providers' API, not a Semantic Kernel invention. Your OpenTelemetry instrumentation survives, because tracing a model call is a cross cutting concern that never belonged to the framework in the first place. Your prompts survive. Your evaluation suite survives. Your cost controls survive. Notice how much of that list is the stuff Series 3 spent six articles on.

What you replace. The Kernel object as your composition root. The planner based orchestration, which was deprecated already and is now formally superseded by agents and function calling planning. The Semantic Kernel specific agent types, which give way to AIAgent and ChatClientAgent. The session or thread state handling for multi turn conversations, which the Agent Framework models with its own abstraction rather than Semantic Kernel's.

What was never the framework's job. Your retrieval pipeline. Your evaluation harness. Your prompt versioning discipline. Your observability spans. Your per tenant cost budgets. Your incident runbooks. None of these were ever Semantic Kernel features, which means none of them are Microsoft Agent Framework migrations. They are your architecture, and architecture is the thing that survives every consolidation, including the next one, which Microsoft will absolutely ship eventually, because they always do.

If your migration spreadsheet has a hundred rows, the honest version probably has fifteen rows in the "replace" column and eighty five rows in the "this was never the framework" column. The panic is proportional to how much architecture you accidentally let leak into framework specific types.

The Decision Tree, Minus the Hand Wringing

Two situations, two answers.

You are starting a greenfield .NET agent today. Start on the Microsoft Agent Framework. It is GA, it is the supported path, the new features land here, and there is no reason to begin a new project on the framework Microsoft just put into maintenance. This is the easy case, and it is the one most articles stop at.

You are maintaining a Semantic Kernel codebase that works. Do not rip it out this sprint. Semantic Kernel is not deleted. It moves to what amounts to maintenance mode: critical bug fixes and security patches keep coming, and Microsoft committed to supporting Semantic Kernel for at least a year after the Agent Framework reached GA. New features go to the Agent Framework now, but your running code does not stop running because a blog post was published. Microsoft even shipped a migration guide for the day you decide to make the move.

The mistake is treating "Semantic Kernel is in maintenance mode" as "Semantic Kernel is on fire, evacuate." It is not on fire. It is a supported v1.x with a v2.0 available, which is the most normal situation in the history of software libraries. You migrate when the cost of staying, the missing new features, exceeds the cost of moving. Not before, and not in a panic because a brand changed.

Failure Modes

These are the ways a .NET team gets the consolidation wrong, ordered roughly by how often I have watched it happen.

• Over coupling to framework types. Your domain logic imports framework specific classes everywhere, so a framework change becomes a thousand file change. The fix is the oldest fix in the book: depend on your own interfaces, keep the framework at the edges, and let the substrate types like IChatClient be the widest thing that leaks inward.
• Betting on deprecated planners. Building new orchestration on the Sequential or Stepwise Planner in 2026, because a tutorial from 2024 used them. They were already superseded by function calling planning before the consolidation. If your design doc names a planner, the design doc is out of date.
• Missing the substrate underneath. Treating the Agent Framework as a black box and never realizing that Microsoft.Extensions.AI is the layer doing the actual model communication. Teams that miss this rebuild logging, caching, and telemetry inside their agent code instead of composing it in the substrate pipeline, which is the subject of the next article.
• Treating the consolidation as a rename. The opposite error: assuming nothing changed and that Microsoft.Agents.AI is just Semantic Kernel with a new logo. The composition root, the orchestration model, and the agent abstractions did genuinely change. It is a reset of the load bearing types, not a find and replace on the namespace.

The first and last failure modes are mirror images. One team panics and rewrites everything; the other team shrugs and rewrites nothing. Both are wrong for the same reason: neither one sorted their code into the three piles first.

The Architect's Checklist

Bring this to the migration review.

1. Have you sorted your codebase into keep, replace, and was never the framework's job?
2. Is your domain logic free of framework specific types, or do they leak into the core?
3. Are you still depending on the Sequential or Stepwise Planner anywhere?
4. Do you know where Microsoft.Extensions.AI and its IChatClient sit in your stack, independent of the agent framework on top?
5. For greenfield work, is the default the Microsoft Agent Framework rather than Semantic Kernel?
6. For existing Semantic Kernel code, do you have a reason to migrate beyond "the brand changed"?
7. Did you confirm Semantic Kernel's actual support window before treating it as urgent?
8. Is your function calling built on the model providers' native tool calling, not a framework planner?
9. Are your evaluation, observability, and cost controls implemented as architecture, so no framework consolidation touches them?
10. Do you have a migration guide and a real timeline, or just anxiety and a Slack thread?

If you can answer most of these cleanly, the consolidation is a Tuesday. If you cannot, the consolidation is not your problem. Your problem is that you let a framework become your architecture, and this was always going to hurt eventually.

The Architect's Mental Model

Frameworks are rented, not owned. Semantic Kernel was a good rental. The Microsoft Agent Framework is a better one. There will be a third, and a fourth, because this field is eighteen months old wearing a suit, and the vendors are still figuring out the abstractions in public.

The teams that sailed through April 2026 are the teams that always treated the framework as one replaceable component sitting on a substrate they understood, surrounded by an architecture they owned. The teams that did not sleep that month are the teams that let Kernel become a load bearing type in their domain model and then acted surprised when the load bearing type got consolidated.

Everything in this series, the unified client layer, the structured outputs, the hosting model, the streaming transport, the reference architecture, is built to be the part you own, on top of whatever framework Microsoft is shipping this quarter. Get the layering right once, and the next consolidation is a press release you read with your coffee, not an incident you staff at midnight.

The framework is the binding on the book. The architecture is the text. Bindings wear out and get replaced. You keep the text.

• • *|

🗓️ Last updated: July 2026

The finance team flags the anomaly on Monday morning. The prior week LLM spend is up by a factor of eleven. The chart is a single weekend spike, starting Saturday evening, ending, well, not ending, running flat at the new ceiling through Sunday and into Monday. Eleven times the budget.

The on call engineer pulls up the traces. One tenant. One workflow. A multi step agent that calls itself with the result of the previous step. The first step retrieves a document, the second summarises it, the third asks "does this answer the user's question?", and on Saturday at 7:42 PM, for one user, the answer was no. The workflow retried. The check failed. The workflow retried. Forty eight hours of model calls. Several thousand dollars of tokens. Zero useful output. The user, meanwhile, gave up after thirty seconds and went to bed, blissfully unaware they had kicked off a small fire.

No error fired in any system. No alert fired because no metric the team watched had a per tenant budget on it. The vendor's tier level rate limit eventually clipped the workflow, but the ceiling was high enough that the cost accumulated faster than anyone noticed. The system did exactly what it was designed to do. The design simply did not include the word "stop."

This is the failure shape that distinguishes LLM system cost from every other line item in a software budget. The cost of a database query has a ceiling, because the query times out. The cost of a CPU loop has a ceiling, because the box maxes out. The cost of a model call has no native ceiling. It is bounded only by the budgets, rate limits, and kill switches you explicitly put in front of it. Build the system without those, and the worst case user is also the most expensive user, by a factor you will not believe until the invoice arrives and ruins your Monday.

This article is about the discipline that keeps that invoice from arriving. It is also the closer for Series 3.

What this article is not

It is not a survey of vendor pricing. Pricing changes; the patterns do not. It is not an argument for the cheapest model in every case. The cost bounded discipline from Series 1 Article 5 covers per call optimisation. This article is about the system level guardrails around all the calls, the structural defences that decide what the maximum loss is, before the optimisation question is even relevant.

The thesis

An LLM system is the only system in your stack where a single user can cost you a thousand times the median, in a single hour, without any component reporting an error. Budgeting, rate limiting, and kill switches are not optimisation. They are the structural defence against your own pricing model. The discipline applies at four layers, per user, per tenant, per workflow, and platform wide, and any layer without a defined ceiling is the layer the next incident will exploit. The incident is very good at finding that layer.

What classical cost discipline looked like

Before LLMs, the cost of a feature was largely deterministic. A web request consumed a known amount of CPU and memory. A database query took roughly the same time every time. The p99 to p50 cost ratio was typically single digit, not hundreds of times.

Cost discipline in that world consisted of three things. Capacity planning sized the box for peak load. Per request resource limits, CPU time, memory caps, query timeouts, prevented any single request from monopolising the system. Classical rate limiting algorithms, the token bucket and the leaky bucket, controlled the volume of incoming work.

Each of those mechanisms protected the system from being overwhelmed. None of them protected the budget. They did not need to. The cost of overwhelming the system was capped at "the system gets slow," not "the bill triples." Slow was the worst case. Slow does not have a balance sheet.

The p99 cost problem

LLM systems break the cost per user distribution, and they break it in a direction nobody warns you about.

A median user might consume a few thousand input tokens and a few hundred output tokens per session. A p99 user, one who triggers a long retrieval context, a multi step agent loop, a verbose model on a thinking mode setting, and several tool calls, might consume hundreds of thousands of tokens in a single session. The ratio between p99 and p50 is not 5x or 10x. It is routinely two or three orders of magnitude. The same endpoint, the same auth, wildly different bills.

Three things drive the gap. Output tokens are typically priced several times higher than input tokens; on hosted frontier models the asymmetry sits in the three to five times range. Context is rebuilt every turn in a multi turn conversation, so a ten turn session does not pay 10x the cost of a one turn session, it pays something closer to the sum of every turn so far, because every turn resends the entire prior conversation. And agentic workflows multiply: a workflow that calls the model five times for one user request pays five times the cost of a single turn call, plus the cost of the tool calls in between.

This is not a tail to be optimised away by tuning the median. The p99 user is structurally different, different code paths, different context patterns, different workflow branches. The cost discipline has to be designed against the p99 user, not the median. Tuning the median user makes the dashboard look nice and changes the invoice not at all.

The first design step is to know what the p99 looks like in your own system, by tenant, by user, by workflow. Without that visibility there is no design, only hope, and hope reliably loses to a recursive loop on a Saturday night.

Per tenant and per user budgets

The single most effective cost guardrail is a budget that lives at the same granularity as the cost. For most B2B systems that is per tenant. For some B2C systems it is per user. For agentic systems it is sometimes per workflow execution.

A budget is three things: a number, a measurement window, and a behaviour when the number is exceeded. The number is the maximum cost the entity is allowed to incur in the window. The window is typically one minute, one hour, one day, or one billing month, depending on what the budget is protecting against. The behaviour is what happens at the ceiling, and it is the part teams forget to specify, which is how they end up with a ceiling that observes the overage politely and does nothing about it.

The four behaviours, in roughly decreasing order of customer impact:

1. Hard reject. Return an error or graceful refusal at the ceiling. Acceptable for free tier abuse protection; uncomfortable for paying customers.
2. Downgrade. Switch to a cheaper model for the remainder of the window. Preserves availability at degraded quality. The fallback must already be eval tested, the same pattern Article 5 covered for incident time mitigation. A fallback nobody tested is a second incident wearing the first one's clothes.
3. Queue with deferred processing. Move the work to an asynchronous batch path. Both major vendors offer batch API pricing at roughly half the synchronous rate, which makes deferred processing a viable budget recovery mechanism for noninteractive workloads.
4. Surface a paywall. Show the user the cost they have consumed and offer a path to a higher tier. Works only when the user knows they are using a metered resource.

What is not a valid behaviour: "do nothing and hope." A ceiling without an enforced behaviour is a number on a dashboard, not a guardrail. Enforcement has to happen at request admission time, before the model is called. A budget enforced by retrospective billing has already been blown by the time it fires. It is not a guardrail at that point, it is a receipt.

Semantic aware rate limits

Classical rate limits count requests. RPM is requests per minute; the leaky bucket admits N requests per second; the token bucket replenishes capacity at a fixed rate. The counted unit is the request, on the assumption that requests are roughly equivalent in cost.

In LLM systems they are not, and that assumption is the whole bug. Two requests to the same endpoint, with the same authentication, can differ in cost by orders of magnitude based on prompt length, model selection, sampling parameters, and downstream tool calls. An RPM counted rate limit lets a tenant consume the maximum allowed budget by sending the smallest allowed number of the largest possible requests. The limit is satisfied. The wallet is not.

Semantic aware rate limits count cost, not requests. The unit is tokens per minute, dollars per minute, or a synthetic cost unit per minute that combines model selection with token volume. The same algorithms, token bucket and leaky bucket, apply unchanged, with the counted unit replaced.

Vendor tokens per minute ceilings are the outermost envelope of this idea, set by the provider at the tier or organisation level. Both major hosted vendors expose tokens per minute and requests per minute as joint limits; whichever one binds first throttles the caller. The internal rate limit a system enforces on its own users should sit well below the vendor ceiling, with enough headroom that one greedy tenant never trips a 429 on the vendor side and incidentally rate limits every other tenant sharing the API key.

The architectural placement matters. A per tenant cost limit enforced inside the application is the only level at which a single tenant can be isolated from the others. Vendor level limits are shared across all tenants on the same key, which means a runaway tenant becomes everyone's incident, and the blameless postmortem now has a much longer attendee list.

Token aware caching

The biggest cost lever in most LLM systems is not the rate limit. It is the cache. The rate limit stops the bleeding; the cache stops you from paying full price for the same answer twice.

Both major hosted vendors expose explicit prompt caching mechanisms. Anthropic uses a cache_control marker placed at a prefix boundary; cache writes are priced above the base input rate and cache reads at roughly a tenth of it. OpenAI provides automatic prompt caching for repeated prefixes with a discount on the cached portion. The mechanic is the same: a long stable prompt prefix, system instructions, tool definitions, retrieval context that does not change between turns, is paid for once at write time and reused at read time for a fraction of the cost.

The savings compound on workloads that reuse prefixes. Agent system prompts, tool catalogues, schema definitions, multi turn conversations with stable system instructions, all have a long stable prefix and a short volatile suffix. Designed for cache friendliness, the same workload pays the input token cost for the stable prefix once per cache lifetime, rather than once per request.

Designing for cache friendliness is a structural choice, not an optimisation. The stable content goes at the start of the prompt. The volatile content goes at the end. Tool definitions are reused verbatim, not regenerated per request. These choices are made at prompt design time; they cannot be retrofitted after the workload is in production without a prompt refactor, which is the polite term for rewriting the thing under deadline.

Token aware caching at the application layer, semantic similarity caches for whole response reuse, is a complementary lever. Prompt caching reduces the cost of computing a response. Semantic caching avoids computing it at all. Both belong in the design.

Kill switches at multiple layers

A cost guardrail is only as good as the kill switch behind it. Five places where a kill switch belongs, in order of granularity from blunt to surgical:

1. The platform kill switch. A flag that disables LLM calls system wide. Used for budget emergencies, vendor outages, or trust and safety incidents. Returns a graceful fallback ("AI features are temporarily unavailable") on every endpoint.

2. The model kill switch. Disables a specific model. Routes requests that would have used it to a fallback. Useful when one model is responsible for a cost runaway or a behavioural regression.

3. The workflow kill switch. Disables a specific agentic workflow or multi step chain. Useful when one workflow is responsible for the cost. The recursive retry case in the opening scene is exactly this category, and the engineer staring at the Saturday trace would have given a great deal for this switch.

4. The tenant kill switch. Disables LLM calls for a specific tenant. Used when one tenant is generating runaway cost and the rest of the system needs to be preserved. Implies a defined customer communication path, because someone has to tell them.

5. The user kill switch. Disables LLM calls for a specific user. The narrowest blast radius. Useful for abuse cases and individual runaway sessions.

Each kill switch is a contract with the system around it: when this switch is off, the system handles the absence gracefully. Building the graceful absence path is the prerequisite. A kill switch that breaks the rest of the product when it fires is one nobody will dare flip during an incident, which means it does not exist when you need it most.

Observability for cost

Cost is a first class telemetry signal. Every model call emits a span (Article 3 of this series), and every span carries token usage in standard OpenTelemetry GenAI fields. Cost is a derived attribute on that span, tokens in plus tokens out, multiplied by the model's pricing, attributed to the calling tenant, user, workflow, and feature.

Three dashboards belong in every LLM system. Cost per tenant, rolled up by hour and day, with anomaly detection on rate of change. Cost per workflow, identifying which agentic paths drive spend. Cost per user within a tenant, surfacing the p99 users whose patterns might break the tenant budget if they grow.

The alerts on those dashboards make the budgets real. A per tenant cost alert that fires when a tenant's hourly burn exceeds a few times its trailing week average catches the weekend runaway in the opening scene within minutes, not after forty eight hours of accumulated bill. The alert should page someone on call. Cost incidents are operational incidents, even though they never turn a single dashboard red.

Failure modes, severity ordered

The patterns that turn a healthy LLM system into a cost incident, worst first:

1. Recursive workflows without bounded depth. The opening scene failure. An agent calls itself with the result of the previous step, the success condition is fuzzy, the loop never converges, the system pays for every iteration. Mitigation: maximum recursion depth, maximum total tokens per workflow execution, and a per workflow cost budget that hard rejects further calls. Series 1 Article 6, Multi Agent Coordination Patterns, covers the broader pattern that turns single agent loops into multi agent cascades; the cost discipline scales accordingly, which is a gentle way of saying it gets worse.

2. Unbounded retries. A model call fails. The system retries with exponential backoff. The retry fails. The system retries again. The retry budget has no ceiling, or the ceiling is per attempt duration rather than per attempt count. The system pays for every attempt. Mitigation: maximum retry count per logical operation, with the count enforced across the whole call graph rather than per individual call site.

3. Runaway agents. An agent in a tool using loop calls a tool, gets a result, decides to call another tool, gets a result, decides to call another tool, indefinitely. Mitigation: maximum tool calls per agent run, maximum total tokens per agent run, and a circuit breaker on tool call rate that fires before the wallet does.

4. Greedy multi turn context. Every turn of a long conversation resends the entire prior conversation. By turn fifty, the input cost per turn is dominated by ancient history that no longer matters. Mitigation: context summarisation, sliding window context, or memory store retrieval that keeps the active context bounded. Series 2 Article 2, Designing a Real Time Coding Assistant, is the canonical high volume per user case where this pattern bites first.

5. Cache cliff. The workload was designed to cache hit, but the cache is invalidated by a small variation in the prefix that nobody intended to vary, a timestamp, a request ID, a session token. The cache hit rate falls off a cliff, input cost rises by an order of magnitude overnight, and the only thing that changed was a field nobody thought counted. Mitigation: cache hit rate as a first class metric, with alerting on rate of change.

6. Wrong model defaulting. The system routes all requests to the most capable model, on the assumption that better is always better. Cheap requests pay frontier prices for no quality gain. Mitigation: model routing by request complexity, with a default to the cheapest model that meets the quality bar.

7. Vendor tier surprise. The team built against the vendor's middle tier; usage grew; the next tier up has a higher floor than the team budgeted for. Mitigation: vendor tier transitions are forecast against projected usage, not discovered by being throttled in production at the worst possible moment.

The Architect's Checklist

1. Every LLM call attributes its cost to a tenant, user, workflow, and feature, in the same span the observability layer captures.
2. Per tenant cost budgets are defined, enforced at request admission time, and survive process restarts.
3. Per tenant rate limits count tokens (or cost), not requests.
4. Every workflow execution has a hard ceiling on total tokens, total cost, and total recursion depth.
5. Retries are budgeted by logical operation, not per call site. Total retry cost is bounded.
6. Tool using agents have a hard ceiling on tool calls per agent run.
7. Multi turn conversations summarise or window context above a defined threshold.
8. Prompts are structured for cache friendliness, stable content prefix, volatile content suffix.
9. Vendor prompt caching is enabled wherever the workload supports it, with cache hit rate monitored.
10. Semantic similarity caches sit in front of the model for response level reuse where business logic permits.
11. Model routing chooses the cheapest model that meets the quality bar for each request class.
12. Kill switches exist at platform, model, workflow, tenant, and user granularity. Each "off" state is graceful.
13. Cost dashboards by tenant, workflow, and user are live. Anomaly alerts on burn rate page someone on call.
14. Vendor tier transitions are forecast quarterly, not discovered at throttle time.

The Architect's Mental Model

LLM systems are the first dependency in the modern stack where the worst case user is also the most expensive user, by a factor large enough to matter on the company's income statement. The discipline this article describes is what closes the gap between worst case behaviour and worst case cost, not by making the worst case impossible, but by making the worst case bounded.

The budgets define the bound. The rate limits enforce it. The caches lower the floor. The kill switches give the operator a way to stop the bleeding. The observability makes any of the above possible at all. Remove any one and the other four develop a blind spot in the exact shape of the thing you removed.

None of these are optimisations. They are structural defences against the system you have already built. Without them, the system has no defined worst case cost. With them, the worst case is a number you chose in advance, which is the only kind of worst case anyone in finance is willing to sign off on.

Classical systems get slower under overload. LLM systems send you the bill.

What's next, closing Series 3 and opening Series 4

This article closes Series 3, Shipping AI Systems, six articles on the operational discipline that turns an LLM prototype into a production system that survives contact with real users, real traffic, real vendors, and real budgets. Evaluation, versioning, observability, rollout, incident response, and now cost. Each topic has a classical analogue in the pre LLM stack. Each one needed adaptation for an artefact whose contract is a probability distribution rather than a typed signature.

The next series, Series 4: AI for the .NET Architect, turns from the general operational discipline of the platform to the specific implementation discipline of building agentic systems on .NET. The Microsoft Agent Framework, Microsoft.Extensions.AI as the unified client layer, structured outputs in C#, hosting long running agents, streaming with IAsyncEnumerable and SignalR, and a reference architecture that ties it together. The same architect's lens; a more concrete stack.

Series 4 starts a new cadence, one article per week, replacing the twice weekly Series 3 sprint. Subscribe at pragmaticstack.hashnode.dev to catch the first .NET article.

🗓️ Last updated: July 2026

The screenshot is on a leadership Slack channel by 9:14 in the morning. A customer asked the assistant a routine policy question. The assistant answered confidently and in detail. The answer was wrong in a way that, if acted on, would have cost the customer real money. The customer screenshotted it, posted it, and tagged the company account. The post is gaining traction. The CEO has asked, in writing, what is being done.

The on call engineer pulls up the dashboards. The system is healthy. Latency normal, error rate zero, no recent deploys. The git log shows nothing in the last week that would touch this surface. The prompt template has not changed. The model version pinned in the manifest is the same revision that was in production yesterday and the week before. The retrieval index is the same. By every metric the platform tracks, nothing is wrong.

The model said the thing. The model is the bug. There is no commit to revert.

This is the failure shape that breaks classical incident response. The on call playbook assumes that an incident has a cause, the cause is in a system you control, and the fix is a code change or a config change that reverses the cause. When the model is the bug, all three assumptions wobble. The cause may not be in any system you control. The fix may not be a code change. And the "revert" may not exist.

This article is about how to run an incident when the failing component is the model itself.

What this article is not

It is not a general incident response tutorial. The classical roles, severity levels, war room structure, and postmortem discipline all still apply. What changes are the triage steps, the mitigation surface, and the shape of the postmortem when the bug lives inside a probability distribution rather than inside your repository.

The thesis

Classical incident response assumes you can find the bad commit and revert it. Model incidents have no commit. The playbook needs three things classical incidents do not: a replay capture that lets you reproduce the failure deterministically from logged inputs, a triage taxonomy that distinguishes between the five places the bug actually lives, and a runtime mitigation layer that lets you change behaviour without a deploy. Get those three right and a model incident becomes a tractable engineering problem. Skip any of them and you end up in the "wait for the vendor" state, with an angry CEO and no plan.

What classical incident response looked like

The classical playbook is well documented and largely universal. The shape, condensed:

1. Detection. An alert fires, a customer reports, a metric breaches.
2. Triage. Severity is assigned, typically SEV1 for a customer facing outage, SEV2 for major degradation, SEV3 and below for partial issues, following the framework PagerDuty publishes openly as part of their incident response guidance.
3. Incident command. A single incident commander coordinates. A communications lead handles internal and external updates. Responders investigate.
4. Diagnosis. Logs, traces, dashboards. A recent change is identified: a deploy, a config push, a dependency upgrade.
5. Mitigation. Revert the change, fail over, scale out, drain traffic. The faster path is almost always reversal of the most recent change.
6. Resolution. The metric returns to baseline. The incident is closed.
7. Postmortem. Following the model formalised in Chapter 15 of the Google SRE book, Postmortem Culture: Learning from Failure, the team writes a blameless retrospective focused on systemic causes rather than individual fault.

This works because classical incidents have causal arrows that point at code. The bug is in a function, a query, a config, a state. The team finds the arrow and breaks it.

What changes when the model is the bug

Three things change at once when the bug lives inside the model.

Detection moves outside the system. The customer facing screenshot in the opener was not detected by any internal metric. The model did not error. The latency was fine. The deterministic plumbing all worked. The bug surfaced because a human read the output and recognised it as wrong. The first signal of a model incident is very often external, a support ticket, a screenshot, a Trust and Safety report, a regulator's email, and the detection latency is the time it takes a customer to notice, not the time it takes an alert to fire.

The "recent change" hunt fails. The triage instinct in a classical incident is to look for the last thing that moved. In a model incident, often nothing moved in any system you control. The model's behaviour shifted because retrieved context shifted, because a downstream prompt exploited an interaction that had always been possible but never triggered, because the hosted version silently updated on a moving alias, or because today is simply the day the dice rolled badly on a failure that was always possible at low probability. Article 2 covers why "what moved" is harder to answer for LLM systems. In an incident, that difficulty becomes time to mitigate.

Mitigation does not look like a revert. Classical mitigation reverses the most recent change. Model incident mitigation is constructive: you ship something new, a tighter system prompt, a content filter, a fallback model, a kill switch on a specific tool call, because there was nothing to revert. Every patch survives into the next incident.

The replay imperative

The single most important capability in model incident response is replay. Given a failure case, the exact prompt the customer sent, the exact retrieval context the system assembled, the exact tool calls and intermediate responses, the exact model version pinned at the time, can you rerun that case offline and reproduce the failure deterministically?

If yes, the incident becomes a reproducible bug, and a reproducible bug is a tractable engineering problem. The team can iterate on prompt changes, fallback configurations, and filter rules against the replay until the bad output goes away, and then test the mitigation against the wider eval suite to confirm no regression elsewhere.

If no, if the system cannot reconstruct what happened well enough to reproduce it, then the team is reduced to guessing. Every proposed fix is a hypothesis with no way to verify before deploy. The incident enters the "wait and see" state, where mitigations are shipped speculatively and the only way to know if they worked is to wait for the next failure. This is a miserable place to run an incident from.

The observability discipline from Article 3 of this series is the prerequisite. Replay requires that every model call captured its full input, full output, model version, sampling parameters, retrieval context, and tool call trace, at the time it happened, retained long enough to still exist when the incident is opened. Sampling based observability is fine for capacity planning. It is fatal for incident response, because the one call you need is invariably the one that was sampled out.

The two practical questions to answer in advance, before the incident:

• What is the maximum gap between failure and reproduction? If a customer reports a bad output from three days ago and the logs only retain twenty four hours of full request bodies, the replay never happens.
• What is the privacy posture? Full fidelity replay implies storing user prompts and model outputs, which has data protection implications. The retention window and access controls need to be defined before the SEV1 ticket opens, not negotiated during it.

The triage taxonomy

When the model is the bug, "the model" is a useful shorthand but a terrible diagnostic category. The actual bug usually lives in one of five distinct places, and the mitigation differs for each. Naming them up front is the difference between a triage that converges and a triage that loops.

1. Hallucination on the model itself. The prompt and the context were reasonable. The model produced a confident, fabricated answer. The bug lives inside the model's training distribution. Mitigation surface: a tighter system prompt with explicit grounding instructions, retrieval enforcement ("only answer from the provided context"), a stricter judge model on the output, or in severe cases a switch to a different model with better grounding behaviour on this class of prompt.

2. Retrieval poisoning or retrieval miss. The model received misleading or insufficient context and answered the question as if the context were correct. The bug lives in the retrieval layer or the source data. Mitigation: rerank, source filtering, source authority weighting, a blacklist on specific documents, a refresh of the index. Multi tenant RAG systems, covered in Series 2 Article 3, have an extra dimension here: the bad context may belong to one tenant and the failure may surface for that tenant only.

3. Prompt regression. A recent prompt change, a tweak to the system instruction, a new tool description, a refactored output schema, interacts badly with a class of input nobody tested. The bug is in the prompt commit. Mitigation: revert the prompt to its previous version, which is the closest model incidents get to a classical revert, then roll the fix out again with the failing case added to the regression suite. Article 2's version discipline patterns are the prerequisite for a clean revert.

4. Vendor side degradation. The hosted model started behaving differently. Either the alias pin moved to a new revision (avoidable, covered in Article 2), or the dated revision the team pinned itself drifted, through capacity issues, regional routing, or a partial outage. Mitigation: the vendor status page is the first stop. If the incident is on the vendor side, the response is communication and partial degradation modes, not engineering. Article 4 covers the deprecation discipline that prevents this from becoming an emergency in the first place.

5. Downstream contract break. The model output is correct, but a system downstream of the model is mishandling it: a parser, a renderer, a tool call adapter, a workflow step. The user sees a broken response, but the model is innocent. This was the opening scene of Article 4. Mitigation lives in the downstream system, not in the model layer. The triage error is to spend an hour tweaking prompts when the actual bug is in a regex three services away.

The triage flow, in practice: capture the replay, walk the five categories in order, eliminate each on evidence. Most incidents in production fall into categories 2, 3, or 5, not category 1. Declaring "the model hallucinated" without ruling out the other four is one of the most common errors in model incident response, and one of the most expensive.

Runtime mitigations

A model incident usually cannot be mitigated by deploy. Or, more precisely, the deploy cycle for a model system, change the prompt, run the eval suite, push to staging, canary, full rollout, is far too slow for an active customer facing incident. The runtime mitigation layer is what bridges the gap between "the incident is open" and "the proper fix is in production."

Three patterns, ordered by how surgical they are.

Filter and reject. A content filter sits between the model and the user. For the duration of the incident, it rejects outputs matching a specific pattern, the offending phrase, a class of factual claim, a tool call to a specific function. Rejected outputs return a graceful fallback. The cost is precision loss and the risk of over blocking. The benefit is that the bad output stops reaching customers immediately.

Fallback to a different model. If the failure is localised to a specific model on a specific class of prompts, route that class to a different model. The fallback may be slower or more expensive, but on this class of prompts it is correct, and during an incident "correct and degraded" beats "fast and wrong" every time. The fallback model has to already be wired in and eval tested for this class. Standing up a new vendor integration during a SEV1 is not feasible, no matter how calmly leadership asks for it.

Kill switch a capability. Disable the tool, the retrieval source, the workflow step, or the model entirely for the affected surface. The kill switch is the bluntest instrument and the most reliable. It assumes that "off" is a defined state the rest of the system handles gracefully, where Article 4's rollback semantics apply here too.

Each of these mitigations is a patch that survives into the next incident unless someone deliberately removes it. Without discipline, the system slowly becomes a thicket of incident era filters that nobody dares touch. Every runtime mitigation needs an owner, an expiry date, and a review cadence.

Postmortems on probabilistic failures

The blameless postmortem template, what happened, when it was detected, when it was mitigated, contributing factors, action items, applies to model incidents largely unchanged. Two adaptations matter.

"Root cause" is sometimes the wrong framing. Classical postmortems converge on a primary contributing factor: a bad commit, a missed alert, a configuration drift. Model incidents often have no single cause. The system was always capable of producing this output, and the conditions to elicit it happened to align today. The honest postmortem says the failure surface was visible in retrospect and the system should have had a filter, a stricter retrieval rule, or a different prompt, not that there was a bug to fix. Forcing a single root cause onto a distributional failure produces postmortems that read as deflection.

Action items need a recurrence probability framing. Classical action items reduce a probability that was effectively one back to effectively zero. Model incident action items reduce a probability that was always non zero. "Add this case to the regression eval suite" is honest. "Fix the model so this can never happen" is not. The postmortem distribution list also widens, since contributing factors typically span the model, retrieval, prompt, downstream consumer, and vendor layers.

Failure modes, severity ordered

The patterns that turn model incidents from manageable into ruinous, roughly worst first:

1. No replay capability. The system cannot reproduce the failure offline. Every mitigation is a guess. The incident closes when the customer stops complaining, not when the cause is understood. Mitigation: full fidelity observability per Article 3, with a retention window matched to incident detection latency.

2. "Blame the user." The triage concludes that the failure was the user's fault: a malformed prompt, an unusual phrasing, an edge case. The action item is to do nothing. The same incident recurs with a different user a week later. The OWASP LLM Top 10 explicitly classes prompt injection as a system risk, not a user error, and the same framing applies to most edge case prompts. The user surface is the production surface.

3. Infinite mitigation. Each incident adds a filter or a prompt patch. None are ever removed. After a year the system is a layer cake of incident era mitigations, the regression suite takes hours, and a new prompt change can no longer be reasoned about. Mitigation: every runtime mitigation has an owner and an expiry date, and the postmortem reviews previous mitigations for retirement.

4. Single source triage. One engineer can read the replay. They become the bottleneck on every model incident, and then they go on holiday. Mitigation: replay tooling exposed to the whole team, with a runbook walking the five triage categories.

5. Model vendor lock in revealed at the worst moment. The incident requires falling back to a different model. There is no other model integrated. The team discovers during the SEV1 that switching vendors is a two week project. Mitigation: at least one alternate model is integrated, eval tested on production traffic patterns, and wired into the routing layer before the incident.

6. "Wait for the vendor" as a strategy. The triage concludes the bug is on the vendor side. The team's response is to wait. No degraded service mode is engaged. Customers continue hitting the failure. Mitigation: every "wait for the vendor" decision is paired with a degraded service mode that is actually shipped during the wait, a filter, a fallback, a feature disablement.

7. Postmortem theatre. The postmortem is filed, the action items are assigned, and none of the action items ship. The next similar incident finds the same gaps. Mitigation: action item completion is tracked at the same cadence as the original incident severity. SEV1 action items have a one week SLA for closure, not a "when we get to it."

The Architect's Checklist

1. Every model call captures full input, full output, model version, sampling parameters, retrieval context, and tool call trace, retained long enough to outlast incident detection latency.
2. A replay harness exists. Any engineer on call can rerun a logged failure case offline against the production model version.
3. The triage runbook walks the five categories explicitly: hallucination, retrieval, prompt regression, vendor degradation, downstream contract break.
4. At least one alternate model is integrated, eval tested, and wired into a routing layer that can fail over during an incident.
5. A runtime filter layer exists between the model and the user, with a way to push a new pattern based block without a deploy.
6. Kill switches exist for individual tools, retrieval sources, workflow steps, and the model itself. Each "off" state is a defined operation the rest of the system handles gracefully.
7. Every runtime mitigation has an owner, a documented incident reference, and an expiry or review date.
8. The incident severity framework explicitly includes model output quality as a SEV class, not just availability and latency.
9. The first external detection channel, support, Trust and Safety, social, has a defined escalation path into on call.
10. Vendor status pages are integrated into the on call dashboard.
11. Postmortem templates for model incidents replace "root cause" framing with "failure surface and recurrence probability."
12. SEV1 and SEV2 action items have closure SLAs, tracked at the same cadence as the original incident.
13. The on call rotation includes someone who can read the replay. The replay tooling is not a single engineer skill.

The Architect's Mental Model

A model incident is a deterministic system incident with the causal arrow pointing into a probability distribution. The classical disciplines, incident command, blameless postmortems, severity classification, action item closure, survive almost unchanged. The deterministic toolchain, find the commit, revert it, monitor, does not.

What replaces it is replay, a triage taxonomy, and a runtime mitigation layer. Replay turns the incident from speculation into engineering. The taxonomy stops the team from looping on "the model is broken." The runtime layer stops the customer from being the next person to discover what the team already knows.

Classical incidents have a commit. Model incidents have a customer.

🗓️ Last updated: July 2026

A support ticket arrives on a Monday morning. A user asked the assistant a billing question. The answer was wrong, confident, and specific enough to do damage. The user followed it, disputed a charge, and escalated. The customer experience lead wants a root cause by end of day.

The on call engineer opens the logs. Request received, latency normal, status 200. The response payload is captured. The session ID is captured. None of it explains anything. There is no prompt, no retrieval context, no token usage, no finish reason, no model version, no judge score: none of the things that would let anyone reconstruct what the system actually did. The deterministic plumbing all worked. The behaviour was the bug. The behaviour was not instrumented.

By noon the team has reverse engineered most of the path by replaying the session. By afternoon they have a hypothesis. By the end of the day they have a fix, almost certainly to the prompt, possibly to the retriever, they cannot fully tell because nothing was captured at the level of detail that would settle it. The ticket gets closed. The next one is already in the queue. Nobody is confident the same class of failure will not recur next week.

This article is about the instrumentation discipline that would have settled the question in twenty minutes.

What this article is NOT. It is not a vendor comparison of observability platforms. It is not "five dashboards every LLM team should build." It is the architectural argument that observability for LLM systems requires capturing different signals at a different granularity than classical observability, and that retrofitting it after a quality incident costs an order of magnitude more than designing it in.

What's Different

Classical observability rests on a stable premise: when something breaks, there is a code path that broke it. The job of the observability stack is to point you at that code path. Stack traces, structured logs, request IDs, distributed traces: all of these are mechanisms for compressing "what executed" into something a human can inspect.

LLM systems break that premise in two specific ways.

First, the bug is often in the output, not in any code path. A model returns a confidently wrong answer. The HTTP request was fine. The retrieval ran. The prompt assembled. The model responded. Every code path executed correctly. The output was wrong. Classical instrumentation has nothing to say about why.

Second, the bug is statistical. One in fifty inputs of a certain shape triggers a refusal that should not have happened. One in two hundred produces a hallucinated entity. The other forty nine are fine. Without a way to slice production traffic by input shape, output shape, and outcome, you cannot find the failure cluster. You can only respond to the individual complaint that surfaced it.

There is a structural consequence that catches teams used to deterministic systems off guard: in an LLM system, the content is the diagnostic data. The actual prompt text, the actual model output, the actual retrieved documents: these are not optional metadata. They are the primary signal. A logging policy that excludes them on the grounds that they look like business data has effectively turned off the observability stack.

The Four Golden Signals, Revisited

The Google SRE book popularised the four golden signals: latency, traffic, errors, saturation. The four still apply to LLM systems. They are no longer sufficient.

Latency remains observable, but the question it answers has changed. Time to first token matters as much as wall clock latency for any streaming interface. Latency is also a function of output length, which is a function of the prompt, which is a function of the user's input. A latency regression often turns out to be an output length regression in disguise: the new model produces longer responses, so requests take longer end to end, even though tokens per second is unchanged. Instrument both.

Traffic is no longer well summarised by requests per second on a route. The useful breakdowns are by prompt template, by model and version, by tenant, by intent class. A spike in cost or latency that is invisible at the route level is often a tenant level or template level event. Per route monitoring will miss it for hours.

Errors is the signal that drops in usefulness the most. A 200 OK response containing a wrong answer is the most expensive error in an LLM system, and the classical error rate cannot see it. The traditional 5xx rate still matters for vendor outages, but it is no longer the primary quality signal. The replacement is a portfolio of secondary signals: retry rate, escalation rate, eval scores against a frozen reference set, signals that approximate behavioural correctness without depending on user explicit feedback.

Saturation is still useful, but the thing that saturates is different. CPU and memory matter less than the vendor side tokens per minute limits, the per tenant rate caps, and the eval pipeline's compute budget. A team running near a vendor tokens per minute ceiling will see latency tail blow out long before any classical saturation alarm fires.

The four golden signals were never wrong. They were specified for a class of system where they were sufficient. LLM systems are not in that class.

Span Design for Chains and Agents

The unit of observation in a classical web service is a request. The unit of observation in an LLM system is a trace: a parent span that covers the whole interaction, with child spans for each LLM call, each retrieval, each tool call, each guardrail check, each downstream service.

Span design matters more than instrumentation library choice. The OpenTelemetry GenAI semantic conventions, increasingly adopted across observability vendors in this space, give standard names to the fields that need to be on every LLM span: gen_ai.system, gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.response.finish_reasons, and the prompt and completion themselves. Adopting the convention is cheaper than inventing your own field names; the cost of being nonstandard surfaces the first time you have to migrate observability vendors or write a cross platform query.

For chains and agents specifically, the parent span is the task, not the request. A user message that triggers a five step plan executed by an agent should produce a single trace with the parent span containing the user's intent, the final response, and the aggregate token and cost figures, along with child spans for every model call, every tool invocation, and every retrieval step in between. Without that hierarchy, debugging a five step failure means stitching five separate request logs together by timestamp, which is the manual reconstruction the on call engineer in the opening scene was doing.

Capture the retrieval context as data on the span, not as a pointer to a document store. A trace that says "retrieved 5 documents (ID 1, ID 7, ID 41, ID 88, ID 102)" is not useful three weeks later when the document store has updated. A trace that captures the actual text the model saw at the time the model saw it is useful indefinitely. Storage is cheap; lossy traces are expensive.

Semantic Clustering

A single complaint is one data point. A cluster of similar complaints is a pattern, and patterns are what you can fix.

The classical clustering surface is the URL or the error code. LLM systems need a different one: cluster by what the user was trying to do and what went wrong. Embedding based clustering of production traces is the dominant technique: group user inputs by semantic similarity, then look at the failure rate or the eval score per cluster. A four per cent global retry rate can hide a twenty eight per cent retry rate inside a single cluster of billing related questions. The dashboard that reported "retry rate is up slightly" misses the structural failure entirely.

Several observability platforms in this space, including Langfuse, Arize, Helicone, and Traceloop, offer some form of trace clustering or trace search. The specific tool matters less than the workflow. A useful workflow is: filter traces by outcome (low eval score, user retry, user escalation), cluster the remaining traces by embedding of the user input, look at the top three clusters by failure density, form a hypothesis. If that workflow is not possible in your stack, requiring SQL queries against a warehouse and a custom embedding job, it will not happen during an incident. Clustering becomes a thing that happens once a quarter in a retrospective.

The structural argument is that production traffic is the most honest curation source you have. The golden set was curated when the product launched. The production trace clusters tell you what the product is actually used for now. An observability stack that does not surface those clusters is hiding the dataset you most want to look at.

PII and the Logging Tradeoff

Classical observability has a clean default: do not log personally identifiable information. Log the user ID, log the request URL, log the latency, never log the form body. The discipline is well understood and compliance teams have decades of precedent to lean on.

LLM systems break the default in a specific way. The prompt is the request body. If the user asked "can you summarise the email from my doctor about my appointment on Tuesday," the prompt contains a medical context that no classical request body would carry. The model's response often contains entities lifted from the prompt: names, dates, locations, account numbers, things the classical observability stack has no policy for.

There is no clean answer. There are deliberate ones:

• Redact at log time. Run a PII detector before persistence; mask the matches; persist the redacted form. The detector is itself a model and has its own miss rate; budget for it and review its outputs periodically.
• Tier the retention. Keep full content for a short debug window of hours to a few days. Keep the redacted form for longer term analytics. Define both windows in writing and enforce them in the storage layer, not in policy alone.
• Gate detail by consent or context. Some traffic carries explicit permission for verbose logging: enterprise contracts, internal traffic, debug environments. Some does not: consumer traffic in regulated domains. The trace pipeline should know the difference and apply it at capture time.

The default of "log everything and figure it out later" is a compliance incident waiting to happen. The default of "log nothing to be safe" is operational blindness. Neither is the answer. The answer is a written policy that engineering, security, and legal all signed, and an instrumentation stack that implements it.

Alerting on Probabilistic Signals

The hardest part of LLM observability is not capture. It is alerting.

Classical alerts are threshold based and mostly correct: when the 5xx rate exceeds one per cent, page the on call. The signal is clean. The action is clear. The false positive rate is low enough to keep the team's attention. LLM signals are noisier, slower, and more correlated with each other. A few patterns work, and a few do not.

• Alert on trend, not on absolute value. Eval scores oscillate. Retry rates fluctuate with traffic mix. A single bad hour is rarely the start of an incident; a two sigma drop sustained across three measurement windows often is. Anchor alerts to rate of change, not to level.
• Alert on the change you cannot explain. The finish reason distribution shifting overnight, say more length finishes and fewer stop finishes, is the kind of signal that is small enough to miss and structural enough to matter. A daily diff on these distributions catches it before the customer experience team does.
• Treat cost as a quality signal. A substantial jump in tokens per request across a tenant cohort, with no change in traffic shape, is almost always a prompt or model regression making the model more verbose. Cost dashboards usually exist for finance reasons; they double as a leading indicator of behavioural change.
• Do not alert on individual prompts. Per request alerting is a paging volume problem with no upside. Aggregate first, then alert.

There is a softer point underneath all of these. Alert volume is finite. Every probabilistic signal alert competes with every deterministic signal alert for the on call's attention. A team that adds twelve new LLM alerts in a quarter without retiring any will arrive at alert fatigue on a schedule. Treat the alert catalogue itself as a constrained resource.

Failure Modes

Observability programs fail in ways that look like success on the dashboard. Seven recurring failure modes, severity ordered:

1. The dashboard says everything is fine. All classical signals are green. The eval pipeline is green. Customer experience tickets are climbing. The dashboard is measuring uptime and the failure is in behaviour. The team built a deterministic dashboard for a probabilistic system.
2. PII in places it should not be. A retention sweep finds full medical context prompts in a six month old trace archive. The compliance impact is real, the remediation is expensive, and the right policy was never written before the instrumentation was turned on.
3. Span explosion. Every LLM call generates dozens of child spans for guardrail checks, embedding lookups, and tool dispatches. The observability bill exceeds the inference bill. The team starts sampling, then sampling more aggressively, and ends up blind on the failures the sampling drops.
4. Trace decoration without trace consumption. The traces are beautiful. Nobody opens them during incidents. The on call instinct is still to read the application logs, because that is the muscle memory. The instrumentation never crosses over from "we built it" to "we use it."
5. Alert fatigue from probabilistic noise. Every drift signal becomes a page. The team learns to acknowledge and ignore. Real drift goes unnoticed for a week. The page volume was the failure; the drift was the symptom.
6. No clustering, only individual complaints. Each escalation is investigated as a one off. The pattern is real but invisible: it would take ten complaints to see it and only three have reached the queue. The eleventh complaint comes in via the regulator.
7. Eval in production lag. The production eval suite runs nightly. The team finds out about drift at the next morning's standup. The cluster of failures that triggered it has been live for sixteen hours.

The Architect's Checklist

Twelve items. If a system fails more than three of these, the observability stack is the thing to fix before the next feature ships.

1. Every LLM call produces a span with prompt template ID, prompt version, model ID, model version, input and output token counts, finish reason, latency, and cost.
2. Retrieval steps capture the actual text retrieved, not pointers to a document store that may have updated.
3. Spans roll up into traces at the task level, so a five step agent run produces one trace with five visible child spans.
4. Trace IDs are propagated into the user facing response or support metadata, so a customer ticket can be traced back to the exact production trace in under a minute.
5. Production traces can be clustered by embedding of the user input, queryable in under a minute by anyone on call.
6. PII handling is written down, signed by security and legal, and enforced in the storage layer, not in a policy document nobody reads.
7. Production eval scores are computed continuously, or at least hourly, against a frozen golden set, with trend visible to on call.
8. Alerts fire on rate of change of probabilistic signals, not on absolute values.
9. The finish reason distribution is monitored as a leading indicator of behavioural change.
10. Per tenant cost and token usage are visible in near real time, so cost regressions surface as fast as latency regressions.
11. The on call runbook contains a "this is an LLM bug, not a code bug" branch that names the dashboards and traces to consult.
12. The team runs at least one debugging drill per quarter using only the observability stack, to confirm the instrumentation is sufficient to reach a root cause without out of band help.

A team that can answer yes to most of these can debug an LLM incident the way classical incidents are debugged: in minutes, with the data, without reverse engineering anything.

The Architect's Mental Model

The instinct from classical observability is that you watch the system. The shift in an LLM system is that you watch the behaviour. The system is a thin wrapper around something that produces behaviour, and the behaviour is where the failures live.

That shift has a cost. Watching behaviour means capturing content, which means handling PII deliberately, which means having policy and infrastructure in place that classical observability did not need. It means treating prompt versions, model versions, and retrieval context as first class fields on every span. It means designing alerts around statistical signals that the on call has to learn to read differently from threshold alerts. None of this is free, and the temptation to skip it on the grounds that the classical stack still works is what produces the support ticket that opens this article.

One question to carry out of this article: if a production trace from twenty minutes ago needed to be retrieved, clustered, and explained to the customer experience team within the hour, could your current stack do it? If not, that is the next gap to close.

A deterministic system tells you what failed through a stack trace. A probabilistic system tells you through a customer's confusion. Observability for the second kind of system is the discipline of catching the failure before the customer's confusion reaches your queue.

🗓️ Last updated: July 2026

The release diff is empty. No commits since Tuesday. No infrastructure changes. The dashboards from a week ago and the dashboards from this morning track different distributions of user behaviour. Somewhere in the system, something changed, and nothing in the git log points to it.

The team starts the usual hunt. Build pipelines: green. Container hashes: identical. Database migrations: none. Feature flags: untouched. The hunt continues for two hours before someone runs the offline eval against this morning's logs and notices that the model's responses have shifted in tone: slightly more cautious, slightly more verbose, slightly less helpful on a particular class of question.

The model is the same model. The prompt is the same prompt. The retrieval pipeline returns the same documents. And yet, the system's behaviour has moved.

What changed?

The team is using a floating model alias, one of those "give us whatever is newest" pointer strings the vendor provides as a convenience. The vendor rolled out a minor revision overnight. The training data cutoff shifted by a few weeks. A safety policy was tightened. None of these moves required the team to deploy anything. None of them registered in source control. And the team had no online eval running continuously that would have caught the shift the moment it happened.

This article is about the discipline that turns invisible changes into visible ones.

What this article is NOT. This is not a prompt engineering guide. It will not teach you how to write better prompts, compare model providers, opine on chain of thought formatting, or grade the relative merits of XML tags versus markdown headings in system messages. This is an article about versioning: treating the prompt as code and the model as a dependency, with all the source control discipline, dependency pinning discipline, and rollback discipline those framings demand. The thesis is structural, not stylistic. Most teams ship LLM systems where the most behaviour defining components live outside the version control loop, and then they wonder why behaviour shifts overnight.

The Thesis

A prompt is code.

A model is a dependency.

A retrieval index is a build artefact.

A function calling schema is an API contract.

Most teams ship LLM systems treating all four of these as configuration: strings in a database table, an alias that points to "whatever the vendor is serving today," an index that gets rebuilt on a cron, a list of tools defined inline somewhere in the orchestration layer. None of those four components has a version in the operational sense. None of them passes through a code review, an eval gated deploy, or a rollback procedure. And yet, between them, they account for almost all the production behaviour of the system.

The fix is to treat each of them with the same operational discipline you would apply to a service in production: source controlled, version pinned, eval gated, rollback safe.

What the Classical Version Looked Like

For two decades, the industry has had a clear answer to "how do we keep production behaviour from changing under us":

• Code lives in source control with a complete history of every change.
• Dependencies are pinned to specific versions in a manifest: package.json, requirements.txt, go.mod, Cargo.toml.
• Lockfiles record the exact resolved version of every transitive dependency.
• Versioning follows a published contract. Semantic versioning is the dominant one, codifying the difference between a patch, a minor, and a breaking change.
• Releases are tagged, immutable, and reproducible. Checking out a specific tagged version gives you the same artefact today as it did six months ago.

This stack rests on a structural assumption: behaviour is a function of code, and code is what source control tracks. As long as the code is pinned, the behaviour is pinned.

LLM systems break that assumption at every layer.

The Four Moving Parts

In a classical service, source control tracks essentially everything that determines behaviour. In an LLM service, four components move outside the source control loop unless you specifically pull them in:

1. The prompt. Often a string in a database row, edited via an admin tool, with no diff, no review, no tag.
2. The model. Referenced by a floating alias that resolves to whatever the vendor is serving today.
3. The retrieval index. Rebuilt on a schedule, with no record of what was indexed when, no diff against the previous build, no rollback.
4. The function calling schema. Defined inline in orchestration code, edited freely, with no notion that a downstream contract depends on its shape.

Each of these is a behaviour defining surface. Each can change in production without a deploy. Each can change in production without anyone noticing.

The versioning discipline applies the same operational pattern to all four: source controlled, version pinned, diffable, rollback safe, eval gated.

Prompts as Code

Treat every prompt as a first class source controlled artefact. That is the floor.

In practice, this means:

• Every prompt lives in source control, either inline in the code or in a structured prompt file alongside the code.
• Every prompt has a semantic version, incremented on change.
• Every prompt change goes through code review, with a visible diff.
• Every prompt change runs against a regression eval suite before it can be promoted.
• The promotion mechanism is the same one used for code: a release artefact, an environment tag, a deploy.

Several mature patterns exist. Microsoft's Prompty format stores prompts as text files with YAML front matter, typed named inputs, and explicit model configuration, modelled explicitly after how source files store code. Tools like LangSmith, Langfuse, PromptLayer, and Helicone offer prompt registries with version history, diff views, and experiment tooling, so that a prompt edit is traceable and reversible.

The pattern matters more than the tool. The pattern is: a prompt is not a configuration value. A prompt is a behaviour defining instruction with operational characteristics, including failure modes, edge cases, latency profile, cost profile, and evaluation surface, that demand the same kind of governance you give to code.

Two design choices are worth being explicit about.

Inline versus registry. Inline prompts in the repo are simplest, naturally source controlled, and trivially reviewable. Registry prompts decouple prompt updates from code deploys, which is useful when prompt iteration needs to happen faster than the release cycle. The tradeoff is that the registry now sits in your operational dependency graph, with all the availability, audit, and rollback concerns that implies. For most systems, inline is the right starting point; move to a registry only when there is a concrete reason that justifies the new dependency.

Templating discipline. Prompts that interpolate runtime variables such as user names, tool outputs, and retrieved documents should make those variables explicit, named, and validated. An untyped template that silently consumes whatever upstream input arrives is the prompt equivalent of dynamic SQL: it works until it doesn't, and the failure is uninspectable. Pin the variables. Validate the shapes. Treat the template as a function signature.

Models as Dependencies

Treat every model the way you would treat a third party library upgrade.

For classical dependencies, the discipline is well understood: pin to a specific version, not a moving tag; read the changelog before upgrading; run a regression suite against the new version before promoting; roll out the upgrade gradually, with a rollback path; subscribe to the vendor's deprecation schedule.

For LLM models, the same discipline applies with one important difference. Classical dependencies, properly pinned, do not change. LLM models, even when pinned, can be deprecated by the vendor on a published schedule, after which the pinned version stops responding. Pinning buys you predictability for a window, not forever.

Two patterns matter.

Pin to dated revisions, not aliases. Major LLM vendors publish dated model revisions with the date embedded in the identifier: something like Claude 3.5 Sonnet 20240620 or GPT 4o 20241120. The dated form is the equivalent of a Git commit SHA. The alias form (claude latest, gpt 4o, the floating convenience pointer) is the equivalent of a floating branch. Production systems should reference the dated form, full stop. If you cannot point at a specific revision string in your config, you are not running pinned.

Track vendor deprecation as a first class risk. Major LLM vendors publish model deprecation pages and version policies, and the trend across vendors is to give months of notice between announcement and shutoff. Treat that calendar the way an enterprise IT department treats an OS support window. Plan the upgrade. Run the eval. Schedule the cutover. Do not be in the position of discovering the deprecation when the API starts returning errors.

When the cutover happens, treat it exactly like any other dependency upgrade. The new model is a new dependency. It needs a regression suite. It needs a shadow traffic comparison. It needs a canary. Do not let the fact that the vendor scheduled it for you bypass the discipline you would apply to a self driven upgrade.

Eval Gated Deploys

The discipline above buys you traceability: every change becomes visible, diffable, attributable. It does not, on its own, buy you safety. Safety comes from gating every change behind an eval that has to pass before the change reaches production.

The shape of the gate matters. For a classical code change, the gate is a unit test suite that exercises every code path. For a behaviour defining change in an LLM system, whether prompt, model, retrieval, or schema, the gate is a regression eval suite that exercises every behaviour the system is contractually required to deliver:

• Held out examples that the previous version handled correctly. Does the new version still handle them?
• Examples that the previous version handled badly. Does the new version improve, or at least not regress?
• Examples designed to surface specific failure modes: hallucination, refusal, off topic drift, schema breakage.
• Examples that probe the system at its operational edges: long inputs, ambiguous inputs, hostile inputs.

The gate is automated. The gate runs on every prompt change, every model change, every retrieval index rebuild, every schema edit. The promotion mechanism refuses to advance if the gate fails. The team does not get to override the gate manually without an audit trail.

The size of the regression suite grows with the system. A young system may start with a few dozen examples; a mature system maintains hundreds, drawn from real production traces, real complaints, and real edge cases that have surfaced over time. The suite is not a static asset: it accretes. Every production incident contributes new examples. Every behavioural surprise becomes a held out test. Every user reported failure becomes a regression case. Over time, the suite becomes the most complete behavioural specification the system has, and the deploy gate that enforces it becomes the most important deployment time safety mechanism in the platform.

A change that does not pass eval does not reach production. That is the rule the discipline reduces to.

Rollback Semantics for Nondeterministic Code

Classical rollback is conceptually simple: redeploy the previous version, behaviour returns to where it was. The artefact is byte identical; the behaviour is byte identical.

LLM rollback is not that.

A model rollback may bring back a deprecated revision the vendor has already decommissioned. A prompt rollback restores the previous string, but if the model has changed independently underneath, the combined behaviour is new. A retrieval index rollback restores the previous index, but the documents in the world have moved on. A schema rollback restores the previous tool definitions, but the agent has already learned, in production, to expect the new ones.

The discipline is to treat rollback as a coordinated operation across all four moving parts. Pin the prompt. Pin the model revision. Pin the index. Pin the schema. Capture all four pins together as a release tag. A rollback restores all four pins atomically.

The harder corollary is that rollback paths atrophy unless they are exercised. A rollback that has never been run in production is a rollback the team has no confidence in. Schedule rollback drills the same way mature platform teams schedule disaster recovery drills: pick a release pin from a few weeks ago, restore all four components atomically in a nonproduction environment, and verify that the regression eval suite still passes. Catch the broken parts of the rollback path before they are needed at 3 AM.

The other corollary is that rollback windows are bounded by what the system can still serve. If the vendor has retired the model revision you rolled back to, rollback no longer works. Treat that horizon, the vendor's deprecation date, as the effective end of your rollback window for that release. Plan accordingly.

Failure Modes

These are the failures the discipline is designed to prevent. Severity ordered, worst first.

Silent vendor upgrade. The team uses a floating model alias. The vendor revises the model in place. Production behaviour shifts overnight with no signal in source control. Detection latency is bounded only by how aggressively the team is running online evals. Mitigation: pin to dated revisions; never reference floating aliases in production.

Prompt drift in a registry. A prompt registry sits outside source control. A team member edits the prompt through an admin UI. There is no diff, no review, no eval gate. The change reaches production immediately. Detection depends on whether someone notices the behaviour change in dashboards. Mitigation: every registry change goes through review; every registry change runs the eval gate; the registry maintains a full version history. A registry edit is not a separate workflow from a code change. It is a code change, just stored elsewhere.

Schema drift across services. The agent expects a tool to return one shape. A downstream service quietly changes the response shape. The agent now produces broken outputs on a specific class of input. The change does not register in any LLM system component; it is a downstream contract break. Mitigation: treat function calling schemas as inter service contracts; version them; add consumer driven contract tests; alert when production traces show schema mismatch errors.

Dependency surprise from prompt included context. A prompt template interpolates retrieved documents. The retrieval index is rebuilt on a cron. New documents enter the index. Some of them carry instructions or content that interact with the prompt in unintended ways. Behaviour shifts. Detection requires correlating the index build timestamp against the behaviour shift timestamp, a join the team is rarely set up to do. Mitigation: version the retrieval index; capture the index version in every trace; treat the index as one of the pinned components in a release.

Lossy rollback. The team rolls back the prompt but not the model, or the model but not the prompt. The resulting combined behaviour is something the team has never tested. The dashboards stabilise but the system is now in a never validated state. Mitigation: rollbacks are coordinated, all four pins together, captured as release tags.

The Architect's Checklist

The shipping bar for prompt and model versioning is structural. Walk this list before declaring a system production ready:

• Every prompt is source controlled, either inline in the repo or in a registry with a full version history visible to the team.
• Every model reference in production is a dated revision, not a floating alias.
• Every prompt change, model change, index rebuild, and schema change runs a regression eval suite before promotion.
• The promotion mechanism refuses to advance on a failed eval, with no informal override path.
• Releases are tagged across all four pinned components: prompt version, model revision, index version, and schema version, so a rollback is a single atomic operation.
• The team subscribes to vendor deprecation schedules and plans upgrades against them as known work, not as surprise work.
• Production traces capture all four pin identifiers, so any behaviour shift can be correlated against a versioned cause.
• Schema changes between services run as consumer driven contracts, alerted on at the trace layer.
• The rollback window is documented, bounded by the earliest vendor deprecation date in the current release pin.

If a system meets this checklist, "the diff is empty and the behaviour isn't" stops being a real failure mode. There is no behaviour defining surface that lives outside the versioning loop.

The Architect's Mental Model

Think of the LLM system as a service whose source code is distributed across four repositories: the prompt repo, the model repo, the index repo, and the schema repo. None of those repositories is one the team owns end to end. The prompt repo lives in the registry. The model repo lives at the vendor. The index repo lives at the pipeline that builds the index. The schema repo is fragmented across the services the agent calls.

The versioning discipline is the discipline of making those four distributed repositories behave as if they were one. Pinned. Tagged. Diffable. Rollback safe. Eval gated.

That is the work. It is not glamorous. It does not surface in product demos. It is, however, what stands between a system that ships reliably and a system that drifts.

A prompt is code. A model is a dependency. Treat them like the things they are, and the diff will stop lying to you.

🗓️ Last updated: July 2026

The team spent six weeks on a model upgrade. Every internal eval is green. The new model beats the old one on the golden set by four points. It beats it on the regression suite by two. The LLM as judge evals are a clean sweep. The deploy gate flips. The new model goes live to ten per cent of traffic.

Seventy two hours later, the customer experience team escalates. CSAT is down three points on the cohort. Refund eligibility questions are getting confidently wrong answers. The old model never made that class of mistake. The eval suite still says everything is fine.

Nothing in the suite was wrong. Every test it ran, it ran correctly. The problem was that the suite measured what was easy to measure: accuracy on a curated set, faithfulness on a synthetic RAG benchmark, a handful of hand picked failure cases. The production workload had drifted past it. The team had a measurement system. They did not have an evaluation system.

This article is about the difference.

What this article is NOT. It is not a survey of every eval framework on GitHub. It is not "ten metrics every LLM team should track." It is the architectural argument that LLM evaluation has to be a portfolio of techniques operating on four different timescales, namely offline, online, regression, and drift, and that shipping with only one of those four is the most common quality failure in the field today.

Requirements: What an Eval System Has to Answer

An eval system is not a metric. It is a body of evidence gathered on different timescales, measuring different properties, that supports a single architectural decision: should this change ship?

Five things any production grade eval system has to answer, ordered by how often you will need them:

1. Has anything regressed? When you upgrade a prompt, a model, a retriever, an embedding model, or an upstream tool, what behaviour, if any, got worse?
2. Is the system getting worse over time without anyone changing anything? Models get deprecated. Vendor backends drift. The world the model was trained on dates. Drift is invisible without an eval system that runs continuously, not just on PR.
3. Are real users having a worse experience than the eval suite predicted? Offline evals measure curated inputs. Production inputs are not curated. The gap between the two is where most shipping mistakes live.
4. Where, specifically, did it get worse? A two point drop on a fifty question golden set tells you very little unless you can cluster the failures by topic, intent, or user cohort.
5. Is the eval itself still measuring the right thing? Golden sets rot. Judges drift. Regression suites grow stale. An eval system without a maintenance discipline becomes a measurement system you can no longer trust.

Notice what is missing: there is no requirement for "a single eval score." A single score is for press releases, not architecture.

The Classical Baseline: Tests Don't Generalise

Classical software testing was built around three assumptions: the system is deterministic, the input space is enumerable, and the correctness of an output is decidable in code.

LLM systems break all three.

• Determinism: Even at temperature zero, two requests to the same model can produce different outputs. The inference stack, including batching, routing, and vendor side caching, is nondeterministic at the system level even when the math is deterministic at the model level.
• Enumerability: The input space is natural language. You cannot enumerate it. You can only sample.
• Decidability: "Is this answer correct?" is sometimes trivially yes or no (math, code execution, retrieval recall) but more often a judgement call. Was the tone right. Did it cite the right source. Did it refuse something it should have refused. Did it overcorrect on something it should have answered.

The classical playbook of unit, integration, and end to end tests does not generalise. It still has a role, especially for the deterministic plumbing around the model: the retriever, the schema validator, the rate limiter, the tool dispatcher. But it is not enough. What replaces it is a portfolio: a set of evaluation techniques, each strong where another is weak, each running at the timescale where it is most useful.

The Four Axes

Every credible LLM eval program operates on four axes. They are not interchangeable, and a program missing any one of them has a class of failure it cannot catch.

Offline evals run before deploy. Curated inputs, known good outputs (or known bad outputs), and a scoring function, whether code, heuristic, LLM as judge, or human review. Cheap to iterate, fast to run, and they answer one question: has the candidate change made anything worse on the inputs we know we care about?

Online evals run in production. They measure what real users experience: thumbs up or down, copy rate, retry rate, escalation rate, task completion rate, latency, refusal rate, downstream business metrics. Slow to converge, statistically noisy, but the only signal that catches the thing your offline suite did not think to test.

Regression suites are a specific kind of offline eval: a curated archive of historical failures, each one a frozen test that prevents a known bug from reappearing. They grow over time. They are the institutional memory of every bad bug the team has shipped and learned from.

Drift detection is the longest timescale axis: continuous or scheduled reruns of the offline suite against a frozen model, prompt, and configuration, to detect when the vendor or the world has changed underneath you. A passing eval on Monday and a failing eval on Friday, with the same prompt, the same code, and the same model name, means something changed at the vendor, or something changed in the eval data itself.

Ship without all four and you are missing a class of failure that someone else's program would have caught.

Offline Eval Done Right

The most common offline eval mistake is treating it as a single technique. It is at least three.

Golden sets are the foundation: a curated collection of inputs paired with expected behaviour. For tasks with deterministic correctness such as extraction, classification, and code generation that can be executed, the scoring is straightforward. For open ended tasks like summarisation, support replies, and recommendations, the expected behaviour is more often a rubric than an answer.

The discipline around golden sets matters more than their size. A 200 item set that is curated, versioned, and reviewed quarterly is more valuable than a 5,000 item set that nobody owns. Two specific practices separate teams that get value from golden sets from teams that do not:

• Adversarial curation. Every shipped bug becomes a golden set entry within a week of the incident. The set grows because the system grew, not because someone bought more annotations.
• Diversity over volume. A golden set that does not include the long tail of user intents will reward models that handle the head and penalise nothing for failing the tail. The tail is where most production complaints originate.

LLM as judge evaluators are the workhorse for open ended outputs. The Zheng et al. 2023 paper that formalised the technique, titled "Judging LLM as a Judge with MT Bench and Chatbot Arena," is required reading. It also names the failure modes, including position bias, verbosity bias, and self enhancement bias, that every team eventually rediscovers the hard way. Three practices that turn an LLM judge from a toy into a working tool:

• Use a stronger model than the one being evaluated. Judging is harder than generating.
• Calibrate against a human judged sample at least quarterly. If the judge and the humans diverge, the judge has drifted.
• Treat the judge prompt as code: versioned, reviewed, eval gated. A new judge prompt is a new eval system; do not treat it as a config tweak.

Component evals are the technique most teams skip. If your system is a chain, meaning retrieve then rerank then synthesise then cite, evaluating the end to end output tells you the system is broken without telling you which component broke it. RAGAS style component evals decompose the end to end metric into per component contributions: faithfulness, context relevance, answer relevance, context recall. The cost is more eval scaffolding. The benefit is that when something regresses, you know where to look.

Online Eval

Offline evals tell you whether the candidate is better on the inputs you thought of. Online evals tell you whether it is better on the inputs your users actually send.

The most useful online signals are not the ones teams reach for first.

• Implicit signals beat explicit signals. The user's next action is more honest than the thumbs up button. Did they rephrase the question. Did they copy the output. Did they retry. Did they escalate to a human. Did they come back tomorrow. These signals are noisier than a survey, but they are not biased toward users who fill in surveys.
• A/B is necessary but slow. Statistical power on a small uplift in a noisy metric can take weeks of traffic. That window is longer than most teams want their model upgrades to wait. The honest answer is to run shorter A/Bs on stronger signals (a clear regression shows up fast) and longer A/Bs on subtler signals (CSAT, retention).
• CSAT lies in known ways. Customers who got the wrong answer confidently rate the experience higher than customers who got the right answer hesitantly. Confidence is a confound. Online evals that do not correct for it overstate the quality of the more confident model, which is often, but not always, the better one.

Online evals require an instrumentation discipline that most teams underbuild. The full prompt, the model and version, the retrieval context, the latency, the cost, and the user's next action all have to be captured, joined, and queryable. If your observability stack cannot answer "which prompts in the last hour produced an output the user immediately retried," your online eval is theatre.

Article 3 in this series covers the instrumentation side in detail.

Regression and Drift Detection

Regression is the easy half. Every previously shipped bug becomes a frozen golden set entry. Every PR that touches the model, the prompt, the retriever, or the schema runs against the regression suite. A failure blocks the merge. This is unglamorous and it works.

Drift detection is the harder half, and the one almost every team gets wrong.

The shape of the problem: nothing on your side changed. Same model name, same prompt, same retrieval. Three weeks later, the eval scores drop. Vendors update their inference stack, change their default safety filters, retire a model version, swap a tokeniser, change a backend GPU SKU. None of those changes appear in your change log. All of them can shift behaviour.

The defence:

• Pin model versions explicitly. A generic floating alias is not a model version. A dated revision string is. The alias tags vendors provide for convenience defeat the entire purpose of versioning.
• Run the offline suite on a schedule. Daily, or at least weekly, against a frozen configuration. Plot the time series. Drift is a trend, not an event.
• Alert on the trend, not the absolute score. Eval scores fluctuate; the eval of the eval is noisier than people expect. A two standard deviation drop sustained over three runs is a signal. A single run dip is not.
• Freeze deploys when drift is detected. Until the source is isolated, every deploy compounds the diagnostic problem. The vendor change log is the first place to look.

Drift detection is also the only eval axis where the vendor can be the bug. Article 5 in this series covers what to do then.

Failure Modes: The Section You Actually Need

Eval programs fail in ways that look fine on dashboards. Seven recurring failure modes, severity ordered:

1. The eval suite becomes the product. Goodhart's Law in action: when a measure becomes a target, it ceases to be a good measure. The model is tuned, the prompt is tuned, the golden set is curated, all to push the eval score up. The score goes up. User satisfaction does not. By the time someone notices, the eval has been the target for months and the team no longer trusts it.
2. Judge collapse. The LLM as judge starts agreeing with itself on outputs it should not agree with. Long winded answers beat concise correct ones. Formal tone beats accurate informal tone. Verbose justifications beat short truthful ones. Without a recurring human calibration check, the judge drifts and the eval drifts with it.
3. The frozen golden set. The set was curated when the product launched. The product is now used differently. The set is still measuring the launch era use case. Nothing in the eval notices that the set itself is stale.
4. The single number trap. The team ships when "the eval" passes. There is no per cluster, per intent, or per cohort breakdown. A regression on a small but high value user segment is averaged away by the rest of the set.
5. The online eval blind spot. Offline evals are green. Online metrics are flat or worse. Nobody investigates why offline and online disagree. The answer is almost always that the offline distribution does not match the production distribution, and nobody is generating new golden cases from production traffic.
6. Latency and cost amnesia. The eval suite measures quality. It does not measure latency or per token cost. The new model is two points better on quality and three times more expensive, and nothing in the eval pipeline forces that tradeoff to the surface before the deploy gate flips.
7. The eval that never goes red. Some team somewhere is running an eval suite right now that has not had a meaningful regression in twelve months. It is not because the system is excellent. It is because the suite has been tuned, over time, to the system. An eval that does not catch regressions does not catch anything.

Cost Economics of Evals

The eval suite has a cost iceberg.

Above the waterline:

• LLM as judge calls, often using a stronger and more expensive model than production
• Compute for retrieval, reranking, and generation on every golden set item, every run
• Storage and infrastructure for the eval platform and the historical eval data

Below the waterline, and bigger:

• Engineer time to curate golden sets, maintain judge prompts, investigate regressions, and rebaseline after model upgrades
• The deploy delay an honest eval pipeline introduces, from a few minutes for cheap partial coverage to several hours for full coverage with multi judge consensus and statistical significance
• The opportunity cost of model upgrades that get blocked while the eval suite is being updated to evaluate them

A useful rule of thumb: if the eval program's running cost is trivially small relative to inference spend, you are probably not running enough evals. If it is consuming a substantial fraction of the inference budget, you are probably measuring inputs that do not matter to production.

The cost of an eval program is real. The cost of not having one is a CSAT regression you cannot explain to the customer experience team, and an architecture conversation you cannot win in the next planning meeting.

The Architect's Checklist

Twelve items. If you cannot say yes to most of them, your eval program has gaps.

1. Every shipped bug has been added to the regression suite within a week of the incident.
2. Golden sets are versioned, reviewed quarterly, and have a named owner.
3. LLM as judge prompts are versioned and calibrated against human review at a defined cadence.
4. The eval suite reports per cluster or per intent breakdowns, not just a single score.
5. Offline evals include component level metrics, not only end to end.
6. Online evals capture implicit signals such as retry, copy, and escalation, not only thumbs up or down.
7. A/B tests have pre registered metrics and minimum detectable effect sizing.
8. Model versions are pinned explicitly, with no use of vendor floating alias tags in production.
9. The offline suite runs on a schedule against a frozen config, and drift is plotted over time.
10. Eval cost is tracked as a line item in the inference budget.
11. The eval pipeline is part of the deploy gate; no model or prompt change ships without passing.
12. The team treats the eval suite as code: reviewed, versioned, and maintained as carefully as the system it measures.

The Architect's Mental Model

Classical software is evaluated against a specification. LLM systems are evaluated against a distribution: the distribution of inputs the users will send, and the distribution of outputs that count as good enough. Neither distribution is enumerable. Both shift over time. Your eval program is the apparatus that keeps you in contact with the parts of both distributions you care about.

The teams that ship probabilistic systems with confidence are the teams that built an eval program before they built the second feature. The teams that struggle are the teams that built features for a year, then tried to retrofit evaluation onto a system that already had a quality problem. At that point, the eval system is asked to diagnose rather than prevent. Those are not the same job.

Offline evals tell you yesterday's truth. Online evals tell you today's. Ship without both, and you are shipping blind.

🗓️ Last updated: July 2026

Picture a workflow that began an hour ago with a customer email. Step 1 classified intent. Step 2 retrieved order history. Step 3, an LLM agent, read both, decided the customer wanted a refund, and called the refund API. The refund succeeded. Step 4 was supposed to send a confirmation email. It crashed. The worker process restarted. Now the engine has a decision to make.

What is the state of this workflow? Has the refund happened? Yes, there is an event in the log that says so. Has the customer been notified? No, that step never completed. Should the engine retry step 4? Yes, after a delay. Should the engine rerun step 3? No, the agent already made the decision, the side effect is committed, and rerunning risks a double refund. Should the engine retry the agent's reasoning if a newer model has been deployed since? That depends on whether the operator asked for strict replay or fresh inference, and the engine has to know the difference.

Every one of those questions has a classical workflow engine answer that has been worked out over twenty years of distributed systems literature. And every one of them gets harder the moment an LLM agent is one of the steps. This article is about the engine that answers them honestly.

A note on framing: the previous article in this series, Designing an LLM Inference Platform, designed the system that serves a single LLM call cheaply and reliably at scale. This article is the layer above it. The inference platform owns GPUs and serves tokens; the workflow engine owns a state machine, an event log, a retry policy, and a schema enforced contract for every step. A workflow engine calls into an inference platform on every agentic step. The two systems are often built and operated by different teams, sometimes by different companies. The unit of work for the inference platform is the call. The unit of work for the workflow engine is the graph. Confuse the two and you build the wrong system; the failure modes of one masquerade as the failure modes of the other.

This is also the closing article of System Design, Reimagined, the place where this entire series joins back up with Architecting Agents. Six classical system design problems redrawn for the probabilistic era ends here, with the realisation that what you have actually been building all along is one thing: a discipline for putting probabilistic computation inside deterministic scaffolding.

The thesis: an agentic workflow engine is a graph, not a caller. Its job is to put probabilistic steps inside deterministic scaffolding: contracts, retries, budgets, replay semantics, and observability, so that the agents inside it can be operated, audited, bounded, and trusted. The hard part is not the agent. The hard part is the graph that contains the agent.

Requirements

Five nonfunctional requirements drive every later decision.

1. Durability across process restarts. A workflow may run for seconds, hours, or weeks. If a worker process crashes mid step, the workflow must resume from the last committed checkpoint without losing work or reexecuting committed side effects. This is the core property of every serious workflow engine. Agentic steps do not get an exemption.

2. Replay determinism, to the extent possible. Operators must be able to take a failed or suspicious run and reexecute it from a known state to understand what happened. For deterministic steps, replay is exact. For LLM steps, replay is approximate at best, and the architecture has to be explicit about which is which.

3. End to end observability of every step. Every step's inputs, outputs, tool calls, model used, prompt, token spend, and elapsed time must be queryable as structured data, not log greps. This requirement is borrowed wholesale from Observability for Agentic Systems in the Architecting Agents series; it is the only way to operate something this stateful.

4. Bounded cost per workflow. A workflow that can spend an unbounded number of tokens or invoke an unbounded number of tool calls is not a workflow; it is a budget incident waiting to be filed. The engine must enforce per workflow ceilings on tokens, tool calls, wall clock time, and dollars.

5. Safe interaction with external systems. When a workflow step writes to a database, calls a payment API, or sends an email, the engine has to guarantee that retries do not double execute and that compensation actions undo correctly if a later step fails. This is classical workflow semantics, but now the decision to call the tool may come from a probabilistic step.

The PM trap requirements, "the agent should figure out the whole workflow from a goal" and "the agent should be able to call any tool it wants," sound powerful and are operational disasters. Both have to be bounded at the engine level even when the contract language does not say so.

The Classical Baseline

The classical workflow engine playbook is well developed. A handful of patterns dominate the field:

• Airflow (2014) and Argo Workflows model workflows as static DAGs defined ahead of time. Excellent for batch and ETL; rigid for dynamic flow.
• AWS Step Functions models workflows as state machines defined declaratively. Good cloud native integration; bounded expressiveness.
• Temporal (and its Uber predecessor Cadence) takes a different approach: workflows are normal code, executed by a deterministic replay engine that rederives state from an event log. This buys arbitrary control flow with strong durability guarantees, at the cost of strict determinism rules on workflow code.
• Prefect and Dagster sit closer to data engineering use cases with strong observability primitives.

What all of these systems share is a small set of assumptions: a step is a function whose behaviour depends only on its inputs; the workflow graph is either static or grown by deterministic logic; and replay is exact because the event log captures every nondeterministic decision.

LLM steps break the first and third of those assumptions. They sometimes break the second one too: when the agent's output decides the next branch, the graph is being co authored by the workflow code and the model. The interesting design work is preserving as much of the classical guarantee set as possible while making honest concessions where the probabilistic steps genuinely change the contract.

Four Cascading Decisions

Decision 1: The step boundary. Define what counts as a single step. The choice has dramatic consequences for durability, retries, and observability. Three common settings:

• Step equals one model call. Each LLM invocation is its own durable step; tool calls are also their own steps. Fine grained, easy to replay, but expensive in checkpoint overhead.
• Step equals one model plus tool turn. The model proposes a tool call, the engine executes it, the result is fed back to the model; the whole exchange is one step. Less checkpoint overhead, but a partial failure inside the step is harder to reason about.
• Step equals a whole agent reasoning block. The agent reasons over multiple internal turns before emitting a result; only the block boundary is durable. Useful for short interactive agents; dangerous for long autonomous ones because a crash mid block loses unbounded work.

There is no universal right answer, but there is a universal wrong answer: leaving this implicit. If your engine does not have a documented step boundary, every team building on it picks a different one and the resulting failure modes are impossible to reason about across the system.

Decision 2: State persistence model. Decide how workflow state is captured and restored. Two patterns dominate.

The event sourced model (Temporal's approach) replays a log of past events to reconstruct workflow state. Pure functions of the event log are deterministic by construction; nondeterministic operations have to be wrapped in primitives that record their outcome to the log.

The snapshot model (Step Functions and most home grown engines) checkpoints the entire workflow state at every step boundary. Simpler to reason about, but the state has to be serialisable and the snapshot has to be cheap to take and load.

Event sourcing is more powerful, especially when paired with a true replay debugger. Snapshots are more pragmatic and play better with LLM steps that emit large free form outputs. The right pick is the one your team can operate; both models are battle tested in production at scale.

Decision 3: Branching model, deterministic graph versus runtime decided graph. Classical workflows have a graph determined at definition time. Agentic workflows often have steps where the model decides what happens next: call this tool, give up and escalate, ask the user, route to a different agent.

A clean architecture makes the agent's branching explicit at the engine level: the agent step returns a structured decision (an enum or a typed branch token), the engine matches it against the workflow definition, and the workflow engine, not the agent, moves the cursor. The antipattern is letting the agent emit free form next step instructions that the engine then has to parse. Anywhere the agent's natural language output becomes the workflow's control flow, the system is fragile.

Decision 4: Workflow level bounds. Decide how cost, time, and risk ceilings are enforced. Per step bounds are necessary but not sufficient; they do not catch loops where the workflow keeps reaching new steps within each step's budget. Workflow level bounds (total tokens, total tool calls, total wall clock, total dollars) must be tracked centrally and enforced by the engine, not by step code.

These four decisions cascade. The step boundary determines what state has to be persisted. The state model constrains what kinds of branching can be replayed. The branching model determines how budgets are enforced (per branch versus globally). Resolve them in order, or watch each one get relitigated three times as the implementation reveals their interactions.

Step Contracts and Schemas

A workflow step has two contracts. The input contract describes what state and arguments the step expects. The output contract describes what it produces and which branches downstream code can take based on that output. Classical workflow engines enforce both with strong types, schemas, and registry checks at deploy time.

Agentic steps strain this. An LLM emits free form text by default. To make it a workflow step, the output has to be coerced into a structured shape. The standard tools are structured output (JSON schema constrained decoding, available in most major model APIs), tool call schemas (OpenAI function calling, Anthropic tool use, and MCP defined tools), and post hoc validators that reject and retry malformed outputs.

The discipline this article inherits from earlier in the series, and from Architecting Safe Tool Calling Agents in particular, is that the schema is the contract. If the agent produces output that does not validate, the engine should not pass it downstream and hope. It should retry the step, fall back to a human in the loop branch, or fail the workflow with a clear schema violation error. Trying to repair invalid output downstream is how silent corruption enters the system.

A practical pattern: every agentic step has a wrapping decorator that handles structured output prompting, JSON schema validation, bounded retries on validation failure, and an explicit fall through branch in the workflow graph. The agent does the reasoning; the wrapper enforces the contract. The workflow engine treats the wrapped step exactly like a deterministic step.

Retry, Idempotency, and Compensation

Three classical workflow primitives apply unchanged in the agentic case. Their importance, if anything, goes up.

Retries need to distinguish between three failure classes: transient (network timeout, rate limit), deterministic (schema violation, validation failure), and content (the model produced a plausible answer that turns out to be wrong). Classical engines collapse this into "retryable or not." Agentic engines have to handle a fourth class, semantically incorrect, which usually is not retryable in the same way and instead needs escalation or human review.

Idempotency is the property that retrying a step does not produce duplicate side effects. The standard technique is an idempotency key passed to the downstream service: the same key submitted twice produces one effect. For agentic steps, the idempotency key has to be derived deterministically from the workflow context, not from the model's output, which may vary across retries.

Compensation is the Saga pattern (Garcia Molina and Salem, 1987): if step N+1 fails after step N has committed a side effect, step N's compensating action is invoked to undo it. The discipline here does not change for agentic workflows, but the surface area does, because agentic workflows tend to have more steps and more side effecting tools than classical ones, and every irreversible side effect needs a compensating action defined alongside it.

The forbidden combination is the one that appears most often in early agentic systems: an LLM step decides to call a tool, the tool succeeds, the next step fails, and there is no compensation defined because the tool was "just something the agent decided to do." If the agent can decide to call the tool, the tool needs a compensation defined at the workflow level. The agent's freedom does not exempt the engine from the saga discipline.

Observability and Replay

Workflow engines have the strongest observability story in distributed systems: every step's input, output, retry count, and elapsed time is captured by construction. Agentic workflows need that plus three more things, and the architecture has to make all three first class.

Prompt and model capture. Every LLM step records the exact prompt, model identifier, temperature, max tokens setting, and any tool definitions in scope. Without this, debugging "why did the agent decide that?" is impossible. With it, post mortems become precise.

Cost capture. Every LLM call records its input tokens, output tokens, cached tokens (if prefix caching applies), and a derived cost in dollars. This rolls up to a per step and per workflow total. Without it, the cost bounded agent discipline from earlier in the series has nothing to enforce against.

Replay semantics, honestly stated. Replay of an agentic workflow has three modes, and the engine should support all three:

• Strict deterministic replay. The engine replays the recorded event log, including the recorded LLM outputs. The workflow takes the same path it took originally. This is what most operators want most of the time.
• Fresh inference replay. The engine reruns every LLM call against the live model. The workflow may take a different path. This is for evaluating whether a fix to the prompt or the model would change the outcome.
• Counterfactual replay. The engine replays the recorded events up to a chosen step, then runs fresh inference from there. This is the most useful debugging mode and the hardest to implement cleanly.

The architectural commitment is naming these three modes and exposing them as first class operations on every workflow. This is Observability for Agentic Systems applied to workflows: structured capture, structured queries, and structured replay. Trace IDs are not enough.

Cost Bounds and Coordination Patterns

Two more concerns pull in callbacks from earlier in the series.

Cost bounds at the workflow level. Cost Bounded Agent Design in Architecting Agents argued for explicit token, tool call, time, and dollar ceilings on every agent. In a workflow engine, those ceilings sit at the workflow level, not the step level. A workflow's budget descends to its steps; a step that exceeds its share returns a budget exceeded signal to the engine; the engine decides whether to retry, escalate, or fail the workflow.

The nonobvious design point is when to charge. Token costs are known after the LLM call returns, but a step that loops internally can run up costs before the engine sees a checkpoint. The pattern that works: every LLM client wrapper increments a workflow level counter on every call and checks against the budget before each call. The check is cheap. The alternative, discovering an overrun after the workflow has completed, does not constitute cost control.

Coordination patterns. Coordinating Multi Agent Systems (the closing article of Architecting Agents) walked the three coordination patterns: hierarchical (planner and worker), peer to peer (debate, voting), and pipeline (sequential specialists). Every one of these is a graph pattern. A workflow engine is the natural home for them.

The architectural consequence is that "multi agent system" is not a separate kind of system from "agentic workflow." They are the same system, viewed at different scales. A multi agent product is a workflow where multiple steps happen to be LLM agents and the workflow defines how they pass work and information between each other. Treating the two as separate frameworks, a multi agent library on top of an unrelated workflow library, is a common architectural mistake that produces two competing state stores, two competing observability pipelines, and two competing budget models. Pick the workflow engine. Run the agents inside it.

Failure Modes

Seven failure modes recur. Ordered by severity:

1. Loop blow up. An agent step decides to call a tool, the tool result feeds back to the agent, the agent decides to call another tool, and the cycle continues without converging. Per step budgets do not catch it because each iteration stays within its share. Fix: hard caps on agent step iteration count, workflow level budget enforcement, and a circuit breaker pattern that bails out after N steps without progress.

2. Schema drift between steps. Step N's output schema changes between deploys; step N+1 still expects the old shape; the workflow fails midway in production. Fix: schema versioning, downstream compatible changes, and contract tests in CI that exercise every step boundary with a snapshot of the prior step's output shape.

3. Orphaned compensations. A side effecting step committed; a later step failed; the compensation did not run because it was not defined or it errored. Fix: every irreversible tool call has a compensation defined and tested; compensations themselves are retried; the engine tracks "compensation owed" state explicitly and surfaces it as a metric.

4. Replay nondeterminism. An operator replays a failed workflow expecting to reproduce the issue; the agent makes a different decision on replay; the bug does not reproduce. Fix: the engine names the three replay modes explicitly, defaults strict replay to using recorded LLM outputs, and offers fresh inference replay only as an explicit operation.

5. Coordination deadlock. Two agent steps wait on each other to produce output; neither does; the workflow stalls. Fix: every agent step has a wall clock timeout, the workflow engine enforces it independently of any per call timeout, and the deadlock surfaces as a timeout exception with diagnostic context.

6. Lost context at step boundaries. Useful state in the agent's reasoning is discarded at the step checkpoint; the next step has to rederive it; the model picks something different the second time. Fix: explicit working memory state passed between steps (a callback to Stateful Memory in Agentic Systems), not relying on the agent to remember across boundaries.

7. Double execution under retry. A tool call succeeded but its acknowledgement was lost; the engine retries the step; the tool runs twice. Fix: idempotency keys derived from workflow context, enforced by the downstream tool or by an idempotency layer the engine owns.

The fashionable failure mode, "the agent went rogue," is downstream of all seven of these. An agent that exceeds its budget, drifts off schema, double executes a tool, or deadlocks looks rogue from the outside. The architectural truth is that it was operating in a system that did not bound it. The engine is the bound.

Cost Economics

Workflow engine cost has three layers that need separate accounting.

The engine layer, workers, schedulers, event log storage, dashboards, is classical infrastructure. Costs are proportional to workflow volume and history retention, well understood from twenty years of workflow engine deployment experience.

The LLM layer dominates everything else. Token spend at every agentic step rolls up to a per workflow cost. For workflows with many agentic steps or long contexts, this number can dwarf the engine layer's cost by a wide margin. The unit metric that matters is dollars per completed workflow, not dollars per step. A workflow that completes for ten cents looks healthy; a workflow that fails halfway after spending ten cents is pure burn.

The tool layer covers external API calls invoked from agentic steps: payment gateways, search APIs, code execution sandboxes, third party data services. Each has its own cost model and its own rate limit story. These costs are easy to underbudget because they sit at the leaves of the graph; the engine has to surface them as a first class line in the per workflow cost breakdown.

The single biggest cost lever is workflow completion rate. A workflow that completes 95 percent of the time at one dollar costs the same per success as a workflow that completes 50 percent of the time at fifty cents, and the second one has half the customer experience. Engineering for completion (good schemas, sound retries, calibrated escalation) is the same as engineering for cost.

Architect's Checklist

A reusable list for designing or reviewing an agentic workflow engine:

1. Define the step boundary explicitly; document it; do not let teams pick per step.
2. Pick the state persistence model, event sourced or snapshot, and stay with it across the codebase.
3. Express agentic branching as structured decisions matched against the workflow graph, never as free form text the engine parses.
4. Enforce workflow level bounds on tokens, tool calls, wall clock time, and dollars; per step bounds are necessary but insufficient.
5. Treat the schema as the contract; reject malformed agentic output at the step boundary, do not repair it downstream.
6. Distinguish four failure classes: transient, deterministic, semantically wrong, and content wrong; handle each one explicitly.
7. Derive idempotency keys from workflow context, not from model output.
8. Define a compensating action for every irreversible side effect; test compensations regularly.
9. Capture every LLM call's prompt, model, settings, tokens, and cost as structured data; first class, not log lines.
10. Support three replay modes, strict, fresh inference, and counterfactual, and name them in the operator interface.
11. Pass working memory between steps explicitly; do not rely on agents to remember across boundaries.
12. Measure dollars per completed workflow; engineer for completion as the primary cost lever.

Architect's Mental Model

A workflow engine is a graph that durably executes steps. An agentic workflow engine is the same thing, with a class of steps whose behaviour is partly chosen at runtime by an LLM. Almost everything that makes the classical version work continues to work. The interesting design is in the seams.

The seams are the same five concerns the Architecting Agents series spent six articles on. Failure modes have to be designed for; they are not optional. State must persist between steps; it is the workflow's memory. Tool calls must be safe under retry; saga discipline applies whether a human or an agent decided to invoke the tool. Observability must be structured down to the prompt and the token; otherwise the system cannot be operated. Costs must be bounded at the workflow level; per step ceilings catch the wrong failure mode. Multi agent coordination is a workflow shape; it is not a separate kind of system.

That symmetry is the point of this series of series. Architecting Agents covered the discipline of building a sound agent. System Design, Reimagined covered the discipline of building sound systems around them. The two pieces compose into one body of work, and the workflow engine is where they touch most cleanly. Everything the first series demanded of an agent, the second series demands of the system that contains agents. The graph is what makes that demand enforceable.

The agent reasons. The workflow remembers, retries, bounds, and proves. Without the second, there is no system.

🗓️ Last updated: July 2026

The interviewer drops a familiar sounding line: "design a multi tenant API platform that serves machine learning models at scale."

Your reflexes fire. Load balancer, autoscaling pool, request queue, observability, rate limits. Classical web service playbook.

Then they qualify it: "large language models, hundreds of tenants, mixed workloads, latency sensitive interactive traffic alongside batch jobs, GPUs are the bottleneck."

The playbook breaks. Almost every assumption that makes a stateless HTTP service easy to scale is inverted here. Requests are large. Latency is variable in a way that depends on the request's output length, not just its input. The bottleneck is GPU memory and bandwidth, not CPU or network. Batching is no longer an optimisation. It is the architecture. And the unit cost is not measured in milliseconds of compute but in dollars per million tokens.

This article walks the system design version of "design an LLM inference platform" the way I would want it walked in a senior interview, or in a real architecture review for a team about to commit a quarter of capital expenditure to GPU clusters.

The thesis: multi tenant LLM serving inverts the web infrastructure playbook. The request is large, the latency is variable by output, the resource is scarce and expensive, and the only way to make the unit economics work is to treat batching as a first class architectural concept rather than a clever optimisation.

Requirements

Five nonfunctional requirements drive every later decision.

1. Mixed workloads, mixed SLOs. The platform serves at least three traffic classes: interactive chat (humans waiting on a stream), batch jobs (offline summarisation, embedding regeneration, evaluation runs), and code completions or other near real time streams. Each class has a different latency SLO and a different throughput appetite. A single fleet with one SLO will either waste GPUs on batch or starve interactive traffic.

2. Latency expressed as a distribution, not an average. P50 hides everything that matters. The numbers that move customer experience are time to first token (TTFT) and inter token latency (ITL) at P95 and P99. Tail behaviour is the product. Dean and Barroso's The Tail at Scale (CACM 2013) framed this for classical distributed systems; LLM serving has the same problem with sharper teeth, because long generations and shared GPUs amplify tail effects.

3. Throughput in tokens per second per GPU. Cost lives here. A serving stack that achieves substantially higher throughput per GPU on a given workload costs proportionally less for the same total output. Every architectural choice, batching strategy, attention implementation, quantisation, caching, is ultimately measured against this number.

4. Isolation between tenants. Noisy neighbours in LLM serving are catastrophic. One tenant submitting a request with a 64K token context and a 4K token generation can stall a shared replica for tens of seconds. Isolation is not a courtesy; it is an SLO commitment.

5. Operability under continuous model churn. New model versions ship constantly. The platform has to handle rolling upgrades, A/B routing, canarying, and rollback without breaking active streams. This sounds routine until you remember that loading a 70B parameter model into GPU memory takes minutes, not seconds.

The product manager wants two more things that look like requirements but are actually traps: "infinite context" and "no rate limits for paying customers." Neither is free. The architecture has to bound both at the platform level even when the contract language does not.

The Classical Baseline

The classical analogue is a multi tenant ML inference service from the pre LLM era: image classification, recommendation scoring, fraud detection. The shape was:

• A model server (TensorFlow Serving, TorchServe, Triton) loads a model into GPU memory once and serves it for many requests.
• Requests are small and uniform, a feature vector or an image tensor. Latency is dominated by network and a single forward pass measured in tens of milliseconds.
• Batching is straightforward: collect requests for a few milliseconds, run them as a single tensor, return results. Padding overhead is negligible because requests are similar sizes.
• Autoscaling tracks GPU utilisation or queue depth. Cold start is annoying but bounded.
• Cost is per inference, easy to model, easy to bill.

LLM serving breaks four of those five points simultaneously. Requests are huge and variable. Latency is no longer a single forward pass; it is hundreds or thousands of forward passes for the decode loop. Batching across requests with different generation lengths wastes most of the compute unless you batch dynamically. And cold start is now measured in minutes because a single model weight file can be hundreds of gigabytes.

That last point, variable generation length, is the wedge that drives almost every interesting architectural decision in this system.

Four Cascading Decisions

Decision 1: Workload tiering and SLO classes. Pick how many traffic classes the platform exposes and what their SLOs are. A reasonable starting point is three: interactive (TTFT under a few hundred milliseconds, ITL bounded for streaming), near real time (completions, embeddings, classification, sub second end to end), and batch (no per request SLO; throughput optimised). The number is less important than the act of committing to tiers. Without tiers, every customer's traffic competes against every other customer's traffic, and the tail is whatever the worst case in the system happens to be at that moment.

Decision 2: Hardware tier and tenancy model. Pick GPU SKUs, reserved versus on demand mix, and whether replicas are single tenant or shared. The practical choices today are H100, H200, A100, and L4 on the NVIDIA side, MI300X on the AMD side, and cloud accelerators including TPU, Trainium, and Inferentia. The decision interacts with the workload tier: batch jobs run fine on older, cheaper GPUs with high memory; interactive workloads benefit from the latest generation chips with faster interconnect. A single tenancy model rarely wins; most mature platforms run dedicated capacity for enterprise tiers alongside pooled capacity for everyone else.

Decision 3: Multi model routing strategy. Pick how many models the platform serves and how requests are routed across them. Three patterns dominate:

• A single model fleet runs one model across many replicas. Simple, but you pay frontier model prices for every request including the ones that did not need it.
• Tiered routing has a small, fast model handle easy requests while harder requests escalate to a large model. Saves cost dramatically when traffic is bimodal, but adds a routing decision that can itself be wrong.
• Multi LoRA serving keeps one base model in GPU memory while many fine tuned adapters are swapped in per request. Research including S LoRA (Sheng et al., 2023) and Punica demonstrated this is viable at scale. Powerful when you serve many tenants who each want their own fine tune, but operationally complex.

The right answer depends on the workload. A consumer chat product is often a single model fleet. A platform serving hundreds of enterprise tenants with their own fine tunes leans toward multi LoRA. A general purpose API often runs tiered routing under the hood.

Decision 4: Isolation and quota enforcement. Pick what guarantees the platform makes per tenant and how they are enforced. Tokens per second quotas, requests per second quotas, concurrent request caps, and maximum context length caps all live here. The naive "rate limit at the gateway" pattern is insufficient: a single in flight request with a 100K token context can monopolise a replica regardless of how many requests per second the gateway lets through. Quotas have to be expressed in the resource the system is actually bottlenecked on: tokens, KV cache pages, GPU seconds.

These four decisions cascade. The workload tiering decision constrains the hardware tier (interactive needs the fast chips). The hardware tier constrains the routing strategy (multi LoRA needs enough GPU memory for the base model plus adapters). The routing strategy constrains the isolation model (tiered routing means a tenant's traffic crosses model boundaries, complicating per tenant accounting). Choose them in order; anything else gets relitigated three times.

Continuous Batching: the Architecture, Not the Optimisation

Static batching, the classical approach where you collect N requests, run them as a single forward pass, return results, then collect the next batch, is unusable for LLM serving. The reason is asymmetric generation length. If one request in a batch of 32 generates 500 tokens and the others generate 50, the GPU is busy holding most of the batch hostage for the duration of the long generation. Throughput collapses.

Continuous batching, introduced by Orca (Yu et al., OSDI 2022) and now standard in serving stacks including vLLM, TGI, TensorRT LLM, and DeepSpeed MII, fixes this by batching at the iteration level instead of the request level. Every decoding step is its own batch. A request that finishes drops out; a new request that arrives joins in. The GPU is never waiting for the longest request in the batch.

The architectural consequences are sharper than they look:

• The "batch" is no longer a unit anyone outside the inference engine sees. Requests do not queue waiting for a batch; they slot into an active iteration.
• The KV cache becomes the dominant memory pressure. Every active request holds tens to thousands of tokens of cached attention state. PagedAttention (Kwon et al., SOSP 2023) was the breakthrough here, treating KV cache like virtual memory pages and allowing tighter packing and lower fragmentation.
• Admission control happens at the page level, not the request level. The scheduler decides whether to admit a new request based on whether there is enough KV cache room to hold its prefix and its expected generation, not on whether the request queue is full.

If a candidate or a colleague says "we'll batch requests every 10 milliseconds" in 2026, that is an architectural smell. Continuous batching has been the standard for several years and the serving frameworks that implement it are open source. The interesting questions are how the scheduler decides which requests to admit, how prefill is interleaved with decode, and how prefix caches are shared across requests, not whether to batch at all.

Prefill and Decode Are Different Workloads

A request to an LLM has two phases. Prefill processes the input prompt in parallel, a compute bound operation dominated by matrix multiplies that scales roughly with the square of context length for attention. Decode generates output tokens one at a time, autoregressively, a memory bandwidth bound operation dominated by KV cache reads that scales linearly with output length.

These two phases have wildly different performance characteristics. Prefill on a long context can take seconds on a single GPU. Decode is fast per token but throws away most of the GPU's compute capacity because the operation cannot saturate matrix multiplies the way prefill does.

Serving stacks have tried two approaches:

• Interleaved scheduling has prefill and decode share the same GPU, with the scheduler picking which to advance each step. The cost: a long prefill can stall decode steps, hurting inter token latency for active streams.
• Disaggregated serving runs prefill and decode on separate GPU pools, with the KV cache shipped between them. Research systems including DistServe (Zhong et al., 2024) and Splitwise (Patel et al., 2024) showed this can substantially improve tail latency for interactive workloads at the cost of extra system complexity.

Which one to pick is workload dependent. Heavy long context workloads benefit from disaggregation. Workloads dominated by short prompts and long generations often run fine on interleaved scheduling. Whichever you pick, the nondecision is the dangerous one: assuming all GPU time is fungible and ignoring the prefill and decode split is how you end up with a platform that passes throughput benchmarks and fails latency SLOs.

Prefix Caching, Speculative Decoding, and the Latency Budget

Two more techniques deserve first class billing because they materially change the cost versus latency tradeoff.

Prefix caching keeps the KV cache state for repeated prompt prefixes across requests, so the prefill work does not have to be redone. System prompts, chat history, RAG injected context, anything that appears at the start of many requests. OpenAI shipped prefix caching publicly in October 2024; Anthropic in August 2024; Google followed. For workloads with high prefix overlap (chat applications, RAG over a stable knowledge base), the cache hit rate often dominates platform economics, and the gap between a platform that exposes prefix caching and one that does not can be substantial.

Speculative decoding (Leviathan et al., ICML 2023; Chen et al., 2023) uses a smaller draft model to propose multiple tokens at once, which the main model then verifies in a single forward pass. When draft acceptance is high, throughput per request improves substantially. When acceptance is low, you have spent compute on the draft and gotten nothing, a small but real tail latency penalty.

The architectural lesson is that the latency budget is not just prefill plus decode time. It is the joint distribution over prefix cache hits, draft acceptance rates, scheduler admission delays, queue wait times, and network. Optimise any one of those in isolation and you can make the others worse. The platforms that consistently hit their SLOs are the ones that instrument all of those terms separately and tune them as a system.

Autoscaling under Variable Latency Workloads

Classical autoscaling tracks CPU utilisation or request rate. Both signals lie for LLM serving.

GPU utilisation reported by nvidia smi measures compute occupancy of the streaming multiprocessors, which is high during prefill and low during memory bound decode. A replica that reads as 30 percent utilised on this metric might be at 95 percent of its actual throughput ceiling because the bottleneck is HBM bandwidth, not compute.

Request rate is similarly misleading. A replica handling 10 requests per second with average generation lengths of 50 tokens is at a completely different load than the same replica handling 10 requests per second with 500 token generations.

The signals that actually drive sound autoscaling decisions:

• Active KV cache pages as a fraction of total. This is the closest thing to memory pressure for an LLM server.
• Time to first token at P95. When TTFT starts climbing, the prefill side is saturating.
• Inter token latency at P95. When ITL climbs, the decode side is saturating.
• Queue depth measured in tokens, not requests. A queue with 100 requests of 100 input tokens each is a different load from a queue with 5 requests of 100K input tokens each.

Autoscale on those, not on the metrics your existing infrastructure platform makes easy to collect.

The other autoscaling trap is cold start. Pulling a 70B model from object storage and loading it into GPU memory can take minutes per replica, not seconds. The platform has to scale ahead of demand, not in reaction to it, which means forecasting, prewarming, and accepting some overprovisioning as the cost of meeting latency SLOs during traffic ramps.

Failure Modes

Seven failure modes recur. Ordered roughly by severity:

1. Head of line blocking from long generations. One request asks for 4,000 output tokens; everyone else in that replica's batch waits. The fix is admission control by expected generation length, separate replicas or separate scheduling tiers for long form workloads, and hard caps on max tokens enforced at the platform level.

2. KV cache out of memory under load spike. A burst of long context requests fills the KV cache faster than active requests can release it. New admissions stall or fail. The fix is page level admission control, a real eviction policy for the prefix cache, and clear backpressure to the gateway.

3. Prefill stall starving decode (or the reverse). A long context prefill monopolises the GPU for several seconds; active streams see their inter token latency spike. The fix is scheduler tuning, chunked prefill (processing long prefixes in pieces), and in the most demanding workloads, disaggregated prefill and decode serving.

4. Queue depth runaway and cascading timeouts. The gateway accepts requests faster than replicas can drain them; clients time out and retry; retries deepen the queue. The fix is bounded queues with explicit reject behaviour (load shedding is a feature, not a failure), backpressure signalled to the gateway, and client SDKs that respect the Retry After header.

5. Cold start on scale out or model rotation. A new replica takes minutes to come online; demand peaks before capacity arrives. The fix is forecast based prescaling, model weight caching on local NVMe, and warm pool replicas held in reserve for known bursty workloads.

6. Routing thrash in tiered systems. The small model is called, fails confidence, escalates to the large model, and does this for most requests, paying the cost of both calls without the savings of either. The fix is to make the router's threshold a real decision boundary, evaluate it offline regularly, and instrument the escalation rate as a first class metric.

7. Model version drift across replicas. A rolling upgrade leaves some replicas on version A and others on version B; users see nondeterministic behaviour across requests. The fix is version aware routing (a request's session sticks to one version for its lifetime) and a clear contract about whether version pinning is a tenant facing guarantee.

The interview favourite failure mode, "what happens when the model itself produces unsafe output?", belongs to the application layer above this platform, not the platform itself. The inference platform's responsibility is to deliver tokens reliably and economically. Content safety, citation grounding, hallucination control: those are upstream concerns. Conflating them is how teams end up with a platform that is bad at both.

Cost Economics

The unit cost of LLM inference is dollars per million tokens, but the unit production cost is GPU seconds, and the conversion between them is where the platform earns or loses its margin.

A useful mental model: take a replica's measured tokens per second throughput, multiply by the seconds in a day, divide by the GPU's effective cost per second, and you get the platform's break even price per token. Compare that to the published price per token of the major API providers and you have a reasonable read on how close to the frontier the platform's engineering is.

The four levers that move that number, in order of typical impact:

• Batching efficiency. The single biggest lever. Continuous batching with good admission control delivers several times the throughput of static batching on realistic workloads.
• Quantisation. Serving a 70B model in FP8 or INT8 instead of FP16 roughly doubles the throughput per GPU at small quality cost. The major open source toolchains (GPTQ, AWQ, FP8 in TensorRT LLM) have made this routine.
• Prefix caching hit rate. For workloads with stable system prompts or repeated RAG contexts, this can cut prefill cost dramatically.
• Hardware tier. The newest accelerators have higher memory bandwidth and faster interconnect; for many workloads this translates into proportionally lower cost per token, but only if the rest of the stack is good enough to use it.

The trap is buying the latest GPUs and running an inefficient serving stack on them. The first lever is worth more than the fourth.

Architect's Checklist

A reusable list for designing or reviewing an LLM inference platform:

1. Express latency targets as TTFT and ITL at P50, P95, and P99. Never as averages.
2. Commit to workload tiers with explicit SLOs; do not let interactive and batch share a queue.
3. Pick a serving stack that does continuous batching with PagedAttention or equivalent, and verify it. Do not trust the marketing.
4. Decide on a prefill and decode strategy: interleaved versus disaggregated, based on the workload's context length profile.
5. Expose prefix caching to the workload, and instrument the cache hit rate as a top line metric.
6. Decide on a multi model strategy: single fleet, tiered, or multi LoRA. Match it to the tenant profile.
7. Quantise where quality allows; measure quality change with an offline evaluation suite, not intuition.
8. Bound max tokens at the platform level. The gateway must enforce it; do not rely on clients.
9. Express quotas in the bottleneck resource: tokens, KV cache pages, GPU seconds. Not just requests per second.
10. Autoscale on KV cache occupancy and TTFT and ITL at P95, not GPU utilisation or request rate.
11. Prewarm replicas based on traffic forecasts; do not scale reactively for models that take several minutes to load.
12. Instrument every failure mode listed above with a dashboard. The list is the runbook.

Architect's Mental Model

The shift from classical web infrastructure to LLM serving is the shift from uniform small requests on cheap commodity hardware to variable large requests on scarce expensive hardware. Almost every assumption has to be reexamined.

Stateless replicas are out: every active request carries kilobytes to megabytes of KV cache state that the replica must hold for the duration of the generation. Static batching is out: the request length variance destroys efficiency. Average latency SLOs are out: the workload is tail dominated and the customer experience is tail determined. Reactive autoscaling on CPU is out: the relevant signals are KV cache pressure and token level latencies, and cold start is measured in minutes.

What replaces them: continuous batching as the architectural unit, prefill and decode treated as distinct workloads, prefix and speculative caches as first class economic levers, workload tiers as the way to keep tail latency contained, and quotas expressed in the bottleneck resource rather than at the gateway.

This article connects to the Architecting Agents series in two specific places. First, the observability discipline from that series is the only way to operate a platform of this complexity: TTFT and ITL histograms, KV cache occupancy gauges, and per tenant token accounting are the instruments that keep the system honest. Second, the cost bounded agent pattern has its mirror here: token budgets in the agent translate directly into GPU seconds on the platform, and the two have to be designed together if the unit economics are going to work.

Throughput pays the bills. Tail latency loses the customer. Build for both, then prove both with a histogram, not an average.