Search for a command to run...
A recursive agent spent the weekend asking itself the same question, failing the same check, and billing you for both. No error fired, because no metric you watched had the word stop in it. This article is about putting it there.
No comments yet. Be the first to comment.
How to ship LLM systems that survive production. Evaluation, versioning, observability, rollouts, incident response, cost guardrails.
The new model swept the golden set, the regression suite, and the LLM as judge evals. Then CSAT dropped. The eval did not lie. The eval measured exactly what it was built to measure. That was the problem.
🗓️ Last updated: August 2026 The architect inherits a brownfield .NET 10 service that classifies incoming customer-support emails into six ticket categories, extracts the order number if there is on
Your senior dev has spent three days hand rolling wrapper classes for logging, caching, and provider switching around the OpenAI SDK. Microsoft already shipped that, called it boring, and put it in the box with the rest of .NET. Surprise.
You bet a year of roadmap on Semantic Kernel. Then Microsoft renamed the bet. Here is what actually changed, what did not, and why the architects who shrugged are the ones who read the contract instead of the brand.
The dashboards are healthy, the error rate is zero, nothing shipped this week, and the model just told a customer something expensive and wrong. There is no commit to revert. Welcome to the incident that classical on call was never designed for.
🗓️ Last updated: July 2026
The finance team flags the anomaly on Monday morning. The prior week LLM spend is up by a factor of eleven. The chart is a single weekend spike, starting Saturday evening, ending, well, not ending, running flat at the new ceiling through Sunday and into Monday. Eleven times the budget.
The on call engineer pulls up the traces. One tenant. One workflow. A multi step agent that calls itself with the result of the previous step. The first step retrieves a document, the second summarises it, the third asks "does this answer the user's question?", and on Saturday at 7:42 PM, for one user, the answer was no. The workflow retried. The check failed. The workflow retried. Forty eight hours of model calls. Several thousand dollars of tokens. Zero useful output. The user, meanwhile, gave up after thirty seconds and went to bed, blissfully unaware they had kicked off a small fire.
No error fired in any system. No alert fired because no metric the team watched had a per tenant budget on it. The vendor's tier level rate limit eventually clipped the workflow, but the ceiling was high enough that the cost accumulated faster than anyone noticed. The system did exactly what it was designed to do. The design simply did not include the word "stop."
This is the failure shape that distinguishes LLM system cost from every other line item in a software budget. The cost of a database query has a ceiling, because the query times out. The cost of a CPU loop has a ceiling, because the box maxes out. The cost of a model call has no native ceiling. It is bounded only by the budgets, rate limits, and kill switches you explicitly put in front of it. Build the system without those, and the worst case user is also the most expensive user, by a factor you will not believe until the invoice arrives and ruins your Monday.
This article is about the discipline that keeps that invoice from arriving. It is also the closer for Series 3.
It is not a survey of vendor pricing. Pricing changes; the patterns do not. It is not an argument for the cheapest model in every case. The cost bounded discipline from Series 1 Article 5 covers per call optimisation. This article is about the system level guardrails around all the calls, the structural defences that decide what the maximum loss is, before the optimisation question is even relevant.
An LLM system is the only system in your stack where a single user can cost you a thousand times the median, in a single hour, without any component reporting an error. Budgeting, rate limiting, and kill switches are not optimisation. They are the structural defence against your own pricing model. The discipline applies at four layers, per user, per tenant, per workflow, and platform wide, and any layer without a defined ceiling is the layer the next incident will exploit. The incident is very good at finding that layer.
Before LLMs, the cost of a feature was largely deterministic. A web request consumed a known amount of CPU and memory. A database query took roughly the same time every time. The p99 to p50 cost ratio was typically single digit, not hundreds of times.
Cost discipline in that world consisted of three things. Capacity planning sized the box for peak load. Per request resource limits, CPU time, memory caps, query timeouts, prevented any single request from monopolising the system. Classical rate limiting algorithms, the token bucket and the leaky bucket, controlled the volume of incoming work.
Each of those mechanisms protected the system from being overwhelmed. None of them protected the budget. They did not need to. The cost of overwhelming the system was capped at "the system gets slow," not "the bill triples." Slow was the worst case. Slow does not have a balance sheet.
LLM systems break the cost per user distribution, and they break it in a direction nobody warns you about.
A median user might consume a few thousand input tokens and a few hundred output tokens per session. A p99 user, one who triggers a long retrieval context, a multi step agent loop, a verbose model on a thinking mode setting, and several tool calls, might consume hundreds of thousands of tokens in a single session. The ratio between p99 and p50 is not 5x or 10x. It is routinely two or three orders of magnitude. The same endpoint, the same auth, wildly different bills.
Three things drive the gap. Output tokens are typically priced several times higher than input tokens; on hosted frontier models the asymmetry sits in the three to five times range. Context is rebuilt every turn in a multi turn conversation, so a ten turn session does not pay 10x the cost of a one turn session, it pays something closer to the sum of every turn so far, because every turn resends the entire prior conversation. And agentic workflows multiply: a workflow that calls the model five times for one user request pays five times the cost of a single turn call, plus the cost of the tool calls in between.
This is not a tail to be optimised away by tuning the median. The p99 user is structurally different, different code paths, different context patterns, different workflow branches. The cost discipline has to be designed against the p99 user, not the median. Tuning the median user makes the dashboard look nice and changes the invoice not at all.
The first design step is to know what the p99 looks like in your own system, by tenant, by user, by workflow. Without that visibility there is no design, only hope, and hope reliably loses to a recursive loop on a Saturday night.
The single most effective cost guardrail is a budget that lives at the same granularity as the cost. For most B2B systems that is per tenant. For some B2C systems it is per user. For agentic systems it is sometimes per workflow execution.
A budget is three things: a number, a measurement window, and a behaviour when the number is exceeded. The number is the maximum cost the entity is allowed to incur in the window. The window is typically one minute, one hour, one day, or one billing month, depending on what the budget is protecting against. The behaviour is what happens at the ceiling, and it is the part teams forget to specify, which is how they end up with a ceiling that observes the overage politely and does nothing about it.
The four behaviours, in roughly decreasing order of customer impact:
What is not a valid behaviour: "do nothing and hope." A ceiling without an enforced behaviour is a number on a dashboard, not a guardrail. Enforcement has to happen at request admission time, before the model is called. A budget enforced by retrospective billing has already been blown by the time it fires. It is not a guardrail at that point, it is a receipt.
Classical rate limits count requests. RPM is requests per minute; the leaky bucket admits N requests per second; the token bucket replenishes capacity at a fixed rate. The counted unit is the request, on the assumption that requests are roughly equivalent in cost.
In LLM systems they are not, and that assumption is the whole bug. Two requests to the same endpoint, with the same authentication, can differ in cost by orders of magnitude based on prompt length, model selection, sampling parameters, and downstream tool calls. An RPM counted rate limit lets a tenant consume the maximum allowed budget by sending the smallest allowed number of the largest possible requests. The limit is satisfied. The wallet is not.
Semantic aware rate limits count cost, not requests. The unit is tokens per minute, dollars per minute, or a synthetic cost unit per minute that combines model selection with token volume. The same algorithms, token bucket and leaky bucket, apply unchanged, with the counted unit replaced.
Vendor tokens per minute ceilings are the outermost envelope of this idea, set by the provider at the tier or organisation level. Both major hosted vendors expose tokens per minute and requests per minute as joint limits; whichever one binds first throttles the caller. The internal rate limit a system enforces on its own users should sit well below the vendor ceiling, with enough headroom that one greedy tenant never trips a 429 on the vendor side and incidentally rate limits every other tenant sharing the API key.
The architectural placement matters. A per tenant cost limit enforced inside the application is the only level at which a single tenant can be isolated from the others. Vendor level limits are shared across all tenants on the same key, which means a runaway tenant becomes everyone's incident, and the blameless postmortem now has a much longer attendee list.
The biggest cost lever in most LLM systems is not the rate limit. It is the cache. The rate limit stops the bleeding; the cache stops you from paying full price for the same answer twice.
Both major hosted vendors expose explicit prompt caching mechanisms. Anthropic uses a cache_control marker placed at a prefix boundary; cache writes are priced above the base input rate and cache reads at roughly a tenth of it. OpenAI provides automatic prompt caching for repeated prefixes with a discount on the cached portion. The mechanic is the same: a long stable prompt prefix, system instructions, tool definitions, retrieval context that does not change between turns, is paid for once at write time and reused at read time for a fraction of the cost.
The savings compound on workloads that reuse prefixes. Agent system prompts, tool catalogues, schema definitions, multi turn conversations with stable system instructions, all have a long stable prefix and a short volatile suffix. Designed for cache friendliness, the same workload pays the input token cost for the stable prefix once per cache lifetime, rather than once per request.
Designing for cache friendliness is a structural choice, not an optimisation. The stable content goes at the start of the prompt. The volatile content goes at the end. Tool definitions are reused verbatim, not regenerated per request. These choices are made at prompt design time; they cannot be retrofitted after the workload is in production without a prompt refactor, which is the polite term for rewriting the thing under deadline.
Token aware caching at the application layer, semantic similarity caches for whole response reuse, is a complementary lever. Prompt caching reduces the cost of computing a response. Semantic caching avoids computing it at all. Both belong in the design.
A cost guardrail is only as good as the kill switch behind it. Five places where a kill switch belongs, in order of granularity from blunt to surgical:
1. The platform kill switch. A flag that disables LLM calls system wide. Used for budget emergencies, vendor outages, or trust and safety incidents. Returns a graceful fallback ("AI features are temporarily unavailable") on every endpoint.
2. The model kill switch. Disables a specific model. Routes requests that would have used it to a fallback. Useful when one model is responsible for a cost runaway or a behavioural regression.
3. The workflow kill switch. Disables a specific agentic workflow or multi step chain. Useful when one workflow is responsible for the cost. The recursive retry case in the opening scene is exactly this category, and the engineer staring at the Saturday trace would have given a great deal for this switch.
4. The tenant kill switch. Disables LLM calls for a specific tenant. Used when one tenant is generating runaway cost and the rest of the system needs to be preserved. Implies a defined customer communication path, because someone has to tell them.
5. The user kill switch. Disables LLM calls for a specific user. The narrowest blast radius. Useful for abuse cases and individual runaway sessions.
Each kill switch is a contract with the system around it: when this switch is off, the system handles the absence gracefully. Building the graceful absence path is the prerequisite. A kill switch that breaks the rest of the product when it fires is one nobody will dare flip during an incident, which means it does not exist when you need it most.
Cost is a first class telemetry signal. Every model call emits a span (Article 3 of this series), and every span carries token usage in standard OpenTelemetry GenAI fields. Cost is a derived attribute on that span, tokens in plus tokens out, multiplied by the model's pricing, attributed to the calling tenant, user, workflow, and feature.
Three dashboards belong in every LLM system. Cost per tenant, rolled up by hour and day, with anomaly detection on rate of change. Cost per workflow, identifying which agentic paths drive spend. Cost per user within a tenant, surfacing the p99 users whose patterns might break the tenant budget if they grow.
The alerts on those dashboards make the budgets real. A per tenant cost alert that fires when a tenant's hourly burn exceeds a few times its trailing week average catches the weekend runaway in the opening scene within minutes, not after forty eight hours of accumulated bill. The alert should page someone on call. Cost incidents are operational incidents, even though they never turn a single dashboard red.
The patterns that turn a healthy LLM system into a cost incident, worst first:
1. Recursive workflows without bounded depth. The opening scene failure. An agent calls itself with the result of the previous step, the success condition is fuzzy, the loop never converges, the system pays for every iteration. Mitigation: maximum recursion depth, maximum total tokens per workflow execution, and a per workflow cost budget that hard rejects further calls. Series 1 Article 6, Multi Agent Coordination Patterns, covers the broader pattern that turns single agent loops into multi agent cascades; the cost discipline scales accordingly, which is a gentle way of saying it gets worse.
2. Unbounded retries. A model call fails. The system retries with exponential backoff. The retry fails. The system retries again. The retry budget has no ceiling, or the ceiling is per attempt duration rather than per attempt count. The system pays for every attempt. Mitigation: maximum retry count per logical operation, with the count enforced across the whole call graph rather than per individual call site.
3. Runaway agents. An agent in a tool using loop calls a tool, gets a result, decides to call another tool, gets a result, decides to call another tool, indefinitely. Mitigation: maximum tool calls per agent run, maximum total tokens per agent run, and a circuit breaker on tool call rate that fires before the wallet does.
4. Greedy multi turn context. Every turn of a long conversation resends the entire prior conversation. By turn fifty, the input cost per turn is dominated by ancient history that no longer matters. Mitigation: context summarisation, sliding window context, or memory store retrieval that keeps the active context bounded. Series 2 Article 2, Designing a Real Time Coding Assistant, is the canonical high volume per user case where this pattern bites first.
5. Cache cliff. The workload was designed to cache hit, but the cache is invalidated by a small variation in the prefix that nobody intended to vary, a timestamp, a request ID, a session token. The cache hit rate falls off a cliff, input cost rises by an order of magnitude overnight, and the only thing that changed was a field nobody thought counted. Mitigation: cache hit rate as a first class metric, with alerting on rate of change.
6. Wrong model defaulting. The system routes all requests to the most capable model, on the assumption that better is always better. Cheap requests pay frontier prices for no quality gain. Mitigation: model routing by request complexity, with a default to the cheapest model that meets the quality bar.
7. Vendor tier surprise. The team built against the vendor's middle tier; usage grew; the next tier up has a higher floor than the team budgeted for. Mitigation: vendor tier transitions are forecast against projected usage, not discovered by being throttled in production at the worst possible moment.
LLM systems are the first dependency in the modern stack where the worst case user is also the most expensive user, by a factor large enough to matter on the company's income statement. The discipline this article describes is what closes the gap between worst case behaviour and worst case cost, not by making the worst case impossible, but by making the worst case bounded.
The budgets define the bound. The rate limits enforce it. The caches lower the floor. The kill switches give the operator a way to stop the bleeding. The observability makes any of the above possible at all. Remove any one and the other four develop a blind spot in the exact shape of the thing you removed.
None of these are optimisations. They are structural defences against the system you have already built. Without them, the system has no defined worst case cost. With them, the worst case is a number you chose in advance, which is the only kind of worst case anyone in finance is willing to sign off on.
Classical systems get slower under overload. LLM systems send you the bill.
This article closes Series 3, Shipping AI Systems, six articles on the operational discipline that turns an LLM prototype into a production system that survives contact with real users, real traffic, real vendors, and real budgets. Evaluation, versioning, observability, rollout, incident response, and now cost. Each topic has a classical analogue in the pre LLM stack. Each one needed adaptation for an artefact whose contract is a probability distribution rather than a typed signature.
The next series, Series 4: AI for the .NET Architect, turns from the general operational discipline of the platform to the specific implementation discipline of building agentic systems on .NET. The Microsoft Agent Framework, Microsoft.Extensions.AI as the unified client layer, structured outputs in C#, hosting long running agents, streaming with IAsyncEnumerable and SignalR, and a reference architecture that ties it together. The same architect's lens; a more concrete stack.
Series 4 starts a new cadence, one article per week, replacing the twice weekly Series 3 sprint. Subscribe at pragmaticstack.hashnode.dev to catch the first .NET article.
