🗓️ Last updated: July 2026
The finance team flags the anomaly on Monday morning. The prior-week LLM spend is up by a factor of eleven. The chart is a single weekend spike, starting Saturday evening, ending — well, not ending — running flat at the new ceiling through Sunday and into Monday. Eleven times the budget.
The on-call engineer pulls up the traces. One tenant. One workflow. A multi-step agent that calls itself with the result of the previous step. The first step retrieves a document, the second summarises it, the third asks "does this answer the user's question?" — and on Saturday at 7:42 PM, for one user, the answer was no. The workflow retried. The check failed. The workflow retried. Forty-eight hours of model calls. Several thousand dollars of tokens. Zero useful output. The user gave up after thirty seconds and went to bed.
No error fired in any system. No alert fired because no metric the team watched had a per-tenant budget on it. The vendor's tier-level rate limit eventually clipped the workflow — but the ceiling was high enough that the cost accumulated faster than anyone noticed. The system did exactly what it was designed to do. The design did not include the words "stop."
This is the failure shape that distinguishes LLM-system cost from every other line item in a software budget. The cost of a database query has a ceiling — the query times out. The cost of a CPU loop has a ceiling — the box maxes out. The cost of a model call has no native ceiling. It is bounded only by the budgets, rate limits, and kill-switches you explicitly put in front of it. Build the system without those, and the worst-case user is also the most expensive user, by a factor you will not believe until the invoice arrives.
This article is about the discipline that keeps that invoice from arriving. It is also the closer for Series 3.
What this article is not
It is not a survey of vendor pricing. Pricing changes; the patterns do not. It is not an argument for the cheapest model in every case. The cost-bounded discipline from Series 1 Article 5 covers per-call optimisation. This article is about the system-level guardrails around all the calls — the structural defences that decide what the maximum loss is, before the optimisation question is even relevant.
Thesis
An LLM system is the only system in your stack where a single user can cost you a thousand times the median, in a single hour, without any component reporting an error. Budgeting, rate-limiting, and kill-switches are not optimisation. They are the structural defence against your own pricing model. The discipline applies at four layers — per-user, per-tenant, per-workflow, and platform-wide — and any layer without a defined ceiling is the layer the next incident will exploit.
What classical cost discipline looked like
Pre-LLM, the cost of a feature was largely deterministic. A web request consumed a known amount of CPU and memory. A database query took roughly the same time every time. The p99/p50 cost ratio was typically single-digit, not hundreds-of-times.
Cost discipline in that world consisted of three things. Capacity planning sized the box for peak load. Per-request resource limits — CPU time, memory caps, query timeouts — prevented any single request from monopolising the system. Classical rate-limiting algorithms — token bucket and leaky bucket — controlled the volume of incoming work.
Each of those mechanisms protected the system from being overwhelmed. None protected the budget. They didn't need to. The cost of overwhelming the system was capped at "the system gets slow," not "the bill triples."
The p99 cost problem
LLM systems break the cost-per-user distribution.
A median user might consume a few thousand input tokens and a few hundred output tokens per session. A p99 user — one who triggers a long retrieval context, a multi-step agent loop, a verbose model on a thinking-mode setting, and several tool calls — might consume hundreds of thousands of tokens in a single session. The ratio between p99 and p50 is not 5x or 10x. It is routinely 100x or 1000x.
Three things drive the gap. Output tokens are typically priced several times higher than input tokens — on hosted frontier models the asymmetry is in the three-to-five-times range. Context is rebuilt every turn in a multi-turn conversation, so a ten-turn session does not pay 10x the cost of a one-turn session — it pays something closer to 55x, because every turn re-sends the entire prior conversation. And agentic workflows multiply: a workflow that calls the model five times for one user request pays five times the cost of a single-turn call, plus the cost of the tool calls in between.
This is not a tail to be optimised away by tuning the median. The p99 user is structurally different — different code paths, different context patterns, different workflow branches. The cost discipline has to be designed against the p99 user, not the median.
The first design step is to know what the p99 looks like in your own system, by tenant, by user, by workflow. Without that visibility there is no design — only hope.
Per-tenant and per-user budgets
The single most effective cost guardrail is a budget that lives at the same granularity as the cost. For most B2B systems that is per-tenant. For some B2C systems it is per-user. For agentic systems it is sometimes per-workflow-execution.
A budget is three things: a number, a measurement window, and a behaviour when the number is exceeded. The number is the maximum cost the entity is allowed to incur in the window. The window is typically one minute, one hour, one day, or one billing month depending on what the budget is protecting against. The behaviour is what happens at the ceiling — reject the request, queue it, downgrade it to a cheaper model, or surface a paywall.
The four behaviours, in roughly decreasing order of customer impact:
- Hard reject. Return an error or graceful refusal at the ceiling. Acceptable for free-tier abuse protection; uncomfortable for paying customers.
- Downgrade. Switch to a cheaper model for the remainder of the window. Preserves availability at degraded quality. The fallback must already be eval-tested — Article 5 covered the same pattern for incident-time mitigation.
- Queue with deferred processing. Move the work to an asynchronous batch path. Both major vendors offer batch-API pricing at roughly half the synchronous rate, which makes deferred processing a viable budget-recovery mechanism for non-interactive workloads.
- Surface a paywall. Show the user the cost they have consumed and offer a path to a higher tier. Works only when the user knows they are using a metered resource.
What is not a valid behaviour: "do nothing and hope." A ceiling without an enforced behaviour is a number on a dashboard, not a guardrail. Enforcement has to happen at request-admission time, before the model is called. A budget enforced by retrospective billing has already been blown by the time it fires.
Semantic-aware rate limits
Classical rate limits count requests. RPM is requests-per-minute; the leaky bucket admits N requests per second; the token bucket replenishes capacity at a fixed rate. The counted unit is the request, on the assumption that requests are roughly equivalent in cost.
In LLM systems they are not. Two requests to the same endpoint, with the same authentication, can differ in cost by orders of magnitude based on prompt length, model selection, sampling parameters, and downstream tool calls. An RPM-counted rate limit lets a tenant consume the maximum-allowed budget by sending the smallest-allowed number of the largest-possible requests.
Semantic-aware rate limits count cost, not requests. The unit is tokens-per-minute, dollars-per-minute, or a synthetic cost-units-per-minute that combines model selection with token volume. The same algorithms — token bucket, leaky bucket — apply unchanged, with the counted unit replaced.
Vendor TPM (tokens-per-minute) ceilings are the outermost envelope of this idea, set by the provider at the tier or organisation level. Both major hosted vendors expose TPM and RPM as joint limits; whichever one binds first throttles the caller. The internal rate limit a system enforces on its own users should sit well below the vendor TPM, with enough headroom that one greedy tenant never triggers a 429 on the vendor side and incidentally rate-limits every other tenant sharing the API key.
The architectural placement matters. Per-tenant TPM enforced inside the application is the only level at which a single tenant can be isolated from the others. Vendor-level RPM and TPM are shared across all tenants on the same key, which means a runaway tenant becomes everyone's incident.
Token-aware caching
The biggest cost lever in most LLM systems is not the rate limit. It is the cache.
Both major hosted vendors expose explicit prompt-caching mechanisms. Anthropic uses a cache_control marker placed at a prefix boundary; cache writes are priced above the base input rate and cache reads at roughly a tenth of it. OpenAI provides automatic prompt caching for repeated prefixes with a discount on the cached portion. The mechanic is the same: a long stable prompt prefix — system instructions, tool definitions, retrieval context that doesn't change between turns — is paid for once at write time and re-used at read time for a fraction of the cost.
The savings compound on workloads that re-use prefixes. Agent system prompts, tool catalogues, schema definitions, multi-turn conversations with stable system instructions — all have a long stable prefix and a short volatile suffix. Designed for cache friendliness, the same workload pays the input-token cost for the stable prefix once per cache lifetime, rather than once per request.
Designing for cache friendliness is a structural choice, not an optimisation. The stable content goes at the start of the prompt. The volatile content goes at the end. Tool definitions are reused verbatim, not regenerated per request. These choices are made at prompt-design time; they cannot be retrofitted after the workload is in production without a prompt refactor.
Token-aware caching at the application layer — semantic-similarity caches for whole-response reuse — is a complementary lever. Prompt caching reduces the cost of computing a response. Semantic caching avoids computing it at all. Both belong in the design.
Kill-switches at multiple layers
A cost guardrail is only as good as the kill-switch behind it. Five places where a kill-switch belongs, in order of granularity from blunt to surgical:
1. The platform kill-switch. A flag that disables LLM calls system-wide. Used for budget emergencies, vendor outages, or trust-and-safety incidents. Returns a graceful fallback ("AI features are temporarily unavailable") on every endpoint.
2. The model kill-switch. Disables a specific model. Routes requests that would have used it to a fallback. Useful when one model is responsible for a cost runaway or a behavioural regression.
3. The workflow kill-switch. Disables a specific agentic workflow or multi-step chain. Useful when one workflow is responsible for the cost — the recursive-retry case in the opening scene is exactly this category.
4. The tenant kill-switch. Disables LLM calls for a specific tenant. Used when one tenant is generating runaway cost and the rest of the system needs to be preserved. Implies a defined customer-communication path.
5. The user kill-switch. Disables LLM calls for a specific user. The narrowest blast radius. Useful for abuse cases and individual runaway sessions.
Each kill-switch is a contract with the system around it: when this switch is off, the system handles the absence gracefully. Building the graceful-absence path is the prerequisite. A kill-switch that breaks the rest of the product when it fires is one nobody will flip during an incident.
Observability for cost
Cost is a first-class telemetry signal. Every model call emits a span (Article 3 of this series), and every span carries token usage in standard OpenTelemetry GenAI fields. Cost is a derived attribute on that span — tokens-in plus tokens-out, multiplied by the model's pricing, attributed to the calling tenant, user, workflow, and feature.
Three dashboards belong in every LLM system. Cost per tenant, rolled up by hour and day, with anomaly detection on rate of change. Cost per workflow, identifying which agentic paths drive spend. Cost per user within a tenant, surfacing the p99 users whose patterns might break the tenant budget if they grow.
The alerts on those dashboards make the budgets real. A per-tenant cost alert that fires when a tenant's hourly burn exceeds three times its trailing-week average catches the weekend-runaway in the opening scene within minutes, not after forty-eight hours of accumulated bill. The alert should page someone on call. Cost incidents are operational incidents.
Failure modes, severity-ordered
The patterns that turn a healthy LLM system into a cost incident, worst-first:
1. Recursive workflows without bounded depth. The opening-scene failure. An agent calls itself with the result of the previous step, the success condition is fuzzy, the loop never converges, the system pays for every iteration. Mitigation: maximum recursion depth, maximum total tokens per workflow execution, and a per-workflow cost budget that hard-rejects further calls. Series 1 Article 6 — Multi-Agent Coordination Patterns — covers the broader pattern that turns single-agent loops into multi-agent cascades; the cost discipline scales accordingly.
2. Unbounded retries. A model call fails. The system retries with exponential backoff. The retry fails. The system retries again. The retry budget has no ceiling, or the ceiling is per-attempt-duration rather than per-attempt-count. The system pays for every attempt. Mitigation: maximum retry count per logical operation, with the count enforced across the whole call graph rather than per individual call site.
3. Runaway agents. An agent in a tool-using loop calls a tool, gets a result, decides to call another tool, gets a result, decides to call another tool, indefinitely. Mitigation: maximum tool calls per agent run, maximum total tokens per agent run, and a circuit-breaker on tool-call-rate that fires before the wallet does.
**4. Greedy multi-turn context. ** Every turn of a long conversation re-sends the entire prior conversation. By turn fifty, the input cost per turn is dominated by ancient history that no longer matters. Mitigation: context summarisation, sliding-window context, or memory-store retrieval that keeps the active context bounded. Series 2 Article 2 — Designing a Real-Time Coding Assistant — is the canonical high-volume per-user case where this pattern bites first.
5. Cache cliff. The workload was designed to cache-hit, but the cache is invalidated by a small variation in the prefix that nobody intended to vary — a timestamp, a request ID, a session token. The cache hit rate falls off a cliff, input cost rises by an order of magnitude overnight. Mitigation: cache-hit rate as a first-class metric, with alerting on rate-of-change.
6. Wrong-model defaulting. The system routes all requests to the most capable model, on the assumption that better is always better. Cheap requests pay frontier prices for no quality gain. Mitigation: model routing by request complexity, with a default to the cheapest model that meets the quality bar.
7. Vendor-tier surprise. The team built against the vendor's middle tier; usage grew; the next tier up has a higher floor than the team budgeted for. Mitigation: vendor-tier transitions are forecast against projected usage, not discovered by being throttled.
The architect's checklist
🗓️ Every LLM call attributes its cost to a tenant, user, workflow, and feature — in the same span the observability layer captures.
🗓️ Per-tenant cost budgets are defined, enforced at request-admission time, and survive process restarts.
🗓️ Per-tenant rate limits count tokens (or cost), not requests.
🗓️ Per-workflow execution has a hard ceiling on total tokens, total cost, and total recursion depth.
🗓️ Retries are budgeted by logical operation, not per call site. Total retry cost is bounded.
🗓️ Tool-using agents have a hard ceiling on tool calls per agent run.
🗓️ Multi-turn conversations summarise or window context above a defined threshold.
🗓️ Prompts are structured for cache friendliness — stable content prefix, volatile content suffix.
🗓️ Vendor prompt-caching is enabled wherever the workload supports it, with cache-hit rate monitored.
🗓️ Semantic-similarity caches sit in front of the model for response-level reuse where business logic permits.
🗓️ Model routing chooses the cheapest model that meets the quality bar for each request class.
🗓️ Kill-switches exist at platform, model, workflow, tenant, and user granularity. Each "off" state is graceful.
🗓️ Cost dashboards by tenant, workflow, and user are live. Anomaly alerts on burn-rate page on call.
🗓️ Vendor-tier transitions are forecast quarterly, not discovered at throttle time.
Mental model
LLM systems are the first dependency in the modern stack where the worst-case user is also the most expensive user, by a factor large enough to matter on the company's income statement. The discipline this article describes is what closes that gap between worst-case behaviour and worst-case cost — not by making the worst case impossible, but by making the worst case bounded.
The budgets define the bound. The rate limits enforce it. The caches lower the floor. The kill-switches give the operator a way to stop the bleeding. The observability makes any of the above possible at all.
None of these are optimisations. They are structural defences against the system you have already built. Without them, the system has no defined worst-case cost. With them, the worst case is a number you chose in advance.
Classical systems get slower under overload. LLM systems send you the bill.
What's next — closing Series 3, opening Series 4
This article closes Series 3 — Shipping AI Systems — six articles on the operational discipline that turns an LLM prototype into a production system that survives contact with real users, real traffic, real vendors, and real budgets. Evaluation, versioning, observability, rollout, incident response, and now cost. Each topic has a classical analogue in the pre-LLM stack. Each one needed adaptation for an artefact whose contract is a probability distribution.
The next series — Series 4: AI for the .NET Architect — turns from the general operational discipline of the platform to the specific implementation discipline of building agentic systems on .NET. Semantic Kernel, Microsoft.Extensions.AI, the .NET LLM client patterns, structured-output schemas in C#, dependency-injection lifetimes for stateful agents, IAsyncEnumerable streaming, and the architectural fit between agentic patterns and the .NET runtime. The same architect's-lens approach; a more concrete stack.
Series 4 starts the new cadence — one article per week — replacing the twice-weekly Series 3 sprint. Subscribe at pragmaticstack.hashnode.dev to catch the first .NET article.
Series 3 progress
Series 3 progress
| # |
Title |
Status |
| 1 |
Evaluating LLM Systems |
Published |
| 2 |
Prompt and Model Versioning |
Published |
| 3 |
Observability for LLM Systems |
Published |
| 4 |
Rolling Out a New Model Safely |
Published |
| 5 |
Incident Response When the Model is the Bug |
Published |
| 6 |
Cost Guardrails for LLM Systems |
You are here — series complete |
