# The Behaviour Was the Bug: Observability for Systems Where Stack Traces Don't Help

> 🗓️ **Last updated: July 2026**

A support ticket arrives on a Monday morning. A user asked the assistant a billing question. The answer was wrong, confident, and specific enough to do damage. The user followed it, disputed a charge, and escalated. The customer experience lead wants a root cause by end of day.

The on call engineer opens the logs. Request received, latency normal, status 200. The response payload is captured. The session ID is captured. None of it explains anything. There is no prompt, no retrieval context, no token usage, no finish reason, no model version, no judge score: none of the things that would let anyone reconstruct what the system actually did. The deterministic plumbing all worked. The behaviour was the bug. The behaviour was not instrumented.

By noon the team has reverse engineered most of the path by replaying the session. By afternoon they have a hypothesis. By the end of the day they have a fix, almost certainly to the prompt, possibly to the retriever, they cannot fully tell because nothing was captured at the level of detail that would settle it. The ticket gets closed. The next one is already in the queue. Nobody is confident the same class of failure will not recur next week.

This article is about the instrumentation discipline that would have settled the question in twenty minutes.

> **What this article is NOT.** It is not a vendor comparison of observability platforms. It is not "five dashboards every LLM team should build." It is the architectural argument that observability for LLM systems requires capturing different signals at a different granularity than classical observability, and that retrofitting it after a quality incident costs an order of magnitude more than designing it in.

* * *

## What's Different

Classical observability rests on a stable premise: when something breaks, there is a code path that broke it. The job of the observability stack is to point you at that code path. Stack traces, structured logs, request IDs, distributed traces: all of these are mechanisms for compressing "what executed" into something a human can inspect.

LLM systems break that premise in two specific ways.

First, the bug is often in the output, not in any code path. A model returns a confidently wrong answer. The HTTP request was fine. The retrieval ran. The prompt assembled. The model responded. Every code path executed correctly. The output was wrong. Classical instrumentation has nothing to say about why.

Second, the bug is statistical. One in fifty inputs of a certain shape triggers a refusal that should not have happened. One in two hundred produces a hallucinated entity. The other forty nine are fine. Without a way to slice production traffic by input shape, output shape, and outcome, you cannot find the failure cluster. You can only respond to the individual complaint that surfaced it.

There is a structural consequence that catches teams used to deterministic systems off guard: in an LLM system, the *content* is the diagnostic data. The actual prompt text, the actual model output, the actual retrieved documents: these are not optional metadata. They are the primary signal. A logging policy that excludes them on the grounds that they look like business data has effectively turned off the observability stack.

* * *

## The Four Golden Signals, Revisited

The Google SRE book popularised the four golden signals: latency, traffic, errors, saturation. The four still apply to LLM systems. They are no longer sufficient.

**Latency** remains observable, but the question it answers has changed. Time to first token matters as much as wall clock latency for any streaming interface. Latency is also a function of output length, which is a function of the prompt, which is a function of the user's input. A latency regression often turns out to be an output length regression in disguise: the new model produces longer responses, so requests take longer end to end, even though tokens per second is unchanged. Instrument both.

**Traffic** is no longer well summarised by requests per second on a route. The useful breakdowns are by prompt template, by model and version, by tenant, by intent class. A spike in cost or latency that is invisible at the route level is often a tenant level or template level event. Per route monitoring will miss it for hours.

**Errors** is the signal that drops in usefulness the most. A 200 OK response containing a wrong answer is the most expensive error in an LLM system, and the classical error rate cannot see it. The traditional 5xx rate still matters for vendor outages, but it is no longer the primary quality signal. The replacement is a portfolio of secondary signals: retry rate, escalation rate, eval scores against a frozen reference set, signals that approximate behavioural correctness without depending on user explicit feedback.

**Saturation** is still useful, but the thing that saturates is different. CPU and memory matter less than the vendor side tokens per minute limits, the per tenant rate caps, and the eval pipeline's compute budget. A team running near a vendor tokens per minute ceiling will see latency tail blow out long before any classical saturation alarm fires.

The four golden signals were never wrong. They were specified for a class of system where they were sufficient. LLM systems are not in that class.

* * *

## Span Design for Chains and Agents

The unit of observation in a classical web service is a request. The unit of observation in an LLM system is a *trace*: a parent span that covers the whole interaction, with child spans for each LLM call, each retrieval, each tool call, each guardrail check, each downstream service.

Span design matters more than instrumentation library choice. The OpenTelemetry GenAI semantic conventions, increasingly adopted across observability vendors in this space, give standard names to the fields that need to be on every LLM span: `gen_ai.system`, `gen_ai.request.model`, `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`, `gen_ai.response.finish_reasons`, and the prompt and completion themselves. Adopting the convention is cheaper than inventing your own field names; the cost of being nonstandard surfaces the first time you have to migrate observability vendors or write a cross platform query.

For chains and agents specifically, the parent span is the *task*, not the request. A user message that triggers a five step plan executed by an agent should produce a single trace with the parent span containing the user's intent, the final response, and the aggregate token and cost figures, along with child spans for every model call, every tool invocation, and every retrieval step in between. Without that hierarchy, debugging a five step failure means stitching five separate request logs together by timestamp, which is the manual reconstruction the on call engineer in the opening scene was doing.

Capture the retrieval context as data on the span, not as a pointer to a document store. A trace that says "retrieved 5 documents (ID 1, ID 7, ID 41, ID 88, ID 102)" is not useful three weeks later when the document store has updated. A trace that captures the actual text the model saw at the time the model saw it is useful indefinitely. Storage is cheap; lossy traces are expensive.

* * *

## Semantic Clustering

A single complaint is one data point. A cluster of similar complaints is a pattern, and patterns are what you can fix.

The classical clustering surface is the URL or the error code. LLM systems need a different one: cluster by what the user was trying to do and what went wrong. Embedding based clustering of production traces is the dominant technique: group user inputs by semantic similarity, then look at the failure rate or the eval score per cluster. A four per cent global retry rate can hide a twenty eight per cent retry rate inside a single cluster of billing related questions. The dashboard that reported "retry rate is up slightly" misses the structural failure entirely.

Several observability platforms in this space, including Langfuse, Arize, Helicone, and Traceloop, offer some form of trace clustering or trace search. The specific tool matters less than the workflow. A useful workflow is: filter traces by outcome (low eval score, user retry, user escalation), cluster the remaining traces by embedding of the user input, look at the top three clusters by failure density, form a hypothesis. If that workflow is not possible in your stack, requiring SQL queries against a warehouse and a custom embedding job, it will not happen during an incident. Clustering becomes a thing that happens once a quarter in a retrospective.

The structural argument is that production traffic is the most honest curation source you have. The golden set was curated when the product launched. The production trace clusters tell you what the product is actually used for now. An observability stack that does not surface those clusters is hiding the dataset you most want to look at.

* * *

## PII and the Logging Tradeoff

Classical observability has a clean default: do not log personally identifiable information. Log the user ID, log the request URL, log the latency, never log the form body. The discipline is well understood and compliance teams have decades of precedent to lean on.

LLM systems break the default in a specific way. The prompt is the request body. If the user asked "can you summarise the email from my doctor about my appointment on Tuesday," the prompt contains a medical context that no classical request body would carry. The model's response often contains entities lifted from the prompt: names, dates, locations, account numbers, things the classical observability stack has no policy for.

There is no clean answer. There are deliberate ones:

*   **Redact at log time.** Run a PII detector before persistence; mask the matches; persist the redacted form. The detector is itself a model and has its own miss rate; budget for it and review its outputs periodically.
*   **Tier the retention.** Keep full content for a short debug window of hours to a few days. Keep the redacted form for longer term analytics. Define both windows in writing and enforce them in the storage layer, not in policy alone.
*   **Gate detail by consent or context.** Some traffic carries explicit permission for verbose logging: enterprise contracts, internal traffic, debug environments. Some does not: consumer traffic in regulated domains. The trace pipeline should know the difference and apply it at capture time.

The default of "log everything and figure it out later" is a compliance incident waiting to happen. The default of "log nothing to be safe" is operational blindness. Neither is the answer. The answer is a written policy that engineering, security, and legal all signed, and an instrumentation stack that implements it.

* * *

## Alerting on Probabilistic Signals

The hardest part of LLM observability is not capture. It is alerting.

Classical alerts are threshold based and mostly correct: when the 5xx rate exceeds one per cent, page the on call. The signal is clean. The action is clear. The false positive rate is low enough to keep the team's attention. LLM signals are noisier, slower, and more correlated with each other. A few patterns work, and a few do not.

*   **Alert on trend, not on absolute value.** Eval scores oscillate. Retry rates fluctuate with traffic mix. A single bad hour is rarely the start of an incident; a two sigma drop sustained across three measurement windows often is. Anchor alerts to rate of change, not to level.
*   **Alert on the change you cannot explain.** The finish reason distribution shifting overnight, say more length finishes and fewer stop finishes, is the kind of signal that is small enough to miss and structural enough to matter. A daily diff on these distributions catches it before the customer experience team does.
*   **Treat cost as a quality signal.** A substantial jump in tokens per request across a tenant cohort, with no change in traffic shape, is almost always a prompt or model regression making the model more verbose. Cost dashboards usually exist for finance reasons; they double as a leading indicator of behavioural change.
*   **Do not alert on individual prompts.** Per request alerting is a paging volume problem with no upside. Aggregate first, then alert.

There is a softer point underneath all of these. Alert volume is finite. Every probabilistic signal alert competes with every deterministic signal alert for the on call's attention. A team that adds twelve new LLM alerts in a quarter without retiring any will arrive at alert fatigue on a schedule. Treat the alert catalogue itself as a constrained resource.

* * *

## Failure Modes

Observability programs fail in ways that look like success on the dashboard. Seven recurring failure modes, severity ordered:

1. **The dashboard says everything is fine.** All classical signals are green. The eval pipeline is green. Customer experience tickets are climbing. The dashboard is measuring uptime and the failure is in behaviour. The team built a deterministic dashboard for a probabilistic system.
2. **PII in places it should not be.** A retention sweep finds full medical context prompts in a six month old trace archive. The compliance impact is real, the remediation is expensive, and the right policy was never written before the instrumentation was turned on.
3. **Span explosion.** Every LLM call generates dozens of child spans for guardrail checks, embedding lookups, and tool dispatches. The observability bill exceeds the inference bill. The team starts sampling, then sampling more aggressively, and ends up blind on the failures the sampling drops.
4. **Trace decoration without trace consumption.** The traces are beautiful. Nobody opens them during incidents. The on call instinct is still to read the application logs, because that is the muscle memory. The instrumentation never crosses over from "we built it" to "we use it."
5. **Alert fatigue from probabilistic noise.** Every drift signal becomes a page. The team learns to acknowledge and ignore. Real drift goes unnoticed for a week. The page volume was the failure; the drift was the symptom.
6. **No clustering, only individual complaints.** Each escalation is investigated as a one off. The pattern is real but invisible: it would take ten complaints to see it and only three have reached the queue. The eleventh complaint comes in via the regulator.
7. **Eval in production lag.** The production eval suite runs nightly. The team finds out about drift at the next morning's standup. The cluster of failures that triggered it has been live for sixteen hours.

* * *

## The Architect's Checklist

Twelve items. If a system fails more than three of these, the observability stack is the thing to fix before the next feature ships.

1. Every LLM call produces a span with prompt template ID, prompt version, model ID, model version, input and output token counts, finish reason, latency, and cost.
2. Retrieval steps capture the actual text retrieved, not pointers to a document store that may have updated.
3. Spans roll up into traces at the task level, so a five step agent run produces one trace with five visible child spans.
4. Trace IDs are propagated into the user facing response or support metadata, so a customer ticket can be traced back to the exact production trace in under a minute.
5. Production traces can be clustered by embedding of the user input, queryable in under a minute by anyone on call.
6. PII handling is written down, signed by security and legal, and enforced in the storage layer, not in a policy document nobody reads.
7. Production eval scores are computed continuously, or at least hourly, against a frozen golden set, with trend visible to on call.
8. Alerts fire on rate of change of probabilistic signals, not on absolute values.
9. The finish reason distribution is monitored as a leading indicator of behavioural change.
10. Per tenant cost and token usage are visible in near real time, so cost regressions surface as fast as latency regressions.
11. The on call runbook contains a "this is an LLM bug, not a code bug" branch that names the dashboards and traces to consult.
12. The team runs at least one debugging drill per quarter using only the observability stack, to confirm the instrumentation is sufficient to reach a root cause without out of band help.

A team that can answer yes to most of these can debug an LLM incident the way classical incidents are debugged: in minutes, with the data, without reverse engineering anything.

* * *

## The Architect's Mental Model

The instinct from classical observability is that you watch the system. The shift in an LLM system is that you watch the *behaviour*. The system is a thin wrapper around something that produces behaviour, and the behaviour is where the failures live.

That shift has a cost. Watching behaviour means capturing content, which means handling PII deliberately, which means having policy and infrastructure in place that classical observability did not need. It means treating prompt versions, model versions, and retrieval context as first class fields on every span. It means designing alerts around statistical signals that the on call has to learn to read differently from threshold alerts. None of this is free, and the temptation to skip it on the grounds that the classical stack still works is what produces the support ticket that opens this article.

One question to carry out of this article: if a production trace from twenty minutes ago needed to be retrieved, clustered, and explained to the customer experience team within the hour, could your current stack do it? If not, that is the next gap to close.

> *A deterministic system tells you what failed through a stack trace. A probabilistic system tells you through a customer's confusion. Observability for the second kind of system is the discipline of catching the failure before the customer's confusion reaches your queue.*

* * *