Search for a command to run...
Series
How to ship LLM systems that survive production. Evaluation, versioning, observability, rollouts, incident response, cost guardrails.
The new model swept the golden set, the regression suite, and the LLM as judge evals. Then CSAT dropped. The eval did not lie. The eval measured exactly what it was built to measure. That was the problem.
Nothing shipped. The git log is clean. The model is behaving differently anyway. In an LLM system, that is not a mystery. It is a versioning failure with a known architecture, and this article is that architecture.
Every code path executed correctly. The HTTP response was 200. The answer was wrong. Classical observability has nothing useful to say about that failure. This article is about the instrumentation discipline that does.
It cleared every eval, aced the canary, and shipped on a Wednesday. By Friday it was mailing blank invoices to customers. The model did not regress. It just quietly stopped honouring a contract nobody had bothered to write down.
The dashboards are healthy, the error rate is zero, nothing shipped this week, and the model just told a customer something expensive and wrong. There is no commit to revert. Welcome to the incident that classical on call was never designed for.
A recursive agent spent the weekend asking itself the same question, failing the same check, and billing you for both. No error fired, because no metric you watched had the word stop in it. This article is about putting it there.
