Evaluating LLM Systems

🗓️ Last updated: July 2026

The team has spent six weeks on a model upgrade. Every internal eval is green. The new model beats the old one on the golden set by four points. It beats it on the regression suite by two. The LLM-as-judge evals are a clean sweep. The deploy gate flips. The new model goes live to ten per cent of traffic.

Seventy-two hours later, the customer-experience team escalates. CSAT is down three points on the cohort. Refund-eligibility questions are getting confidently wrong answers. The old model never made that class of mistake. The eval suite still says everything is fine.

Nothing in the suite was wrong. Every test it ran, it ran correctly. The problem was that the suite measured what was easy to measure — accuracy on a curated set, faithfulness on a synthetic RAG benchmark, a handful of hand-picked failure cases — and the production workload had drifted past it. The team had a measurement system. They did not have an evaluation system.

This article is about the difference.

What this article is NOT. It is not a survey of every eval framework on GitHub. It is not "ten metrics every LLM team should track." It is the architectural argument that LLM evaluation has to be a portfolio of techniques operating on four different timescales — offline, online, regression, drift — and that shipping with only one of those four is the most common quality failure in the field today.

Requirements: What an Eval System Has to Answer

An eval system is not a metric. It is a body of evidence — gathered on different timescales, measuring different properties — that supports a single architectural decision: should this change ship?

Five things any production-grade eval system has to answer, ordered by how often you will need them:

1. Has anything regressed? When you upgrade a prompt, a model, a retriever, an embedding model, or an upstream tool, what behaviour, if any, got worse?
2. Is the system getting worse over time without anyone changing anything? Models get deprecated. Vendor backends drift. The world the model was trained on dates. Drift is invisible without an eval system that runs continuously, not just on PR.
3. Are real users having a worse experience than the eval suite predicted? Offline evals measure curated inputs. Production inputs are not curated. The gap between the two is where most shipping mistakes live.
4. Where, specifically, did it get worse? A two-point drop on a fifty-question golden set tells you very little unless you can cluster the failures by topic, intent, or user cohort.
5. Is the eval itself still measuring the right thing? Golden sets rot. Judges drift. Regression suites grow stale. An eval system without a maintenance discipline becomes a measurement system you can no longer trust.

Notice what is missing: there is no requirement for "a single eval score." A single score is for press releases, not architecture.

The Classical Baseline (Tests Don't Generalise)

Classical software testing was built around three assumptions: the system is deterministic, the input space is enumerable, and the correctness of an output is decidable in code.

LLM systems break all three.

• Determinism: Even at temperature zero, two requests to the same model can produce different outputs. The inference stack — batching, routing, vendor-side caching — is non-deterministic at the system level even when the math is deterministic at the model level.
• Enumerability: The input space is natural language. You cannot enumerate it. You can only sample.
• Decidability: "Is this answer correct?" is sometimes trivially yes-or-no (math, code execution, retrieval recall) but more often a judgement call. Was the tone right. Did it cite the right source. Did it refuse something it should have refused. Did it overcorrect on something it should have answered.

The classical playbook — unit, integration, end-to-end tests — does not generalise. It still has a role, especially for the deterministic plumbing around the model (the retriever, the schema validator, the rate limiter, the tool dispatcher). But it is not enough. What replaces it is a portfolio: a set of evaluation techniques, each strong where another is weak, each running at the timescale where it is most useful.

The Four Axes

Every credible LLM eval program operates on four axes. They are not interchangeable, and a program missing any one of them has a class of failure it cannot catch.

Offline evals run before deploy. Curated inputs, known-good outputs (or known-bad outputs), and a scoring function — code, heuristic, LLM-as-judge, or human review. Cheap to iterate, fast to run, and they answer one question: has the candidate change made anything worse on the inputs we know we care about?

Online evals run in production. They measure what real users experience: thumbs up or down, copy rate, retry rate, escalation rate, task-completion rate, latency, refusal rate, downstream business metrics. Slow to converge, statistically noisy, but the only signal that catches the thing your offline suite did not think to test.

Regression suites are a specific kind of offline eval: a curated archive of historical failures, each one a frozen test that prevents a known bug from reappearing. They grow over time. They are the institutional memory of every bad bug the team has shipped and learned from.

Drift detection is the longest-timescale axis: continuous (or scheduled) re-runs of the offline suite against a frozen model, prompt, and configuration, to detect when the vendor or the world has changed underneath you. A passing eval on Monday and a failing eval on Friday — same prompt, same code, same model name — means something changed at the vendor, or something changed in the eval data itself.

Ship without all four and you are missing a class of failure that someone else's program would have caught.

Offline Eval Done Right

The most common offline-eval mistake is treating it as a single technique. It is at least three.

Golden sets are the foundation: a curated collection of inputs paired with expected behaviour. For tasks with deterministic correctness — extraction, classification, code generation that can be executed — the scoring is straightforward. For open-ended tasks (summarisation, support replies, recommendations), the expected behaviour is more often a rubric than an answer.

The discipline around golden sets matters more than their size. A 200-item set that is curated, versioned, and reviewed quarterly is more valuable than a 5,000-item set that nobody owns. Two specific practices separate teams that get value from golden sets from teams that do not:

• Adversarial curation. Every shipped bug becomes a golden-set entry within a week of the incident. The set grows because the system grew, not because someone bought more annotations.
• Diversity over volume. A golden set that does not include the long tail of user intents will reward models that handle the head and penalise nothing for failing the tail. The tail is where most production complaints originate.

LLM-as-judge evaluators are the workhorse for open-ended outputs. The Zheng et al. 2023 paper that formalised the technique (Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena) is required reading. It also names the failure modes — position bias, verbosity bias, self-enhancement bias — that every team eventually rediscovers the hard way. Three practices that turn an LLM judge from a toy into a working tool:

• Use a stronger model than the one being evaluated. Judging is harder than generating.
• Calibrate against a human-judged sample at least quarterly. If the judge and the humans diverge, the judge has drifted.
• Treat the judge prompt as code: versioned, reviewed, eval-gated. A new judge prompt is a new eval system; do not treat it as a config tweak.

Component evals are the technique most teams skip. If your system is a chain — retrieve, then rerank, then synthesise, then cite — evaluating the end-to-end output tells you the system is broken without telling you which component broke it. RAGAS-style component evals (faithfulness, context relevance, answer relevance, context recall) decompose the end-to-end metric into the contributions of each link. The cost is more eval scaffolding. The benefit is that when something regresses, you know where to look.

Online Eval

Offline evals tell you whether the candidate is better on the inputs you thought of. Online evals tell you whether it is better on the inputs your users actually send.

The most useful online signals are not the ones teams reach for first.

• Implicit signals beat explicit signals. The user's next action is more honest than the thumbs-up button. Did they rephrase the question. Did they copy the output. Did they retry. Did they escalate to a human. Did they come back tomorrow. These signals are noisier than a survey, but they are not biased toward users who fill in surveys.
• A/B is necessary but slow. Statistical power on a five-per-cent uplift in a noisy metric can take weeks of traffic. That window is longer than most teams want their model upgrades to wait. The honest answer is to run shorter A/Bs on stronger signals (a clear regression shows up fast) and longer A/Bs on subtler signals (CSAT, retention).
• CSAT lies in known ways. Customers who got the wrong answer confidently rate the experience higher than customers who got the right answer hesitantly. Confidence is a confound. Online evals that do not correct for it overstate the quality of the more confident model — which is often, but not always, the better one.

Online evals require an instrumentation discipline that most teams underbuild. The full prompt, the model and version, the retrieval context, the latency, the cost, and the user's next action all have to be captured, joined, and queryable. If your observability stack cannot answer "which prompts in the last hour produced an output the user immediately retried," your online eval is theatre.

Article 3 in this series covers the instrumentation side in detail.

Regression and Drift Detection

Regression is the easy half. Every previously-shipped bug becomes a frozen golden-set entry. Every PR that touches the model, the prompt, the retriever, or the schema runs against the regression suite. A failure blocks the merge. This is unglamorous and it works.

Drift detection is the harder half, and the one almost every team gets wrong.

The shape of the problem: nothing on your side changed. Same model name, same prompt, same retrieval. Three weeks later, the eval scores drop. Vendors update their inference stack, change their default safety filters, retire a model version, swap a tokenizer, change a backend GPU SKU. None of those changes appear in your change log. All of them can shift behaviour.

The defence:

• Pin model versions explicitly. "gpt-4" is not a model version. "gpt-4-turbo-2024-04-09" is. Same for every other vendor — latest tags are convenience features that defeat the entire purpose of versioning.
• Run the offline suite on a schedule. Daily, or at least weekly, against a frozen configuration. Plot the time series. Drift is a trend, not an event.
• Alert on the trend, not the absolute score. Eval scores fluctuate; the eval-of-the-eval is noisier than people expect. A two-standard-deviation drop sustained over three runs is a signal. A single-run dip is not.
• Freeze deploys when drift is detected. Until the source is isolated, every deploy compounds the diagnostic problem. The vendor change-log is the first place to look.

Drift detection is also the only eval axis where the vendor can be the bug. Article 5 in this series covers what to do then.

Failure Modes (The Section You Actually Need)

Eval programs fail in ways that look fine on dashboards. Seven recurring failure modes, severity-ordered:

1. The eval suite becomes the product. Goodhart's Law in action: when a measure becomes a target, it ceases to be a good measure. The model is tuned, the prompt is tuned, the golden set is curated, all to push the eval score up. The score goes up. User satisfaction does not. By the time someone notices, the eval has been the target for months and the team no longer trusts it.
2. Judge collapse. The LLM-as-judge starts agreeing with itself on outputs it should not agree with. Long-winded answers beat concise correct ones. Formal tone beats accurate informal tone. Verbose justifications beat short truthful ones. Without a recurring human-calibration check, the judge drifts and the eval drifts with it.
3. The frozen golden set. The set was curated when the product launched. The product is now used differently. The set is still measuring the launch-era use case. Nothing in the eval notices that the set itself is stale.
4. The single-number trap. The team ships when "the eval" passes. There is no per-cluster, per-intent, or per-cohort breakdown. A regression on a small but high-value user segment is averaged away by the rest of the set.
5. The online-eval blind spot. Offline evals are green. Online metrics are flat or worse. Nobody investigates why offline and online disagree. The answer is almost always that the offline distribution does not match the production distribution, and nobody is generating new golden cases from production traffic.
6. Latency-and-cost amnesia. The eval suite measures quality. It does not measure latency or per-token cost. The new model is two points better on quality and three times more expensive, and nothing in the eval pipeline forces that trade-off to the surface before the deploy gate flips.
7. The eval that never goes red. Some team somewhere is running an eval suite right now whose lowest score in twelve months is 94 per cent. It is not because the system is excellent. It is because the suite has been tuned, over time, to the system. An eval that does not catch regressions does not catch anything.

Cost Economics of Evals

The eval suite has a cost iceberg.

Above the waterline:

• LLM-as-judge calls, often using a stronger and more expensive model than production
• Compute for retrieval, reranking, and generation on every golden-set item, every run
• Storage and infrastructure for the eval platform and the historical eval data

Below the waterline, and bigger:

• Engineer time to curate golden sets, maintain judge prompts, investigate regressions, and re-baseline after model upgrades
• The deploy delay an honest eval pipeline introduces — between a few minutes (cheap, partial coverage) and several hours (full coverage, multi-judge consensus, statistical significance)
• The opportunity cost of model upgrades that get blocked while the eval suite is being updated to evaluate them

The numbers vary by workload, but a useful heuristic: budget the eval program at roughly five to fifteen per cent of your inference spend. Less than five per cent and you are flying blind on regressions. More than fifteen per cent and your eval program is probably measuring inputs that do not matter to production.

The cost of an eval program is real. The cost of not having one is a CSAT regression you cannot explain to the customer-experience team — and an architecture conversation you cannot win in the next planning meeting.

The Architect's Checklist

Twelve items. If you cannot say yes to most of them, your eval program has gaps.

1. Every shipped bug has been added to the regression suite within a week of the incident.
2. Golden sets are versioned, reviewed quarterly, and have a named owner.
3. LLM-as-judge prompts are versioned and calibrated against human review at a defined cadence.
4. The eval suite reports per-cluster or per-intent breakdowns, not just a single score.
5. Offline evals include component-level metrics, not only end-to-end.
6. Online evals capture implicit signals — retry, copy, escalate — not only thumbs up or down.
7. A/B tests have pre-registered metrics and minimum-detectable-effect sizing.
8. Model versions are pinned explicitly, with no use of vendor latest tags in production.
9. The offline suite runs on a schedule against a frozen config, and drift is plotted over time.
10. Eval cost is tracked as a line item in the inference budget.
11. The eval pipeline is part of the deploy gate; no model or prompt change ships without passing.
12. The team treats the eval suite as code: reviewed, versioned, and maintained as carefully as the system it measures.

The Architect's Mental Model

Classical software is evaluated against a specification. LLM systems are evaluated against a distribution — the distribution of inputs the users will send, and the distribution of outputs that count as good enough. Neither distribution is enumerable. Both shift over time. Your eval program is the apparatus that keeps you in contact with the parts of both distributions you care about.

The teams that ship probabilistic systems with confidence are the teams that built an eval program before they built the second feature. The teams that struggle are the teams that built features for a year, then tried to retrofit evaluation onto a system that already had a quality problem — which, at that point, the eval system is asked to diagnose rather than prevent. Those are not the same job.

Offline evals tell you yesterday's truth. Online evals tell you today's. Ship without both, and you are shipping blind.

What's Next

This is the first article in Shipping AI Systems — the third series in The Pragmatic Stack, focused on the operational discipline that turns a working prototype into a system that survives production.

The series ahead: