LLM Evaluation
LLM evaluation ("evals") is how you measure whether an LLM application actually works — and, crucially, whether a prompt change, model upgrade, or new retrieval step made it better or worse. Without evals, you're shipping on vibes: a tweak that fixes one case silently breaks five others, and you find out from users.
Evaluation is the discipline that turns LLM development from art into engineering. It's the difference between "this prompt feels better" and "this prompt raised faithfulness from 82% to 91% on our 300-case set with no regressions." This page covers how to build eval sets, score outputs (including with the model itself as judge), and wire evals into your development loop.
TL;DR
- You cannot improve what you don't measure. Evals convert subjective "seems better" into a number you can track across changes.
- Build a representative eval set early — real inputs, expected behaviors, and the failure cases that bit you. Grow it every time something breaks in production.
- Score with the right tool: exact/structural checks for deterministic outputs, LLM-as-judge for open-ended quality, human review for the highest-stakes calls.
- Split offline evals (a fixed dataset, run on every change) from online evals (production monitoring on live traffic).
- LLM-as-judge is powerful but biased — calibrate it against human labels, use a strong judge model, and give it a concrete rubric.
- Run evals in CI so a prompt or model change can't merge if it regresses your key metrics.
Quick Example
The core loop: a dataset of cases, a task function, and a scorer. Here, an LLM-as-judge grades faithfulness of a RAG answer:
Run this on every prompt or model change and watch the number.
Core Concepts
The Anatomy of an Eval
Every eval has three parts:
The scorer produces a per-case result; you aggregate into metrics (accuracy, faithfulness rate, average score) and track them across versions.
Offline vs Online Evaluation
- Offline evals run a fixed dataset through your task on demand — before merging a change, on a schedule, or in CI. They answer "did this change help or hurt?" Deterministic and repeatable.
- Online evals score a sample of live production traffic — measuring what users actually experience. They catch drift, edge cases your dataset missed, and real-world distribution shifts.
You need both. Offline catches regressions before ship; online catches the reality your dataset didn't anticipate. Feed online failures back into the offline set.
What to Measure
The metric must match the task:
Building an Eval Set
A good eval set is representative, labeled, and versioned. Practical guidance:
- Start small and real. 20–50 real inputs beat 1,000 synthetic ones. Pull from actual logs, support tickets, or user sessions.
- Include the failures. Every time something breaks in production, add that case to the set. Your eval set is a growing record of "things that must keep working."
- Cover the distribution. Easy cases, hard cases, edge cases, adversarial cases. If 10% of real traffic is non-English, 10% of your set should be too.
- **Label with expected behavior, not just expected text.** For open-ended tasks, the "answer" is a rubric ("must cite the refund policy, must not promise a specific date"), not a golden string.
- Version it. The eval set is code. Review changes to it — silently deleting a hard case to make a number go up is cheating yourself.
💡 Tip: Don't wait for a "complete" eval set. Ship 30 cases this week and grow it. An imperfect eval you run on every change beats a perfect one you never build.
LLM-as-Judge
For open-ended outputs (summaries, chat answers, generated code explanations), there's no golden string to match against. LLM-as-judge uses a model to grade outputs against a rubric — scalable and surprisingly correlated with human judgment when done well.
Making the Judge Reliable
- Give a concrete rubric, not "is this good?" Break quality into specific, checkable criteria ("Does the answer cite a source? Is every claim supported by the context?"). Vague prompts produce noisy scores.
- Prefer binary or low-cardinality judgments. "Faithful: yes/no" is more reliable than "rate 1–10." Humans and models both struggle to place a consistent 7 vs 8.
- Use a strong judge model. A frontier model (e.g. Claude Opus 4.8) is a better judge than the model being evaluated. Judging is easier than generating, but not free.
- Calibrate against humans. Hand-label 50 cases, run the judge on them, and check agreement. If the judge disagrees with humans 30% of the time, fix the rubric before trusting it at scale.
- Score each criterion independently. A judge asked to grade twelve things at once produces mush. One criterion per call (or per structured field) is more reliable.
Known Biases
LLM judges have documented biases: position bias (favoring the first option in pairwise comparisons — randomize order), length bias (favoring longer answers — control for it in the rubric), self-preference (favoring outputs from the same model family), and verbosity/formatting halo effects. Calibration against human labels is how you catch these.
Best Practices
Evaluate the Whole Pipeline and Its Parts
A RAG system can fail at retrieval or at generation. Measure each: retrieval quality (did the right chunks come back?) separately from answer quality (given the chunks, was the answer faithful?). A single end-to-end number hides where it's broken.
Track Metrics Over Time, Not in Isolation
A 91% faithfulness score means nothing alone. The value is the trend: 82% → 91% after a change is a win; 91% → 85% is a regression that should block a merge. Store results per version and plot them.
Put Evals in CI
Wire your offline eval into the pipeline: on every PR that touches a prompt, model, or retrieval config, run the eval set and fail the build if a key metric drops below threshold. This is the single highest-leverage practice — it makes regressions impossible to merge accidentally. See CI/CD.
Re-Run Evals on Every Model Upgrade
A new model version can change behavior in ways your prompt didn't anticipate — different verbosity, different tool-use rate, different formatting. Never swap models in production without running your full eval set first.
Keep a Held-Out Set
If you tune prompts against your eval set repeatedly, you overfit to it. Keep a held-out set you evaluate against occasionally to check that gains are real, not just eval-set memorization.
Common Mistakes
Judging With a Vague Rubric
Testing Only Happy Paths
An eval set of easy questions gives a comforting number and zero signal. If your set has no adversarial inputs, ambiguous cases, or out-of-scope questions, it won't catch the failures that matter.
Shipping Model Upgrades Without Re-Evaluating
FAQ
How many eval cases do I need?
Start with 20–50 real, well-chosen cases and grow from there. Representativeness matters more than volume — a small set that covers your real distribution and known failure modes beats a huge synthetic one. Add a case every time production surprises you.
Is LLM-as-judge trustworthy?
Trustworthy after calibration. Hand-label a sample, measure agreement with the judge, and fix the rubric until it's high. Use a strong judge model, ask for binary judgments against concrete criteria, and randomize option order to fight position bias. Uncalibrated, it produces confident-looking noise.
What's the difference between evals and unit tests?
Unit tests assert deterministic behavior (add(2,3) == 5). Evals measure distributions of non-deterministic output against thresholds ("faithfulness ≥ 90% across 200 cases"). Both belong in CI, but evals tolerate variance and track metrics rather than asserting exact equality. See Unit Testing.
How is this different from model evaluation in ML?
Model evaluation in classic ML measures a trained model against test data with metrics like accuracy and AUC. LLM evaluation measures an application built on a pre-trained model — the prompt, retrieval, tools, and orchestration around the model — and leans heavily on LLM-as-judge for open-ended output. They share statistics but differ in what's under test.
Should I evaluate offline or online?
Both. Offline evals (fixed dataset, run in CI) catch regressions before you ship. Online evals (scoring sampled production traffic) catch drift and real-world edge cases your dataset missed. Feed online failures back into the offline set.
Related Topics
- Prompt Engineering — What evals let you optimize with confidence
- RAG — The pipeline most in need of faithfulness and relevance evals
- AI Agents — Evaluating multi-step trajectories and tool-call correctness
- Model Evaluation — The classic-ML counterpart: metrics for trained models
- CI/CD — Where offline evals belong: gating every change
- Observability — Online evaluation is monitoring for LLM apps
- AI Guardrails — Adversarial eval sets for safety and injection resistance