LLM Evaluation in Production: Why Vibe Checks Fail (2026)

Why Fora Soft wrote this playbook

We’ve shipped production LLM features since GPT-4 was new: ALDA, an AI course generator whose institutions serve 500,000+ students a year, built on the OpenAI Assistant API with a “Chain of Inquiry” prompting method; retrieval-augmented search inside BrainCert (100K+ customers, 500M+ classroom minutes); and voice agents on OpenAI Realtime and LiveKit Agents. Fora Soft is a software development company that has delivered 250+ projects since 2005 with a 50-engineer in-house team, and evaluation is built into how we do AI integration.

Here’s what every one of those had in common once it hit real users: the demo score was never the production score. A model upgrade that looked like a free win regressed three edge cases. A one-word prompt tweak broke tool-calling for a whole intent. The only way we caught these before customers did was an evaluation setup: a golden dataset, automated metrics, and a gate in CI that refuses to merge a regression.

This guide is that setup, written down: the five ways to measure LLM quality, where LLM-as-a-judge helps and where it lies, how to build the golden dataset everyone skips, and what the whole thing costs. It’s the playbook we hand a team that is running an LLM app in production and finding out about quality drops from support tickets.

What is LLM evaluation?

LLM evaluation is the practice of measuring the quality of a language-model application’s outputs against defined criteria, repeatably, so that every prompt, model or retrieval change is testable instead of guessed. In production it has three jobs: score outputs on a fixed dataset, catch regressions before release, and confirm that quality metrics actually track the business outcome you care about.

It is not the same as model benchmarking. Leaderboards like GPQA or MMLU-Pro rank base models on generic tasks (older ones like MMLU are now saturated above 88%, so they barely separate frontier models). LLM evaluation ranks your app on your task: the support bot answering your refund policy, the RAG search over your docs. A model that tops every leaderboard can still fail your use case, and only your own eval will show it.

Think of it the way a serious team treats testing: unit tests for prompts, integration tests for chains, and a regression suite in CI. If that framing is new, our non-functional-requirements checklist puts evaluation where it belongs — alongside latency, security and observability as a shipping requirement, not a nice-to-have.

Why vibe checks fail in production

A vibe check is asking the team “does this look good?” on a handful of queries. It works for a v0 demo and nothing past it. Three failure modes, all silent:

1. Sample bias. The team tries the queries it can think of — well-formed, in-domain, happy-path. Real users send typos, half-sentences, languages nobody planned for, and edge interpretations. The long tail is exactly where quality breaks, and a vibe check never visits it.

2. The anchor effect. Once everyone is happy with v1, every later change gets judged against v1, not against the user’s need. Slow degradation hides in plain sight. We’ve watched six months of “small” prompt and retrieval tweaks quietly drop long-tail accuracy by 15% with no one the wiser, because each step looked fine next to the step before it.

3. No regression testing. Someone bumps the model, edits a prompt, adds a retrieval filter. With no automated suite, the only feedback channel is customer complaints — and most unhappy users churn without ever filing one. You find out from the revenue chart, months late.

The fix is boring and it works: automated, golden-dataset-based, regression-tested evaluation wired into CI. Put it in once and every change becomes a measured change. Quality drift shows up in a PR comment instead of a churn report.

The five categories of LLM evaluation

There is no single “LLM score.” Evaluation splits into five method families, each catching a different class of bug. Mature deployments run three or more, cheapest-and-fastest first.

Figure 1. The five evaluation families. Signal quality and cost both rise from 01 to 05.

Which metrics actually catch bugs

Picking metrics is where teams waste the most effort. A practical rule: cheap automated metrics as fast gates, LLM-as-judge for anything open-ended, and human labels to keep the automated ones honest.

Reference-based (BLEU, ROUGE). They compare output to a reference string. Useful for translation and summarisation where a reference exists; weak for open-ended generation, where many correct answers share few words with the reference. Treat a low BLEU as “look closer,” not “wrong.”

Reference-free (classifiers). Toxicity, PII leakage, format/JSON validity, semantic similarity to the retrieved context. No ground truth needed, cheap enough to run on every request in production. This is your first line for filtering obviously-bad output.

Model-graded (LLM-as-judge). A separate model scores the output against criteria you write (“is this grounded in the context?” “does it answer the question?”). Powerful, cheap, and the workhorse of modern evaluation, with real biases we cover next.

RAG and agents need their own metrics. Generic scores miss retrieval and tool-call bugs entirely, so RAG evaluation uses RAGAS and agent evaluation uses tool-call accuracy. Both get dedicated sections later in this guide.

LLM-as-a-judge and its three biases

LLM-as-a-judge means using a strong model to grade another model’s output. It works better than most engineers expect: in the MT-Bench study, a GPT-4 judge agreed with human preference about 85% of the time in pairwise comparisons — higher than the ~81% agreement between two humans (Zheng et al., NeurIPS 2023). The G-Eval method reached a 0.514 Spearman correlation with human summarisation scores using chain-of-thought grading (Liu et al., 2023). That is good enough to run at scale for pennies.

Figure 2. A judge model matches human preference well — once you correct for three documented biases.

The catch is that judges are biased in ways you can measure and mitigate:

Position bias. In pairwise scoring, judges favour whichever answer appears first. Fix: score both orderings and average, so order cancels out.

Verbosity bias. Judges reward longer answers even when a shorter one is better — research finds this is a stronger pull than position bias. Fix: length-normalise, or cap answer length before grading.

Self-preference bias. A judge rates its own model family’s outputs higher, correlated with how familiar (low-perplexity) the text looks to it (Self-Preference Bias in LLM-as-a-Judge, 2024). Fix: judge with a different model family than the one that generated the answer — grade Claude output with GPT, or the reverse.

One rule holds across all three: never ship LLM-as-judge as your only metric. Anchor it against a small human-labelled set and at least one business metric, and re-check that anchor when you change the judge model.

RAG evaluation with RAGAS

Retrieval-augmented generation needs retrieval-aware metrics, because a RAG answer can be wrong for two very different reasons: the retriever fetched the wrong context, or the model ignored the right context. The open-source RAGAS framework splits this cleanly with four core metrics.

Figure 3. Two RAGAS metrics diagnose retrieval, two diagnose generation — so you fix the right half.

Context precision. Of the chunks you retrieved, how many are actually relevant, and are they ranked high rather than buried under noise? Target > 0.80. Low precision means your retriever is dragging in junk.

Context recall. Did retrieval find all the chunks that contain the answer? This one needs a reference set to compute, and it is the most diagnostic signal for retrieval quality — low recall means the answer was never in the context to begin with.

Faithfulness. Is every claim in the answer supported by the retrieved text? This is your hallucination detector. Target > 0.85. A faithful-but-irrelevant answer still fails, which is why you pair it with the next one.

Answer relevancy. Does the answer actually address the question asked, or wander off-topic? Target > 0.85. RAGAS computes these with an LLM judge under the hood, so the judge-bias rules above still apply. Run them on a fixed 100–200-question golden set and alert on any drop — the retrieval architecture this evaluates is in our RAG for video and audio guide.

Voice and agent evaluation

Voice agents and tool-using agents have quality dimensions a text app doesn’t, and skipping them is how a “working” agent still frustrates users.

Latency budget. For voice, track voice-to-voice latency at p50 (aim < 800 ms) and p95 (< 1.4 s), per turn, and alert on drift. A correct answer that arrives late feels broken.

Tool-call accuracy. Did the agent call the right tool with the right arguments? Score it per tool; aim above 96% success. This is the single most common place agents fail silently — the language looks fine, the action is wrong.

Interruption handling. When a user barges in mid-sentence, does the agent stop within ~200 ms? Track barge-in latency and false interruptions separately.

Containment and trace replay. Containment rate — calls resolved without a human handoff — is the business metric that matters for voice. Capture the full transcript (and audio) on every call so you can replay it later into the golden set or a regression run. The same discipline covers tool-use agents; see our MCP server guide and AI agent development work for the architecture.

How to build a golden dataset

The golden dataset is a fixed set of query–answer pairs you evaluate against every time. It is the hardest step and the highest-return one: done well, it is your most valuable LLM-engineering asset; done badly, every score it produces is noise. Five steps.

Step 1: define use cases. List 5–10 things your app actually does. For each, write 3–5 representative queries. That gives you ~30–50 happy-path items.

Step 2: mine edge cases. Pull the queries that surprised the team from production logs or beta sessions — misspellings, multilingual input, very long and very short prompts, edge interpretations. Aim for another 30–50. This is the half a vibe check never covers.

Step 3: expert annotation. For each query, a domain expert — not the engineer who built the feature — writes the correct answer or marks acceptable ones. Budget $1–$5 per item for general domains, $5–$20 for medical, legal or other specialised work.

Step 4: review and refine. The engineering team reviews labels for consistency. Disagreements are a gift: they expose ambiguity in the use-case definition itself. Iterate until labels are stable.

Step 5: maintain it. Add fresh queries from production each quarter, retire ones that no longer reflect the product, and re-label if the domain shifts. A golden set goes stale in about six months if you don’t feed it.

Regression testing in CI/CD

The highest-return move in this whole guide: run the eval suite automatically on every PR that touches a prompt, retrieval logic or model choice, and block the merge if scores regress past a threshold. This is the difference between observability (you find out) and prevention (it never ships).

Figure 4. The eval gate: a regression is blocked in the PR, not discovered in production.

What to gate on. Top-line metrics only — RAGAS faithfulness, answer relevancy, and one or two custom domain scores. Set a quality floor (faithfulness stays > 0.85) and a regression threshold (no metric drops more than ~3% from main). Gate on too many metrics and the pipeline turns into noise.

The pipeline. PR opens → CI runs the suite (5–15 minutes for a 100-question set) → results post as a PR comment plus a status check → the PR is mergeable only if the thresholds pass. A failing run tells the author exactly which scores dropped.

Cost control. Cache identical prompt/output pairs across runs; run the full suite on release branches and a smaller smoke suite on feature branches; batch non-blocking metrics into a nightly run. A 100-question RAGAS run costs about $0.50–$2 in judge tokens, so this stays cheap even at an active pace.

Production trace replay. Sample 5–10% of live traces into a “recent production” eval set and run it nightly. It catches drift the static golden set can’t, because production keeps inventing new inputs.

The eval tooling matrix for 2026

The eval tooling market grew up fast — Braintrust raised an $80M Series B in February 2026 at an $800M valuation, a sign that LLM observability and evaluation are now their own category. Here is how the tools we reach for compare. Prices are 2026 list and move often; confirm before you commit.

What LLM evaluation costs

A realistic first year runs about $4,050 for a single-feature RAG app, split across three cost buckets: a one-time golden-dataset build, recurring run cost, and tooling. Here is the worked version, all figures 2026 order-of-magnitude rather than a quote.

Figure 5. A realistic first-year eval budget — and what one caught regression is worth against it.

Setup. A 150-item golden set at $3/item of expert labelling is $450. Wiring in a platform and writing the first suite is one to two engineer-weeks. Specialised domains multiply the labelling cost three to five times.

Run. A daily 100-question run at ~$1.50 is about $540 a year. CI gating at ~120 PRs times $0.60 adds ~$72. Together, run cost is a rounding error against engineer time.

Tooling. A managed platform on a flat plan (Braintrust Pro) is $2,988 a year; a self-hosted OSS stack (Langfuse plus RAGAS) is near-zero licence plus ~$50/mo of infrastructure.

Add it up: $450 + $540 + $72 + $2,988 ≈ $4,050 for year one. Set that against the return: the mini-case below reversed roughly $40,000/month in churn the client had pinned on bot quality. One regression caught before customers feel it pays for the year. Eval is insurance, and the premium is small. Because we use Agent Engineering internally (AI agents doing much of the build), our own setup runs faster and cheaper than a hand-built one, but even priced conservatively the maths favours doing it.

Mini case: eval stops a silent quality slide

A B2B voice-AI support platform (under NDA, roughly 3M customer calls a month) came to us in late 2025 with climbing churn and one recurring complaint: “the bot used to work, now it gives wrong answers.” The team had spent six months tuning prompts and upgrading models with no evaluation setup, so no one could say which change caused what.

The four-week intervention. Week 1, we built a golden set of 240 query–answer pairs across eight use cases by mining six weeks of real calls, labelled by their own support managers. Week 2, we wired in Braintrust plus RAGAS, ran a baseline against current production, and set quality thresholds. Week 3, we put an eval gate on every PR. Week 4, we ran a six-month historical regression to find the exact prompt change that had started the slide — grading each version with a judge from a different model family to dodge self-preference bias.

Outcome. Faithfulness on the golden set had fallen to 0.71 after the bad prompt change; rolling it back and re-tuning brought it to 0.89. Customer-reported quality issues dropped ~60% the next month, and the churn the team had attributed to bot quality (about $40,000/month) stopped climbing within six weeks. The eval suite now catches that class of regression in the PR. Want the same audit on your app? Book a 30-min call and bring a few traces.

Pick your eval stack in five questions

Q1. Stack type? LangChain-heavy: LangSmith. Mixed or framework-agnostic: Braintrust. Self-hosted or compliance-bound: Langfuse or Arize Phoenix.

Q2. How many LLM features? One: open-source RAGAS or DeepEval is plenty. Two or three: a managed platform starts paying back. Five or more: a managed platform is no longer optional.

Q3. Compliance posture? Standard: any cloud option. HIPAA, SOC 2 or EU residency: self-host the eval data inside your VPC.

Q4. Team size? One to three engineers: code-first OSS. Four to fifteen: a managed UI earns its keep. Fifteen-plus: enterprise tier with role-based access.

Q5. Production volume? Under 10k calls/day: any tool. Over 100k/day: prioritise strong sampling and cost controls, or the eval bill outgrows the app. Still unsure? That is exactly the audit we run for free.

When not to over-invest in eval

Evaluation is insurance, and you can over-insure. Skip a full eval setup in three cases: pre-product-market-fit prototypes you rewrite weekly, apps with no golden data to label yet, and deterministic or low-stakes output a schema or regex already settles. Here is where our AI engineering teams say “not yet”:

Pre-product-market-fit prototypes. If you’re changing the whole app weekly and have fewer than a handful of users, a 20-line spot-check script beats a golden dataset you’ll throw away next sprint. Build the suite when the app stops moving daily.

No golden data yet. Gating CI before you can label “good” just blocks merges on noise. Ship tracing first, collect real traces for a few weeks, then build the golden set from them.

Deterministic or low-stakes output. If a schema check, regex or exact match already settles correctness, don’t reach for an LLM judge; it’s slower, costlier and less reliable than the deterministic check you already have.

The honest version: start with tracing and a tiny reference set, and grow the eval setup as the stakes grow. Over-building eval on a toy is as much a waste as shipping a payments flow with none.

Five pitfalls to avoid

1. LLM-as-judge as the only metric. Its biases (position, verbosity, self-preference) mean a single judge score can drift with the judge model. Always pair it with a reference-based or business metric and periodic human labels.

2. A golden dataset that goes stale. Add fresh production queries quarterly. Without a refresh you’re grading yesterday’s app against yesterday’s inputs.

3. No CI integration. An eval that runs nightly catches regressions a day late; an eval that gates PRs catches them before merge. The first is a dashboard, the second prevents drift.

4. Engineers labelling their own data. The people who built the feature label charitably. Use domain experts or real users as proxies, so the labels are rigorous instead of hopeful.

5. Treating eval as a one-time project. Stand the suite up as v0, then iterate the dataset, metrics and gates as the product evolves. Eval is a habit, not a milestone.

KPIs worth tracking

Quality KPIs. Top-line eval scores (faithfulness > 0.85, answer relevancy > 0.85), the regression rate (share of PRs that fail the gate), and time-to-detect for production drift (target under 24 hours).

Business KPIs. Customer-reported quality issues per month (should trend down), containment rate for voice or task-success rate for text agents, and churn you can attribute to LLM quality.

Reliability KPIs. Eval-pipeline uptime (99%+) and the false-positive rate on quality gates (under 10% — gates that block good PRs burn developer trust fast).

FAQ

What is LLM evaluation?

LLM evaluation is measuring the quality of a language-model application’s outputs against defined criteria, repeatably, so every prompt, model or retrieval change is testable. In production it scores outputs on a fixed golden dataset, catches regressions before release, and confirms quality metrics track the business outcome.

How do I evaluate an LLM app without reference answers?

Use reference-free metrics: an LLM-as-judge scoring against a rubric, semantic similarity to the retrieved context, and classifiers for toxicity, PII or format. RAGAS combines several of these. They are less precise than reference-based scoring but apply everywhere.

Is RAGAS enough on its own?

For RAG apps, RAGAS covers retrieval and generation quality well with four metrics: context precision, context recall, faithfulness and answer relevancy. Pair it with a tracing platform for production observability, and Braintrust, LangSmith or DeepEval for CI gating.

Braintrust or LangSmith?

Braintrust suits non-LangChain, multi-feature teams and its flat $249/mo (no per-seat) plan; LangSmith suits LangChain-heavy stacks that want framework-native tracing and start free. Pick by stack alignment and team size, not features — both are strong.

Can a model evaluate its own output?

Yes — that’s the LLM-as-judge pattern — but a model shows self-preference bias toward its own family’s outputs. Best practice is to judge with a different model family (grade GPT output with Claude, or the reverse) and re-check against human labels periodically.

How big should my golden dataset be?

100–200 query–answer pairs for a v1. Under 50 has too much variance to trust; above 500 the marginal value drops, so refresh quarterly instead of over-investing in one huge static set.

How much does production LLM evaluation cost?

A realistic first year is around $4,050 for a single-feature RAG app: ~$450 golden-dataset labelling, ~$540 daily runs, ~$72 CI runs and ~$2,988 for a managed platform. Self-hosting OSS tools cuts tooling to near-zero plus infrastructure.

How is LLM evaluation different from model benchmarking?

Benchmarks (GPQA, MMLU-Pro, SWE-bench) rank base models on generic tasks. LLM evaluation measures your app on your task and data. A model that tops every leaderboard can still fail your use case, and only your own eval will show it.

How long does it take to set up production eval?

Greenfield: 2–3 weeks for a golden dataset, platform integration and CI gates. Retrofitting an existing app is 4–6 weeks because of golden-dataset construction. Reusing patterns from prior LLM apps, we typically deliver in 3–4 weeks.

Should I evaluate every LLM call in production?

No — sample. A 5–10% eval-score sampling rate balances cost and coverage; trace 100% for debugging. Raise sampling to 100% only on high-risk paths like medical, financial or legal advice.

What to read next

Ready to ship LLM apps that hold quality?

LLM evaluation is what separates apps that improve from apps that quietly rot. Vibe checks miss the long tail, normalise to the team’s taste and never regression-test — so quality drifts until churn shows up. The fix is a golden dataset, three or more of the five metric families, LLM-as-judge used with its biases corrected, and an eval gate in CI that blocks the regression before it merges.

None of it is expensive: about $4,050 for a first year, against churn measured in tens of thousands a month. Build the golden set, pick the tooling by stack and compliance, gate the pipeline, and re-check against human labels. That is the whole playbook — and it’s the one we run on our own production LLM work.