AI Evals: How to Measure AI Quality Before and After Launch

“It worked in the demo” is the most expensive sentence in AI projects. Demos use ten friendly questions. Production brings thousands of real ones — misspelled, multilingual, angry, sometimes hostile — plus model updates you never asked for and a knowledge base that changes every week.

That is why quality, not cost, is the top barrier for AI agents. In LangChain’s State of Agent Engineering 2026, 32% of teams name quality as the main obstacle to production, ahead of latency at around 20% (LangChain survey). Gartner expects more than 40% of agentic AI projects to be cancelled by the end of 2027 (Gartner). The teams that make it share one habit: they measure quality systematically, before and after launch. That habit is called evals.

This guide explains what evals are, how to build a first eval set in days, which scoring methods to use for which job, and how to keep measuring once real customers arrive.

What evals are — and why unit tests are not enough

An eval is a repeatable test of AI behaviour: a fixed set of inputs, a definition of what a good output looks like, a scoring method and a threshold. Classic software tests check that a function returns the expected value. AI systems are different:

• Outputs vary. The same question can produce different, equally correct wordings — and occasionally a wrong one.
• Changes have side effects. A prompt tweak that fixes refund questions can break shipping questions. A provider’s model update can change tone, length or tool use overnight.
• “Almost right” is the typical failure. In the 2025 Stack Overflow developer survey, 66% of developers named AI answers that are “almost right, but not quite” as their top frustration, and 46% said they distrust AI accuracy (Stack Overflow). Almost right is exactly what a demo will not reveal.

A mature setup has three layers:

1. Offline evals on a curated “golden” dataset during development.
2. Release gates that rerun the evals automatically on every change to prompts, models, tools or retrieval — and block the release on regression.
3. Online evals that score a sample of real production conversations continuously.

Step 1: Define “good” in business terms

Evals start with the business outcome, not with model metrics. For a customer support agent, the outcome is a correctly resolved ticket. For a document pipeline, it is a correctly booked invoice. For a voice receptionist, it is an appointment with the right time, name and service.

This matters commercially, too. Platforms already charge per outcome: Intercom Fin $0.99 per resolution (Intercom), Zendesk $1.50–2.00 per automated resolution (Macha), HubSpot $0.50 per resolved conversation since April 2026 (Resolve247). Analysts at HFS warn that the definition of a “resolved” interaction will be the next battleground in service-level agreements, and that it has to be written into contracts and made verifiable (HFS Research). If you pay or get paid per outcome, your eval definitions become part of the contract.

Break the outcome down into measurable dimensions:

• Correctness: facts, numbers, prices and policies match the source of truth.
• Groundedness: every claim is supported by retrieved documents; nothing is invented.
• Action accuracy: the right tool was called with the right parameters — order ID, amount, date.
• Policy compliance: no forbidden promises, the required AI disclosure is present, escalation happens when it should.
• Tone and language: answers come in the customer’s language and your brand voice.
• Latency and cost: within the budget for the channel.

Step 2: Build a golden dataset in days, not months

You do not need thousands of examples to start. A few dozen to a couple of hundred carefully chosen cases catch most regressions. Good sources:

• Real history. Past tickets, emails, call transcripts and documents, anonymised. They contain the questions customers actually ask, in the words they actually use.
• The long tail. Rare but expensive cases: cancellations, complaints, legal threats, VIP customers, mixed languages.
• “Must refuse” and “must escalate” cases. Out-of-scope requests, medical or legal advice, an angry customer asking for a human. An agent that answers everything is a liability.
• Adversarial cases. Attempts to extract the system prompt, talk the agent into discounts or smuggle instructions in via documents — see our guide to prompt injection defenses.
• Awkward formats. Scanned PDFs, photos of invoices, voice recordings with background noise.

For each case, write down the expected result: facts that must appear, statements that must not appear, the expected tool call with parameters, or the expected extracted fields. Version the dataset in Git like code, and add every production failure to it. The dataset should grow with every incident.

Step 3: Choose the right scoring method

Two tips. First, use deterministic checks wherever you can and save the LLM judge for what genuinely needs judgement. Second, calibrate the judge: have people label 50–100 cases, compare their labels with the judge’s scores, and refine the rubric until they agree. Never let a model grade its own answers with the same prompt.

Voice deserves special care. According to AssemblyAI, speech recognition in real conditions typically performs two to three times worse than in lab benchmarks (AssemblyAI), and telephone audio makes it worse still. Names, addresses, phone numbers and dates are where voice agents fail most often — so test exactly those.

Step 4: Thresholds and release gates

An eval without a threshold is a report nobody reads. Define per dimension what is acceptable, and keep averages separate from critical failures.

Here is an example eval plan for a customer support agent. The thresholds are illustrative; set your own according to risk.

Run the full suite on every change: prompt, model, tool definition, retrieval setting, knowledge base import. Block the release if a critical metric drops. When you compare models, run the same dataset and look at quality, latency and cost side by side. That is exactly the evidence an LLM gateway needs to move traffic to cheaper models safely.

Step 5: Keep measuring after launch

Launch day is when the real dataset starts arriving. Online evals close the loop:

• Score a sample of live conversations with the same rubric — daily for new systems, weekly once they are stable. Automated scoring makes full coverage affordable: manual QA in contact centres typically reviews only 1–2% of calls (AssemblyAI). As a price reference, Retell charges $0.10 per minute for automated QA of voice agents (Retell).
• Track behavioural signals. Escalation rate, repeat contacts within 48 hours, thumbs-down ratings, abandoned conversations, manual corrections in back-office tools.
• Watch for drift. A stale knowledge base is the most common cause of rising hallucinations a few months after launch; provider model updates come second. Re-run the offline suite after every update.
• Use confidence thresholds. Let the system hand over to a human when it is unsure. One CRM agent vendor, for example, escalates automatically when retrieval relevance drops below 0.40 (NOVA). The exact number matters less than having one and tuning it with data.
• Feed failures back. Every serious production error becomes a new test case, so it can never silently return.

What good results look like

Evals are what turn vague promises into numbers. A few published examples — all vendor-reported, so read them as direction rather than benchmarks:

• Fractional AI reports 80% fewer hallucinations in an AI system built for Zapier (Fractional AI).
• Vstorm cites a deployment where accuracy rose from 44% to 98% (Vstorm).
• Salesforce found that 70% of service organisations using AI agents see a measurable impact within the first 60 days (Salesforce).

What these cases share is not a better model but a measured loop: baseline, change, re-measure.

Who owns the evals

Evals fail most often for organisational reasons, not technical ones. Three roles make them work:

• A business owner — the head of support, operations or sales — decides what “good” means, signs off thresholds and accepts the trade-off between automation rate and risk.
• Domain experts — experienced agents, accountants, dispatchers — label cases and review samples. Two hours a week from the right person is worth more than a thousand synthetic examples.
• An engineering owner maintains the harness, the release gates and the dashboards, and makes sure every incident turns into a test case.

Put the monthly eval report on the same agenda as other operational KPIs. When quality numbers sit next to cost and volume, decisions about models, prompts and automation scope become fact-based instead of anecdotal.

Tooling: start simple

You can start with a spreadsheet of test cases and a script that calls your system and scores the outputs. As volume grows, add tracing and evaluation tooling. Langfuse or Helicone for traces and scores, Sentry for errors and PostHog for product signals are a common combination in production AI packages (HouseofMVPs).

Whatever you use, keep datasets, rubrics and results in systems you own, not only inside a vendor’s dashboard. After your data itself, they are the most valuable asset of an AI project: they let you switch models, vendors or agencies without starting from zero.

Typical mistakes

• Testing on the demo questions. If the dataset is easy, the score is meaningless.
• Grading text but not actions. A polite answer with the wrong refund amount is a failure.
• Hiding critical failures in averages. A 97% average can include a data leak.
• Uncalibrated LLM judges, or a model grading its own outputs.
• No refusal or escalation cases, so the agent is rewarded for answering everything.
• Evals once before launch, never again. Models, data and customers keep changing.
• Unversioned prompts and datasets, which make it impossible to know what caused a regression.

Conclusion

Evals are the cheapest insurance in AI. A golden dataset of real cases, a handful of well-chosen metrics with thresholds, automatic release gates and continuous scoring of live traffic turn an impressive demo into a system you can trust — and price. Most stalled pilots we see do not lack a better model; they lack a way to prove that the current one works. If that sounds familiar, our guide on getting AI pilots to production is the natural next read.

Our Systems team builds eval suites, release gates and quality dashboards for support, voice and document agents. Book a free 15-minute audit and we will review how your AI quality is measured today and where the blind spots are.