Evaluating an LLM app is more than checking whether a few answers sound good. This guide gives you a repeatable workflow for building test cases, scoring correctness and relevance, estimating latency and cost, reviewing safety, and deciding when a prompt, model, retrieval pipeline, or agent workflow is ready for production.
Overview
An LLM application can fail in several different ways. A chatbot may answer fluently but provide incorrect information. A RAG application may retrieve the wrong documents even when the generation step is strong. An AI agent may complete simple tasks while becoming unreliable when tools are unavailable or a user request is ambiguous. A single quality score cannot describe all of these risks.
A useful LLM evaluation separates the application into observable dimensions:
- Task success: Did the application produce the result the user needed?
- Correctness: Is the answer accurate according to a reference answer, source document, or business rule?
- Relevance: Did it address the request without unnecessary or unrelated content?
- Grounding: For RAG systems, are claims supported by the retrieved context?
- Completeness: Did it include the required fields, steps, or caveats?
- Safety and policy adherence: Did it avoid disallowed actions, sensitive disclosures, or unsafe behavior?
- Operational performance: How long did it take, how often did it fail, and what resources did it consume?
The goal is not to create a perfect benchmark. The goal is to make changes measurable. When you revise a system prompt, switch models, alter retrieval settings, or add a tool to an agent, the same evaluation set should show whether the change improved the application or merely changed its style.
For background on structuring prompts before testing them, see the prompt engineering guide. If you are testing prompts specifically, the LLM prompt testing framework provides a useful companion process.
How to estimate
Start with a small, representative test set rather than a large collection of easy examples. A practical first version may contain 20 to 50 cases distributed across the main tasks your application performs. Include ordinary requests, boundary cases, incomplete inputs, adversarial wording, and examples that previously caused failures.
For each case, record the input, expected behavior, and evaluation criteria. Then run the same cases against the current version and any proposed alternative. A simple weighted quality estimate is:
Overall score = (correctness × weight) + (relevance × weight) + (completeness × weight) + (safety × weight)
Use weights that reflect the consequences of failure. For a customer-support assistant, correctness and policy adherence may matter more than concise wording. For an extraction workflow, field accuracy and valid output format may matter more than conversational relevance. Keep the scoring scale stable; for example, use zero to two points for each dimension:
- 0: Fails the requirement or creates a material problem.
- 1: Partially meets the requirement and needs review or correction.
- 2: Meets the requirement with no material issue.
Estimate operational performance separately. Track median latency, high-percentile latency if available, error rate, average input and output size, and cost per successful task. The last measure is often more useful than cost per request because a cheap response that requires human correction may not be economical.
For an approximate cost model, use:
Estimated usage cost = (input units × input rate) + (output units × output rate)
Apply this calculation to a typical request, then multiply it by expected request volume. For applications with retrieval, include the added input from retrieved passages. For agents, estimate the number of model calls and tool-related turns per task rather than assuming every request uses one call.
Do not treat an automated evaluator as the final authority. Model-based grading can help process large test sets, but ambiguous, high-risk, or surprising results should receive human review. The most reliable workflow combines deterministic checks, automated semantic checks, and a sampled human audit.
Inputs and assumptions
A maintainable evaluation record should make its assumptions visible. A spreadsheet is sufficient for a small project, while a JSON or database schema is easier to version as the application grows. At minimum, store the following fields:
{
"case_id": "support-014",
"task_type": "policy_question",
"input": "Can I change the delivery address after ordering?",
"reference_answer": "Explain the address-change process and its limits.",
"required_behavior": [
"Answer from approved policy content",
"State uncertainty when order status is unavailable",
"Do not invent a guarantee"
],
"retrieved_context_ids": ["policy-12", "policy-19"],
"version": {
"prompt": "support-v4",
"model": "model-version-a",
"retrieval": "retrieval-v2"
},
"scores": {
"correctness": 2,
"relevance": 2,
"grounding": 1,
"safety": 2
},
"latency_ms": 1840,
"review_status": "human_reviewed",
"failure_reason": "One claim was not supported by retrieved context"
}Separate test data by purpose. A development set helps you iterate on prompts and code. A validation set helps compare candidate versions. A small, protected regression set should remain unchanged so that improvements do not hide old failures. If you repeatedly edit every test while optimizing, the score can become a reflection of your tuning process rather than application quality.
Make assumptions explicit before calculating results. Define what counts as a successful task, which fields are mandatory, whether partial answers pass, how refusals are scored, and how missing source evidence is handled. For RAG evaluation, distinguish retrieval failure from generation failure. If the correct passage was never retrieved, changing the prompt alone is unlikely to solve the problem. The related guide on semantic search versus keyword search can help when retrieval quality is the limiting factor.
For agents, add tool-call validity, argument accuracy, task completion, unnecessary-step rate, and recovery behavior. A response can be linguistically excellent while the underlying action is wrong. Log traces and intermediate tool results, not only the final message; the LLM observability guide covers this broader view.
Worked examples
Example 1: A support chatbot
Suppose a support chatbot has 30 test cases. You assign correctness a weight of 0.4, relevance 0.2, completeness 0.2, and policy adherence 0.2. Each dimension receives a score from zero to two. If the average scores are 1.6, 1.8, 1.4, and 1.9, convert each to a proportion of the two-point maximum:
Estimated quality = (0.8 × 0.4) + (0.9 × 0.2) + (0.7 × 0.2) + (0.95 × 0.2) = 0.81
This 0.81 estimate is not a universal pass mark. It is a decision aid. You might accept it for a low-risk internal assistant while requiring a higher result, plus manual review, for customer-facing policy answers. More importantly, the dimension scores show where to improve: completeness is weaker than policy adherence, so adding a response checklist or improving the source material may be more useful than changing the model.
Example 2: A RAG knowledge assistant
Imagine 40 questions drawn from internal documentation. The application retrieves five passages for each question. You measure whether at least one relevant passage appears, whether the answer is supported by those passages, and whether it answers the user’s question. If retrieval succeeds in 34 cases but the final answer is grounded in only 29, the pipeline has two separate rates: 85% retrieval coverage and 72.5% grounded-answer coverage.
The gap suggests that generation, context selection, or citation behavior needs attention. Review the 11 failures individually. Some may result from irrelevant passages, others from the model ignoring relevant context, and others from questions that the documentation does not answer. These categories should lead to different fixes.
Example 3: Cost and latency for an agent
Assume a task usually requires one planning call, two tool-related calls, and one final response. Estimate usage for all four calls, not just the visible answer. Record the average and worst observed number of turns, then calculate cost under both a typical and a heavier workflow. Also record latency for each stage. A tool that adds little cost but creates a long delay may still affect user experience; a tool that reduces hallucinations but increases calls may be worthwhile for high-value tasks.
Compare versions using the same cases and report quality, cost, latency, and failure categories together. This prevents a narrow optimization, such as reducing token usage while increasing unsupported answers.
When to recalculate
Run the full evaluation whenever you change the model, system prompt, retrieval method, chunking strategy, tool definitions, output schema, safety rules, or major application code. Recalculate cost when provider pricing, model limits, average context size, or expected traffic changes. Recheck latency after infrastructure changes or when agent workflows gain additional steps.
Use a smaller smoke test for every deployment and a broader regression run on a planned schedule. Add a new regression case whenever a production failure reveals a missing scenario. Record the failure, expected behavior, fix, and the version in which it was resolved. Do not delete difficult cases simply because a newer prompt performs well on them; they are valuable evidence of risk.
Review the evaluation design itself when the product changes. A test set built for a chatbot may not cover tool authorization, state management, or recovery behavior in an agent. Likewise, a RAG benchmark should be updated when the knowledge base, document structure, or user population changes.
To put this workflow into practice, create a versioned test file, select representative cases, define scoring rules before reviewing outputs, and run a baseline. Then change one major variable at a time, compare quality with cost and latency, inspect failures by category, and promote only the changes that improve the behavior you actually need. That discipline turns prompt testing from an informal demo into an ongoing evaluation system for reliable LLM app development.
