LLM Evaluation Guide: Build a Reliable Prompt Testing and Regression Framework

Overview

LLM evaluation is the process of measuring whether an application produces acceptable results for its intended use. In practice, that means testing more than whether an answer sounds fluent. A useful evaluation suite checks task quality, factuality, format compliance, safety, latency, and operating cost.

Prompt testing becomes valuable when an application has repeatable inputs and a defined output. Examples include classifying support tickets, extracting fields from documents, summarizing articles, generating product descriptions, or answering questions over a retrieval system. Without a test set, prompt optimization often becomes a sequence of subjective spot checks. A change may improve one example while quietly damaging several important cases.

A regression framework gives every change a consistent comparison point. Store representative inputs, expected properties, model settings, prompt versions, outputs, and scores. Then run the same cases against a new prompt, model, retrieval configuration, or parsing rule before release.

This approach complements broader guidance in How to Evaluate an LLM App and the practical examples in LLM Prompt Testing: A Practical Evaluation Framework.

How to estimate evaluation results

Start by turning the evaluation into a scorecard rather than a single pass-or-fail judgment. A simple scorecard can combine several dimensions:

• Task quality: Did the response complete the requested task?
• Factuality: Are claims supported by the supplied context or source material?
• Format compliance: Does the output follow the required schema, labels, or length limits?
• Safety: Does the application avoid disallowed, unsafe, or inappropriate behavior for its use case?
• Latency: How long does a request take, including retrieval, model generation, and post-processing?
• Cost: What is the estimated spend per request and over the expected request volume?

For a binary metric, calculate the pass rate as:

pass rate = passing test cases / total test cases

For a scored rubric, use:

average score = sum of case scores / number of evaluated cases

If you combine metrics into one decision score, document the weights. For example:

overall score = (quality × 0.50) + (factuality × 0.25) + (format × 0.15) + (safety × 0.10)

The numbers in this example are assumptions, not universal standards. A customer-facing answer may require a high factuality threshold, while an internal brainstorming tool may place more emphasis on usefulness and speed. In many applications, safety and hard format failures should be treated as release blockers instead of being averaged away.

For cost estimation, record the input and output token assumptions separately:

estimated request cost = (input tokens × input rate) + (output tokens × output rate)

When provider pricing is expressed in larger token units, convert consistently before calculating. For a monthly estimate:

monthly cost = estimated request cost × monthly request count

Keep pricing variables outside the test logic so they can be updated without changing historical evaluation results. This makes it easier to compare a model change with a pricing change.

Inputs and assumptions

A useful evaluation dataset is small enough to run frequently and broad enough to expose realistic failure modes. Begin with a golden dataset containing cases that the application must handle consistently. Each record should include:

• A stable test case ID
• The user input or source document
• Relevant context, retrieved passages, or tool results
• Expected facts, fields, or answer characteristics
• A rubric or pass condition
• Optional severity, category, and owner fields

Include ordinary cases, edge cases, ambiguous inputs, long inputs, missing information, malformed data, and adversarial requests relevant to your product. A dataset made only of easy examples can produce an inflated quality estimate.

Separate objective checks from judgment-based checks. Objective checks can validate JSON parsing, required keys, exact labels, citation presence, prohibited strings, or numerical ranges. Rubric-based checks can assess relevance, completeness, tone, and whether a response answers the question without unsupported additions.

LLM-based graders can help with open-ended quality, but they should not be treated as unquestionable truth. Use clear grading instructions, provide the relevant reference answer or evidence, and periodically review grader decisions manually. For high-impact workflows, sample both passes and failures for human review.

Track the assumptions that affect the estimate:

• Model and model version
• System prompt, user prompt, and few-shot examples
• Temperature and other generation settings
• Retrieval settings, such as chunking and number of results
• Average input and output length
• Expected request volume
• Latency measurement method
• Failure-handling and retry behavior

Version these inputs together. A prompt change can appear successful when the real improvement came from a different retrieval setting, a changed test case, or a more permissive parser. For retrieval-heavy applications, the production RAG tutorial provides useful context for evaluating the retrieval layer separately from generation.

Worked examples

Example 1: Structured extraction

Suppose an invoice extraction application has 40 golden test cases. A baseline prompt produces valid JSON for 38 cases, extracts all required fields correctly for 34, and introduces an unsupported value in 3 cases.

The baseline results are:

• Format pass rate: 38 / 40 = 95%
• Complete extraction rate: 34 / 40 = 85%
• Unsupported-value rate: 3 / 40 = 7.5%

A revised prompt raises valid JSON output to 40 cases and complete extraction to 36, but unsupported values rise to 5. The revision improves format and completeness while worsening factual reliability. It should not be approved solely because its average score is higher. Investigate the five cases, add similar examples if they represent a recurring failure, and consider a release rule that blocks any increase in unsupported values above an agreed threshold.

Example 2: Cost and latency comparison

Assume a test run contains 100 cases. The baseline averages 1,200 input tokens and 300 output tokens per request. A candidate prompt averages 1,500 input tokens and 360 output tokens. If the candidate also requires an additional retry for 8% of cases, estimate its effective request volume as:

100 × 1.08 = 108 request-equivalents

Use the provider’s current input and output rates as variables rather than embedding unverified prices in the test report. Calculate baseline and candidate costs with the same formula, then compare the difference. Also record median and high-percentile latency if your monitoring system supports them. An average latency can conceal a small group of requests that are too slow for an interactive workflow.

For prompt optimization, the best candidate is not automatically the one with the highest quality score. A modest quality improvement may be worthwhile if it reduces retries and manual review; a larger improvement may be unsuitable if it materially increases latency or cost. Make that trade-off explicit in the release decision.

When to recalculate

Run the full regression suite before deploying a new prompt, model, retrieval configuration, tool definition, output schema, or safety rule. Also run targeted tests when changing application code that affects truncation, parsing, retries, caching, or conversation history.

Recalculate cost estimates whenever model pricing, token limits, average input length, output length, request volume, or retry rates change. Recalculate latency estimates after infrastructure changes, provider changes, retrieval changes, or modifications to streaming and timeout behavior.

Refresh the golden dataset when real users reveal a new failure pattern. Do not replace old cases without preserving them; regression tests are most useful when previously observed failures remain in the suite. Review test coverage on a regular schedule and remove cases only when the underlying product behavior is intentionally retired.

For a practical workflow, store each run with a timestamp, code commit, prompt version, model identifier, dataset version, configuration, metric results, failed case IDs, and cost assumptions. Compare the candidate against the current production baseline, not only against an older experiment. A simple release checklist is:

1. Run deterministic format and safety checks.
2. Run rubric-based quality and factuality checks.
3. Compare pass rates and failure severity with the baseline.
4. Estimate cost and latency using current, clearly labeled assumptions.
5. Review important failures manually.
6. Record the decision and keep the evaluation artifacts.

As your application grows, connect these tests to production observability and user feedback. The LLM observability guide covers how logs, traces, and feedback can reveal cases that belong in the next dataset revision. A disciplined evaluation loop turns prompt engineering from one-off experimentation into measurable model optimization.