RAG Tutorial: Build a Production-Ready Retrieval-Augmented Generation App

Overview

RAG combines information retrieval with text generation. Instead of asking a language model to answer only from its learned parameters, the application searches a private or changing knowledge base and includes relevant passages in the model request. The model then uses those passages to formulate an answer, ideally with citations or links back to the original material.

A practical RAG system usually contains these stages:

Documents → Parse and clean → Chunk → Embed → Store in index
                                                ↓
User question → Rewrite or classify → Retrieve → Rerank → Build context
                                                            ↓
                                                   LLM answer + citations
                                                            ↓
                                                     Logs and evaluation

The pipeline has two distinct paths. The ingestion path prepares documents before a user asks a question. The query path interprets a question, searches the index, constructs a bounded context window, and generates a response. Keeping these paths separate makes it easier to update documents, reprocess failed files, and diagnose whether a problem came from retrieval or generation.

Start with a narrow use case and a defined answer policy. For example, an internal policy assistant might answer only from approved documents, cite the document title and section, and say that it cannot find supporting information when retrieval is weak. Clear boundaries are more useful than a general instruction such as “be accurate.” For prompt design patterns, see the practical prompt engineering guide.

What to track

1. Source quality and ingestion health

Track every document as an identifiable record with a stable source ID, title, version or modification time, access scope, and ingestion status. Store enough metadata to answer basic operational questions: Was this file processed? Which parser handled it? When was it last indexed? Which chunks belong to it?

Monitor the number of documents discovered, successfully processed, skipped, and failed. A sudden change can indicate a broken connector, an unsupported file type, a permissions problem, or an accidental deletion. Preserve the original file or source reference where possible so that a failed record can be reprocessed without manual reconstruction.

2. Chunking and index coverage

Chunking affects both retrieval and answer quality. Chunks that are too small may separate a heading from its explanation; chunks that are too large may contain several unrelated topics and consume the context window. Begin with boundaries that respect document structure, such as headings, paragraphs, list items, and table rows. Use overlap carefully rather than treating it as a substitute for meaningful segmentation.

Track chunk counts by document and, when practical, inspect a sample of chunks after each ingestion run. Look for truncated sentences, duplicated text, empty chunks, lost table labels, and metadata that is missing from the searchable record. If a document changes substantially, re-embedding it is usually safer than trying to patch individual chunks without a clear versioning strategy.

3. Retrieval quality

Log the query, retrieval method, filters, returned source IDs, similarity or ranking values, and the final passages provided to the model. Do not log sensitive content indiscriminately; apply redaction and access controls before storing traces.

Use a small evaluation set of realistic questions with expected source documents. Useful retrieval measures include whether the expected source appears in the top results and whether the returned passages contain enough information to support an answer. Review both successful and failed queries. A high similarity score does not guarantee that a passage answers the question, especially when the query is vague or the corpus contains near-duplicates.

Compare vector search with keyword or hybrid retrieval for exact names, product codes, legal phrases, and error messages. The guide on semantic search versus keyword search provides a useful way to decide which approach fits a query type.

4. Answer quality and citations

Track whether the answer is supported by the retrieved context, whether it follows the requested format, and whether citations point to the correct source. Separate these questions:

• Grounding: Does the answer stay within the evidence?
• Completeness: Does it address the important parts of the question?
• Relevance: Does it avoid unrelated retrieved material?
• Refusal behavior: Does it acknowledge insufficient evidence instead of inventing an answer?
• Usability: Can a reader verify the claim from the citation?

Track latency, token usage, errors, empty results, and user feedback alongside quality. A response can be factually acceptable but too slow or too expensive for its intended workflow. Observability guidance on LLM logs, traces, and feedback loops can help structure these records.

5. Security and access boundaries

Authorization must be applied before retrieval results reach the model. Associate each document and chunk with an access scope, then filter by the authenticated user's permissions during search. Do not rely on the system prompt to prevent disclosure of content that should never have been retrieved.

Test cross-tenant and role-based access explicitly. Also consider prompt injection inside documents: retrieved text should be treated as untrusted data, not as instructions that can override the application's system rules. Keep instructions separate from source content and define what the model should do when a document contains commands aimed at the assistant.

Cadence and checkpoints

A RAG application needs two kinds of maintenance: event-driven maintenance when the corpus or product changes, and scheduled maintenance that catches gradual drift.

At every ingestion run

• Confirm that expected sources were discovered.
• Record successful, skipped, and failed documents.
• Check chunk counts and representative samples.
• Verify that metadata, permissions, and source links were preserved.
• Run a small smoke-test query set before publishing the new index.

Weekly or after a meaningful release

Review failed queries, low-confidence retrievals, user corrections, citation complaints, and latency outliers. Add recurring failure patterns to the evaluation set. If a prompt, embedding model, chunking rule, reranker, or retrieval filter changes, compare the new version with the previous one rather than judging it from a few anecdotal examples.

Monthly

Run the complete regression set. Review retrieval coverage, grounded-answer rate, refusal behavior, latency, cost proxies, and error rates by query category. Sample answers manually because automated scores can miss confusing citations, subtle omissions, or answers that are technically supported but poorly framed.

Quarterly or after a major corpus change

Reassess the architecture. Check whether the document types, access model, query patterns, and response requirements have changed. Revisit chunking, metadata filters, hybrid search, reranking, context limits, and the choice of generation model. Keep a dated record of the evaluation set and configuration so that future comparisons remain meaningful. For a broader workflow, use the LLM app evaluation framework.

How to interpret changes

When answer quality drops, isolate the stage before changing the prompt. If expected sources are absent from the top results, investigate query rewriting, chunk boundaries, metadata filters, indexing freshness, and the retrieval method. If the correct passages are present but the answer is wrong, inspect context ordering, instructions, output constraints, and the model's handling of uncertainty.

If retrieval metrics improve while user satisfaction falls, the system may be returning too much context, citing poorly, or answering with unnecessary detail. If latency increases without a quality gain, compare the number of retrieved chunks, reranking work, embedding calls, and generation length. If failures appear only for certain teams or document groups, inspect authorization filters and corpus coverage rather than applying a global prompt change.

Use controlled comparisons. Change one major variable at a time, preserve the old configuration, and evaluate on the same fixed questions. Keep separate scores for retrieval and generation so a single blended number does not hide the cause. A simple test record can include: question, user role, expected sources, retrieved sources, supported claims, citation result, latency, and reviewer notes. Prompt testing methods described in this evaluation framework can be adapted for RAG regression tests.

Common fixes include improving document parsing before increasing model size, adding metadata filters before adding more retrieved chunks, rewriting ambiguous queries, using hybrid retrieval for exact terminology, and requiring the model to quote or cite supporting passages. These are hypotheses to test, not universal rules. The right change depends on the failure pattern.

When to revisit

Revisit this RAG implementation on a monthly cadence for operational health and at least quarterly for architecture and evaluation review. Return sooner when any of the following occurs:

• A source system changes its format, API, permissions, or update schedule.
• A large document collection is added, removed, or reorganized.
• Users report unsupported answers, missing citations, or access-control problems.
• The application prompt, embedding model, reranker, generation model, or chunking strategy changes.
• Latency, token usage, error rates, or empty-result rates move materially from the established baseline.
• A new workflow requires structured output, multilingual retrieval, tables, images, or more demanding citations.

End each review with three concrete artifacts: an updated evaluation set, a short list of observed failure modes, and a dated record of configuration changes. Then run the smoke tests, inspect representative traces, and decide whether the next action belongs in ingestion, retrieval, prompting, access control, or monitoring.

A production-ready RAG app is not finished when the first answer looks convincing. It is ready when its evidence path is inspectable, its access boundaries are enforced, its failures are measurable, and its quality can be checked again after the corpus and application change. Treat this article as a recurring checklist, not a one-time setup guide, and the system will be easier to improve without losing reliability.