Production-ready RAG: beyond the demo

A retrieval-augmented generation demo can look impressive after an afternoon. A system people can trust with real work requires a different level of engineering.

Retrieval is the product

When an answer is poor, the language model often receives the blame. In practice, the source content, chunking, metadata, and retrieval strategy usually determine the ceiling for quality.

Start with a representative question set. For each question, identify the source passages a capable person would need. That dataset becomes the basis for evaluating retrieval independently from generation.

Build an evaluation loop

Production teams need to know whether a change improved the system. Track at least two layers:

1. Retrieval: Did the system find the relevant evidence?
2. Generation: Did the response use that evidence correctly and answer the question?

Automated scoring is useful, but it should be calibrated against human review. Sample real queries, protect sensitive data, and make feedback easy to inspect.

def evaluate_case(case, retriever, generator):
    documents = retriever.search(case.question)
    answer = generator.answer(case.question, documents)
 
    return {
        "recall": source_recall(documents, case.expected_sources),
        "groundedness": groundedness(answer, documents),
        "relevance": answer_relevance(answer, case.question),
    }

Permissions must travel with content

Enterprise retrieval cannot flatten authorization. A document a user cannot open in the source system should not appear in a generated answer.

Carry access-control metadata through ingestion, indexing, retrieval, and citation. Filter before generation, and test for permission leakage as an explicit security property.

Make uncertainty visible

A useful assistant should be able to say that it lacks enough evidence. Require citations, distinguish source facts from synthesis, and define a fallback when retrieval confidence is low.

Good interaction design helps users judge an answer:

• link directly to supporting material;
• expose freshness and ownership when relevant;
• make it easy to report a weak or incorrect response;
• never invent a citation to fill a gap.

Operate the whole pipeline

Content changes, embedding models change, and user questions drift. Monitor ingestion failures, index freshness, retrieval latency, model cost, and evaluation trends.

A production RAG system is a data product, a search system, and an AI application at the same time. Treating all three seriously is how a prototype becomes dependable infrastructure.