Retrieval augmented generation, or RAG, combines information retrieval with language model generation. The system finds relevant external material, includes it in the model's context and asks the model to answer using that evidence.
The RAG paper by Lewis and colleagues introduced an influential architecture combining a generator with retrieved passages. Today, RAG commonly describes application pipelines using document stores, search services and language models.
The RAG pipeline from indexing to answering
At indexing time, the system parses documents, divides them into usable passages and stores searchable representations. Metadata should include source identity, version, permissions and any dates that affect applicability.
At query time, the system searches within the user's authorised scope, ranks candidate passages and selects evidence that fits the context budget. The generator receives the question and selected passages. The application then checks the response and its citations.
Changes to documents, permissions and deletions must reach the index. A correct search algorithm can still return an obsolete policy if the ingestion process leaves old versions active.
RAG has a preparation workflow and a request workflow. The preparation work creates the searchable material that later requests use.
Chunk documents without separating rules from exceptions
Chunking determines the units available for retrieval. A fixed character cut can separate a refund rule from the exception that changes its meaning.
Prefer meaningful boundaries such as sections, clauses or paragraphs where the document supports them. Include titles and headings when they help identify the passage. For long tables, repeat relevant column headers in each chunk so the rows remain interpretable.
Overlap can preserve information near a boundary, but it also creates duplicates. Compare chunking strategies using actual questions and known supporting passages. There is no universally correct chunk size.
Embeddings, keyword search and hybrid retrieval
An embedding represents text numerically for similarity search. Similarity is useful for matching different phrasings, but it is not a probability that a passage answers the question.
Queries and documents need compatible representations. Some systems use the same encoder; others use paired query and document encoders trained for a shared space. Changing the embedding system usually requires rebuilding compatible document vectors.
Keyword retrieval helps with exact identifiers such as product codes and error strings. Hybrid search combines lexical and vector results. Reciprocal rank fusion is one way to combine ranked lists without assuming that their raw scores are comparable; Microsoft's RRF documentation explains its use in a hybrid search system.
Rerank a candidate set before building context
A reranker evaluates candidates more closely against the question. A common design uses a cross encoder to score query passage pairs, although other reranking methods exist.
The first retrieval stage must include the needed evidence. Reranking cannot promote a document absent from its candidate set. Measure candidate coverage before spending time optimising the final order.
After reranking, remove redundant results and retain linked evidence when needed. A high scoring general rule without its applicable exception may still produce an incorrect answer.
Measure hit rate, recall and precision correctly
These retrieval metrics answer different questions:
| Metric | Meaning for one query |
|---|---|
| Hit rate at k | Whether at least one relevant item appears in the first k results |
| Recall at k | Relevant items retrieved in the first k, divided by all relevant items |
| Precision at k | Relevant items in the first k, divided by k |
Suppose a question has three relevant passages and the top four results contain two of them. Hit rate is 1 for that query, recall is 2/3 and precision is 2/4. Average per query values using a stated method. Handle queries with no relevant items separately.
If each query has exactly one relevant passage, average hit rate and average recall coincide. That special case should not obscure the distinction.
Evaluate retrieval and generation separately
Assume 62% of questions receive sufficient evidence, and the generator answers 90% of those correctly. If correctness requires retrieved evidence and every evidence missing case fails, overall success is 0.62 × 0.90 = 55.8%.
That is an illustrative conditional calculation, not a universal ceiling. A system may correctly abstain, find another source or answer from other authorised context. Define success and evidence sufficiency before applying the arithmetic.
Track answer correctness, citation support and appropriate abstention alongside retrieval metrics. A stronger generator may use evidence better, while query rewriting or retrieval changes may improve coverage. Diagnose the failed stage before changing the prompt.
RAG and fine tuning in the same application
RAG is often suitable for information that changes or requires source attribution. Fine tuning can improve task behaviour, format or domain performance when suitable training data is available.
They can be combined. Neither eliminates access control requirements or hallucinations. Start with the smallest design that meets measured requirements, then add complexity in response to a specific failure.