This lab builds and evaluates the retrieval stage of a RAG application. It uses a fictional policy collection containing current documents, an outdated rule and a private note. The deliverable is a retrieval report and, optionally, a grounded model answer.
The starter uses lexical word overlap so every ranking decision can be inspected. It is a baseline, not an implementation of embeddings, BM25 or hybrid search.
Run the retrieval baseline
Download the Python lab bundle, extract it and run the following command from its directory with Python 3.10 or later.
python rag_lab.py --k 2
The script writes retrieval.json and context.json under lab output/rag. No model or network access is used by this command. A new output directory is required for each run.
The corpus and labelled queries are near the top of rag_lab.py. Inspect them before changing the algorithm. Each document has an ID, active version flag, access scope and text.
Compare unfiltered and filtered retrieval
The script evaluates an intentionally unsafe unfiltered baseline against a filtered version. The unfiltered path exists only to demonstrate the effect of stale or unauthorised candidates in a fictional corpus.
The filtered path excludes inactive and staff only documents before ranking. Optional generation always uses this filtered path.
Inspect forbidden_retrievals in the report. A relevant looking private note is still an invalid result for a public policy query. A high similarity score cannot grant access.
In a real application, the access decision must use the authenticated user's current permissions. A hardcoded public scope is sufficient only for this exercise.
Read hit rate, recall and precision separately
For each query, the report records the retrieved IDs and these measures:
| Field | Interpretation |
|---|---|
| hit_at_k | At least one labelled relevant passage was retrieved |
| recall_at_k | The share of labelled relevant passages retrieved |
| precision_at_k | Relevant passages divided by the requested k |
| evidence_complete | Every passage required by this fixture was retrieved |
| forbidden_retrievals | Stale or private documents in the result |
The second query requires both the cancellation rule and the returns process. Retrieving one can produce a hit while leaving recall at one half and evidence_complete false.
The warranty query has no relevant passage. Its recall and hit values are null because there is no positive evidence set to retrieve. Evaluate that case as an abstention or false positive problem, not by dividing by zero.
When fewer than k results are returned, this starter still divides precision by k. State that convention in the report; a measure dividing by returned count is a different metric.
Change the retrieval budget
Run two more experiments:
python rag_lab.py --k 1 --out lab-output/rag-k1
python rag_lab.py --k 4 --out lab-output/rag-k4
Compare evidence coverage, irrelevant results and selected context. Increasing k can recover a missing clause while adding material the answer does not need.
The starter does not measure model tokens. Count the actual assembled context with the tokenizer or counting interface for the model selected for generation. Word count is only a rough proxy.
Add a new policy exception and a query that needs it. Label the necessary passages before tuning retrieval. Then test a structural chunking change or a different ranking method against the same labels.
Generate an answer with the selected evidence
Optional live mode uses the llm_client.py adapter. Set ANTHROPIC_API_KEY outside the source files and supply an available model ID.
python rag_lab.py --live --model YOUR_MODEL_ID --out lab-output/rag-live
This makes at most one paid generation request, capped at 512 output tokens. If retrieval returns no passages, the script produces an insufficient evidence result without calling the model.
The response must contain outcome, answer and citations. The application checks object fields, types, outcome consistency and whether every citation ID belongs to the selected documents.
These checks do not establish that the cited passage supports the claim. The result deliberately marks claim_support as requires_review. Compare each material claim with its cited evidence and record the judgement.
Test a retrieval failure and a generation failure
For a retrieval failure, remove an essential passage from the active corpus or set k too low. Determine which part of the answer becomes unsupported.
For a generation contract failure, replace the adapter response in a local test with an object citing an invented ID. The validator should reject it. The included unit test demonstrates this without a network call.
Also test a question that shares policy vocabulary but asks for an absent fact. The generator should abstain even when retrieval returns related passages. A non empty result set is not proof that the answer is present.
Acceptance criteria and submitted artefacts
Submit the corpus, relevance labels, reports at different k values and a diagnosis of at least one failure. Include the generated answer and claim level review if live mode was used.
The filtered path must return no inactive or unauthorised documents. The report must distinguish partial evidence from complete evidence and explain how unanswerable cases are handled.
A useful extension replaces lexical overlap with hybrid retrieval and reranking while retaining the evaluation cases. Improvements should be demonstrated through those cases, with any additional latency, cost and index maintenance requirements recorded.