FREE RAG RETRIEVAL FOUNDATIONS LEARNING PATH

WikiFree original practice workbook

The evidence engine

CASE STATUS: fictional/synthetic teaching material. Do not present the example as real research, employment experience or measured business performance.

OUTCOME
Build a local retrieval prototype that returns source IDs, quotes bounded evidence and abstains when the small document collection cannot answer.

BEFORE YOU START
Basic Python lists, dictionaries and string handling. No embeddings subscription or model API.

CASE
The fictional Maple Makerspace provides three original notes: D1 says weekday opening hours are 09:00–17:00; D2 says workshops require advance booking; D3 says tool loans last seven days. The notes contain no holiday hours, prices or personal records.

INPUTS
Document,Approved fact
D1,Weekday opening hours: 09:00–17:00
D2,Workshops require advance booking
D3,Tool loans last seven days


ENTRY CHECK

Question: Can retrieval supply facts absent from the collection?

Self-check: No.

Question: Does a citation automatically support every sentence?

Self-check: No. Check the claim against the cited text.

Question: Is keyword overlap the same as semantic embedding search?

Self-check: No. It is a simpler baseline.

STAGE 1: Define the evidence collection

Prerequisite: Basic Python lists, dictionaries and string handling. No embeddings subscription or model API.

Instruction: A retrieval system needs a bounded collection with source IDs and version dates. Preserve the text that supports a response; an unlabeled excerpt is hard to verify later. Separate source coverage from answer quality. If the collection says nothing about holidays, a fluent holiday answer would still be unsupported.

Practice task: Create the three-document collection and a coverage table listing answerable and unanswerable questions.

Deliverable: A versioned source inventory.

Acceptance check: Holiday hours and prices are explicitly marked absent.

If stuck: Do not fill missing coverage with plausible guesses or general knowledge.

Record your work, decisions and evidence:




Record your result against the acceptance check:



STAGE 2: Choose a retrieval baseline

Prerequisite: Bring your previous output: A versioned source inventory.

Instruction: Start with a simple keyword-overlap retriever before adding embeddings. Normalize case and punctuation consistently, keep document IDs and show the matched terms. This baseline will miss paraphrases and can over-rank incidental word matches. Its transparency makes those failures teachable. Keep retrieval scoring separate from any rule deciding whether evidence is sufficient to answer.

Practice task: Implement the supplied token-overlap function and rank the documents for hours, booking and loan questions.

Deliverable: A ranked retrieval result with source IDs.

Acceptance check: The expected topic document appears first for the three direct questions; scores and ties are visible.

If stuck: A top-ranked result is not automatically relevant enough to answer.

Record your work, decisions and evidence:




Record your result against the acceptance check:



STAGE 3: Compare chunking choices

Prerequisite: Bring your previous output: A ranked retrieval result with source IDs.

Instruction: A chunk should preserve enough context to interpret its facts. Splitting a short note into isolated words destroys meaning; joining all unrelated policies makes citations vague. Our three notes are already small coherent units. For a longer document, retain the section title and source ID with each chunk and test whether the answer-bearing sentence survives the split.

Practice task: Keep the three baseline notes intact. Add a two-paragraph synthetic policy and compare paragraph chunks with one combined chunk.

Deliverable: A chunking decision log.

Acceptance check: Every chunk retains its parent source and the complete fact needed for its test question.

If stuck: If a number is separated from its unit or condition, revise the boundary.

Record your work, decisions and evidence:




Record your result against the acceptance check:



STAGE 4: Compose a bounded response

Prerequisite: Bring your previous output: A chunking decision log.

Instruction: The prototype returns an extract from a supported note and its ID; it does not use a language model to invent a synthesis. Add a conservative topic/coverage check and an explicit missing-information response. Retrieved instructions are data, not commands. A future generative layer would require additional faithfulness testing; this prototype establishes only the local evidence path.

Practice task: Implement source-linked extract responses for the three supported topics. Return an abstention for holiday hours and an injected instruction.

Deliverable: A cited extractive-answer function.

Acceptance check: Each supported answer names the correct ID; unsupported questions receive no invented fact.

If stuck: Do not cite D1 for holiday hours merely because it contains the word hours.

Record your work, decisions and evidence:




Record your result against the acceptance check:



STAGE 5: Evaluate coverage and failure

Prerequisite: Bring your previous output: A cited extractive-answer function.

Instruction: Separate retrieval success, evidence support and abstention quality. Test direct questions, paraphrases, an absent topic and a malicious instruction placed in a document. Record observed behavior honestly; a simple baseline may fail a paraphrase. Improvement begins with identifying that failure, not relabeling the question until it passes.

Practice task: Run the supplied evaluation set and add two paraphrases. Write one improvement proposal tied to an observed failure.

Deliverable: An evaluation report and improvement backlog.

Acceptance check: Every result includes the input, expected source or abstention, actual output and a support judgment.

If stuck: Do not claim real-world RAG reliability from a tiny hand-built note collection.

Record your work, decisions and evidence:




Record your result against the acceptance check:



WORKED CASE
The question about loan duration can return D3 and seven days. The question about holiday opening must abstain because D1 covers weekdays only. A retrieval score cannot expand the scope of that source.

FINAL PROJECT
Deliver the notes, retriever, response rules, evaluation cases and README. Change D3 to a ten-day loan policy in a versioned copy, rerun the relevant tests and demonstrate that the answer cites the updated source.

REQUIRED CRITICAL CHECK
Answers remain within the cited source scope and absent information produces an explicit abstention.

SCORING
Score each of the five stage acceptance checks from 0–16 (0 missing/incorrect; 8 partially supported; 16 complete and evidenced). Score the final integrated project from 0–20 (0 missing; 10 incomplete handover; 20 complete, reproducible and consistent with the case). Target: at least 80/100 AND the critical check must pass. A total score never overrides a failed critical check.

SCOPE LIMIT
This is an extractive retrieval baseline, not a full generative RAG service, an embedding benchmark or production document security.

SOURCES AND ACCESS
Public videos and standard Python are sufficient. No external model API, vector database subscription or private document upload is required.

Free RAG Course: Retrieval, Vector Search and AI Agents
WikiFree: https://wikifree.com/courses/free-rag-course-retrieval-vector-search-and-ai-agents/
Official source: https://www.youtube.com/playlist?list=PLOspHqNVtKADc8E1JLd_kBBPdEBDdmwsR
Why selected: IBM videos explain retrieval motivation, retrieval types, document handling and evidence trust. The small local keyword and extraction prototype is original practice.

Stage 1 selected lessons:

2. Is RAG Still Needed? Choosing the Best Approach for LLMs (11:10)
https://www.youtube.com/watch?v=UabBYexBD4k&list=PLOspHqNVtKADc8E1JLd_kBBPdEBDdmwsR&index=2

Stage 2 selected lessons:

15. Top 3 RAG Retrieval Strategies: Sparse, Dense, & Hybrid Explained (8:29)
https://www.youtube.com/watch?v=r0Dciuq0knU&list=PLOspHqNVtKADc8E1JLd_kBBPdEBDdmwsR&index=15

21. Unlocking Smarter AI Agents with Unstructured Data, RAG & Vector Databases (6:41)
https://www.youtube.com/watch?v=sMQ5R92F86o&list=PLOspHqNVtKADc8E1JLd_kBBPdEBDdmwsR&index=21

Stage 3 selected lessons:

20. Why RAG Solutions Fail with Complex Documents & Vector Databases (7:46)
https://www.youtube.com/watch?v=xc63tFIIfeA&list=PLOspHqNVtKADc8E1JLd_kBBPdEBDdmwsR&index=20

Stage 4 selected lessons:

23. Building Trustworthy AI Agents with Hybrid RAG (4:25)
https://www.youtube.com/watch?v=l9LfC98tE_4&list=PLOspHqNVtKADc8E1JLd_kBBPdEBDdmwsR&index=23

Stage 5 selected lessons:

6. RAG vs Agentic AI: How LLMs Connect Data for Smarter AI (10:01)
https://www.youtube.com/watch?v=fB2JQXEH_94&list=PLOspHqNVtKADc8E1JLd_kBBPdEBDdmwsR&index=6

REUSE
Original WikiFree teaching text and synthetic fixtures are supplied for personal learning and adaptation with attribution. Provider lessons remain on their original sites and retain their own terms. No provider certificate is included.