WikiFree learning path 30 · Intermediate
The evidence engine
Retrieve evidence before you compose an answer.
Build a local retrieval prototype that returns source IDs, quotes bounded evidence and abstains when the small document collection cannot answer.
Video duration uses the selected lessons only. Reading and practice times are planning estimates; repeat difficult work as needed.
A clear starting point
Who this is for
Basic Python lists, dictionaries and string handling. No embeddings subscription or model API.
Your practical result
Build a local retrieval prototype that returns source IDs, quotes bounded evidence and abstains when the small document collection cannot answer.
This is an extractive retrieval baseline, not a full generative RAG service, an embedding benchmark or production document security.
Access and cost
Public videos and standard Python are sufficient. No external model API, vector database subscription or private document upload is required.
The project pack contains an original workbook, case inputs, an evidence log and reference notes. Some technical paths include starter code and a runnable reference. Read the pack README before starting.
Three-minute readiness check
Answer first, then open each explanation. A missed answer identifies what to revisit; it does not stop you from starting.
Can retrieval supply facts absent from the collection?
No.
Does a citation automatically support every sentence?
No. Check the claim against the cited text.
Is keyword overlap the same as semantic embedding search?
No. It is a simpler baseline.
The case you will work on
The fictional Maple Makerspace provides three original notes: D1 says weekday opening hours are 09:00–17:00; D2 says workshops require advance booking; D3 says tool loans last seven days. The notes contain no holiday hours, prices or personal records.
| Document | Approved fact |
|---|---|
| D1 | Weekday opening hours: 09:00–17:00 |
| D2 | Workshops require advance booking |
| D3 | Tool loans last seven days |
These are supplied synthetic inputs, not observations from a real organization. Label assumptions and any additional data you create.
Keep an evidence trail
Save each stage output in the project folder. In evidence_log.csv, record the decision, the artifact or test that supports it, what failed, and the revision you made.
Your route from learning to evidence
- Stage 1Define the evidence collectionA versioned source inventory.55 minutes of original practice
- Stage 2Choose a retrieval baselineA ranked retrieval result with source IDs.80 minutes of original practice
- Stage 3Compare chunking choicesA chunking decision log.65 minutes of original practice
- Stage 4Compose a bounded responseA cited extractive-answer function.85 minutes of original practice
- Stage 5Evaluate coverage and failureAn evaluation report and improvement backlog.70 minutes of original practice
- Final projectMake the evidence reviewableConnect the five stage outputs and test a change.240 minutes estimated
Progress stays in this browser. Tick a stage only after its evidence check passes. This is a personal checklist, not a certificate.
Why these source courses
The path uses selected material from courses already published on WikiFree. The original exercises below connect that material to one practical outcome. Follow only the listed assignments; completing the full source courses is optional.
IBM
Free RAG Course: Retrieval, Vector Search and AI Agents
IBM videos explain retrieval motivation, retrieval types, document handling and evidence trust. The small local keyword and extraction prototype is original practice.
Access: Free Video
Certificate: Free video viewing; IBM certification is not included
Learn it, apply it, check it
Work in order. Every stage produces something the next stage uses. Revisit the repair instruction when a check fails.
Define the evidence collection
Basic Python lists, dictionaries and string handling. No embeddings subscription or model API.
The idea to use
A retrieval system needs a bounded collection with source IDs and version dates. Preserve the text that supports a response; an unlabeled excerpt is hard to verify later. Separate source coverage from answer quality. If the collection says nothing about holidays, a fluent holiday answer would still be unsupported.
Watch the selected lessons
Selected video time: 11m. Open this course on WikiFree.
Your 55-minute lab
Create the three-document collection and a coverage table listing answerable and unanswerable questions.
Save: A versioned source inventory..
Evidence check
Holiday hours and prices are explicitly marked absent.
If it fails: Do not fill missing coverage with plausible guesses or general knowledge.
Choose a retrieval baseline
Bring your previous output: A versioned source inventory.
The idea to use
Start with a simple keyword-overlap retriever before adding embeddings. Normalize case and punctuation consistently, keep document IDs and show the matched terms. This baseline will miss paraphrases and can over-rank incidental word matches. Its transparency makes those failures teachable. Keep retrieval scoring separate from any rule deciding whether evidence is sufficient to answer.
Watch the selected lessons
- 15. Top 3 RAG Retrieval Strategies: Sparse, Dense, & Hybrid Explained8:29
- 21. Unlocking Smarter AI Agents with Unstructured Data, RAG & Vector Databases6:41
Selected video time: 15m. Open this course on WikiFree.
Your 80-minute lab
Implement the supplied token-overlap function and rank the documents for hours, booking and loan questions.
Save: A ranked retrieval result with source IDs..
Evidence check
The expected topic document appears first for the three direct questions; scores and ties are visible.
If it fails: A top-ranked result is not automatically relevant enough to answer.
Compare chunking choices
Bring your previous output: A ranked retrieval result with source IDs.
The idea to use
A chunk should preserve enough context to interpret its facts. Splitting a short note into isolated words destroys meaning; joining all unrelated policies makes citations vague. Our three notes are already small coherent units. For a longer document, retain the section title and source ID with each chunk and test whether the answer-bearing sentence survives the split.
Watch the selected lessons
Selected video time: 8m. Open this course on WikiFree.
Your 65-minute lab
Keep the three baseline notes intact. Add a two-paragraph synthetic policy and compare paragraph chunks with one combined chunk.
Save: A chunking decision log..
Evidence check
Every chunk retains its parent source and the complete fact needed for its test question.
If it fails: If a number is separated from its unit or condition, revise the boundary.
Compose a bounded response
Bring your previous output: A chunking decision log.
The idea to use
The prototype returns an extract from a supported note and its ID; it does not use a language model to invent a synthesis. Add a conservative topic/coverage check and an explicit missing-information response. Retrieved instructions are data, not commands. A future generative layer would require additional faithfulness testing; this prototype establishes only the local evidence path.
Watch the selected lessons
Selected video time: 4m. Open this course on WikiFree.
Your 85-minute lab
Implement source-linked extract responses for the three supported topics. Return an abstention for holiday hours and an injected instruction.
Save: A cited extractive-answer function..
Evidence check
Each supported answer names the correct ID; unsupported questions receive no invented fact.
If it fails: Do not cite D1 for holiday hours merely because it contains the word hours.
Evaluate coverage and failure
Bring your previous output: A cited extractive-answer function.
The idea to use
Separate retrieval success, evidence support and abstention quality. Test direct questions, paraphrases, an absent topic and a malicious instruction placed in a document. Record observed behavior honestly; a simple baseline may fail a paraphrase. Improvement begins with identifying that failure, not relabeling the question until it passes.
Watch the selected lessons
Selected video time: 10m. Open this course on WikiFree.
Your 70-minute lab
Run the supplied evaluation set and add two paraphrases. Write one improvement proposal tied to an observed failure.
Save: An evaluation report and improvement backlog..
Evidence check
Every result includes the input, expected source or abstention, actual output and a support judgment.
If it fails: Do not claim real-world RAG reliability from a tiny hand-built note collection.
A worked reference to compare
The question about loan duration can return D3 and seven days. The question about holiday opening must abstain because D1 covers weekdays only. A retrieval score cannot expand the scope of that source.
Compare your reasoning as well as the result. If your answer differs, state the assumption or input that explains the difference. Reference notes are for checking after an attempt; copying them is not evidence of independent work.
Your final project
Your brief
Deliver the notes, retriever, response rules, evaluation cases and README. Change D3 to a ten-day loan policy in a versioned copy, rerun the relevant tests and demonstrate that the answer cites the updated source.
Time box: 4h 00m estimated. Extend it if a gate needs repair.
Hand in these artifacts
- A versioned source inventory.
- A ranked retrieval result with source IDs.
- A chunking decision log.
- A cited extractive-answer function.
- An evaluation report and improvement backlog.
- A short README: outcome, how to inspect or run the work, assumptions and limitations.
- A before-and-after change test and the completed evidence log.
Run a final review
- Use the supplied reference to reconcile the baseline result.
- Choose one relevant input or constraint to change. Write your expected effect before testing it.
- Run the calculation, walkthrough or test again and save the actual result.
- Ask a peer to inspect the evidence, or perform a separate self-review. Label which review you used.
- Score the work using the rubric and repair every critical failure.
Review rubric · 100 points
| Criterion | Points | Evidence to inspect |
|---|---|---|
| Practical outcome and case reasoning | 25 | Build a local retrieval prototype that returns source IDs, quotes bounded evidence and abstains when the small document collection cannot answer. |
| Five stage checks | 30 | Award 6 points for each stage only when its stated gate is demonstrably met. |
| Traceable evidence | 20 | Artifacts, inputs and calculations or tests are reproducible; synthetic and simulated evidence is labelled. |
| Revision after a changed condition | 15 | Change one input or constraint, predict the effect, test it and record the result. |
| Handoff quality | 10 | A reviewer can find the files, follow the instructions and understand the remaining limitations. |
Completion target: at least 80/100, all five stage gates passed, and every critical issue below corrected. This is a learning benchmark, not an accredited qualification.
Critical issue: Answers remain within the cited source scope and absent information produces an explicit abstention.
Before you go further
What will I be able to produce?
Build a local retrieval prototype that returns source IDs, quotes bounded evidence and abstains when the small document collection cannot answer.
Is the whole path free?
Public videos and standard Python are sufficient. No external model API, vector database subscription or private document upload is required.
Do I need to finish every source course?
No. Follow the selected assignments at each stage. Source courses provide context, while the original case, labs and final review build the practical result. Extra study is optional.
What counts as completion?
Complete all five stage checks, earn at least 80/100 on the final rubric and correct every critical issue. This is an extractive retrieval baseline, not a full generative RAG service, an embedding benchmark or production document security.
Questions & discussion
Share a useful question or correction. Comments appear after moderation. Please avoid personal or sensitive information.