knowledge · intermediate · 40 min
Measure retrieval before you tune it
Put a number on how often the right page is found, then change one thing and see if the number moves.
HANDS-ON LAB · YOU RUN IT ON YOUR MACHINE · 4 STEPS
-
Write 20 questions, each with the file that holds the answer
Run this to check
wc -l evalset.jsonlYou pass when
20 lines. Include five that use different words from the page.
Files you write
- evalset.jsonl
-
Print recall at 4 and the mean reciprocal rank
Recall at 4: how often the right page is in the four chunks you fetch. Reciprocal rank: 1 if it is first, 1/2 if second, and so on.
Run this to check
uv run --with requests --with "psycopg[binary]" eval.py evalset.jsonlYou pass when
Two numbers between 0 and 1, and a list of the questions that missed.
Files you write
- eval.py
-
Change one thing: chunk size, overlap, or a rerank step
Run this to check
uv run --with requests --with "psycopg[binary]" eval.py evalset.jsonl --label afterYou pass when
The same two numbers for the new setup.
Files you write
- ingest.py
- eval.py
-
Write the before and after in a table and keep or revert the change
Run this to check
cat RESULTS.mdYou pass when
Two rows and a one-line decision. A change that does not move the number is reverted.
Files you write
- RESULTS.md
No transcript here on purpose. This page does not show terminal output, because model answers differ from run to run. Each step says what a pass looks like; your own terminal is the proof.