LPractice & Learn AI
reliable · intermediate · 50 min

Grade with a judge you have checked

Use a model to grade answers, and prove first that it grades the way you do.

Start step 1 ↓
HANDS-ON LAB · YOU RUN IT ON YOUR MACHINE · 4 STEPS
  1. Step 1 of 4

    Grade 40 claim decisions yourself: right or wrong

    Run this to check

    wc -l labels.csv

    You pass when

    41 lines: a header and 40 rows of claim, decision and your grade. Make about a quarter of them wrong decisions.

    Files you write

    • labels.csv
  2. Step 2 of 4

    Write a judge prompt and grade the same 40 with a model

    Run this to check

    uv run --with openai --with scikit-learn judge.py labels.csv

    You pass when

    A 2 by 2 table of your grade against the judge's, with precision and recall.

    Files you write

    • judge.py
    • JUDGE.md
  3. Step 3 of 4

    Print agreement beyond chance (Cohen's kappa)

    Plain agreement flatters a judge when most answers are right. Kappa takes out the agreement you would get by luck.

    Run this to check

    uv run --with openai --with scikit-learn judge.py labels.csv --kappa

    You pass when

    A number up to 1. Below 0.6, read the rows you disagree on, tighten the judge prompt and run again.

    Files you write

    • JUDGE.md
  4. Step 4 of 4

    Decide which mistake is worse: paying a bad claim or refusing a good one

    Run this to check

    cat COSTS.md

    You pass when

    A price on each mistake, and the judge setting you chose because of it.

    Files you write

    • COSTS.md
No transcript here on purpose. This page does not show terminal output, because model answers differ from run to run. Each step says what a pass looks like; your own terminal is the proof.