LPractice & Learn AI
reliable · advanced · 60 min

Let it improve itself, and prove it

Have a model propose changes to your prompt from its own failures, and accept a change only when held-back data says it helped.

Start step 1 ↓
HANDS-ON LAB · YOU RUN IT ON YOUR MACHINE · 4 STEPS
  1. Step 1 of 4

    Split 60 graded examples into 40 to learn from and 20 held back

    Run this to check

    wc -l dev.jsonl heldout.jsonl

    You pass when

    40 and 20 lines, with no example in both.

    Files you write

    • dev.jsonl
    • heldout.jsonl
  2. Step 2 of 4

    Score the current prompt on the held-back set three times

    The same prompt scores differently on each run. Until you know by how much, you cannot tell an improvement from luck.

    Run this to check

    uv run --with openai improve.py --baseline

    You pass when

    Three scores and their spread. Any gain smaller than that spread is noise.

    Files you write

    • improve.py
  3. Step 3 of 4

    Show the improver the failures from the 40 and ask for one prompt change

    Run this to check

    uv run --with openai improve.py --propose

    You pass when

    A changed prompt saved as a candidate, with one sentence on what it changes.

    Files you write

    • prompts/candidate.md
  4. Step 4 of 4

    Score the candidate on the held-back 20 and decide

    Run this to check

    uv run --with openai improve.py --decide && grep -c heldout improver.log

    You pass when

    ACCEPT or REJECT with both scores. The count is 0: the improver never saw the held-back set.

    Files you write

    • improve.py
    • improver.log
No transcript here on purpose. This page does not show terminal output, because model answers differ from run to run. Each step says what a pass looks like; your own terminal is the proof.