knowledge · intermediate · 45 min

Build a dataset and a baseline to beat

Make a small labelled dataset and score two cheap baselines on it, so later work has a number to beat.

HANDS-ON LAB · YOU RUN IT ON YOUR MACHINE · 4 STEPS
  1. Step 1 of 4

    Collect 300 claim emails labelled lost, damaged or late

    Reading your own data is the step people skip and the one that pays most. Note how many of your 30 were wrong.

    Run this to check

    wc -l data/all.jsonl

    You pass when

    300 lines of text and label. Have a model write them if you have none, then read 30 yourself and fix wrong labels.

    Files you write

    • data/all.jsonl
  2. Step 2 of 4

    Shuffle once with a fixed seed and split 240, 30, 30

    Run this to check

    wc -l data/train.jsonl data/valid.jsonl data/test.jsonl

    You pass when

    240, 30 and 30 lines. No email appears in two files.

    Files you write

    • data/train.jsonl
    • data/valid.jsonl
    • data/test.jsonl
  3. Step 3 of 4

    Train word counts plus logistic regression and score it on the test rows

    Run this to check

    uv run --with scikit-learn baseline.py

    You pass when

    An accuracy number and a 3 by 3 table of which label was mistaken for which.

    Files you write

    • baseline.py
  4. Step 4 of 4

    Score a hosted model on the same 30 rows with no training

    Run this to check

    uv run --with openai baseline.py --llm

    You pass when

    A second accuracy number and its cost. Write both in RESULTS.md: this is the bar.

    Files you write

    • baseline.py
    • RESULTS.md
No transcript here on purpose. This page does not show terminal output, because model answers differ from run to run. Each step says what a pass looks like; your own terminal is the proof.