knowledge · intermediate · 45 min
Build a dataset and a baseline to beat
Make a small labelled dataset and score two cheap baselines on it, so later work has a number to beat.
HANDS-ON LAB · YOU RUN IT ON YOUR MACHINE · 4 STEPS
-
Collect 300 claim emails labelled lost, damaged or late
Reading your own data is the step people skip and the one that pays most. Note how many of your 30 were wrong.
Run this to check
wc -l data/all.jsonlYou pass when
300 lines of text and label. Have a model write them if you have none, then read 30 yourself and fix wrong labels.
Files you write
- data/all.jsonl
-
Shuffle once with a fixed seed and split 240, 30, 30
Run this to check
wc -l data/train.jsonl data/valid.jsonl data/test.jsonlYou pass when
240, 30 and 30 lines. No email appears in two files.
Files you write
- data/train.jsonl
- data/valid.jsonl
- data/test.jsonl
-
Train word counts plus logistic regression and score it on the test rows
Run this to check
uv run --with scikit-learn baseline.pyYou pass when
An accuracy number and a 3 by 3 table of which label was mistaken for which.
Files you write
- baseline.py
-
Score a hosted model on the same 30 rows with no training
Run this to check
uv run --with openai baseline.py --llmYou pass when
A second accuracy number and its cost. Write both in RESULTS.md: this is the bar.
Files you write
- baseline.py
- RESULTS.md
No transcript here on purpose. This page does not show terminal output, because model answers differ from run to run. Each step says what a pass looks like; your own terminal is the proof.