Pick a model with numbers
Run the same five tasks on three models and choose one from a table of speed, cost and quality.
-
Write five real tasks from your own work, one per line
Public leaderboards rank models on other people's tasks. Five of your own tell you more than any of them.
Run this to check
wc -l prompts.jsonlYou pass when
Five lines. Each is something you would really ask, with an answer you can judge.
Files you write
- prompts.jsonl
-
Write bench.py: every prompt on one local and two hosted models, timed
Run this to check
uv run --with openai bench.py && column -s, -t results.csv | head -20You pass when
15 rows, each with model, prompt number, seconds, tokens in, tokens out and cost.
Files you write
- bench.py
- results.csv
-
Score every answer from 1 to 5 without looking at the model name
Hide the model column while you score. Knowing which model wrote an answer changes how you grade it.
Run this to check
uv run bench.py --summaryYou pass when
One line per model: average score, average seconds, total cost.
Files you write
- results.csv
-
Write down your pick and the number that decided it
Run this to check
cat DECISION.mdYou pass when
Three lines: the model, the reason as a number from your table, and what would make you switch.
Files you write
- DECISION.md