LPractice & Learn AI
harness · intermediate · 35 min

Choose and run open models

Measure what a smaller, more compressed copy of a model costs you in quality and gives you in speed and memory.

Start step 1 ↓
HANDS-ON LAB · YOU RUN IT ON YOUR MACHINE · 4 STEPS
  1. Step 1 of 4

    Pull the same model at 4-bit and 8-bit

    Quantising stores each weight in fewer bits. The file shrinks and runs faster; some accuracy is lost. How much is what you measure.

    Run this to check

    ollama pull llama3.2:3b-instruct-q4_K_M && ollama pull llama3.2:3b-instruct-q8_0 && ollama list

    You pass when

    Two entries for the same model. The 8-bit one is clearly larger on disk.

  2. Step 2 of 4

    Time both on one prompt

    Run this to check

    ollama run llama3.2:3b-instruct-q4_K_M --verbose "Write a haiku about parcels"

    You pass when

    After the answer, a block of timings with an eval rate in tokens per second. Run it for both and note the two rates.

  3. Step 3 of 4

    See what is loaded and how much memory it takes

    Run this to check

    ollama ps

    You pass when

    The loaded model with its size in memory and whether it runs on the GPU.

  4. Step 4 of 4

    Run your five prompts from lab 12 on both and score them blind

    Run this to check

    cat QUANT.md

    You pass when

    A table with size, memory, speed and average score for each, and the one you would ship.

    Files you write

    • QUANT.md
No transcript here on purpose. This page does not show terminal output, because model answers differ from run to run. Each step says what a pass looks like; your own terminal is the proof.