Quantisation: a primer

Block quantisation, K-quants, and the importance matrix

Large language models are large

A model with 70B parameters at FP16:

70B × 2 bytes ≈ 140 GB

Quantisation hones in on the FP16 assumption, and asks: but what if every weight didn't need 16 bits?

A 4-bit representation theoretically takes only a quarter as much storage:

70B × 0.5 bytes ≈ 35 GB

The catch is that useful quantisation isn't just "store every weight in 4 bits." The techniques used to do this efficiently evolved in stages:

Block quantisation (Q4_0)
K-quants (mixed-precision schemes like Q4_K_M)
Importance-matrix quantisation (imatrix)

...and following from that came 'dynamic' quantisation, which builds on the imatrix idea.

Stage 1: Block quantisation (`Q4_0`)

You can't just round every FP16 weight to the nearest 1-of-16 bucket — weight magnitudes vary a lot across a tensor, and a single global scale would waste most of the 4-bit range on the few large outlier weights while crushing everything else to zero.

The fix: group weights into small blocks (32 weights per block in the original scheme) and give each block its own scale factor. For Q4_0, every weight in the block is represented as:

w ≈ q × block_scale

where q is a signed 4-bit integer (16 possible levels) and block_scale is one FP16 (or similar) value shared by all 32 weights in that block. This is why it's called "0" — no zero-point offset, just a scale, so it assumes weights are roughly symmetric around zero (usually a safe assumption post-normalization). Q4_1 adds a zero-point term as well, letting the block represent an asymmetric range, at the cost of one extra stored value per block.

The overhead here is real but small: for 32 weights at 4 bits (16 bytes) plus one scale value (2 bytes), you're paying ~18 bytes per 32 weights instead of a naive 16 — call it 4.5 bits/weight effectively rather than a clean 4. This "effective bits per weight, including block metadata" number matters later when comparing formats.

Stage 2: K-quants — smarter blocks, not just smaller ones

Block quantisation alone still treats every part of the model identically. The "K-quants," introduced into llama.cpp, made two changes:

Superblocks: weights are grouped into superblocks of 256, each containing 8 sub-blocks of 32. Each sub-block gets its own scale, but those scales are themselves quantised and stored more efficiently at the superblock level — a scale-of-scales, essentially, which cuts metadata overhead versus naively storing a full-precision scale per 32-weight block.
Per-tensor-type precision rules: instead of quantising every tensor in the model to the same bit-width, the K-quant naming scheme (Q4_K_S, Q4_K_M, Q4_K_L) applies a fixed, hand-designed rule about which tensor types get bumped up:

Q4_K_M
Q4 → 4-bit quantisation
K  → K-quant block scheme (superblocks), not the older Q4_0/Q4_1 style
M  → "Medium": a specific, static allocation — most tensors at Q4_K,
     certain layer types (feed-forward down-projections, output layers)
     bumped to Q6_K

The rule is static. "Bump ffn_down and output to Q6_K" was decided once by the llama.cpp maintainers, based on general intuition about which tensor types tend to be more sensitive across models broadly. It does not ask whether this particular tensor, in this particular model, is actually sensitive — it applies the same heuristic everywhere.

Stage 3: the importance matrix (imatrix)

Rather than rounding every weight purely by magnitude, you run a small calibration dataset through the full-precision model and record which weights actually get activated strongly and often. Weights that consistently drive large activations are treated as more important, and quantisation error is minimised for them preferentially, even within a single quant format. This is separate from K-quants — you can imatrix-quantise a Q4_K_M— but it is the input that Dynamic quantisation depends on, because Dynamic needs a per-layer signal for which layers matter, and the imatrix, plus its calibration dataset, is that signal.