Aug 2026 draft
what actually happens during quantization?
taking weights stored in 16 bits down to 8 or 4 without wrecking the model. the whole trick is one scale factor and tolerating small errors.
the basic move
a weight matrix in fp16 is just a lot of 16-bit floats. quantization replaces each float with a small integer plus enough metadata to approximately recover the original. the simplest version: find the largest absolute value in a group of weights, call it amax, and set scale = amax / 127. then every weight becomes round(w / scale), an int8 between -127 and 127. to use it, you multiply back: w ≈ q * scale. storage halves, and the matmul can run in int8 with the rescale folded in at the end.
the error on any single weight is at most half a quantization step, which is scale / 2. when weights in a group are similar in magnitude, that error is tiny relative to the values. the entire scheme stands or falls on that grouping decision.
a concrete example
say one row of weights is [0.03, -0.61, 0.44, -0.08]. amax is 0.61, so scale = 0.61 / 127 ≈ 0.0048. the quantized row is [6, -127, 92, -17]. dequantized, that is [0.029, -0.61, 0.442, -0.082]. the small values survive fine here because nothing in the group dwarfs them.
now poison the group: append one outlier weight of 3.2. scale jumps to about 0.025, and 0.03 quantizes to 1, which dequantizes to 0.025. same small weight, five times the error, caused by a neighbor it has nothing to do with. that is the whole game: outliers stretching the scale and crushing everything small around them.
why activations are the hard part
weights are known ahead of time, so you can measure them, group them per channel, and pick scales offline. activations only exist at runtime and their ranges swing with the input. worse, a few activation channels are consistently huge outliers (often 50 to 100 times the median channel), so one per-tensor scale for activations destroys the small channels exactly the way my poisoned example did.
the fixes all amount to finer grouping: per-channel scales, per-group scales over blocks of 32 or 128 values, or keeping the outlier channels in higher precision and quantizing the rest. each step down in bits (8 to 4 to 2) forces finer groups to hold quality, and finer groups mean more stored scales eating back your savings. there is no free setting, only a frontier.
what surprised me
what surprised me is how well 4-bit weights work at all. my intuition said throwing away 75 percent of the bits must visibly break the model, but the errors are roughly random and small relative to typical activation magnitudes, so they partially wash out across a large matmul instead of compounding. the model degrades gracefully rather than falling off a cliff, which is not what i expected from lossy compression applied hundreds of times in a row.
what i would do differently: stop eyeballing outputs and measure. the honest check is perplexity on real text before and after quantizing, plus one downstream task, at 16, 8, and 4 bits with the same calibration data. i have the mechanism clear in my head now. the numbers to back it up are the missing half of this note.
all notes