Skip to content
LLM Serving Glossary

Quantization (making models smaller)

Quantization (quant) — Storing model weights (and sometimes activations/KV cache) in fewer bits than the original 16, trading a little quality for a lot less memory and often more speed. “4-bit is not free” — quality loss is real and must be validated.

Number formats —

  • BF16 / FP16 — 16-bit floats; the “full quality” baseline for inference.
  • FP8 — 8-bit float in two sub-formats: E4M3 (4 exponent, 3 mantissa bits — better precision, smaller range) and E5M2 (5 exponent, 2 mantissa bits — larger range, won’t overflow in gradients). vLLM’s --kv-cache-dtype fp8_e4m3 uses E4M3 for KV cache (the tighter range is fine for cached values); weights sometimes get E5M2 for training stability. Halves memory vs 16-bit with usually negligible quality loss — a common default for serving big models.
  • FP4 (e2m1) — 4-bit float; e2m1 is the bit layout (2 exponent, 1 mantissa bit). Very small, but quality risk is significant — blamed for one model’s degraded output. Only 16 representable values, so FP4 is never used alone: weights are stored in small blocks that share a scale factor, and the two competing formats below differ exactly in how that block scaling works.
  • MXFP4 — the open OCP Microscaling (MX) standard: blocks of 32 values sharing one E8M0 scale (8-bit, exponent-only — powers of two). Cross-vendor: supported on NVIDIA Blackwell and AMD hardware. MXFP8 is the 8-bit sibling in the same MX standard, sometimes paired with MXFP4 weights for activations.
  • NVFP4 — NVIDIA’s own variant, Blackwell-only: smaller blocks of 16 values with a finer FP8 (E4M3) scale, plus a per-tensor FP32 scale on top. The finer/smaller-block scaling tracks outliers better, so it usually loses less quality than MXFP4 — at slightly more storage overhead.
  • Ternary / 1.58-bit / BitNet — the extreme end: every weight is just −1, 0, or +1 (log₂3 ≈ 1.58 bits of information). The step below is binary / 1-bit (weights only ±1). Requires training the model that way from the start; rewards are tiny files and multiplication-free inference.
  • INT8 / INT4 — 8-/4-bit integers.
  • W4A16 / W8A8 — W = how the stored weights are quantized, A = the precision the actual math runs at. Weights are the fixed numbers sitting in GPU memory; activations are the in-flight values computed from your input as it flows through the model. W4A16: weights stored in 4-bit to save memory, but unpacked to 16-bit for every multiplication — so it saves memory and memory bandwidth, not compute. W8A8: weights and activations in 8-bit, so the GPU can use its faster 8-bit math units — saves memory and compute, but quantizing activations is riskier for quality (the SmoothQuant problem).

GGUF quant types (Q4_0, Q4_1, Q4_K_M, Q8_0, IQ2_XS, …) — The naming scheme for quantization levels of GGUF files (llama.cpp / Ollama). Reading the name: Q<bits>_<variant> — the number is bits per weight, the suffix is how the scaling works:

  • Q4_0 / Q5_0 / Q8_0 — the original (“legacy”) scheme: weights are grouped in blocks of 32, each block stores one scale factor. Simple, slightly lossier.
  • Q4_1 / Q5_1 — same, but each block stores a scale and an offset (minimum), recovering a bit more accuracy for a bit more size. So Q4_1 ≈ slightly bigger + slightly better than Q4_0.
  • K-quants (Q3_K, Q4_K, Q5_K, Q6_K) — the newer scheme: “super-blocks” with smarter, finer-grained scales, and mixed precision — important layers get more bits. The _S / _M / _L suffix (e.g. Q4_K_M) picks small/medium/large within that: how many layers get the higher-precision treatment. Q4_K_M is the common “sweet spot” recommendation.
  • IQ-quants (IQ2_XS, IQ3_M, …) — “importance-matrix” quants for extreme compression (≲3-bit): calibration data decides which weights deserve precision. Usable where plain Q2/Q3 would fall apart, at some CPU speed cost. The size ladder within one bit level runs XXS < XS < S < M < L (bigger = more precision); IQ4_NL is the “non-linear” variant whose value grid is spaced to fit weight distributions better than evenly-spaced levels.
  • TQ1_0 / TQ2_0 — ternary types for BitNet-style 1.58-bit models (weights −1/0/+1, see the ternary entry above); only meaningful for models trained ternary, not a way to shrink a normal model.
  • Recipe prefixes/suffixes (UD-, _XL, …) — a prefix like UD- (Unsloth Dynamic) or an extra suffix like _XL marks a quanter’s own recipe on top of a standard type: typically dynamic per-layer bit allocation, with sensitive layers (embeddings, lm-head) kept at higher precision. The standard type after the prefix (UD-Q4_K_XL → Q4_K) still tells you the base format; the rest is the uploader’s tuning.

Rule of thumb: higher number = better quality + bigger file; at equal bits, IQ > K > _1 > _0 in quality. Q8_0 is near-lossless; Q4_K_M is the typical quality/size compromise; below Q3 expect visible degradation.

Dequantization (dequant) — Unpacking quantized weights back to a computable format at inference time, done per matrix multiply by a GPU kernel. Its cost varies by quant type and GPU architecture — which is why a bigger Q8_0 can decode faster than a smaller IQ quant on hardware where the IQ dequant kernels are expensive. Benchmark quants on your own hardware; smaller ≠ faster.

QAT (Quantization-Aware Training) — The model was trained knowing it would be quantized, so the quantized version loses much less quality (e.g. Gemma’s QAT checkpoints). Naming pattern worth decoding: a repo called …-qat-q4_0-unquantized is not a contradiction — it holds the full-precision weights that came out of a QAT run targeting q4_0, published so you can do the final quantization to that format yourself.

Pruning — The other compression axis besides quantization: delete parts of the model (layers, neurons, experts) instead of shrinking the numbers. Structured pruning (width/depth pruning) removes whole layers or hidden dimensions, usually followed by distillation to heal quality — NVIDIA’s Minitron recipe, behind the smaller Nemotron models. Composable with quantization.

REAP (Router-weighted Expert Activation Pruning) — Cerebras’ one-shot expert pruning for MoE models: drop the least-useful experts (scored by router gate values × activation norms) — no retraining, ~50% fewer experts at near-lossless quality on code/generative tasks. A -REAP- infix in a model name (e.g. GLM-4.6-REAP-252B-A32B) means such a pruned checkpoint: total params shrink, active params stay the same. The paper also argues pruning beats expert merging (fusing similar experts into one), which loses routing granularity.

Calibration — For post-training quantization: running sample data through the model to choose good scaling factors. Bad calibration → subtle quality bugs (e.g. a “miscalibrated logit tail” at temperature 1.0).

Quantization toolkits/formats — compressed-tensors (-ct) (checkpoint format used by RedHatAI et al.), AutoRound (Intel), AWQ, GPTQ, llm-compressor, modelopt (NVIDIA TensorRT Model Optimizer — what NVIDIA’s own NVFP4/FP8 checkpoints on HuggingFace are made with), GGUF (llama.cpp’s file format — see the GGUF quant types above).