AttnQuant metrics

See every published attnquant number for throughput, quality, and compression, with the device it was measured on and the command that reproduces it.

Every number in these docs is a measurement, and every measurement names the device and configuration that produced it. This section collects the results in three families:

• Throughput: how much faster attnquant decodes than the built-in path, per model, quantization, and concurrency.
• Quality: how output parity with the original model is verified, and what NVFP4 quantization costs on gsm8k and MMLU-Pro.
• Compression: what NVFP4 quantization saves, and how theta-bundle size scales with rank.

The headline results

Measured on one NVIDIA B200 with capture pinned explicitly on both sides of every comparison:

Rows use different harness configurations and different units, so compare within a row rather than across. All Gemma 4 31B rows are /v1/chat/completions at concurrency 1, reproduced on three independent B200s that agree to within 2%. The Gemma 3 1B row’s raw gap is about 8%, inside that model’s ~20% minimum detectable effect, which is why it reads as no reproducible difference; see How the numbers are measured.

The speed comes from coverage. AttnQuant quantizes all 410 projections and reads 19.295 GB per token; NVIDIA’s NVFP4 checkpoint quantizes 180, keeps every self_attn block in bf16, and reads 31.482 GB. Matching that coverage with ATTNQUANT_QUANT_IGNORE=self_attn returns throughput to baseline, which is the third row above. The trade is about 0.41 pp of MMLU-Pro; see quality.

On Gemma 3 1B in bf16 the same mechanism shows up more sharply, because MAX’s built-in Gemma3ForCausalLM can’t enable capture at all: forcing it raises NotImplementedError: Gemma3Inputs does not define model ABI buffers. AttnQuant implements the missing buffer ABI and turns capture on, worth +30.1% (248.9 vs 191.3 tok/s, pooled over 2 capture-on and 4 capture-off server sessions). With capture matched off, the two paths show no reproducible difference: this model’s spread runs 30-57% within a cell and is session-level, so its minimum detectable effect is about 20%, not the 5% that holds on the 31B. Output quality holds throughout: logit-parity checks verify attnquant is native-equivalent to the original model.

How the numbers are measured

All serving numbers come from max benchmark against a live max serve endpoint on one NVIDIA B200. That pairing is the canonical harness, and every published figure names its harness. Its resolution is model-dependent: about 5% on Gemma 4 31B, but only about 20% on Gemma 3 1B, whose spread is session-level and doesn’t shrink with more prompts per run, so 1B figures are pooled across multiple independent server sessions per cell (4 for the capture-off cells, 2 for the capture-on cell). The batch-1 max generate path reads higher but is disqualified for NVFP4 comparisons: 18-34% raw trial spread, bimodal across processes. Analytic metrics (reconstruction fidelity, bundle size) need no GPU and reproduce deterministically.

Reproduce the numbers

Every metric comes from the test harness in tests/. The analytic metrics run without a GPU; the serve-based metrics need a GPU and a Hugging Face token with the Gemma 3 license:

# Analytic metrics (no GPU, deterministic):
pixi run pytest tests/test_benchmark.py -v -k "ThetaBundleSize" -s
pixi run pytest tests/test_decompose.py -v -s

# Serve-based metrics (need a GPU and an HF token with the Gemma 3 license):
pixi run pytest tests/test_benchmark.py -v -m serve -k Throughput -s
pixi run pytest tests/test_quality.py -v -m serve -s

Each metrics page names the device and configuration its numbers were measured on, so a number is never separated from the conditions that produced it.