Quality

How attnquant output quality is verified, from logit parity at the shipped operating point to paired MMLU-Pro at n=12032 showing that full-coverage NVFP4 most likely costs about 0.4 pp beyond MLP-only quantization.

AttnQuant output is native-equivalent to the original model at the shipped operating point. At passthrough (G = num_layers: one W0 per layer), every projection rebuilds its original weight exactly, so the served graph computes the same function as the built-in path, and the parity test suite verifies it. This page defines the verification method, reports where it holds, and measures what NVFP4 quantization costs on downstream benchmarks.

Native-equivalent at passthrough

At G = num_layers each layer keeps its own weight as W0, the per-row scale s lands near 1, and the low-rank delta is near zero, so the reconstructed weight is the original weight and generation matches the original model. This holds for both models on the B200:

• Gemma 3 1B (google/gemma-3-1b-it, 26 layers): at G=26 it generates correct English matching the original, for example The capital of France is Paris. Parity and quality tests pass.
• Gemma 4 31B (google/gemma-4-31B-it, 60 layers): at G=60 greedy decode, parity, and quality tests all pass.

Native-equivalent means logit-parity, not bit-exact: the attnquant model’s output logits match the original within a max logprob delta below 1.0, and greedy (argmax) tokens agree on at least 80-90% of positions. Small numeric differences are expected from reordered floating-point reductions; the parity thresholds bound them.

Two metrics report parity:

• Greedy-decode match runs the same prompts through the attnquant and original models at temperature 0 and reports the fraction of matching output tokens.
• Logprob delta compares each model’s logprob for the same tokens and reports the gap against the logit-parity threshold.

One implementation rule keeps parity intact when you port a new model: AttnQuantMLP resolves the activation from config.hidden_activation (gelu_tanh for Gemma 3 and Gemma 4) rather than hardcoding one. A wrong activation breaks output even when every weight is correct; see the extending guide.

Verify it yourself

Run the parity and quality suites against a live server:

pixi run pytest tests/test_quality.py -v -m serve -s

Or smoke-test generation directly at passthrough:

ATTNQUANT_BLOCKS=26 pixi run python run_max.py generate \
  --model google/gemma-3-1b-it --custom-architectures attnquant \
  --prompt "The capital of France is" --max-new-tokens 16

What NVFP4 quantization costs on downstream benchmarks

Logit parity covers the bf16 path, where the rebuilt weight equals the original. NVFP4 changes the numerics deliberately, so it needs a separate answer measured on tasks. These runs use lm-eval on one NVIDIA B200, on Gemma 4 31B, and the two NVFP4 configurations differ in what they quantize as well as how:

Every configuration runs the same items, so the informative comparison is paired. The paired deltas use McNemar’s test on identical items, in percentage points:

The setup validates against a published score

Built-in bf16 measures 84.88 ± 0.32 on MMLU-Pro against NVIDIA’s published 85.25 for the same model, within the run’s 95% margin of error. The harness, prompt format, and scoring reproduce an external reference rather than an internal convention, which is what makes the deltas worth reading.

The attnquant implementation is accuracy-neutral

The bf16 control separates the attnquant graph from the quantization scheme, and it comes back clean. Against built-in bf16, attnquant bf16 is 0/0 discordant on gsm8k (not one of the 1319 problems changed outcome in either direction) and +0.05 ± 0.11 pp on MMLU-Pro. Reconstruction, fusion, and the precompute-outside path cost nothing measurable. Whatever the NVFP4 rows give up is the quantization scheme, not attnquant.

What quantizing attention costs

AttnQuant quantizes all 410 projections, the 230 attention projections included. NVIDIA’s NVFP4 checkpoint doesn’t: its quantization_config.ignore list keeps all 60 self_attn layers in bf16. (It also names lm_head, which changes nothing because tie_word_embeddings: true makes the tied head bf16 on both paths.) The contrast is therefore attention coverage alone, and it buys 1.63x fewer weight bytes per token (19.295 GB against 31.482 GB) with no calibration set.

It costs about 0.41 pp of MMLU-Pro. The measurement sits just inside a conventional null at p = 0.056, but the evidence reads as a real effect the sample couldn’t quite resolve:

• The 95% CI upper bound is +0.001 pp; it contains zero by a thousandth of a point.
• The point estimate, −0.41 pp, is the same size as the −0.44 pp that MLP-only NVFP4 costs, which the identical test called significant at p = 0.025.
• The three contrasts are additive: −0.44 (MLP alone) plus −0.41 (adding attention) gives −0.85, and the directly measured total is −0.848 at p = 6x10⁻⁵.

Discordance is treatment-driven rather than scheduling noise. Greedy output on this stack varies with forward-pass shape, so a fourth arm bounds that: a different compiled graph over byte-identical weights at the same concurrency differs by only 37 discordant items (0.31%) against the coverage contrast’s 629 (5.2%). Noise accounts for at most about 6% of it.

In practice this is a trade you can take or decline. Full coverage buys roughly 9% non-speculative throughput, 17% under speculative decoding, and 1.63x fewer weight bytes, for 0.41 pp of MMLU-Pro on top of the 0.44 pp NVFP4 already costs on the MLP, or −0.85 pp total against bf16. Set ATTNQUANT_QUANT_IGNORE=self_attn to reproduce the NVIDIA checkpoint’s coverage instead; the accuracy returns and so does the baseline throughput.

One caveat on attribution: the tree carries both the shared-weight_scale_2 fix and can_use_fused_mlp=True. The fused-MLP half is measured accuracy-neutral (−0.025 pp, p = 0.742), so it’s not contributing here; separating the scale-sharing half would need a per-projection-ws2 arm, which hasn’t been run.

Research: why shared configurations need training

Settings below passthrough (G < num_layers) share one W0 across layers to shrink the bundle (see compression). The measurements below show why those settings aren’t shipped: the one-shot analytic decomposition leaves a residual that a small low-rank delta can’t absorb, so output quality degrades below the usable bar at every shared setting tested. Closing that gap is a training or distillation problem, and the data here scopes it.

Reconstruction error

The predictor is the relative Frobenius error of the reconstructed weight against each true weight (see the reconstruction formula), averaged over all layers and projection families, measured on the actual safetensors.

For Gemma 3 1B (original projection weights total 1778.91 MB in fp32):

At G=1 the reconstruction is 95% off at rank 4 and still 45% off at rank 256. Even mild sharing (G=13, two layers per band) is 66% off at rank 4. G=26 is exactly zero because it’s passthrough. The residual is dominated by the large MLP matrices (up_proj, 4096x1152, always the worst family).

Gemma 4 31B shows the same pattern: at G=1 the average Frobenius error is 90.7%, at G=5 it’s 60.9%, and only passthrough reaches 0%.

The residual is full-rank

The residual weight - s·W0 has no low-rank structure to exploit: rank 1024 captures only 62% of its energy, with no knee in the spectrum.

Alternating least squares barely improves the shared block, confirming the constraint is the residual’s rank structure, not the choice of W0:

What this means

Analytic decomposition alone can’t produce a shared configuration with production-quality output: reaching one requires a training or distillation step that fits the shared blocks and deltas to an objective. That’s the research direction these measurements mark out. For weight compression today, use NVFP4 quantization: 3.18x smaller whole-model weights at passthrough, with the measured accuracy cost above rather than exact parity.

Reproduce the sweep

Sweep the (G, rank) grid on Gemma 3 1B:

for G in 1 2 13 26; do for r in 4 32 64; do
  ATTNQUANT_BLOCKS=$G ATTNQUANT_RANK=$r pixi run python run_max.py generate \
    --model google/gemma-3-1b-it --custom-architectures attnquant \
    --prompt "The capital of France is" --max-new-tokens 16
done; done

Reproduce the Frobenius error tables with pixi run python over the real safetensors and core.decompose.relative_frobenius_error() (see decompose).