Compression

NVFP4 quantization delivers 3.18x whole-model weight compression today, plus the theta-bundle size analysis that maps the trained-sharing research direction.

AttnQuant compresses weights today through NVFP4 quantization: 4-bit weights with per-block float8 scales, enabled with one environment variable. On Gemma 4 31B, whole-model weight bytes drop from 61.4 GB to 19.3 GB. Decode gets faster too, but by less than the byte cut implies. This page covers the measured NVFP4 results, then the theta-bundle size analysis that maps how much further trained weight sharing could go.

NVFP4: 3.18x smaller weights

Set ATTNQUANT_QUANT=nvfp4 on the precompute-outside path and attnquant quantizes each effective weight at load time:

The per-element cost of NVFP4 is 0.5625 B: 0.5 B of packed 4-bit weight plus one float8_e4m3fn block scale per 16 elements, which is why the ratio is 3.18x and not 4x. Per-token weight-read bytes equal the resident totals: lm_head is tied to embed_tokens, so it’s resident once but read in full for every token’s logits. These figures are text-only and exclude the 1.15 GB vision tower.

The two NVFP4 paths don’t read the same bytes, and this is the more important comparison:

That’s 1.63x fewer bytes per token against NVIDIA’s NVFP4 checkpoint, achieved with no calibration set. The coverage-matched control below shows the byte difference is exactly what produces the measured throughput advantage: matching the NVIDIA checkpoint’s coverage removes it; see throughput.

The compression is 3.18x rather than 4x because the fp8 per-block scales cost 0.5 bit/param on top of the 4-bit weights, and the embeddings and lm_head stay in bf16. An earlier figure of “RSS 246 GB to 76 GB, 4x smaller” described host-RSS load-time peaks (transient buffers during decomposition and quantization), not resident weight bytes; a 183 GB B200 couldn’t have held a 246 GB bf16 model.

The byte cut buys less throughput than it might appear to promise. Against the built-in NVFP4 path, attnquant reads 1.63x fewer bytes per token and measures +9.4%, because a large fraction of a batch-1 decode step is byte-invariant (host gap plus attention, norms, and elementwise work), and the NVFP4 kernels themselves achieve only about 47% of peak bandwidth against 57-65% for the bf16 families. Quantization is a footprint win first and a throughput win second. See throughput for the decomposition.

The extra footprint saving is not free on accuracy. AttnQuant quantizes all 410 projections including attention, while NVIDIA’s NVFP4 checkpoint keeps every self_attn layer in bf16 (its tied lm_head is bf16 on both paths). The result is 19.295 GB of weights re-read per token against 31.482 GB, 1.63x fewer bytes, with no calibration set. Measured with lm-eval on paired MMLU-Pro at n=12032, adding attention coverage costs −0.41 pp (95% CI [−0.816, +0.001], p=0.056) on top of the −0.44 pp that NVIDIA’s MLP-only quantization already costs, for a total of −0.85 pp against bf16. The attention half sits just inside a pre-registered null, but its point estimate matches an effect the same test called significant, and the three deltas are additive. Read it as a real cost this sample couldn’t resolve, not as zero. See quality for the paired tables and the reasoning.

Quantization runs once at load: a cold pass over the 31B model’s 410 projections takes 1782 s (~30 minutes, of which 1199.6 s is the precompute itself), and a later start that hits the quantized-weight disk cache takes roughly 30 s. See configuration for the flag.

One footprint note for the bf16 fast path: without quantization, peak GPU memory measures about 1.9x the built-in baseline (32.7 vs 17.6 GB on Gemma 3 1B). NVFP4 recovers most of that.

Theta-bundle size: the sharing headroom

Beyond quantization, the theta bundle itself defines a much larger compression target: if many layers share one W0, the bundle can shrink to a few percent of the original weights. The numbers below measure that headroom at real Gemma 3 1B dimensions (26 layers, hidden_size=1152, head_dim=256, intermediate_size=4096, float32). Reproduce them with pixi run pytest tests/test_benchmark.py -k ThetaBundleSize -s.

The ratios assume a single shared W0 per family (G=1), the maximal-sharing setting. The bundle grows with rank because only U and V scale with it; W0 and s are fixed:

The full sweep puts the original projection weights at 1778.91 MB (float32):

Where the bytes go

Breaking the G=1 rank-32 bundle into its four components shows why the ratio stays low: the shared block W0 is stored once per family, the scale s is negligible, and only the low-rank factors U and V carry per-layer cost.

W0 is nearly half the bundle at rank 32 and stays fixed regardless of rank, which is why dropping to rank 4 barely moves the total: the low-rank factors shrink but the shared block doesn’t.

Reaching the sharing headroom needs training

Those single-digit ratios are the target, not a shipped setting: reaching them with production-quality output requires a training or distillation step that fits the shared blocks to the model, because the analytic one-shot decomposition leaves too large a residual at shared settings (the quality page has the measurements). Until then, NVFP4 is the shipped compression path. With trained sharing on top of NVFP4-quantized shared blocks, the combined headroom is roughly 8.5x over bf16.