Throughput

Measured tokens per second for Gemma 4 31B on one B200 (92 built-in, 101 attnquant, 290 with speculative decoding), and where each number comes from.

Every figure here is tokens per second on one NVIDIA B200, Gemma 4 31B, max serve + max benchmark at concurrency 1 on /v1/chat/completions, median of ≥5 retained trials across ≥2 server sessions. Each is reproduced on at least two idle machines that agree to within 2%.

Results

Speculative decoding is much the largest lever on this model, at 2.7-2.9x over non-speculative, and it’s orthogonal to everything else on this page. The attnquant advantage is roughly the same size with it (+17.2%) and without it (+9.4%).

Device graph capture accounts for about +3% on either path and isn’t the difference between them: built-in measures 92.0 off / 93.8 on, attnquant 100.6 off / 104.9 on. The architecture declaration only sets the default; --device-graph-capture overrides it on either path.

Where the throughput comes from

AttnQuant quantizes all 410 projections to NVFP4. NVIDIA’s checkpoint nvidia/Gemma-4-31B-IT-NVFP4 quantizes 180: its quantization_config.ignore list keeps all 60 self_attn blocks in bf16. (It also names lm_head, which changes nothing: tie_word_embeddings: true makes the tied head bf16 on both paths.) The two paths therefore read different amounts of memory per token:

The third row is the control. Set ATTNQUANT_QUANT_IGNORE=self_attn so attnquant quantizes exactly what NVIDIA quantizes, and the advantage disappears: 92.6 against 92.0, ranges overlapping, where full coverage leads by +9.4% with disjoint ranges. The same control under MTP gives 251.0 against 247.5: 91.9% of the speculative advantage is coverage too.

At ATTNQUANT_BLOCKS=60 the decomposition reconstructs the original weights bit-for-bit, so it adds no approximation and no runtime work; the speed is entirely the byte reduction. The comparison isn’t precision-matched, though: attnquant runs less precision than the baseline, which is what the accuracy figures below price.

What it costs

Full coverage is a trade, not a free lunch. Paired MMLU-Pro at n=12032:

Full coverage costs about 0.41 pp of MMLU-Pro on top of the 0.44 pp that MLP-only NVFP4 already costs. Quality has the paired tables. To decline the trade, set ATTNQUANT_QUANT_IGNORE=self_attn.

Against vLLM

Same box, same benchmark client driving both engines, same checkpoint, prefix caching disabled on both sides. Medians over 48 trials per arm across two sessions with the sweep order reversed. Aggregate output tokens/sec.

MAX’s built-in path trails vLLM by 18% at concurrency 1, narrowing to 6% at concurrency 64. With the two decode fixes in upstream/patches applied, it leads by 10.5% at c=32 and 9.9% at c=64, and is at parity below that.

Inter-token latency is close throughout: 15.84 ms against 15.20 at c=64, a 4% difference. The aggregate gap tracks GPU utilization instead: MAX runs at 76-80% across the sweep against vLLM’s 89-96%.

Time to first token

enable_in_flight_batching lets prefill ride along with the decode batch instead of running in its own iterations. run_max.py turns it on for you; see configuration.

With the flag on, MAX’s median TTFT is well below vLLM’s at every concurrency above 8. Two limits: it does nothing at c=1 and c=4, where there is no decode batch to join, and it moves the median only, with p99 TTFT changing by less than 7%.

Reproduce a cross-engine comparison

max benchmark --flush-prefix-cache POSTs /reset_prefix_cache, which vLLM answers with a 404 that the client logs as a warning and otherwise ignores (benchmark_serving.py:936). With a fixed seed the same prompts repeat every trial, so an unflushed engine’s cache hit rate climbs into the high tens of percent while the other engine serves every prefill. Disable prefix caching at the server on both sides and read the hit rate back from each server’s metrics.

A useful check: compute the prefill FLOP floor for your shape and compare it to the measured TTFT. Gemma 4 31B costs 57.47 GFLOP/token, so a 64x141-token wave is 519 TFLOP and can’t complete faster than about 105 ms on a B200. A TTFT at or below that floor means something is being skipped.

Gemma 3 1B and Gemma 4 31B bf16

On Gemma 3 1B the built-in Gemma3ForCausalLM can’t capture at all, because Gemma3Inputs defines no model ABI buffers, so there capture genuinely is a capability attnquant adds rather than a flag the built-in path leaves unset. The 1B spread is 30-57% within a cell, driven by session-level rather than sample-level variance, putting the minimum detectable effect at 20-25%; on the 31B the MDE is about 5%.

Provenance

All numbers below come from one NVIDIA B200 running the software stack in this table:

The canonical harness

Every decode number on this page comes from max serve plus max benchmark against the live endpoint. Nothing else is quotable, because nothing else resolves finely enough to see the effects being claimed.

Bring up the server, then drive it at concurrency 1:

pixi run max benchmark \
  --model google/gemma-4-31B-it \
  --backend modular --endpoint /v1/chat/completions \
  --host localhost --port 3100 \
  --max-concurrency 1 \
  --dataset-name random \
  --random-input-len 128 --random-output-len 128 \
  --num-prompts 8 --temperature 0

The reporting rules that go with it:

• Run at least 5 trials spread across at least 2 independent server sessions. Within-session repeats don’t catch process-level bimodality.
• Report the median tok/s and mean_tpot_ms together. A tok/s figure without its TPOT hides which end of the step moved.
• Quote the device_graph_capture line from the serve log alongside the number, and pin capture explicitly on both sides of any comparison rather than letting the architecture default decide. Mismatched capture is the single most common way to produce a wrong answer here.
• Publish the endpoint with every number: Without speculation the two endpoints agree to within ~1%. Under speculative decoding they don’t: the same cells measure +64% (built-in) and +138% (attnquant) higher on /v1/chat/completions than on /v1/completions, because raw random token ids are far harder for a drafter to predict than the same ids inside a chat template. That’s enough to reverse the sign of a attnquant-vs-built-in comparison: +16.9% on chat, −19.2% on completions. Every number on this page is chat.
• Name the harness every time. The model-only max generate path reads about 35% higher on the same configuration and isn’t comparable.
• Never difference a speculative and a non-speculative measurement: On this stack the non-speculative path isn’t the γ=0 limit of the speculative one; its step sits ~0.8-0.9 ms above the speculative fit’s zero-pass intercept. Subtracting across the two families has produced three separate wrong results here, including a “speculation costs only 2.3%” figure whose true value is ~11%.
• Byte-identity is a valid correctness gate only within one fixed configuration at concurrency 1: Greedy output on this stack isn’t deterministic across forward-pass shape. On one server, non-speculatively, at temperature 0, sequential requests, 4-concurrent and 24-concurrent all produce different output, and two runs at 24-concurrent differ from each other. The cause is near-ties (at 21 of 22 divergent positions only two candidate tokens ever appear, median separation 0.005 nats) resolving differently under different reduction orders; fp8 KV was tested as the suspected amplifier and refuted. Consequences: an md5 comparison across two different configurations was never an equality test, at concurrency above 1 md5 gates are invalid in either arm, and accuracy comparisons must use an eval score rather than output identity.
• A speculative throughput number quoted beside a non-speculative baseline carries an unmeasured accuracy caveat: MTP is distribution-preserving by construction, so at temperature 0 every emitted token is an argmax over the target’s logits and acceptance can’t change the text. But its practical accuracy on this checkpoint hasn’t been evaluated, and its output legitimately differs from the non-speculative path for the reason above.
• Say which metric a number is. median_output_throughput is a per-request median that includes prefill; mean_itl_ms is pure decode. They disagree by up to 2.5% and their ratio isn’t constant across configurations. Quote throughput for a serving decision and the step time for a mechanism claim.
• Don’t compare numbers across machines unless both were measured idle. Two uncontended B200s agree to within 2%; a contended one read 14% low and produced every superseded figure on this page.

Demonstrated resolution on Gemma 4 31B is 2.6-4.8% within a session and 2.5% between sessions, which puts the minimum detectable effect at about 5% on that model. Anything smaller isn’t a result.

That bar doesn’t transfer to Gemma 3 1B. On the 1B the same harness shows 30-57% within-cell spread, and pooling 4 independent server sessions per cell doesn’t collapse it because the variance is session-level: raising --num-prompts from 8 to 64 leaves it unchanged. Treat the minimum detectable effect on Gemma 3 1B as about 20-25%, and pool across sessions before quoting anything.

Settled baselines for the built-in NVFP4 path on Gemma 4 31B:

That baseline is reproduced on three independent B200s, which agree to within 2% when all are idle: 91.19, 91.44 and 93.11 tok/s. Run the box idle: a machine carrying other GPU work read 14% low on the same cell.

For Gemma 3 1B, run the same max serve + max benchmark pairing against google/gemma-3-1b-it, repeated across at least 4 separate server sessions per configuration, because a single session doesn’t resolve anything on this model:

pixi run max benchmark \
  --model google/gemma-3-1b-it \
  --backend modular --endpoint /v1/chat/completions \
  --host localhost --port 3100 \
  --max-concurrency 1 \
  --dataset-name random \
  --random-input-len 128 --random-output-len 128 \
  --num-prompts 8 --temperature 0

How the decode step breaks down

The model-only decode harness reads higher, and doesn’t reproduce

The batch-1 max generate path reports a higher number than the serve harness because it excludes server-side per-request work. The rows below come from the earlier campaign and have not been re-measured on a verified-idle box, so read them as within-harness comparisons only:

An earlier published figure of 120.0 tok/s for the attnquant NVFP4 configuration doesn’t reproduce; the re-measurement is 112.39 tok/s, 6.3% lower. The built-in figures on this harness do reproduce (NVFP4 107.8 to 108.69, +0.8%; bf16 83.4 to 82.98, -0.5%).

Treat this harness as disqualified for NVFP4 comparisons. Raw trial spread is 18-34%, and the results are bimodal across processes at fixed shapes with no clock throttling, well outside the ~5% effect the comparison is trying to resolve. Use max serve + max benchmark instead.

What quantization actually saves

Weight bytes on Gemma 4 31B:

That’s 3.18x whole-model, 3.56x projections-only. It’s not 4x: NVFP4 costs 0.5625 B/element (0.5 B of packed weight plus one float8_e4m3fn scale per 16 elements). These come from the safetensors headers and the loaded tensor manifest, and are text-only, excluding the 1.15 GB vision tower.

An earlier figure of “RSS 246 GB to 76 GB, 4x smaller” described host-RSS load-time peaks, not weight sizes. Those peaks are dominated by transient buffers during decomposition and quantization, and 246 GB was never a resident weight footprint: the B200 has 183 GB.

Quantization runs once at load. Measured on an idle B200: a cold start takes 1782 s to reach a serving-ready server, of which 1199.6 s is the precompute over all 410 projections. A later start that hits the quantized-weight disk cache takes roughly 30 s.

What a decode step is actually made of

The byte cut doesn’t convert into throughput one-for-one. Against the built-in NVFP4 path, attnquant reads 1.63x fewer bytes per token (19.295 GB against 31.482 GB) and measures +9.4%. Against bf16 the byte cut is 3.18x. The reason the conversion is so far from linear is below: roughly half a batch-1 decode step doesn’t scale with weight bytes at all.

An nsys study decomposes that directly, on a 12.29 ms built-in NVFP4 step:

The byte-invariant GPU row breaks down as attention 0.77 ms, norms 0.89 ms, elementwise 0.44 ms, and activation quantization 0.31 ms. The four rows are measured independently and overlap slightly (they sum to about 107% of the step), so read the shares as approximate.

Byte-invariant cost is the host gap plus byte-invariant GPU work: 5.77 ms, or 47% of the step. About half a batch-1 decode step is where extra bytes can’t reach, so treating it as a pure bandwidth-utilization problem overstates what quantization can buy.

The byte-based roofline is consistent with that, and it can now be stated exactly rather than estimated, because the per-token byte counts are derived from the loaded tensor manifests rather than assumed:

Both paths sit far below their own byte ceiling, attnquant further below at 24% of it, because cutting bytes raises the ceiling without touching the byte-invariant half of the step. Neither is bandwidth-bound at batch 1, and a faster quantized GEMM isn’t the lever it looks like.

More pointedly: the bf16 roofline is 130.3 tok/s, and neither NVFP4 path reaches it. Built-in measures 92.0 and attnquant 100.6. So on this stack, quantization hasn’t yet bought throughput that bf16’s own bandwidth ceiling would have forbidden. The byte cut is real and the resulting speedup is real, but the machine is leaving so much on the table that bytes aren’t currently what limits it. vLLM reaches the same ceiling from the other direction, running this checkpoint 18.4% faster at concurrency 1 and 6.1% faster at concurrency 64; both engines are far from the byte limit.

The NVFP4 MLP kernels are the least bandwidth-efficient on the step

Achieved bandwidth by weight family, from the same profile:

Cutting MLP bytes 3.56x therefore doesn’t buy a 3.56x MLP speedup: the kernels reading the smaller weights run at the lowest fraction of peak of any family on the step.

Measured per kernel rather than per family, the two MLP GEMMs differ sharply, and part of the gap is configuration rather than kernel quality:

Both improvements are tuning-table rows, not kernel changes: Gemma 4’s shapes have no entry in _get_tuning_list_sm100_nvfp4(), so they fall to a fallback whose k_group_size is chosen for legality rather than speed. What remains after tuning is genuine kernel headroom: down runs 42 CTAs on 148 SMs at 28% occupancy, and does so identically in a leading third-party kernel library, so roughly 2x is available to whoever implements split-K.

It’s not a dequantization pass. There is no dequant kernel at all: 0 launches, 0.00% of GPU time. The block-scaled GEMM consumes fp4 weights directly, so the hypothesis that dequantization doubles HBM traffic is ruled out.

Host overhead: dispatch is not the cost, but a blocking sync was

Two measurements bound how much of the host gap is per-launch dispatch, and both say none of it:

• Forcing device graph capture put 99.1% of kernels into graphs and moved the host gap from 3316 µs to 3318 µs: no change. Whatever the 3.32 ms is, it’s not per-launch dispatch cost.
• A py-spy study places ~9.5 ms of the step inside MAX’s native _execute_device_tensors, of which at least 2666 µs is genuine GPU wait. Removable Python is ~1.0-1.6 ms/step, not the ~2.8 ms a naive reading suggests. A mechanically correct caching change that removed 262 µs/step of real host work measured −1.7% throughput with overlapping envelopes.

Neither of those measures the cost that turned out to matter, because both look at diffuse host time: interpreter work and launch overhead spread across the step. Profiling with nsys --cuda-graph-trace=node finds the idle concentrated instead in two discrete gaps of ~1290 µs and ~864 µs per step, together 94% of all GPU idle, both in the sampling tail.

They trace to one line: ctx.synchronize() in fused_token_sampling (max/kernels/src/nn/topk.mojo:2029), a blocking full-stream sync backing a NaN check whose flag buffer valid_buf is write-only, with no kernel reading it. Removing it measures −17.88% step time with byte-identical output, cuStreamSynchronize going 1161 → 0, and GPU idle 2.281 → 0.138 ms per step.

Aggregate host time and launch counts can both look healthy while a single serialising wait dominates the step. Measure the gaps, not the totals.

Gemma 4 31B in bf16: parity

In bf16 the precompute-outside path uses vanilla Linear/StackedLinear/MLP classes with the reconstructed weight stored as .weight, making the compiled graph identical to the base model’s. The benchmark uses max benchmark with 50 prompts, 512 input tokens, and 128 output tokens at concurrency 50, and request throughput ties: 13.14 req/s attnquant vs 13.15 req/s built-in.

One requirement for parity: keep the overlap scheduler enabled (don’t set supports_overlap_scheduler=False), or throughput drops about 5%.

The load-time profile is the same cold-start cost measured above: 1782 s cold, roughly 30 s once the quantized-weight disk cache is warm.

Speculative decoding

Multi-token prediction is the largest lever on this model by a wide margin. Serving the NVFP4 target with google/gemma-4-31B-it-assistant as the drafter, γ=3, capture on both sides:

For scale, every other effect on this page is in the single-digit-to-17% range. If you serve this model and can tolerate a drafter, MTP is the first thing to turn on.

AttnQuant does compose with MTP, but only because the framework registers a second architecture to make it work. MAX rewrites the target architecture name to UnifiedMTPGemma4ForCausalLM before resolving it whenever a drafter is supplied, so an architecture registered only under the pre-rewrite name is silently bypassed: the server comes up, accepts the attnquant environment variables, and serves MAX’s built-in model. attnquant registers under both names for exactly this reason.

Acceptance is a workload property, not an implementation gap. Measured accept_len is 2.75-2.78 at γ=3 on random 128/128, and 3.237 on natural chat at γ=4, against Google’s published 4.86 for this pair. Logit-verified against the HuggingFace Gemma4AssistantForCausalLM reference on identical prompts, MAX scores 3.237 against the reference’s 3.127, 3.5% ahead, with both decaying at the same rate per draft position. The same reference drafter reaches accept_len 5.000 on a verbatim-copy prompt, so the range for this drafter is roughly 3.1 to 5.0 depending entirely on how predictable the text is. Any comparison against a published acceptance figure has to match the workload first. Three candidate explanations were tested and refuted: a bf16-drafter-versus-NVFP4-target mismatch (+0.085), typical-acceptance versus greedy rejection (delta 0.001 at temperature 0), and an unmasked 262K-vocab drafter head (the checkpoint has no centroid tensors).

The drafter costs about 0.39 ms per pass, measured across γ=1/3/7 with residuals under 0.04 ms, roughly 11% of the step at γ=3. Derive that from a γ sweep rather than by subtracting a non-speculative step from a speculative one; the two paths don’t share a per-step baseline.

The advantage doesn’t scale with γ. Both paths pay the same per-pass cost, so the ratio erodes slightly as γ grows: step-time ratios run 1.133 at γ=1, 1.134 at γ=3, 1.119 at γ=7. Treat the attnquant advantage under speculation as a constant mid-teens percent, not something that compounds.

MTP output isn’t bit-identical to the non-speculative path, and that’s expected rather than a defect: at temperature 0 every MTP token is an argmax over the target’s logits, so acceptance can’t change the text. Greedy output on this stack isn’t stable across forward-pass shape (see the harness rules). MTP’s practical accuracy on this checkpoint hasn’t been evaluated, so a speculative throughput number quoted beside a non-speculative baseline carries that open caveat.

Gemma 3 1B in bf16

AttnQuant enables device graph capture on a path that cannot otherwise use it

The capture-isolation pass pools 4 independent server sessions per capture-off cell on the canonical harness at c=1 (tree SHA 8cf0526), with the output-token count identical (6733) in every cell:

Capture on versus off on the attnquant path is +30.1% (Mann-Whitney z=−4.28), and the direction holds in every session. At a ~20% minimum detectable effect that clears the bar comfortably.

The built-in path can’t answer it. Forcing capture on MAX’s Gemma3ForCausalLM raises NotImplementedError: Gemma3Inputs does not define model ABI buffers (pipeline_model.py:205): the architecture doesn’t implement the buffer ABI that serve-side capture requires. AttnQuant implements it as AttnQuantGemma3Inputs.buffers and declares supports_device_graph_capture=True (models/gemma3/arch.py:87). So on Gemma 3 1B, capture is a capability attnquant adds, not a flag someone forgot to set. That’s a stronger claim than the 31B case, where the built-in path can be forced into capture and gains about 0%.

With capture matched off, there is no reproducible advantage

The two capture-off cells put attnquant 8.0% below the built-in path (Mann-Whitney z=−2.28). Don’t quote that as a result. The sign isn’t stable: an earlier two-session pass on an identical configuration read +16.1%, and the next read −13.5%. Within-cell spread ran 30-57% even after 8x the measurement mass, and raising --num-prompts from 8 to 64 didn’t shrink it, because the variance is session-level rather than sample-level.

What the data supports is the negative: with capture matched off, attnquant shows no reproducible advantage over the built-in path on Gemma 3 1B. Fusion still cuts the precompute-outside graph from 21 kernel launches per layer to 8, but that saving is below this model’s measurement floor.

Figures from the earlier harness, not re-verified

Three Gemma 3 1B figures published before the capture-isolation pass came from a different harness on a different tree and were not re-measured. They’re kept here, labelled, rather than restated as current results:

• Single-stream decode, 392 built-in vs 439 tok/s attnquant: Superseded by the table above and not comparable to it: the absolute magnitudes differ because the harnesses differ, so the two sets must not be mixed in one table.
• Concurrent throughput at c=32, 63.8 built-in vs 68.0 req/s attnquant (+6.5%), from max benchmark --max-concurrency 32 --num-prompts 160, median of 3 runs per mode. The original write-up rested on three within-session runs having non-overlapping ranges; the session-level variance measured above means within-session repeats can’t establish that. Unconfirmed, and there is no re-measured counterpart.
• TTFT at batch 1, 10.5 ms built-in vs 13.6 ms attnquant: Also from the earlier harness and not re-measured.

The runtime reconstruction path does not reach dense parity

Everything above runs the shipped precompute-outside path. The runtime path, with ATTNQUANT_PRECOMPUTE_OUTSIDE unset so W0 and the low-rank delta are rebuilt in the graph on every forward pass, was measured separately on Gemma 3 1B bf16, canonical harness, 5 sessions per cell, at matched capture. This is the tree where attnquant_qkv_gemm is wired up, issuing 8 GEMM launches per layer against vanilla’s 4.

A pre-registered gate (ratio ≤ 0.80 with the CI upper bound below 0.90) classifies the capture-off cells STILL-SLOW. Plainly: rebuilding the weight at runtime is materially slower than serving a dense weight, and the shipped fast path is precompute-outside.

Two qualifications go with it:

• How much the launch-count fix helped is unresolved. The confidence interval spans both the older stale-tree ratio of 0.48 and the 0.758 predicted for the fixed tree. What the interval does resolve is that the path doesn’t reach parity.
• The capture-on row isn’t a win. Vanilla gemma3 can’t run with capture at all (NotImplementedError: Gemma3Inputs does not define model ABI buffers), so 251.7 tok/s is compared against vanilla with capture off, and the interval [0.758, 1.227] contains 1.0. Read it as a tie, not a gain.

The two ATTNQUANT_BLOCKS settings also don’t separate: G=1 versus G=26 measures 1.035x with capture off (CI [0.780, 1.345]) and 1.090x with capture on (CI [0.996, 1.245]). Band sharing has no measured throughput effect on this model: it’s a bundle-size dial, not a speed dial.

Both negatives are limited by the same thing. Session-level spread on the 1B harness runs 30-57% and doesn’t shrink with more prompts, which puts the real minimum detectable effect here at about 20-25% rather than the ~5% that holds on the 31B.

Decode, TTFT, and TPOT by rank and path

The two reconstruction paths are math-identical, so any difference here is pure performance. The GEMM path (stock max.graph matmuls) is faster than the custom-op path at every rank on the B200: 48% faster at rank 4 (213.9 vs 144.6 tok/s) and over 10x faster at rank 32 (166.1 vs 15.0 tok/s), because custom-op reconstruction is launch-bound while GEMM lowers to MAX’s device matmul kernels. This is why the auto-select routes NVIDIA GPUs to GEMM.

With ATTNQUANT_GEMM unset on the B200 (a non-Metal accelerator), auto-select picks the GEMM path at any rank and delivers GEMM-path throughput:

The scorecard, side by side

Capture isolation on the left-hand axis, and the one cost that measures cleanly on this model, peak GPU memory on the bf16 fast path:

The memory bar reflects the measured peak of the bf16 fast path (32.7 vs 17.6 GB on Gemma 3 1B). NVFP4 quantization recovers most of that by shrinking the weights 3.18x.

Related

• Serving configuration: what ATTNQUANT_PRECOMPUTE_OUTSIDE and ATTNQUANT_QUANT do
• Reconstruction: the math and path selection
• GPU acceleration: capture, fusion, and the profiling data