Runtime configuration

Canonical reference for every attnquant environment variable and config.json field, including the fast decode path and NVFP4 quantization.

AttnQuant reads its behavior from two places: config fields in the model’s config.json, and environment variables that override those fields at runtime. Environment variables win. They apply to both the decomposition adapter and the served model graph, so the theta bundle and the served weights stay consistent. This is the one place to look up a flag or field; other pages link here rather than restate them.

If you serve on an NVIDIA GPU, start with these settings:

• Set ATTNQUANT_BLOCKS to the model’s layer count (26 for Gemma 3 1B, 60 for Gemma 4 31B) for the passthrough operating point, where output matches the original model. The config default is 1, which is an analysis setting, not a serving setting.
• Set ATTNQUANT_PRECOMPUTE_OUTSIDE=1 for the fastest decode.
• Leave ATTNQUANT_GEMM unset so device-aware selection picks the GEMM path.
• Add ATTNQUANT_QUANT=nvfp4 for 3.18x smaller whole-model weights on large models.

Environment variables

Each variable below overrides the corresponding config field for both decomposition and serving:

ATTNQUANT_PRECOMPUTE and ATTNQUANT_PRECOMPUTE_OUTSIDE are different flags, so set the one you mean. ATTNQUANT_PRECOMPUTE builds W_eff inside the graph and recomputes the delta on every forward pass. ATTNQUANT_PRECOMPUTE_OUTSIDE builds it once at load time and never recomputes it, which is what you want for decode.

ATTNQUANT_PRECOMPUTE_OUTSIDE

When set to 1, each layer’s effective weight W_eff = s · W0 + U Vᵀ is materialized once at load time, outside the graph (see execution paths). Every projection then runs a single x @ W_effᵀ matmul, the same launch count as the built-in dense path, with Q/K/V fused into one matmul. That takes the path from 21 kernel launches per layer to 8. This is the recommended setting on NVIDIA GPUs.

If a layer’s theta keys can’t be matched to a shape- and band-compatible W0 at load, the precompute fails immediately with the offending layers listed. That almost always means ATTNQUANT_BLOCKS differs between decomposition and serving. Set it once on the command line so both stages read the same value.

Peak GPU memory on this path in bf16 measures about 1.9x the built-in baseline (32.7 vs 17.6 GB on Gemma 3 1B). ATTNQUANT_QUANT=nvfp4 recovers most of that by shrinking the weights 3.18x.

To serve on the fast path at passthrough:

ATTNQUANT_BLOCKS=26 ATTNQUANT_PRECOMPUTE_OUTSIDE=1 \
  pixi run python run_max.py serve \
    --model google/gemma-3-1b-it \
    --custom-architectures attnquant \
    --port 3100

run_max.py registers attnquant as a MAX custom architecture and forwards everything after the subcommand to max, so --max-length, --device-memory-utilization, and other max serve flags pass straight through.

In-flight batching is on by default

run_max.py appends --enable-in-flight-batching to serve unless you have already decided: an explicit --enable-in-flight-batching or --no-enable-in-flight-batching, or a --config-file, always wins.

MAX ships this off (enable_in_flight_batching defaults to False), which makes context encoding and token generation mutually exclusive. An arriving wave of requests then drains through several prefill-only iterations while decode stalls, and every request in the wave pays that drain in its time-to-first-token.

Measured on one B200, Gemma 4 31B NVFP4, medians over 48 trials per arm across two independent sessions:

It’s a median-latency fix. TTFT p50 falls 49.2% at c=64 and 54.7% at c=32, while aggregate throughput and inter-token latency stay flat within about 1%.

The tail isn’t improved. p99 TTFT moves −6.7% at c=64 and +2.6% at c=32, so a service level written against p99 rather than p50 doesn’t see this flag.

At concurrency 1 there’s never a decode batch for prefill to join, so the flag makes no difference; it matters only if you serve concurrent traffic.

ATTNQUANT_QUANT

When set to nvfp4 (with ATTNQUANT_PRECOMPUTE_OUTSIDE=1), attnquant quantizes each W_eff to NVFP4 at load time: packed 4-bit weights with per-block (1, 16) float8 scales instead of bf16. Whole-model weight bytes shrink 3.18x (on Gemma 4 31B, from 61.395 GB in bf16 to 16.474 GB of NVFP4 plus 2.819 GB of unquantized bf16 embeddings and tied lm_head), and decode measures 100.6 tok/s on one B200, +9.4% over the built-in NVFP4 path’s 92.0 tok/s (max serve + max benchmark, concurrency 1, /v1/chat/completions). It also reads 1.63x fewer bytes per token than the built-in path, which keeps all 60 self_attn blocks in bf16. The tied lm_head is bf16 on both paths, so the difference is attention coverage alone, and that coverage is where the throughput comes from, at a cost of about 0.41 pp of MMLU-Pro. Set ATTNQUANT_QUANT_IGNORE=self_attn to decline the trade, which also removes the speedup.

Quantization runs once at load (the measured first load for the 31B model’s 410 projections was ~51 minutes), and the quantized state dict is cached to disk under ~/.cache/attnquant/quantized_weights/, which cut the next load to ~22 minutes:

ATTNQUANT_BLOCKS=60 ATTNQUANT_QUANT=nvfp4 ATTNQUANT_PRECOMPUTE_OUTSIDE=1 \
  pixi run python run_max.py serve \
    --model google/gemma-4-31B-it \
    --custom-architectures attnquant \
    --port 3100

NVFP4 is a footprint win first: the 1.90x cut in bytes moved per decode step buys only 1.19x on the serve path, because 47% of a profiled step is byte-invariant and the NVFP4 kernels themselves reach only 47% of peak bandwidth. See GPU acceleration.

Config fields

The attnquant config class reads these from config.json and inherits them alongside every standard field of the base architecture. Defaults are backward compatible, so an unmodified Gemma repo (whose config.json doesn’t carry these keys) loads with sensible values:

ATTNQUANT_RANK and ATTNQUANT_BLOCKS must match between decomposition and serving. Because both stages read the same environment variables, setting them once on the command line keeps the theta bundle and the model graph in sync.