How attnquant works

The theta bundle representation, the analytic decomposition that produces it from a trained checkpoint, and the reconstruction identity that rebuilds each weight at inference time.

AttnQuant replaces each linear projection’s stored weight with a compact theta bundle and rebuilds the projection from it. This page covers what the bundle holds, how it’s computed from a trained checkpoint with no training step, and the identity that rebuilds the weight. For how the rebuild executes on a GPU, see execution paths.

The theta bundle

A theta bundle replaces one projection’s dense [out_dim, in_dim] weight with three pieces:

• W0 (shared block): one dense block reused across the layers of a projection family, stored once per family, not once per layer.
• s (per-row scale): a per-output-row scale of W0, stored per layer.
• U, V (low-rank delta): two thin factors of rank r whose product U · Vᵀ corrects W0 back toward the layer’s original weight, stored per layer.

A projection family is one of the linear projections that repeats in every decoder layer: q_proj, k_proj, v_proj, o_proj in attention, and gate_proj, up_proj, down_proj in the MLP. Because W0 is shared across a family, a layer stores only the small s, U, and V tensors plus a reference to the family’s one W0.

Each piece has a fixed shape and scope:

The reconstruction identity

The bundle rebuilds the weight as:

weight ≈ diag(s) · W0 + U · Vᵀ

Applying that weight to an activation x gives the projection output without ever forming the dense matrix:

y = s · (W0 x) + U (Vᵀ x)

This is the one canonical formula in attnquant, and every execution path computes it exactly. Execution paths covers the three ways it runs (precompute-outside, GEMM, custom op) and how the fastest one is selected for your accelerator.

How the decomposition produces s, U, and V

The decomposition is analytic and one-shot, computed in core/decompose.py when the checkpoint loads. There’s no training step and no gradient descent, which is why attnquant works on any trained checkpoint in a single pass. For one projection weight and its shared block W0, it runs two steps.

First, it fits a per-output-row scale by least squares, scaling each row of W0 to best match the corresponding row of the target weight:

s[o] = (weight[o] · W0[o]) / (W0[o] · W0[o])

Second, it takes the residual left over after that scaled shared block and factors it into a rank-r product with a truncated SVD:

resid = weight − diag(s) · W0
U, V  = truncated_svd(resid, rank=r)     so  U · Vᵀ ≈ resid

The SVD is randomized (Halko et al., 2009) for large matrices, falling back to a full np.linalg.svd when min(out_dim, in_dim) <= 256 or when the oversampled probe would capture the whole matrix. U absorbs the singular values (U = U_svd · S), so callers use U @ V.T directly. When r = 0, U and V are empty and the weight is a pure scaled shared block.

W0 itself is the mean of the projection weights across the layers that share it. How many layers share one W0 is the shared_blocks dial (G), described in banded sharing.

The passthrough operating point

Production serving runs at passthrough: G = num_layers, one W0 per layer. Then each W0 is that layer’s own weight, the residual is zero, and the reconstructed weight equals the original exactly, which is why attnquant output matches the original model. Set it with ATTNQUANT_BLOCKS=<num_layers> (26 for Gemma 3 1B, 60 for Gemma 4 31B); the config default is G=1, an analysis setting for decomposition-error studies.

Native-equivalent means logit-parity, not bit-exact: the output logits match the original model within a max logprob delta below 1.0, and greedy tokens agree on more than 80-90% of positions. Small numeric differences come from reordered floating-point reductions; quality defines the verification method.

Next steps

• Execution paths: how the identity runs on GPU, and which path is fastest where.
• GPU acceleration: where the measured speedups come from, across capture, fusion, and NVFP4.
• Banded sharing: the G dial between one shared block and one per layer.
• Quickstart: serve a model with this machinery in a few commands.