Quickstart

Install the MAX toolchain with pixi, then generate and serve Gemma 3 1B through attnquant in three steps.

This page takes you from a clean checkout to a served model. You’ll install the pinned MAX toolchain, generate a token from Gemma 3 1B through attnquant, and start an OpenAI-compatible endpoint.

AttnQuant registers as a MAX custom architecture, so the commands are the same max generate and max serve you already know, invoked through the run_max.py wrapper. Two environment variables configure it:

• ATTNQUANT_PRECOMPUTE_OUTSIDE=1 turns on the fast path: attnquant rebuilds each weight once at load time, so every projection runs a single matmul with the same launch count as the built-in dense path, plus Q/K/V fusion on top.
• ATTNQUANT_BLOCKS=26 sets the passthrough operating point for Gemma 3 1B’s 26 layers, so each rebuilt weight equals the original and output matches the original model.

Before you start

You need the following:

• pixi: the project ships a pixi.toml and lockfile that pin modular (MAX and Mojo) plus the Python dependencies.
• A Hugging Face token with access to google/gemma-3-1b-it, saved at ~/.cache/huggingface/token.
• An NVIDIA GPU for the fast path. Apple silicon (Metal) and CPU also produce correct output; the measured throughput numbers are from an NVIDIA B200.

Install the environment

Resolve the pinned MAX toolchain from the project root:

cd attnquant
pixi install

Generate a token

Pass --custom-architectures attnquant so MAX loads the attnquant version of Gemma 3, and set the two attnquant variables:

ATTNQUANT_BLOCKS=26 ATTNQUANT_PRECOMPUTE_OUTSIDE=1 pixi run python run_max.py generate \
  --model google/gemma-3-1b-it \
  --custom-architectures attnquant \
  --prompt "Explain attnquant inference in one sentence." \
  --max-new-tokens 64

At load time, attnquant decomposes the checkpoint’s 26 layers into theta bundles and rebuilds each projection weight. You’ll see the decomposition and load logs, then the generated text streamed to stdout:

AttnQuant inference rebuilds each layer's weight from a small theta
bundle at load time instead of storing the full dense matrix.

The output is native-equivalent to the original Gemma 3, verified by logit-parity: the output logits match the original model within a max logprob delta below 1.0. See quality for how parity is verified.

Serve the model

The same wrapper starts an OpenAI-compatible server. The project convention is port 3100:

ATTNQUANT_BLOCKS=26 ATTNQUANT_PRECOMPUTE_OUTSIDE=1 pixi run python run_max.py serve \
  --model google/gemma-3-1b-it \
  --custom-architectures attnquant \
  --port 3100 \
  --max-length 4096 \
  --device-memory-utilization 0.9

Once the server reports it’s ready, call it like any MAX endpoint:

curl http://localhost:3100/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "google/gemma-3-1b-it",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

The response is standard OpenAI chat-completion JSON, with the assistant reply under choices[0].message.content:

{
  "choices": [
    {"message": {"role": "assistant", "content": "Hi there! How can I help?"}}
  ]
}

Next steps