AWQ — Activation-Aware Weight Quantization

AWQ — Activation-aware Weight Quantization, Lin et al., 2023 — is the spiritual successor to GPTQ and the practical default for INT4 weight-only quantization in 2024–2026. The insight is one line: in a quantization-error sense, not all weights matter equally. A weight multiplied by a large activation contributes more to the layer output than a weight multiplied by a small activation, so quantization error on the high-activation weights dominates the layer-output error.

If you can identify those salient weight channels and protect them with a per-channel scale before rounding, you get most of GPTQ’s quality with none of its second-order math.

The setup

Take a linear layer , where and . The output’s sensitivity to a perturbation at column is

So if input dimension has a large typical magnitude , then quantization error on the entire -th column of shows up amplified at the output. Empirically, in transformer linear layers, this dominates: a small fraction of input channels — typically less than 1% — carry magnitudes 10–100× larger than the typical channel, and those channels concentrate the error.

Activation magnitudes across input channels (per-channel mean |x_j|):

  |
  |          o    <- "salient" channels: high mean |x_j|, dominate error
  |
  |
  |  o   o  o    o    o
  |o o o    o   o  o      o   o    o
  | o    o    o     o o  o   o   o
  +---------------------------------------> channel j
              <-- ~99% typical -->

GPTQ handles this by computing the full Hessian and using a closed-form update to compensate downstream columns when each column is rounded. AWQ takes a more direct approach: scale the salient channels up before quantization (so they round to a finer grid in absolute terms), then absorb the inverse scale into the activation pathway.

The math

AWQ’s central trick is an algebraic identity:

Here is a vector of per-input-channel scaling factors. Multiplying by shrinks the weights along the salient channels (so they round to INT4 with smaller absolute error), and multiplying by expands those activations by the same factor. The product is unchanged in floating point — it’s a mathematical identity.

Without AWQ scaling:

   x  --(quantize W)-->  W_q * x
                          ^
                          |  large quant error on salient channels
                          |  (their large weights round to coarse grid)


With AWQ scaling (s_j large for salient channel j):

   x  --(* s)-->  s*x  --(quantize W * 1/s)-->  W'_q * (s*x)  =  W_q * x  (smaller error)
                                ^
                                |  W' = W/s has smaller weights on salient channels
                                |  -> they round more accurately at INT4
                                |
                                + s*x is computed at FP16, no precision lost there

In practice is large for high-activation channels and ≈1 for the rest. The salient channels’ weights become smaller before quantization (better rounding); their activations become larger at runtime (computed in FP16, no precision lost). The non-salient channels are essentially untouched.

Choosing the scales

AWQ doesn’t try to derive analytically. Instead it does a grid search: parameterize where is the per-channel mean activation magnitude on a calibration set, and search on a small grid (typically 20 values) to minimize the layer-output reconstruction error.

For each linear layer:
  1. Run ~16-128 calibration samples, capture activation X.
  2. Compute per-channel mean magnitude: m_j = mean_k |X[k, j]|
  3. For alpha in [0.0, 0.05, 0.10, ..., 1.0]:
       s = m^alpha                         (per-channel scaling vector)
       W' = W * diag(1 / s)                (apply inverse scale to weights)
       W'_q = quantize_INT4_per_channel(W')
       err(alpha) = || X * W  -  (X * diag(s)) * W'_q ||_F^2
  4. Pick alpha* that minimizes err. Store s* and the quantized W'_q.
  5. At runtime: dequantize W'_q to FP16, fuse the diag(s) with the input
                 (or with the previous layer's output projection for free).

That’s the entire algorithm. No Hessian inversion, no Cholesky factorization, no per-column ordering decisions. The grid search costs O(20) forward passes per layer, which is dominated by the matmul cost — calibration runs in seconds rather than minutes.

How it compares to GPTQ

Method	Compensation	Calibration set	Time per layer	INT4 quality (ΔPPL on LLaMA)
Naive PTQ	none	none	seconds	+5 to +20 (often unusable)
GPTQ	Hessian-based per-column update	~128 samples	1–10 minutes	+0.3 to +1.0
AWQ	Pre-quantization per-channel scale	16–128 samples	seconds	+0.2 to +0.7

Both methods address the same fundamental issue — outlier channels that wreck naive scaling — but from different angles. GPTQ minimizes residual error after a fixed quantization grid via a closed-form weight update. AWQ rotates the problem upstream by changing the scale before quantization happens, so the grid becomes an easier target.

The practical consequences:

• AWQ calibrates 5–10x faster than GPTQ. No Hessian computation, no Cholesky, no per-column iteration.
• AWQ matches or beats GPTQ in published benchmarks on LLaMA-2 and LLaMA-3 family models, by 0.1–0.3 perplexity at INT4.
• AWQ kernels are at least as fast. The awq_marlin and TensorRT-LLM AWQ kernels are state-of-the-art for INT4 weight-only INT4 GEMM on Hopper.
• AWQ is per-channel by construction. GPTQ uses per-channel grids too, but AWQ’s scaling step is the per-channel knob — it makes per-channel a first-class concept rather than an implementation detail.

Where AWQ ships

When AWQ is not enough

The decision tree:

• INT4 weight-only on Hopper / Ampere? AWQ. (GPTQ is fine if you already have checkpoints.)
• INT4 weights and INT8 activations on Hopper? AWQ + SmoothQuant, in that order.
• FP4 weights + FP4 activations on Blackwell? NVFP4 . AWQ’s scaling trick is being absorbed into the NVFP4 quantization tooling.
• QLoRA-shaped 4-bit fine-tuning? NF4 . AWQ targets weight-only inference, not fine-tuning workflows.
• CPU / Apple Silicon inference? GGUF K-quants. AWQ has GPU-targeted kernels; GGUF has CPU-targeted kernels.

AWQ won the post-GPTQ era of INT4 weight-only quantization and now sits as the production default. The 2026 question isn’t “GPTQ or AWQ?” — it’s “AWQ or NVFP4?”, and the answer depends on your hardware.