GPTQ — Hessian-Based Post-Training Quantization

GPTQ — Frantar et al., 2022 — is a layer-by-layer post-training quantization algorithm that pushed transformers to usable INT4 weight-only quantization for the first time. The key idea is borrowed from a 1990s pruning literature: when you change one weight, update the remaining weights to compensate rather than treating each weight independently. GPTQ formalizes that as a closed-form Hessian-weighted correction and makes it tractable for transformer-sized layers via Cholesky factorization and column blocking.

For roughly two years (mid-2023 through 2024) GPTQ was the dominant INT4 format on Hugging Face — millions of checkpoints — and the inference kernels in AutoGPTQ, ExLlama(V2), and vLLM were the first to make INT4 weight-only serving practically faster than FP16.

The setup

The reconstruction loss for a single linear layer is the sum-of-squares error between the FP16 layer’s output and the quantized layer’s output, taken over a calibration set of activation rows:

Naive PTQ rounds each weight independently to the nearest INT4 level. GPTQ instead picks the quantization order and the compensation update to the still-unquantized columns so that the layer-output error stays small across the calibration set.

Algorithm flow

For each layer l, with weight matrix W (d_in x d_out) and calibration X (K x d_in):

  1. Forward K calibration inputs through the model up to layer l, capture X.

  2. Compute Hessian:
        H = 2 * X^T X / K  +  lambda * I       (lambda = small ridge)
        H_inv = Cholesky-based inverse, processed column-block by column-block

  3. Walk columns of W in some order (often left-to-right within blocks of 128):
       for each column j:
         a. Quantize: q_j = round_to_INT4_grid( w_j )           (per-column scale)
         b. Compute the per-element error:  eps_j = w_j - q_j
         c. Update remaining columns to compensate:
                W[:, k>j]  -=  eps_j  *  H_inv[j, k>j] / H_inv[j, j]
         d. Mark column j frozen.

  4. Output quantized W_q.  Activations remain FP16.

Key implementation details:

• Block of 128. Columns are processed in blocks of 128. Inside a block, the Hessian inverse is computed lazily; across blocks, only the relevant submatrices are touched.
• Cholesky for stability. Direct would be unstable on near-singular Grams. GPTQ Cholesky-factorizes once and re-uses the factor.
• Per-channel grids. Each output column gets its own scale (and optional zero-point) — that scale is what q_j = round_to_INT4_grid(w_j) uses.

Why the compensation update works

When you pin column of to a discrete value , every output dimension that depends on input now has an error contribution . The other columns of — whose weights are still continuous — can be perturbed slightly to cancel most of that contribution in the output projection. GPTQ writes down exactly the perturbation that minimizes layer-output MSE under the constraint that is fixed at , and does it for free with one row-vector matrix-multiply per column.

What you actually get

On LLaMA-class models, GPTQ-INT4 typically lands within 0.5–1 perplexity of the FP16 baseline, with the larger gap on the smaller models (7B feels INT4 quantization more than 70B does). Concretely:

Model	FP16 PPL	GPTQ-INT4 PPL	Δ
LLaMA-7B	5.68	6.13	+0.45
LLaMA-13B	5.09	5.40	+0.31
LLaMA-30B	4.10	4.45	+0.35
LLaMA-65B	3.53	3.84	+0.31

(WikiText-2 perplexity, group size 128 — representative of the published results, exact numbers vary by reproduction.)

For most production tasks the perplexity gap translates to a 1–2% drop on downstream evals — usable, and often invisible inside generation noise. Smaller groups (32 instead of 128) cut the gap further at the cost of slightly more scale-overhead bits.

Software & kernels

The GPTQ format is more than a quantization algorithm — it’s a file layout with matching INT4 GEMM kernels. The split:

• Quantization side. AutoGPTQ, gptqmodel, Optimum-GPTQ. Run once on the FP16 model + calibration set, produce a packed INT4 checkpoint. Takes minutes to hours depending on model size.
• Inference side. ExLlamaV2 (the canonical fast kernels), AutoGPTQ kernels, vLLM’s gptq_marlin kernel (fastest in 2025 — fuses dequantize-into-FP16-matmul on Hopper/Ampere). Handle the packed INT4 layout, per-channel scales, optional act-order permutation.
• Group size 128 by default. Smaller groups = better quality, more scale overhead, slightly slower kernels. Group size 32 is the high-quality variant; group size –1 (per-tensor) is the legacy fast variant.

GPTQ vs the rest

The rough hierarchy in 2026:

• AWQ — the practical default for new INT4 weight-only quantizations. Simpler, faster, equal or better quality.
• GPTQ — equally usable, dominant on disk, slightly more legacy. Still a fine choice when you already have GPTQ checkpoints or kernel-level optimizations you trust.
• NF4 — for QLoRA-shaped fine-tuning workflows. Different objective (Gaussian-fitted levels, no calibration data) but the same INT4-weight-only deployment shape.
• MXFP4 / NVFP4 — for Blackwell-class hardware. Compresses activations as well as weights, with native tensor-core support.
• GGUF K-quants — for CPU / Apple-Silicon inference. Different ecosystem, parallel evolution.

When to still pick GPTQ

You’re running on Hopper or Ampere, you want INT4 weights with FP16 activations, you have a calibration set and don’t mind ten minutes of quantization time, and you want kernels that have had three years of optimization tuning. Or you’re loading an existing GPTQ checkpoint and don’t want to re-quantize.

For everything else — new quantizations on Blackwell, training, end-to-end FP4 inference — GPTQ has been superseded. It earned its place as the format that proved INT4 weight-only is a viable serving target, then handed the baton.