ReadPaper Blog
KronQ: LLM Quantization via Kronecker-Factored Hessian
KronQ addresses post-training quantization for large language models by correcting a key limitation of GPTQ-style methods: they use input activation covariance but ignore output-side gradient sensitivity. The paper introduces a Kronecker-factored Hessian view in which quantization error depends on both activation covariance and gradient covariance, enabling better incoherence processing and mixed-precision allocation without retraining. Its main implication is that ultra-low-bit LLM compression, especially 2-bit weight-only quantization, can be made substantially more stable for models such as LLaMA-2 and LLaMA-3.
Source: KronQ: LLM Quantization via Kronecker-Factored Hessian
Research question
The paper studies how to compress large language models through post-training quantization while preserving model quality at very low bit-widths. Its central problem is that practical PTQ methods must operate without retraining, using only a small calibration set, yet large models with billions of parameters are highly sensitive to poorly allocated rounding error. KronQ frames this problem through the layer-wise Hessian of the quantization objective and argues that existing second-order quantizers are missing an important part of that curvature. The proposed method uses a Kronecker-factored Hessian approximation, written as an activation covariance factor and a gradient covariance factor, to model both input-side and output-side sensitivity. This matters because LLM deployment is constrained by memory footprint and inference latency, and better PTQ can make low-bit LLaMA-family models more usable on commodity and edge hardware.
Why old methods fall short
The paper’s critique of prior compensation-based PTQ focuses on GPTQ and GPTAQ, which build their quantization objectives from the input activation covariance H_X. In the standard objective, the error term compares W X with its quantized counterpart and therefore treats H_X as a proxy Hessian for the layer. KronQ argues that this implicitly assumes the output-side factor H_G is the identity, meaning all output channels are treated as equally important. The paper reports that the normalized diagonal entries of the gradient covariance vary by orders of magnitude across Q, K, V, and O projections in LLaMA-2-13B, showing that this equal-sensitivity assumption is unrealistic. GPTAQ improves GPTQ by correcting input drift from sequential layer-wise quantization, but the paper emphasizes that GPTAQ still inherits the same input-only Hessian limitation.
Core idea
KronQ’s core idea is to incorporate gradient covariance into PTQ through the Kronecker-factored approximation H ≈ H_X ⊗ H_G. For a linear layer y = W x, the per-sample gradient factorizes as g xᵀ, so the empirical Fisher form of the Hessian can be approximated by E[xxᵀ] ⊗ E[ggᵀ] under the K-FAC independence assumption. This yields a quantization objective of the form tr[H_G ΔW H_X ΔWᵀ], and the paper extends it with GPTAQ’s asymmetric input-drift correction. A notable technical result is that H_G cancels algebraically in the column-wise OBS weight compensation update, so KronQ can preserve the efficiency of the GPTAQ-style solver. The gradient covariance is therefore used where it changes the quantization pipeline most effectively: in output-aware incoherence processing and inter-layer mixed-precision scoring.
Evidence check
The method section develops two concrete uses of the Kronecker-factored objective beyond simply rewriting the loss. First, KronQ introduces bidirectional incoherence processing, extending the usual input-side random rotation to the output dimension by using the gradient covariance, with the goal of reducing weight magnitude variance across both input and output dimensions. This builds on the QuIP-style observation that incoherent weight and Hessian structures lower quantization error, but it adds output-side Hessian information that prior rotation-based methods leave unmodeled. Second, KronQ derives an inter-layer mixed-precision sensitivity metric based on the product tr(H_G) · tr(H_X). This score can distinguish sublayers that share identical input statistics, such as Q, K, and V projections, because their gradient covariances can differ. The paper presents this as a practical improvement because H_G is obtained from a single backward pass over the calibration set rather than by costly Hessian sketching.
One thing to remember
The empirical claim of the paper is that adding output-side gradient covariance yields the largest gains in the hardest quantization regimes. The authors evaluate KronQ on LLaMA-2 and LLaMA-3 models ranging from 7B to 70B under W2, W3, and W4 settings, including weight-only and weight-and-activation quantization. The excerpt highlights a particularly strong result for 2-bit weight-only quantization on LLaMA-3-70B: GPTQ and GPTAQ diverge or produce degenerate WikiText-2 perplexities above 2000, while KronQ reaches 7.93 perplexity. The broader implication is that second-order PTQ should not rely solely on activation statistics when output channels have heterogeneous gradient sensitivity. KronQ’s takeaway is that a Kronecker-factored Hessian can add the missing output-side information while keeping the base quantization update computationally practical.
