Contextual Utility of Quantization Moves in Extreme Low-Bit LLMs

A better reconstruction is not always a better model

Post-training quantization changes a finite code to reduce weight or activation reconstruction error, while the model is ultimately judged by predictive loss and task performance. The paper asks how to tell whether a legal code move is actually useful at a fixed bit width.

Utility depends on the displacement of the move. A gradient evaluated only at the start misses curvature along the path; evaluating at the move midpoint gives a better sign estimate for endpoint loss. Low-bit Llama-3.2 experiments test this distinction.

Quantization moves interact

Enumeration over legal quantized states shows that combined utility is approximately quadratic. Small pairwise interactions can still change the best multi-objective tradeoff, and the same move can switch from helpful to harmful in a different state.

Midpoint readings repair local choices. For larger combinations, the method reevaluates from states that are actually reached and verifies exact endpoints; endpoint-evaluated beam search finds sparse updates that outperform larger one-shot changes.

From static scores to state-aware optimization

The work moves quantization optimization to the level of individual code changes: define the objective, evaluate each move along its path, and recombine moves as the state evolves. Experiments connect these choices to task accuracy and held-out perplexity.