Turning reasoning off did not just save tokens. It exposed a quantization regression.
Quantizing a local model and then turning its reasoning chain off are two separate decisions, taken for the same reason: cost. They are usually validated separately, if at all, and almost never against each other. That is a mistake, and here is the measurement that shows why.
The comparison
The pair is a sparse local model, 26b-a4b, which carries 26B parameters but activates only about 4B of them per token, compared against its int4 quantization-aware build. Both see the same problems, generated with a seed, with the answer computed in Python so the gold is exact. The comparison is paired on the same items and read with an exact McNemar test (McNEMAR, 1947), which is the right test when two systems are evaluated on an identical set and what matters is the items on which they disagree.
Run it with the model’s reasoning chain enabled and the result is unambiguous.
| Reasoning | n | Original | Quantized (QAT int4) | Verdict |
|---|---|---|---|---|
| ON | 40 | 0.95 | 0.95 | tie |
| OFF | 120 | 0.7833 | 0.6833 | p = 0.012, quantized is worse |
With reasoning on, the two are identical. The obvious reading is that the quantization cost nothing, and that reading is what most teams would ship.
What appears when the chain comes off
Reasoning off is not an exotic configuration. It is how the model runs when cost matters, which is the same pressure that motivated quantizing it. So I ran the same pairing again with the chain disabled, at n = 120.
The original scores 0.7833. The quantized variant scores 0.6833. Ten points of accuracy, and the paired test puts it at p = 0.012. The discordant pairs are the part worth staring at: on 16 items the original was right and the quantized variant was wrong, against only 4 the other way. That asymmetry is the whole result. It is not noise in a near-tie; it is a lopsided disagreement.
The interpretation, stated carefully
The damage was not created by turning reasoning off. The damage was created by quantization, and it was being compensated for by the reasoning chain. A model that can spend tokens re-deriving an intermediate result can absorb a degraded weight representation; the chain is doing repair work that never appears in the score.
Which means that with reasoning on, I was not measuring the quantization. I was measuring the quantization plus the reasoning chain that was covering for it. The tie at 0.95 is a true number and a misleading one at the same time: true about that configuration, misleading about the component I was trying to evaluate.
The practical order of operations follows directly. If you validated your quantization in a reasoning-on configuration and later disabled reasoning to cut token cost, you did not make one change. You made two, and you only tested the first one in a regime that no longer exists. The regression is running now, and nothing in your dashboard will show it, because the benchmark that approved the quantization was run under the configuration that hid it.
What this does not establish
The evaluation domain is multi-step arithmetic. It was chosen because it discriminates, which an easier factual set did not, but it is one domain and the size of the effect will not transfer as a number to other tasks. The direction of the finding, that a reasoning chain can mask a weight-level regression, is the part that generalizes; the ten points are not a constant.
There are also two confounders specific to this pair, and they are mine to declare. The comparison mixes the quantization method with a difference in prompt template between the two variants, so isolating the quantization requires repeating the run with an identical template. And the model is sparse, which may be why it took the damage at all: a dense model given the same int4 treatment came through it as the strongest model in the study. Both caveats are developed in the companion article on the three verdicts.
Neither dissolves the finding here, because the finding is about the regime, not about the cause. The same template difference and the same architecture are present in the reasoning-on run, where the pair ties at 0.95. Whatever is doing the damage, the reasoning chain was covering for it, and turning the chain off is what made it visible.
Source. Own measurement, RTX 4090, local inference, temperature zero. Multi-step arithmetic generated with a seed, gold computed in Python, identical items across models, exact paired McNemar.
Reference. McNEMAR, Q. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, v. 12, n. 2, p. 153-157, 1947. DOI: 10.1007/BF02295996.