A test that cannot fail anyone cannot pass anyone
The first evaluation set I wrote is the one everybody writes: a handful of factual questions and some easy arithmetic, ten items, quick to run, easy to read. Every model I put through it scored at or near a perfect one. I remember being pleased, briefly, and then realising what the result actually said.
It said nothing. A test on which every candidate passes has not measured the candidates. It has measured its own difficulty.
The saturated set
| Variant | Accuracy, easy set (10 items) |
|---|---|
e2b |
0.90 |
e2b-it-qat |
1.00 |
26b-fast |
1.00 |
26b-a4b-it-q4_K_M |
1.00 |
26b-a4b-it-qat |
1.00 |
Four of five variants at a clean 1.0, the fifth at 0.9. There is no ordering here that survives a second look, and no rejection is possible: the set cannot fail anyone. Which means, and this is the part that took me a moment, that the set cannot pass anyone either. An approval is only informative if a rejection was on the table. If the instrument produces “pass” regardless of the input, its output carries no information about the input.
That is the trap of an internal benchmark. It is written by the person who wants the migration approved, it is validated on the models that already work, and it converges to a rubber stamp precisely because nothing ever fails it.
The metric I had to throw away
The next attempt was to compare variants against each other directly, measuring how often two variants produce the same output on the same prompt. Exact agreement came out at 0.125 for both pairs I tested.
That number looks like a signal and is not one. The variants differ in prompt template and in how their outputs are rendered, so two models can reason identically and still disagree on the exact string. An agreement metric under those conditions measures formatting, not capability, and it does so with enough noise to bury whatever capability difference exists underneath. I discarded it. It could not distinguish “these models think differently” from “these models punctuate differently”.
Discarding a metric you already computed is uncomfortable and it is the correct move more often than the literature suggests.
The set that can reject
What finally discriminated was multi-step arithmetic: expressions generated from a seed so the set is reproducible, with the answer computed in Python so the gold is exact and no model is grading itself. The same items go to every model, and the comparison is read with an exact paired McNemar test (McNEMAR, 1947), which uses only the items on which two models disagree.
| Variant | Accuracy, multi-step arithmetic (n=120, reasoning off) |
|---|---|
e2b |
0.2833 |
e2b-it-qat |
0.3000 |
26b-fast |
0.7833 |
26b-a4b-it-qat |
0.6833 |
31b-it-qat |
0.8917 |
Sixty points of spread, from 0.28 to 0.89, on models that the easy set could not separate at all. This set has the property the first one lacked: a model can fail it. And because a model can fail it, a model passing it means something.
It earned its keep immediately. It is the set that showed a quantized variant losing to its original at p = 0.012, and the set that showed a different quantized variant tying its original at p = 0.82. Neither of those verdicts was visible in the saturated set, where both pairs sat at 1.0 and looked equivalent.
What this does not establish
Arithmetic is a proxy. It discriminates, it is cheap to generate, and its gold is exact, which is why it works as an instrument. It is not a claim about the tasks these models will actually run, and a variant that wins on arithmetic is not thereby the right choice for extraction, or for summarisation, or for anything else. A discriminating benchmark tells you the models are different. It does not tell you which difference matters for your workload; only a task-shaped evaluation does that.
The point is narrower and harder to escape: before an evaluation set can tell you anything, it has to be able to say no.
The transferable part
Before you trust an internal benchmark, ask it one question: has anything ever failed this? If nothing has, you do not have a measurement. You have a ritual. And the first model that would have failed it will not be caught in your evaluation. It will be caught in production, by your users, in the configuration you never tested.
Source. Own measurement, RTX 4090, local inference, temperature zero. Easy set: 10 factual and arithmetic items. Discriminating set: multi-step arithmetic generated with a seed, gold computed in Python, identical items across models, exact paired McNemar. Agreement measured as exact string match at fixed output length.
Reference. McNEMAR, Q. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, v. 12, n. 2, p. 153-157, 1947. DOI: 10.1007/BF02295996.