When the grader and the graded make the same mistake: field hallucination in document extraction
The cheap way to build a labelled set for a structured extraction task is to run your largest model over the documents and treat its output as the reference. Everything downstream then measures the smaller, faster candidates against that reference. It is fast, it scales, and it quietly answers the wrong question.
I built exactly that set to decide which local vision variant could take over structured field extraction from consumer invoices, and then I hand-checked a small sample of it. The sample changed the metric.
The failure the automatic label could not see
The task is to read a document image and return a fixed set of fields as JSON, at temperature zero, with reasoning off. Among the fields is a tax rate. Some document formats itemize that rate; consumer telecom documents do not carry it at all.
Instructed to extract the tax rate from a document that does not contain one, the largest model did not return null. It returned a plausible rate.
That is the failure mode: not a misread digit, but a fabricated field. And the automatic reference could not see it, because the automatic reference came from the same model. The grader and the graded made the same mistake, in the same direction, and the agreement metric stayed green. A benchmark built that way does not measure whether the model is right. It measures whether two runs of the same weights agree, which they do, including when both are wrong.
The only thing that caught it was a gold set checked by hand. Five documents were enough to document three distinct fabrications.
What the hand-checked set measured
With a gold set you can score two different things, and they are not the same axis. The first is accuracy on the fields that are present in the document. The second is abstinence on the fields that are absent: how often the model returns a value where the truth is null.
| Variant | Correct on fields present | Fabricated on fields absent | Fabricated tax rate | Latency per document |
|---|---|---|---|---|
e2b-it-qat |
6/15 (40%) | 11/20 (55%) | 0/3 | 1.87 s |
26b-fast |
9/15 (60%) | 10/20 (50%) | 2/3 | 2.70 s |
31b-it-qat |
14/15 (93%) | 10/20 (50%) | 2/3 | 5.13 s |
Read the table twice. The largest variant is far and away the best extractor of fields that exist: 14 of 15, against 9 and 6 for the others. It is also fabricating half of the fields that do not exist, 10 of 20, exactly like the mid-size variant. Its accuracy advantage is real and its abstinence advantage is zero.
If you rank these three variants by extraction accuracy alone, you promote the model that invents the most confident fictions. The ranking is not wrong on its own terms. The terms are wrong.
The metric that follows from this
The promotion criterion for the benchmark changed to two conditions, both binding:
- accuracy on fields present in the gold, and
- hallucination rate on fields absent in the gold, defined as the fraction of null-in-gold fields that the variant returns filled.
A variant is promotable only if it is accurate on what is there and silent about what is not. A model that scores well on the first and badly on the second is not a good extractor with a small defect. It is a system that will inject fabricated values into whatever consumes its output, and it will do so with the same confidence it applies to the fields it read correctly.
What this does not establish
The gold set is five documents. Those are counts, not estimates with a confidence interval, and I am reporting them as counts. Their purpose is to test whether the automatic reference was adequate, and five documents were sufficient for that, because the automatic reference asserted zero fabrications where the hand check found three. Extending the gold set, particularly to the formats that do not itemize the field and therefore induce the fabrication, is the next step and it is not done.
The domain is one document family, and the finding is about a mode of failure, not about a rate that transfers to other tasks. And the underlying corpus contains personal financial records, so no values, identifiers, service names or dates from it appear here or anywhere outside a private repository. What is publishable is the method and the failure, not the data.
The transferable part
If you are validating a small model against a large one, your number is agreement, not accuracy. The two coincide only when the large model is right, and the case where it is systematically wrong is precisely the case your benchmark was supposed to catch.
Hand-check a small sample. It does not need to be large. It needs to exist.
Source. Own measurement, local vision variants on an RTX 4090, temperature zero, reasoning off, same prompt and same rendered images across variants. Gold set of 5 hand-checked documents; counts as reported.