The model copies the number correctly and then concludes wrong: move the reasoning into code
An earlier article in this series measured a model inventing a field that did not exist on the page, and argued for a metric that catches it: the hallucination rate on absent fields. Measuring is the right first move. It is also, on its own, an unhappy place to stay, because it scales badly. If your pipeline produces one automatic reading for each of five thousand records, you cannot hand-check the output, and a detector that flags one percent of records still leaves you with fifty defects and no way to tell which fifty.
So the second move is to stop detecting the failure and start making it impossible. This is a report on doing that, in a public panel that generates an interpretive reading for each of the 5513 Brazilian municipalities using a 2B local model.
What a small model does when it has to reason about a number
Hand a 2B model a record of raw figures and ask it to work out what they mean, and it makes three errors reliably. It inverts the direction of a comparison, so a low percentile rank comes out as “high dependency”. It inflates intensity, calling something high where the data says low. And it expands acronyms wrong, inventing a plausible full name for an initialism it does not know.
Notice what these three have in common, because the distinction turns out to matter more than I first realised. They are failures of reasoning about the numbers, not of writing down the numbers. The model’s prose is fine. Its direction logic and its arithmetic are not, and no amount of prompt tuning made them reliable at this size.
That observation is the whole design, and, as the ablation at the end of this article shows, it is also a sharper observation than the one I built on top of it.
The model writes, the harness reasons
The pipeline separates the two jobs and gives the model only the one it is good at.
Everything that requires being right is computed deterministically, in Python, before the model sees anything. For each municipality the harness resolves the comparison direction and hands over the correct word for it, computes the qualitative dependency level from the percentile rank so the model never has to infer intensity, compares fiscal ratios against the median of the same population size class rather than the national mean, spells out services in full so no acronym has to be expanded, and derives the candidate fiscal observations by rule.
What reaches the model is a small, self-contained block of facts already resolved. Its job is to turn resolved facts into readable prose. Nothing it can get wrong is left for it to get wrong.
The number as a symbol
Here is the part that does the work.
Every displayable value, every number, percentage, distance, population and city name, is a textual symbol in braces. The model emits only the symbol. Substitution by the canonical value is mechanical and happens after the model is done.
{IDM} -> "IDM 0,45"
{IDHM} -> "IDHM 0,65 (Censo 2010)"
{PERC} -> "34%"
{FPM} -> "21%"
{POLO} -> "Cacoal (RO)"
{DIST} -> "83 km"The model writes “the municipality is more dependent than {PERC} of the country and orbits {POLO} at {DIST}”. The harness produces the published sentence. The value, the label and the format are never the model’s responsibility, which removes numeric hallucination, swapped labels and inconsistent formatting in one move rather than three.
The self-labelling is the detail that makes it airtight. {IDM} does not expand to 0,45; it expands to IDM 0,45, carrying its own name. So the model cannot write “IDM {IDHM}” and produce a swapped index, because the label travels with the value and the model never handles either.
And now the audit is trivial. The gate runs before substitution, and asks two questions of the model’s raw output:
- Is there any digit written outside a symbol? If yes, the model tried to author a number. Reject.
- Is there a symbol that does not exist in this record’s dictionary? If yes, the model invented one. Reject.
A clean output has zero of both. This is a regular expression, not a judgement call, and there is no threshold to tune. That is the difference between prevention and correction: a wrong number is not scored, argued about or tolerated at some rate. It is unrepresentable.
The gate does the rest of the reasoning too
The same deterministic critic rejects the other failure modes, each by rule, on the pre-substitution text:
| Rule | Rejects |
|---|---|
| moral | judgement of the administration (good, bad, efficient) |
| causal | causal vocabulary (because, leads to, results in) |
| ordem | instructions to the city hall |
| intensidade | intensity above the level the harness computed (suppressed when the level really is high, since then the strong word is correct) |
| digito_vazado | any digit outside a symbol |
| simbolo_invalido | a symbol that does not exist for this record |
| fatia_ausente | a required output field left empty |
A record that fails is retried. A record that keeps failing is marked as having no reading and is published without one rather than with a bad one. Across the full run of 5513 municipalities, the count of records published without a reading is zero.
Two findings worth keeping
A dedicated field beats a synthesis. The model fills a schema field faithfully. It drops content when it has to stitch several ideas into one field. So anything that cannot be lost gets its own mandatory field in the schema rather than a clause inside a summary. This was more effective than a self-revision loop, which at 2B is both marginal and fragile: a critic that matches on strings breaks the moment the model paraphrases.
The ceiling is editorial judgement. What a 2B cannot do, even fully mediated, is decide what is most worth saying: hierarchise, notice the notable, choose the lead. That limit is real and it is not closed by better prompting. It is acceptable here because the output is labelled to the reader as automatically generated, and the panel says so on every reading. If it were not acceptable, the answer would be a larger model on the subset that comes out flat, not more prompt engineering on the small one.
The controlled ablation: is the harness doing the work?
The design rests on a claim I had not tested: that the deterministic mediation is what holds the quality, and that the model on its own would not. So I tested it.
The design is paired. The same 120 municipalities, drawn with a seed, go through both arms on the same model, the same strict schema and the same engine as production, google/gemma-4-E2B-it served through vLLM with continuous batching, at temperature zero. First pass only, no retry, because the question is what the model produces, not what the harness repairs. One thing differs between the arms: whether the facts arrive resolved.
- With harness (production): the model receives hooks in which the direction is already resolved, the dependency level already computed and the services already spelled out, and it emits symbols.
- Without harness: the model receives the raw record, the percentile as a bare fraction, the fiscal ratios, the median for its size class, the service flags as acronyms, and it has to do the reasoning itself.
Both arms are scored by the same deterministic check against the record, on the text as published: a number that matches nothing derivable from the record, an inverted percentile comparison, or a dependency level that contradicts the percentile.
That last one is why the comparison is not rigged. Even with the harness, the level is still a word the model has to write; it is handed the answer, but it could still write the wrong one. If the harness were doing nothing, both arms would fail alike.
| Arm | Readings with a factual defect | Rate |
|---|---|---|
| With harness | 0 / 120 | 0.0% |
| Without harness | 36 / 120 | 30.0% |
Thirty-six discordant pairs, all in the same direction. Not one case where the harness arm failed and the bare arm did not. Exact McNemar, p = 2.9 x 10⁻¹¹.
What breaks, and it is not what I expected
Look at the composition of the failures, because it is sharper than the headline.
Of the 36 defective readings without the harness, 35 are a wrong dependency level and only 2 are an ungrounded number. The model is not inventing figures. It is copying them faithfully and then drawing the wrong conclusion from them.
The pattern is almost comic once you see it. At the 69th percentile, which is high dependency, it writes “intermediate dependency”. At the 31st, which is low, it writes “intermediate”. At the 4th percentile, which is about as low as the country goes, it writes “intermediate” again. It collapses everything toward the middle, because ranking a value against a distribution is a reasoning step, and that is the step a 2B model cannot take.
The zero in the other arm is not an artifact of the model staying silent, and I checked. With the harness, the model asserts a dependency level in 120 out of 120 readings and gets it right every time, because it is not deciding anything: it is repeating a word the harness computed. Without the harness it asserts a level in only 89 of 120, and of those 89 it gets 35 wrong, an error rate of 39% among the cases where it commits to an answer. The other 31 simply omit the level, which is its own failure of the task and which I did not even count against it.
So the answer to the question in the section title is yes, and the effect is not marginal. Remove the mediation and three in ten published readings carry a factual defect. Keep it and the rate is zero at this sample size.
What this establishes and what it does not
It establishes that the reasoning, not the writing, is where a 2B model fails on this task, and that moving the reasoning into deterministic code removes the failure rather than reducing it. The three error modes that motivated the design were not folklore; the level error is now measured, at 39% of the readings that commit to a level.
It does not establish that the symbol layer contributes to that number. The symbols guarantee that the value, the label and the format are never the model’s to get wrong, and the audit that enforces it is a regular expression that runs on every record rather than on a sample. That is a property of the construction, not an effect size, and I am no longer going to present it as one.
The domain is one panel over one national dataset, with public official sources and no personal data. The detectors were validated against planted defects before the run: a fabricated number, an inverted comparison and a contradicted level are each flagged, and clean text passes. An instrument that cannot fail anything cannot be trusted when it passes everything, which is the argument of another article in this series and applies here to my own measurement.
The transferable part
The instinct, when a small model gets facts wrong, is to make it try harder: a better prompt, a self-critique loop, a bigger model. The measurement here points somewhere cheaper.
Separate the two jobs. Ask what in your task requires being right, and move that into code, where it is deterministic, testable and free. Then hand the model what is left, which is usually the writing, and it will do that part well even at 2B. The reasoning is what it cannot do; the prose is what it can.
Numbers as symbols is the same principle applied to the last mile: the model was never adding value to the digit, so do not let it own the digit. You lose nothing, and you gain a failure mode a regular expression can prove absent on every record rather than on a sample.
And then test the separation, rather than assuming it. I built this pipeline believing the symbols were the load-bearing part. They are not. The mediation is, and I only know that because I ran the arm without it.
The panel this came from is public at idm.pcbrom.com, and every automatic reading on it is labelled as automatically generated.
Source. Own implementation, interpretacao/ in the IDM panel project: deterministic harness, symbol dictionary, validation gate. Model google/gemma-4-E2B-it, served in batch through vLLM with continuous batching on an RTX 4090. Artifact of 2026-07-08: 5513 records, zero without reading. Paired ablation of 2026-07-13, interpretacao/ablacao_harness.py: 120 municipalities drawn with a seed, with and without the mediation, same model and schema, temperature zero, first pass without retry, exact McNemar; detectors validated against planted defects before the run. Public data from official Brazilian sources; no personal data.