Pedro Carvalho Brom

HomeBlog

Confidence is not a promise: what a 0.97 from a decision model licenses

A typed decision model returns 0.97 and the code acts on it as if it were a guarantee. A reference audit of two typed deciders, now on arXiv, measures what that number licenses, why a threshold chosen by point estimate breaks its own promise in 147 of 190 evaluations, and what a caller can do instead.

Code repositorypcbrom/typed-deciders-audit

Picture the line of code. A model reads a customer message, picks one of five intents and returns 0.97. Above 0.95 the ticket is routed with no human in the loop, below it a person takes a look. Somebody signed off on that threshold with a sentence like “we only automate when the model is at least 95% sure”, and that sentence is now a promise the system makes to everyone downstream.

I spent the last weeks measuring whether promises like that hold. The result is a paper now on arXiv, Confidence Is Not a Promise: A Reference Audit of Typed Decision Models (arXiv:2610.36311), released with the code, the frozen answers of both models and the proofs. Every figure below comes from the paper, which carries the tables, the protocol and the proofs behind each one.

What the number means

A typed decision model takes a state and a list of options supplied by the caller and returns a probability distribution over those options. The audit compares two of them on the same questions. Laya has open weights and was measured in five checkpoints, as released and fine-tuned. Jev 1.13 is available only through an API. Most measurements use MASSIVE (FITZGERALD et al., 2023), intent questions with five options, in Portuguese and English, and in 52 locales for the language front.

Everything else rests on one distinction. A 0.97 licenses a statement about groups. Among many answers given with that confidence, about 97% are right, provided the model is calibrated on data like yours. It does not certify the single answer in front of you. Confidence does separate right from wrong answers well, with an AUROC from 0.80 to 0.96, and that is exactly why it is so tempting to read it as a guarantee.

Confidence as a detector of a correct answer, on MASSIVE in Portuguese (left) and English (right). Each curve shows the share of correct answers a threshold accepts for each share of wrong answers it lets through. Every model sits well above the diagonal of a random guess, which is why the number feels like a guarantee. In the paper the same figure comes with reliability diagrams and the calibration of every checkpoint.

The promise, measured

The gate in the opening scene promises an accuracy of at least 95% among the cases it decides alone. To test such a promise, the 2,000 items of each locale stood in for the population. A calibration sample was drawn from them thousands of times, a threshold was fitted on each sample, and the true accuracy of that threshold was computed on the whole population. A violation is a true accuracy below the target.

Choose the threshold by point estimate, the lowest one whose observed accuracy on the calibration sample clears the target, and wherever the target binds the promise breaks in 39% to 53% of the draws. That is close to a coin flip, and it is not bad luck. A threshold picked because its estimate just clears the target sits, almost by construction, where the true accuracy is as likely to fall below as above. Across the 190 exchangeable evaluations of the paper, the point rule breaks the promise in 147.

How often a promise of 99% breaks, for five rules and six model configurations, with error bars at the one-sided 95% upper bound. The grey bars are the point rule, which breaks the promise in close to half of the calibration draws. The four rules built on a bound stay under the dashed tolerance of 0.05. The paper adds what each rule pays for that, in decisions per 1,000 items.

Replace the estimate by a lower bound and the picture changes. Learn then Test (ANGELOPOULOS et al., 2025) keeps the promise in all 190, with a formal guarantee. A Beta lower-bound rule with an ascending scan keeps it in 183 and decides more items on average than Learn then Test in 128 evaluations and fewer in none, but without the guarantee, and after a correction for multiple comparisons no gain in decisions survives on the static front. Almost the whole effect comes from a single change, a bound in place of an estimate.

What is decided alone against what is right, MASSIVE in Portuguese, promise of 99%, over 1,000 random splits of the items. The dark line is every possible threshold on the held-out half. Grey dots are thresholds chosen by point estimate, scattered on both sides of the dashed target. Blue triangles are thresholds chosen by the Beta rule, and the shaded bands mark the coverage at which it kept the promise (green) and broke it (red). With Jev, the Beta rule qualifies no threshold at all in 608 of the 1,000 splits and decides nothing. The paper shows where the remaining failures come from, at high coverage, where the curve flattens against the target.

The bound has a price worth knowing before deployment. A threshold needs a minimum number of accepted calibration items before any bound can clear the target, 58 at 95%. With fewer labels, the honest gate abstains.

A promise does not travel

The guarantee covers items exchangeable with the calibration set, and real traffic rarely is. Ordinary typing noise costs a few points of accuracy, and a few points are enough to break a promise calibrated on clean text. Calibrated in English, Jev keeps a promise of 95% in 8 of 51 locales.

The promise of 95% across the 51 locales of MASSIVE other than English. Left, the violation rate when the threshold is calibrated in English, against the accuracy each locale loses relative to English. Jev keeps the promise only where the loss stays below about one point. Right, the number of locales where the promise holds. Calibrating in each locale recovers it for Laya with the bounded rules, at the price of fewer decisions per 1,000 items.

Even the small shift from the sampled test rows of MASSIVE to its dev partition broke the bounded rules in two of six cases, Learn then Test included. On those new items, a fine-tuned Laya breaks a promise of 99% in 30% of the draws under Learn then Test.

The same holds across domains. In four further domains, a fine-tuned Laya needs from 250 to 8,000 labels to be non-inferior to Jev, and on LIAR2 no tested count was enough.

Label curves in four further domains. The fine-tuned Laya (blue) against Jev without labels (orange), as labelled items grow from 250 to 8,000. Gold where the fine-tune beats Jev, green where it is non-inferior within 2 points, red where it is neither. The paper reports the same curves with three seeds and Jev’s calibration in each domain.

Time adds its own shift. Under drift, the Bayesian retraining loop I preregistered did not meet its criterion. It avoided needless retraining where the change did not hurt the model, at a cost equivalent to plain recalibration, while re-qualifying the threshold in every window kept the promise more often where the change did hurt. On the monthly series of M3 (MAKRIDAKIS; HIBON, 2000), neither model predicted the direction of the next value better than a constant answer, and exponential smoothing was right in 67% of the decisions.

What the number cannot tell you

Confidence also does not pick out faithful explanations. Removing the words an explanation ranks first does make a decision fall faster than removing words at random, so explanations carry information on average. Decision by decision, though, the rank correlation between the probability and the fidelity of the explanation, measured by its deletion curve, is at most 0.38 in absolute value.

Deletion curves, mean over 1,000 utterances. Removing the words an explanation ranks first (solid) makes the decision fall faster than removing words at random (dotted), so the explanation carries information on average. What the average hides, whether a confident decision also has a more faithful explanation, is the question the paper takes up next.

Exact coalition games of single decisions still show what an explanation can say through an API that rounds to two decimals, and the empty state, the options with no message at all, reveals how much the options alone decide. On harder distractors drawn from the same scenario, the whole advantage of the fine-tuned checkpoints over Jev sat on items that an empty message already got right. Where the empty message was wrong, the differences were −0.2 and −0.6 points, with intervals around zero.

What to do with it

For code that acts on a model’s confidence, the paper reduces to four habits. Calibrate on data that looks like your traffic, noise and language included. Choose the threshold with a lower bound, never with the point estimate. Check the promise again when the traffic changes, and re-qualify the threshold where reviews are affordable. Audit the decisions the gate takes alone, because those are the ones nobody else will look at. The paper closes with a table that maps ten situations a caller meets, from few calibration labels to a model behind an API, to what was measured and what to do, and it is the part I would read first.

So, the threshold in your pipeline. Was it the one whose estimate just cleared the target? If the target binds, it is close to a coin flip that its true accuracy sits below what was promised, and nobody will notice until a number downstream does.

---

Paper. BROM, P. C. Confidence Is Not a Promise: A Reference Audit of Typed Decision Models. arXiv:2610.36311, 2026. Available at: https://arxiv.org/abs/2610.36311.

Code and data. Code, derived measurements, preregistration notes, Lean 4 proofs and the frozen answers with a datasheet, archived on Zenodo under DOI 10.5281/zenodo.23676316.

References.

ANGELOPOULOS, A. N.; BATES, S.; CANDÈS, E. J.; JORDAN, M. I.; LEI, L. Learn Then Test: calibrating predictive algorithms to achieve risk control. The Annals of Applied Statistics, v. 19, n. 2, p. 1641-1662, 2025. DOI: 10.1214/24-AOAS1998.

FITZGERALD, J. et al. MASSIVE: a 1M-example multilingual natural language understanding dataset with 51 typologically-diverse languages. In: ANNUAL MEETING OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS, 61., 2023, Toronto. Proceedings […]. Toronto: ACL, 2023. p. 4277-4302. DOI: 10.18653/v1/2023.acl-long.235.

MAKRIDAKIS, S.; HIBON, M. The M3-Competition: results, conclusions and implications. International Journal of Forecasting, v. 16, n. 4, p. 451-476, 2000. DOI: 10.1016/S0169-2070(00)00057-1.

The source is open on GitHub: github.com/pcbrom/typed-deciders-audit. I write about this kind of work, scientific method, statistics, and AI applied with rigor, on LinkedIn: linkedin.com/in/pcbrom.