Blog

Pedro Carvalho Brom

Technical notes on statistics, generative AI and the R language.

The model copies the number correctly and then concludes wrong: move the reasoning into code

LLM

evaluation

local-models

prompt-engineering

data

A 2B model given a raw record copies the numbers faithfully and then draws the wrong conclusion from them: at the 4th percentile it still says intermediate. So the pipeline moves the reasoning into deterministic code and leaves the model only the prose, with numbers as symbols the harness substitutes. The paired ablation puts a number on it: remove the mediation and 30% of readings carry a factual defect, against 0% with it, McNemar p = 2.9e-11. What breaks in a small model is not transcription. It is ranking.

Jul 13, 2026

A test that cannot fail anyone cannot pass anyone

LLM

evaluation

quantization

local-models

statistics

My first evaluation set was ten easy factual and arithmetic questions. Every model scored at or near 1.0. The set had zero discriminating power, so the approval it handed out was worthless. This is how the benchmark was rebuilt until it could reject a model, and why an agreement metric had to be thrown away on the way.

Jul 13, 2026

When the grader and the graded make the same mistake: field hallucination in document extraction

LLM

evaluation

quantization

local-models

OCR

Labelling an extraction benchmark with your largest model is the cheap path, and it hides the failure that matters most: the model fills in a field that does not exist on the page. Only a hand-checked gold set caught it. The measured consequence is that the most accurate extractor is also the one that invents half the absent fields.

Jul 13, 2026

The same quantization recipe, three different verdicts

LLM

evaluation

quantization

local-models

statistics

One int4 QAT recipe, three local models, three outcomes: no difference, significantly worse, and significantly better. The best model in the whole study is itself a QAT build, which kills the easy story that quantization degrades quality. What separates the failure from the successes is not size; the failing model is the sparse one, and it is also the one with a template confounder. Quantization preserving quality is a result, not a property, and this is what it is conditional on.

Jul 13, 2026

Turning reasoning off did not just save tokens. It exposed a quantization regression.

LLM

evaluation

quantization

local-models

statistics

With reasoning on, a quantized model and its original tie exactly. With reasoning off, the same pair separates by ten points of accuracy, McNemar p = 0.012. The regression was there all along; the reasoning chain was paying for it. If you cut reasoning to cut cost and did not revalidate the quantization, you inherited the damage.

Jul 13, 2026

Context compression cuts 60 to 92 percent of your tokens, until the needle looks like noise

LLM

context-engineering

evaluation

statistics

agents

opensource

An LLM context compressor can drop most of a tool output or log with zero loss on retrieval, and still fail catastrophically on one specific shape of input: a critical fact disguised as a routine line. This post measures both the win and the failure on a local 31B model, shows why the average sells a number the tail does not honor, and gives a routing rule that keeps the gain without the risk.

Jun 29, 2026

gpumetropolis 0.4: a guided tour of a Bayesian toolkit that declares where it loses

MCMC

Bayesian

statistics

extreme-value

gpumetropolis

opensource

Version 0.4 turns gpumetropolis from a fast sampler into a full Bayesian workflow: Differential Evolution for correlated targets, a Cramer-Rao reference that refuses to answer when the posterior is multimodal, decision and model-comparison verbs, and three real-data cases. This is the guided reference, one worked figure per tool, with the honest head-to-head benchmark as its spine: a package that names the regimes where it loses, not only the ones where it wins.

Jun 29, 2026

When a bimodal posterior lies: parallel tempering from a toy model to Oxford’s temperature extremes

MCMC

Bayesian

statistics

extreme-value

gpumetropolis

opensource

A pooled MCMC density can look perfectly bimodal while every chain is stuck in one mode and the answer is wrong. This post shows the trap on a toy model, the one diagnostic that catches it, the parallel-tempering fix in gpumetropolis, and why it is not academic: real extreme-value data sits exactly on this multimodal geometry, and no R package fits a Gumbel mixture in one call.

Jun 16, 2026

harness: bringing the code agent back inside RStudio

RStudio

agents

LLM

tooling

opensource

A new R package that launches a command-line coding agent in an RStudio terminal tab, pre-configured for one of seventeen professional roles, with curated skills, a system prompt, a folder layout, and an audit-first execution gate. With a hands-on walkthrough of the first session, how the seventeen roles are wired, the RStudio editor bridge, and a worked example that runs three coders on the same task.

Jun 9, 2026

harness: trazendo o agente de código de volta para dentro do RStudio

RStudio

agents

LLM

tooling

opensource

Um novo pacote R que abre um agente de código de linha de comando em uma aba de Terminal do RStudio, pré-configurado para um de dezessete papéis profissionais, com skills curadas, system prompt, layout de pastas e gate de execução por auditoria. Com um walkthrough prático da primeira sessão, como os dezessete papéis estão ligados, a ponte de editor do RStudio, e um exemplo trabalhado que roda três coders na mesma tarefa.

Jun 9, 2026

From CRAN to R-universe: the 10 MB ceiling, the policy line, and what stays human

CRAN

runiverse

packaging

Rust

agents

opensource

An honest account of a CRAN submission that did not become a release. What the cran_publisher automation handled, what it learned from the pretests, and where it stopped because no automation crosses a policy line. With the size arithmetic, the reviewer’s request, and the move to R-universe.

Jun 4, 2026

Do CRAN ao R-universe: o teto de 10 MB, a linha de política, e o que continua humano

CRAN

runiverse

packaging

Rust

agents

opensource

Um relato honesto de uma submissão ao CRAN que não virou release. O que a automação cran_publisher tratou, o que ela aprendeu com os pretests, e onde ela parou porque automação alguma atravessa uma linha de política. Com a aritmética de tamanho, o pedido do revisor, e a mudança para o R-universe.

Jun 4, 2026

autoresearch v1.1: closing the propose-apply-measure loop

LLM

agents

Python

optimization

opensource

What changed in autoresearch v1.1: the loop now drives a headless coding agent on every iteration, removing the human step that used to sit between propose and measure. With an illustrative TSP run, the two permission postures, and the plug-in critic backends shipped in the same release.

Jun 4, 2026

autoresearch v1.1: fechando o loop propor-aplicar-medir

LLM

agents

Python

optimization

opensource

O que mudou no autoresearch v1.1: o loop passa a chamar um agente de código sem cabeça em cada iteração e elimina a etapa humana que ficava entre propor e medir. Com uma run ilustrativa de TSP, as duas posturas de permissão, e os backends de critic plugáveis no mesmo release.

Jun 4, 2026

community-skills v0.2.0: a queryable CRAN graph, an assisted package publisher, and a skills hub

CRAN

packages

tooling

Tools for the R ecosystem: a queryable graph of CRAN dependencies, an assisted package-submission pipeline, and a hub that wraps CRAN packages as typed, agent-callable skills.

May 16, 2026

pcbrom.com is live

I want my agent to fit a mixed effects model in R. How many tokens does that cost?

CRAN

LLM

agents

tooling

A practical look at the token cost of letting an LLM agent call CRAN packages, and how a typed skill contract cuts it. With a benchmark, an honest methodology, and an ROI estimate.

May 15, 2026

Categories

Blog