Context compression cuts 60 to 92 percent of your tokens, until the needle looks like noise
A density plot is the first thing people trust after a long run, and a headline percentage is the first thing people trust after a compression benchmark. Both are easy to be fooled by. “Sixty to ninety-five percent fewer tokens with no quality loss” is the kind of average that sells a tool. It is also true, on the inputs the average was measured over. The question this post answers is what the average leaves out, because that is where an agent pipeline actually breaks.
The setup is mundane and it is most of an agent’s context budget. A coding or research agent does not spend its window on prose. It spends it on tool outputs: JSON blobs, log dumps, API responses, the same field names repeated a thousand times. Context compression promises to squeeze that redundancy before it reaches the model. The tool under test here is the open-source headroom-ai compressor (version 0.27.0), run end to end against a local model so the token counts are the model’s own, not a proxy.
What the average reports
I measured token reduction by content type, on gemma-4-31B-it (Q4_K_M with multi-token prediction, llama.cpp), counting tokens through the model’s own tokenizer rather than a characters-over-four estimate. The default router does not treat all text alike:
type token reduction handling
log 91.8% log compressor
json 62.0% structured compressor
code 0.0% protected by default
prose 0.0% passthrough
conversation 0.0% passthroughSo the first correction to the headline: the claimed 60-to-95 range is real, landing at 62 to 92 percent in my runs, but it is restricted to structured and redundant content. Prose and conversation are not compressed at all in the default configuration, and code is deliberately protected. The win lives exactly where agent context rots, in JSON and logs, which is convenient, but the scope matters before anyone quotes the number.
Reduction is only half of it. Cutting tokens is worthless if the model can no longer answer. So the second measurement plants a known fact in each input, a needle, and checks whether the model still recovers it from the compressed context, against the full context and against naive truncation at the same token budget. Scoring is by presence of the gold answer in the model’s output, temperature zero, twenty items per cell, with a McNemar paired test on the discordant pairs (McNEMAR, 1947).
On plain retrieval the compressor pays nothing:
task full compressed truncated compressed vs truncated
json retrieval 1.00 1.00 0.65 p = 0.016
log retrieval 1.00 1.00 0.65 p = 0.016
json multi-hop 1.00 1.00 1.00 not distinguishableCompressed context matches full context exactly, one for one, no discordant pairs, while keeping 62 to 92 percent of the budget. It also beats naive truncation with a significant margin, and truncation is the honest baseline here: it is what you do when you just clip the context to fit. The multi-hop row matters too, because it shows the compressor does not sever a two-step reference chain. If the answer needs fact A to find fact B, both survive.
This is a good tool. The average is not lying.
What the tail does
Now the input the average never sees. Take a log where the needle is not an anomaly, not an ERROR line, not a stack trace, but a single mundane INFO line that happens to carry the fact you will be asked about, sitting among hundreds of near-identical INFO lines. This is the adversarial case, built on purpose to probe one hypothesis: that a log compressor keeps what looks salient and discards what looks like boilerplate.
task full compressed truncated
log adversarial 0.70 0.05 0.25
p = 0.0002 vs fullAccuracy collapses from 0.70 on the full context to 0.05 on the compressed one. Naive truncation, the dumb baseline, scored 0.25, five times better than the smart compressor. The compressor threw the needle away because the needle was shaped like noise. The paired test puts this at p = 0.0002, which is not a fluctuation. It is the failure mode the adversarial task was designed to catch, and it caught it cleanly.
This is the point of the post, and it is a measurement-discipline point before it is an engineering one. Two numbers describe the same tool: a near-perfect score on retrieval and a near-zero score on the adversarial log. Report the mean across content and you publish something near the top. Report the tail and you expose a regime where the tool is worse than doing nothing clever. The average sells 60 to 92 percent. The tail is where the risk lives, and the risk is not hypothetical, it is 0.05.
A reader who works with heavy-tailed data will recognize the structure. The expected value is a poor summary when the distribution has a mode that the mean glides over. A compressor that is excellent on the bulk and catastrophic on a thin, identifiable slice has exactly that shape. You do not learn it from the headline. You learn it by building the input that targets the failure and measuring what breaks.
The rule that keeps the win
None of this argues against compression. It argues for routing by content type and for knowing which fields can look like boilerplate. Three practical conclusions came out of the audit.
First, lossless beats lossy wherever it is available. The structured compressor used for JSON is byte-recoverable: the original reconstructs exactly, and it still cut 62 percent of the JSON budget in the measurement above. That is the safe default, a large saving at no fidelity cost, with one operational caveat noted below. The more aggressive log compressor is lossy by design, and that is the one the adversarial case punishes.
Second, compress logs only where the routine lines you are discarding are genuinely irrelevant. If a specific INFO line, a config value, or one entry among many identical ones can be the thing you later query, the lossy log path is not safe for that context. Protect those fields or keep them out of the compressor.
Third, the natural insertion point is a hook on the tool output itself, a PostToolUse step or an MCP wrapper that compresses outputs before they enter the model’s context. JSON goes through the lossless path freely. Logs go through the lossy path only under the constraint above. Prose, conversation, and code pass through untouched, which is what the default already does.
What this is and is not
The honest limits, because the tail lesson applies to my own numbers too. This is twenty items per cell on a single local model, enough to make the adversarial collapse unambiguous (p = 0.0002) and the retrieval parity convincing, not enough to chase a precise effect size on the borderline cells. The lossless JSON path is byte-recoverable but its packed form is not itself re-parseable JSON, so a downstream step that expects to json.parse the compressed text needs the unpack first. Nested JSON was not tested. None of these caveats touch the central result: the same compressor that earns its place on structured retrieval will silently delete a fact that wears the uniform of a routine line, and only an input built to find that will tell you.
The average is a sales pitch. The tail is the spec.
Source. Headroom compression audit, phases 1 to 3, own measurement on gemma-4-31B-it with headroom-ai 0.27.0, paired McNemar evaluation, twenty items per content type.
Reference. McNEMAR, Q. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, v. 12, n. 2, p. 153-157, 1947. DOI: 10.1007/BF02295996.