# Your Agent Passed the Eval. Its Retrieval Calls Still Tripled.

An AI agent can pass the task and still fail the engineering evaluation.

That happens when a context-compression policy preserves the final outcome but changes how the agent gets there. The agent still completes the benchmark. It also spends far more of its interaction budget re-querying tools for state it had already discovered.

A new preprint, [*What Does Context Compression Cost an Agent?*](https://arxiv.org/html/2608.16370v1), demonstrates this gap in a controlled setting. Its useful conclusion is narrow: **task completion can remain statistically unchanged while state reacquisition rises sharply.**

Passing the task is not the same as preserving the work profile.

## Completion hides a second axis

Most context-management evaluations ask a necessary question: does the agent still finish after older context is summarized, filtered, or removed?

Completion compresses the whole trajectory into one outcome. It cannot tell us whether the agent used 20 retrieval calls or 60, whether it repeatedly read the same resource, or whether recovery consumed so much of the horizon that one harder task would have failed.

The paper makes that missing dimension explicit. Under a fixed 24-turn horizon, it separates tool calls into:

- **retrieval:** calls that obtain environment state;
- **execution:** calls that perform the task's operations.

At the prespecified 5x compression point, completion changes were not statistically significant in any of six model-regime comparisons. Retrieval increased in all six and remained significant in five after Holm correction. The sharpest cell used GPT-5.5 in the study's high-reacquisition regime: completion moved from 80% to 85%, with no detected difference (`p = 1.0`), while mean retrieval calls rose from 21.0 to 63.9 (`p = .002`). The confidence interval for completion was wide and included zero, so the result is not evidence that compression improved success. It is evidence that completion-only scoring discarded a large cost difference.

![Two agent trajectories reach the same completion point, but the compressed-context path loops repeatedly through state retrieval before execution.](https://cdn.hashnode.com/res/hashnode/image/upload/v1787288966049/e0a79e1c-4e16-4e20-a01e-c34c2cee4cc6.png)

## The agent was recovering state, not doing more work

The tool decomposition matters because extra interaction is not automatically waste.

An agent might make more calls because it checks its work, explores a better plan, or completes more useful operations. That is not what dominated this experiment.

In the DeepSeek high-reacquisition severity sweep, retrieval calls increased from 22.2 with full context to 63.2 at 10x sliding compression. Execution calls fell from 17.3 to 13.9. The agent was not attempting more task work. It was spending more of the same horizon reconstructing state that compression removed.

The failure becomes visible in completion only after recovery consumes enough of the interaction budget. In that sweep, retrieval was already significantly higher at 5x compression; completion did not show a significant decline until the more aggressive 10x condition.

Completion is therefore a lagging indicator for this failure mode.

## What survives matters more than the ratio

The study also compares two ways to spend the same 5x context budget.

A sliding-window policy deleted older turns. A deterministic extractive summary retained observed state facts while discarding earlier reasoning text. With DeepSeek in the high-reacquisition regime, the sliding condition recorded 72% completion and 55.1 retrieval calls. The fact-preserving summary recorded 83% completion and 19.5 retrieval calls, close to the full-context condition's 80% completion and 19.0 retrieval calls.

This is a designed control, not proof that arbitrary LLM summaries are lossless. It supports a more practical point: **the nominal compression ratio tells us less than which execution-relevant state survives.**

The oracle intervention strengthens that explanation. When externally queryable task state was restored to the sliding condition, total tool calls fell from 72.9 to 35.8 and completion recovered from 66% to 80%. Restoring the missing state removed much of the re-querying, which gives the reacquisition mechanism causal support inside this environment.

## Add a work-profile regression test

A compression change should be evaluated as a paired runtime intervention, not only as a token-saving feature.

For each representative task:

1. Hold the model, task set, tools, and interaction horizon fixed.
2. Compare full context with the candidate compression policy.
3. Record completion or graded progress alongside total interaction cost.
4. Classify tool calls as retrieval or execution where tool semantics permit it.
5. Track repeated reads of the same resource or state.
6. Measure turns, tokens, latency, billed cost, termination reason, and budget exhaustion in the real deployment.
7. Restore deliberately dropped state as a control when causal attribution matters.

The exact metric should follow the environment. A rising retrieval-to-execution ratio is useful only when those tool categories are semantically clear. In other systems, duplicate resource reads, cache misses, or repeated database queries may expose the same pattern more directly.

A sensible approval rule is:

> Do not accept a compression policy solely because completion is non-inferior. Require no material regression in success-adjusted interaction cost, or document why the extra recovery work fits the deployment's latency, cost, and reliability limits.

## The boundary is part of the result

This paper does not establish that compression is broadly harmful.

Its main environment is a synthetic planning benchmark. The horizon is fixed at 24 turns. Only three model families are tested, and only DeepSeek receives the full severity sweep. Tool-call count is an interaction proxy, not measured latency, money, or user experience. The summary control is deterministic and fact-preserving, not a general evaluation of learned or abstractive compressors.

Most importantly, the retrieval surge did not reproduce in the paper's ALFWorld probe. In that environment, relevant state could generally be re-observed directly, and sliding compression produced approximately no change in retrieval-like actions. Completion was also very low and dominated by task difficulty, so the probe should not be read as a broad capability result.

That boundary makes the engineering lesson more precise. Compression cost is not a property of compression alone. It depends on what state disappears, how the environment exposes that state again, and how much interaction the agent must spend to recover it.

The preprint is new, single-author work, and an earlier version was accepted at a COLM 2026 workshop. Independent replication and broader production evidence are still needed. Related work points in the same measurement direction: [*Control Under Compression*](https://arxiv.org/abs/2608.01056) reports context-specific reliability frontiers, while [*Token Reduction Is Not Cost Reduction*](https://arxiv.org/abs/2607.12161) finds that removing tool-output tokens did not reliably reduce provider-billed cost in its Claude Code evaluation. Neither independently confirms this paper's exact mechanism or magnitudes.

## Evaluate the path, not only the finish line

Context compression can be a sound runtime decision. [ACON](https://arxiv.org/abs/2510.00615), for example, reports lower peak-token use while preserving or improving task success across its evaluated agent environments. The question is not whether agents should compress context.

The question is whether the compression policy preserved useful behavior under the deployment's budget.

If an agent finishes only after spending three times as much on retrieval, the completion score has told the truth. It just has not told enough of it.