A Tool Result Is Evidence, Not Ground Truth

Connecting an agent to a database does not remove hallucination risk.

It gives the system a second source that can be stale, malformed, unauthorized, or simply wrong—and a protocol that may make the model trust it more than it should.

Tools do not eliminate the trust problem. They move the trust boundary outside the model.

A new preprint makes that failure measurable. Its result is narrow, but the production lesson is not: a clean tool payload should be treated as evidence, not as truth.

MemToC tests the conflict that ordinary tool benchmarks skip

Many tool-use evaluations ask whether a model can select a tool, form valid arguments, and incorporate the returned value. Those are important interface checks. They usually do not establish whether the tool value deserves to be believed.

MemToC, submitted on August 26, 2026, separates those questions. The researchers first elicit a closed-book answer, verify whether it is correct, then hold the question fixed while presenting the model with one of four tool conditions: a correct result, a plausible incorrect result, an error payload, or no result.

The released benchmark contains 6,504 episodes built from 542 quality-controlled factual questions. The main findings cover four instruction-tuned open-weight models in the 7–9B range: Llama 3.1 8B Instruct, Gemma 2 9B IT, Qwen 2.5 7B Instruct, and Mistral 7B Instruct v0.3.

This is a preprint, not a peer-reviewed general law about every agent. But it tests a system property that production evaluations often leave implicit: what happens when parametric memory and a tool disagree, and one of them is known to be wrong?

The models followed the channel more than the correctness

When the model's initial answer was verified correct and the tool returned a plausible wrong answer, correct-answer retention was only 6.5–17.1% across the four instruction models. When the tool was correct, the models followed it 86.0–93.1% of the time.

The asymmetry could be mistaken for sensible tool trust. The harder condition shows why it is not enough. When both the model's answer and the tool output were wrong, the models still repeated the incorrect tool result in 78.4–86.0% of eligible cases.

The source arrived through the tool interface, and that presentation carried authority even when correctness did not.

The conflict was also quiet. Two annotators reviewed 120 incorrect-tool responses for an explicit acknowledgment that the tool disagreed with the model's own knowledge. None acknowledged it. The paper reports a 95% upper bound of 3% for that measured behavior.

That matters because silent deference is harder to monitor than visible uncertainty. A wrong answer with a “tool verified” shape can pass through logs, reviewers, and downstream agents without producing an obvious conflict signal.

A valid JSON response can still be epistemically invalid

Tool integration encourages a useful engineering discipline: typed inputs, schemas, authentication, retries, and explicit error handling. But interface validity and factual validity are different properties.

A response can be:

• valid against the schema and stale by six hours;
• authorized for the caller and scoped to the wrong account;
• returned with HTTP 200 and computed from incomplete data;
• syntactically precise and based on a failed upstream dependency;
• correct when produced and obsolete when consumed.

The model often cannot infer those conditions from the payload alone. A neat object with a successful status can look more authoritative than uncertain internal knowledge, even when the object carries no provenance or freshness.

This is not only a model-alignment problem. It is a system-contract problem. If the application strips timestamps, source identity, confidence, error semantics, or data lineage before passing the result to the model, it removes the evidence needed for sensible arbitration.

The strongest case for trusting tools still holds

Agents should usually prefer an authoritative live system over memorized training data. Account balances, inventory, permissions, current prices, deployment status, and private business records do not belong in model weights. A model that routinely argues with those systems would be unreliable in a different way.

The MemToC results do not justify blanket tool distrust. A policy that preserves internal answers at all costs would fail whenever the tool is the current source of truth. The paper's mitigation results show this trade-off directly: prompting or fine-tuning could improve resistance in some settings while damaging correct-tool following, tool-error abstention, or another reliability dimension.

The paper's preregistered asymmetric criterion required correct-answer retention to improve without a detected reduction in correct-tool following. Supervised fine-tuning met it on two of four instruction-tuned backbones. Other changes often shifted one error mode into another.

So the goal is not “trust memory” or “trust tools.” It is to make source quality observable enough that the system can distinguish the cases.

Build a trust contract around every consequential tool

Start with provenance. A tool result should identify the source, relevant timestamp, scope, and whether the value is direct, cached, inferred, or aggregated when those distinctions matter. Do not turn every payload into a legal document, but do not hide the fields needed to judge it.

Then define the failure policy by consequence:

The evaluation should inject failures deliberately. Return a plausible wrong value, stale data, a permission error, a timeout, an empty success, and conflicting outputs from two sources. Measure at least four behaviors together: correct-tool use, harmful deference, error abstention, and explicit conflict acknowledgment.

Optimizing only correct-tool following rewards blind compliance. Optimizing only resistance rewards stubbornness. The useful target is calibrated arbitration under a declared cost of error.

For high-impact workflows, deterministic validation should sit outside the language model whenever possible. Check authorization in code. Enforce amount and date bounds before a transaction. Compare identifiers against the active account. Require approval for irreversible actions. The model can interpret evidence; it should not be the only component deciding whether the evidence is admissible.

The limitations keep the claim bounded

MemToC studies short factual questions and controlled synthetic substitutions. Its main results cover open-weight models, mostly 7–9B, not current proprietary frontier systems. The paper also reports meaningful sensitivity to prompt wording and presentation, with no stable cross-model ordering across three formulations.

Those constraints prevent a universal claim that every production agent will defer at the same rate. They do not support dismissing the failure mode.

The reasonable inference is narrower: tool deference cannot be assumed to track tool correctness, and a model's behavior under one prompt is not a stable reliability property. Each deployed model, prompt, tool contract, and consequence class needs its own conflict tests.

The verdict

“Grounded by tools” describes an architecture. It does not certify an answer.

Once a tool enters the loop, its provenance, freshness, failure modes, and presentation become part of the model-system evaluation. If the only test is whether the agent used the returned value, the evaluation can reward exactly the behavior that makes a plausible bad payload dangerous.

A reliable agent does not merely call tools successfully. It knows when the evidence conflicts, exposes that conflict, and has permission to stop.