The Trust Gap: Why Model Evaluation Is Not Agent Evaluation

Abstract

This position paper synthesizes two recent academic findings - the system-scaling thesis (Gu, 2026) and the reasoning-depth limitation of context-window models (Lee et al., 2026) - with commercial and regulatory developments in AI compliance (ISO 42001, NIST AI RMF, EU AI Act) to argue that the agent economy needs a new evaluation paradigm: harness-level vetting that scores all six components of agent behavior, not just model accuracy.

We propose a practical framework for this evaluation, the AgentVet Harness Scorecard, and map it to compliance requirements that enterprises will increasingly demand.

1. Model Scores Are Harness Scores

When a benchmark reports that an agent achieves X% on SWE-bench or GAIA, that score is not a model score. It is a model-plus-harness score. The same model with a different tool interface, memory system, or orchestration loop produces different results.

SWE-agent demonstrated this clearly: redesigning the agent-computer interface alone - while holding the underlying model fixed - substantially improved benchmark accuracy. Yet the field continues to report results as if they reflect model capability alone.

Any evaluation that doesn't separate model capability from harness design is measuring the wrong unit of analysis.

2. The Reliability Gap

A single successful run doesn't prove an agent works. It proves it worked once.

Recent analysis shows that agents that achieve high single-shot pass rates can collapse under pass^k - the probability of succeeding on k independent rollouts. An agent that passes 80% of the time on a single run may pass only 50% of the time over three independent attempts. This reliability gap is invisible to benchmark leaderboards but critical for production deployment.

Trust requires reliability, not just capability. An agent that sometimes works is not an agent you can deploy.

3. The Reasoning Depth Problem

Bigger context windows don't solve the context problem. Transformer attention dilutes over long inputs - models prefer evidence at the start or end, missing the middle. Hybrid SSM-attention models compress old context into fast weights, but Lee et al. (2026) show that these models fail when the required reasoning depth increases, even when the information to store is held constant.

The bottleneck isn't memory capacity. It's computation over memory. Converting transient context into useful internal representations requires work, and one pass isn't enough. Their solution: a sleep-like consolidation mechanism where the model performs multiple offline recurrent passes over accumulated context before clearing its cache.

Long-running agents degrade. An agent that performs well at the start of a session may perform poorly hours later as context accumulates and reasoning depth requirements increase. This is a failure mode that no current benchmark tests for.

4. The Six-Component Harness Framework

Following Gu (2026), agent performance emerges from the interaction of six components:

Model scaling improves R. System scaling improves M, C, S, O, and G. The industry has invested heavily in R and neglected the other five. This creates the trust gap: agents that score well on benchmarks but fail in production because their memory is unreliable, their context construction is sloppy, their skill routing is insecure, their orchestration is fragile, or their governance is absent.

5. The Compliance Imperative

Three regulatory forces make the Governance component non-optional:

The market is shifting from “is AI safe?” to “prove it's safe.” Companies that can't demonstrate compliance will lose contracts, fail audits, and face regulatory penalties.

6. The AgentVet Harness Scorecard

We propose a practical evaluation framework that scores all six harness components on a 1-10 scale:

7. From Evaluation to Certification

The Harness Scorecard is not just an evaluation tool. It's a certification pathway:

Monitoring without certification is just observation. Certification without monitoring is just paperwork. Together, they're trust.

8. The Research Agenda

The field needs new benchmarks that evaluate systems, not models. We propose:

Conclusion

The agent economy is built on a false premise: that model evaluation is sufficient. It is not. Models are components. Agents are systems. Evaluating systems as if they were components produces misleading confidence, hidden reliability gaps, and unaccounted risk.

The academic evidence is clear. Gu (2026) shows that harness design determines agent behavior as much as model capability. Lee et al. (2026) show that reasoning depth - not just memory capacity - degrades without adequate computation over context. The regulatory evidence is equally clear. ISO 42001, NIST AI RMF, and the EU AI Act all demand system-level governance that model-only evaluation cannot provide.

The trust gap is real. It is measurable. It is the defining opportunity of the agent economy.