The AI Accuracy Race Is Costing You Four Times What It Should
Your team ran an evaluation last quarter. You picked the agent configuration with the highest factuality score. It topped the leaderboard, it looked responsible, and it went into production. What you did not see in that report is that the second-place system answered just as well, at a quarter of the compute cost, in a quarter of the time. The leaderboard had no column for that.
Researchers at Shanghai Jiao Tong University published a paper in July 2026 that names this problem precisely and builds a method to fix it. Their system, MAS-HQ (Multi-Agent System Hallucination Quest), introduces a scoring protocol that treats compute as a real cost rather than a free resource. The headline finding is concrete: a Best-of-4 sampling agent posts an H-Score of 0.9169 and would top any standard factuality leaderboard, but it consumes 6.32 million tokens and 35,570 seconds to do it. Their competing Q-Agent posts an H-Score of 0.9103, a 0.0066 gap, while using 1.36 million tokens and 8,830 seconds. Once cost is factored in, the leaderboard winner is the worse deployment choice, and the paper provides the math to prove it every time. The gains from MAS-HQ are described by the authors themselves as small but consistent across 100 independent trials, and the ranking reversal pattern holds for frontier models including GPT-5 and Gemini-2.5-Pro, where raw factuality scores already cluster near the ceiling.
Why Treating Compute as Free Has Been Distorting Every Evaluation You Have Run
The core problem is architectural, not incidental. Standard factuality benchmarks score accuracy on a scale from zero to one and report that score in isolation. There is no mechanism to penalize a system for running four inference passes when one would do, or for consuming six million tokens when one and a half million produce a result that is 99.3% as accurate. The result is what the paper calls the over-optimization trap: a system can buy a higher raw score by throwing more compute at a problem, and the leaderboard rewards it.
This is not a theoretical edge case. The authors demonstrate it empirically across five LLM families, including Qwen-Max, Deepseek-V3, Gemini-2.0-Flash, and Grok-3-beta. The reversal is not a quirk of one model's architecture. It is a structural feature of any evaluation that separates accuracy from cost.
The scale of the distortion matters for real budgets. Best-of-4 sampling uses approximately four times the tokens, four times the API calls, and four times the wall-clock latency of a single-pass agent. In a high-volume production environment, that multiple is the difference between a system that fits inside an operational budget and one that does not. As the paper states directly: in high-frequency trading or real-time news intelligence, an agent that delivers equally credible insights on half the budget delivers greater operational value.
What the Q-Score Actually Measures, and What Breaks at Each Parameter
The Q-Score formula is: Q-Score = (1/N) multiplied by the sum of (α times H-Score minus β times P), where H-Score is the factuality metric, P is a normalized operational cost penalty aggregating token count, API calls, review iterations, and wall-clock runtime, and α and β are deployment-specific weights. The paper uses α=1 and β=0.01 as the main setting and sweeps β to show what happens at the extremes.
| β Setting | What It Does | Deployment Consequence |
|---|---|---|
| β = 0 | Protocol collapses to static axis; cost is free again | Equivalent to a standard leaderboard; ranking reversals invisible |
| β = 0.01 | Balanced setting; reversal visible; review counts disciplined | Q-Agent A wins on Q-Score (0.5217) despite lower raw H-Score |
| β = 0.02 | Cost weight dominates; H-Scores suppressed by penalty | Quality is penalized excessively; over-corrects against review |
This parameter dependency is the most important honest caveat in the paper. There is no canonical value of β. An organization must set it based on its own deployment SLA, specifically how much it values factuality versus how much it is willing to pay in compute. Setting β too low recovers the same failure mode MAS-HQ was designed to eliminate. Setting it too high penalizes systems for doing quality work.
The scoring axis only becomes cost-honest when β is above zero, which means the first decision any team implementing this protocol must make is a deliberate one about what accuracy is worth at their specific operating cost.
The Architecture That Makes Competition Do the Disciplining
MAS-HQ does not just add a cost term to an existing score. It restructures evaluation as a head-to-head competition between two agents processing the same passage set in opposite orders, with partial visibility into each other's activity. The competitive structure is what forces resource discipline to emerge from the agents' own decision-making rather than from an external hard cap.
The Q-Agent implementation has four components that work in sequence:
- The Policy Agent receives the current state and, optionally, a partial snapshot of the competitor's state, then decides whether to continue generating, trigger a review pass, or end the session.
- The Summarization Agent executes the primary task, isolated from Q-Score awareness so it cannot game the metric by producing deliberately short output.
- The Review Agent refines the lowest-scoring summary at additional resource cost, triggered only when the Policy Agent judges the review worth the penalty.
- The Evaluation Agent scores each summary against the factuality detector and tracks all resource consumption metrics, feeding that information back to the Policy Agent.
The vision mechanism is the design element that makes the competition non-trivial. When one agent chooses to run a review pass, a snapshot of its state, specifically its worst H-Score and total tokens consumed, is disclosed to its opponent. The opponent can then adjust its own policy with that information. Table 9 in the paper confirms that without this mechanism, and without opposite processing orders, both agents produce identical Q-Scores. The match collapses to a tie. Both conditions are necessary to generate asymmetric, discriminative outcomes.
The Frontier Model Result That Static Leaderboards Cannot Show
The finding most likely to be underweighted in a quick read is the frontier model result. When GPT-5 competes against GPT-5, the agent with the higher raw factuality score, 0.9312, loses to the agent with the lower score, 0.9278, on Q-Score, 0.5312 versus 0.5381. The same pattern holds for Gemini-2.5-Pro: the agent scoring 0.9264 on raw factuality loses to the agent scoring 0.9221 once cost is counted, 0.5264 versus 0.5310.
This matters because the standard defense of leaderboards at the frontier level is that when models are so close in raw accuracy, ranking is just noise. The MAS-HQ result suggests the opposite: when raw factuality saturates near the ceiling, cost becomes the only remaining axis that differentiates systems. Two GPT-5 configurations are indistinguishable on factuality. They are not indistinguishable on compute. A static leaderboard has nothing to say about that distinction. MAS-HQ exists precisely for that case.
The statistical robustness is worth noting. Across 100 independent trial repetitions of the GPT-4o-mini match, Q-Agent A's advantage holds at 0.5216 plus or minus 0.0009 versus 0.5131 plus or minus 0.0009. The gap of 0.0085 is approximately 9.4 times the standard deviation on either agent. The result is not sensitive to randomness.
Where This Logic Does Not Hold, and Why That Matters
The paper includes a SimpleQA generalization test on exact-match factoid questions, and the result inverts the summarization finding. On that task, the winner is the agent that invests more resources for higher accuracy, not the agent that conserves them. GPT-4o scores approximately 38.5% exact match accuracy. GPT-4o-mini scores approximately 1.2%. More resources spent correlates with better outcomes, not worse ones.
The authors frame this as evidence that the core trade-off between performance and cost remains present across task types, which is accurate. But the direction of the efficient outcome reverses depending on the task structure. In summarization, a single careful pass captures most of the factuality available. In hard factoid retrieval, brute-force attempts produce genuine gains that justify the cost. Any organization applying MAS-HQ needs to calibrate β and interpret results task-by-task, not assume the efficiency dynamic is universal.
There is also a structural limitation the paper is explicit about. Q-Score is a match-specific payoff, not a standalone model score. Two different opponents produce two different Q-Scores for the same model. To produce an intrinsic per-model ranking, you would need tournament or Elo-style aggregation across many matches. The paper acknowledges this directly and leaves it to future work. That gap matters for any team trying to build a procurement checklist from a single evaluation run.
What This Changes for Teams Buying or Building AI Systems Right Now
The most immediate implication is not technical. It is how organizations frame their evaluation criteria before a procurement decision or a production deployment.
Any team currently selecting an LLM configuration based on published factuality leaderboard scores is operating with an incomplete signal. The leaderboard tells you what accuracy is achievable. It does not tell you what accuracy costs at each tier. The Best-of-4 result in this paper, 0.0066 better factuality at four times the compute, is a concrete illustration of what that gap looks like in practice.
The protocol itself is implementable now. It requires a factuality detector, a corpus of representative passages from your target task, and a decision about β that reflects your actual deployment cost tolerance. The choice of agent architecture appears to matter less than the scoring structure: Table 7 shows a standard ReAct agent produces nearly identical outcomes to the custom Q-Agent, which the authors interpret as evidence that resource-efficient behavior emerges from the incentive structure, not from architectural specifics.
The longer-term implication points in a different direction. As frontier model factuality scores converge toward the ceiling, the question of which system to deploy will increasingly be decided on cost per unit of quality rather than quality alone. MAS-HQ provides the first reproducible method to make that trade-off visible. The organizations that build cost-aware evaluation into their model selection process now will not be surprised when the next leaderboard release reshuffles rankings they thought were settled.
The leaderboard told you which system was most accurate. It never told you what that accuracy was worth.
Agents Applied is a weekly newsletter for executives and senior technologists navigating the practical realities of AI deployment. If someone forwarded this to you, you can subscribe at agentsapplied.com.