On July 24, ARC Prize verified Claude Opus 5 at 30.16% on the ARC-AGI-3 public set. On August 21, NVIDIA reported the same model at 100.00 on the same set. The weights did not change. The code around them did.
In between, MIT did the same thing (August 5), a group led by Impossible Research got to 98.98 (July 15), and OpenAI tripled GPT-5.6 Sol's score by flipping two API settings (July 29). Then Microsoft published a framework that trains the model through the harness (August 18), and Google published one that gives the environment a harness of its own (August 20).
In July I wrote that self-editing harnesses have a provenance problem. This month the problem moved up a level: the benchmark score itself has no provenance.
TL;DR: On ARC-AGI-3's public set, the spread between "model in the official harness" and "model in the best harness" is 25 to 70 points, on a benchmark designed to resist exactly this. None of the 100s are verified on the private set, and every author says so. Microsoft's Agent Lightning v1.0 runs RL with the deploy-time harness owning the loop, so the harness is becoming part of the weights, and its reward-hacking section is the checklist my July post warned about. A benchmark number without a harness version, memory state and action budget attached is a self-reported claim with an unmarked type. Unmarked means self-reported.
ARC-AGI-3 scores agents with RHAE (Relative Human Action Efficiency). Per level, score = (humanbaselineactions / aiactions)^2, with the ratio capped at 1.15x the human baseline. Game scores are level-weighted averages, you must finish the last level to get full credit, and the overall number is the mean over games. A 100.00 means the agent finished every level at least as efficiently as a first-time human.
| Harness | Who | Date | Model | Public RHAE | Actions | Verified by ARC Prize | |---|---|---|---|---|---|---| | Official ARC Prize harness | ARC Prize | Jul 24 | Claude Opus 5 (high) | 30.16% | n/a | yes | | Official harness, default settings | OpenAI | Jul 29 | GPT-5.6 Sol (max) | 13.3% | n/a | no | | Official harness + retained reasoning + compaction | OpenAI | Jul 29 | GPT-5.6 Sol (max) | 38.3% | 6x fewer output tokens | no | | Schema | Impossible Research (+ UC Berkeley, CMU) | Jul 15 | Opus 4.8 / Fable 5 | 98.98 | n/a | no | | VISTA | MIT (Han, Hu, Qiu, Wu, He) | Aug 5 | Claude Opus 5 | 100.00 | 7,542 (humans: 17,135) | no | | AVO | NVIDIA | Aug 21 | Claude Opus 5 | 100.00 | 6,624 | no |
The number that matters is not in the table. It is the gap between the first row and the last: 70 points, same model, same 25 games, same metric.
The official harness is not a neutral baseline. OpenAI's write-up quotes ARC's intent: an "intentionally generic harness, without tools or special features" built to make "model shortcomings more visible." In practice it discarded all private reasoning after each game action and used a rolling truncation window, so older actions vanished as history grew. Retaining reasoning and enabling compaction took Sol from 13.3% to 38.3% and cut output tokens by 6x. The harness was wiping the model's mind between moves.
So the leaderboard measures "model plus a harness built to expose the model." The 100s measure "model plus a harness built to cover for the model." Neither measures the model, and nobody has isolated which part of the 70 points is which.
The authors are unusually honest about this. NVIDIA: the AVO-versus-VISTA comparison "should not be interpreted as a controlled ablation," and the results "should not be interpreted as a direct measurement of the performance contribution of AVO." VISTA: the models "were released after the public ARC-AGI-3 games," overlap cannot be excluded, and "the private set remains the real test of generalization." Schema: "no frozen-harness or held-out-performance claim." Every 100 on that table is a public-set number on games the models may have seen in training.
Read the harness papers side by side and the same three components appear under different names.
Memory. VISTA keeps a "lossless visual memory" of every past observation. AVO carries forward "prior implementations, evaluation results, compiler and profiler outputs, and accumulated reasoning." OpenAI's two settings are memory settings: keep the reasoning, compact instead of truncate.
Supervision. AVO runs a monitor that watches "the broader trajectory for stagnation or repeated unproductive cycles and can redirect the main agent." That is the layer that turns a model that gives up into an agent that does not.
An action budget. RHAE squares the efficiency ratio, so wasted moves are punished quadratically. AVO's headline against VISTA is 12% fewer actions. That is a harness optimization target, not a model property.
In July I split harness work into two piles: compensatory layers that patch what the model cannot do yet, and protective layers that constrain what it must not do. I predicted pile one depreciates with every model release. All three components above are pile one, and on a benchmark built to resist static tricks they are currently worth 25 to 70 points with the newest frontier models. Either my prediction is early or it is wrong about magnitude. I will take the second reading until the private-set numbers say otherwise.
One more thing about compaction, since it is the setting that tripled OpenAI's score. In my preregistered compaction experiment, the same operation produced 3.47% false proceeds on irreversible-action gates: the agent went through a gate it should have stopped at, because the compacted context no longer carried the provenance the gate depended on. Not a contradiction. ARC-AGI-3 scores task completion; my gates scored whether the agent still knew why it was allowed to act. Compaction improves the first, degrades the second, and a benchmark only sees the first.
Agent Lightning v1.0 (arXiv, August 18) names something the July thread never got to: RL where the harness is not a bystander. In their words, "the harness owns this loop, while the training engine observes only a sequence of LLM request-response pairs." The deploy-time scaffold (mini-SWE-agent in their coding runs) executes the task inside Kubernetes; the trainer sits behind a gateway that looks like a normal LLM endpoint and collects the traffic.
The result is real: Qwen3.5-9B goes from 41.8% to 56.4% on SWE-bench Verified, a 14.6-point gain from about 6,000 examples filtered out of SWE-smith's 59,136 tasks across 128 repositories, in roughly 3,500 lines of framework code.