OpenAI Says Two Settings Tripled Its ARC-AGI-3 Score — ARC Prize Says That Isn't the Same Test

OpenAI published a post today with an unusually specific claim: turning on two settings in its Responses API tripled GPT-5.6 Sol's score on ARC-AGI-3, taking it to 38.3% on the public task set while using roughly six times fewer output tokens per game. On one game it cleared all six levels — a game where, on the official leaderboard, no frontier model had passed level one.

That 38.3% would sit comfortably above the 30.2% that Claude Opus 5 posted to take the ARC-AGI-3 record less than a week ago. OpenAI has not framed it as a leaderboard claim, but plenty of coverage has.

The problem is that the two numbers were not produced the same way, and ARC Prize — which runs the benchmark — says that is exactly the point.

First, which number is which

Coverage of this has muddled two different baselines, so it is worth separating them.

Model	Reasoning effort	Official ARC-AGI-3 score
Claude Opus 5	High	30.16%
Fable 5	—	16.6%
GPT-5.6 Sol	Max	7.78%
Claude Opus 4.8	—	1.52%

Those are ARC Prize's own published figures for the public set. The 7.8% is GPT-5.6 Sol's official leaderboard result, and it was the previous state of the art before Opus 5 displaced it.

OpenAI's own post, however, describes a baseline of 13.3% from running the official harness itself, which is the figure the word "tripled" refers to — 13.3% to 38.3% is roughly 3×, whereas 7.8% to 38.3% would be closer to 5×. Several outlets have used 7.8% and 13.3% interchangeably as "the official score". They are not the same measurement, and the most consistent reading is that 7.8% is the leaderboard entry while 13.3% is OpenAI's in-house reproduction. We could not confirm that split directly against OpenAI's post, so treat the reconciliation as inference rather than something the company stated.

What the two settings actually do

Neither setting is exotic. Both are generally available API features.

Behaviour	Official harness	OpenAI's setup
Model's private reasoning between moves	Discarded after every action	Retained across steps
History that exceeds the window	Rolling truncation of the oldest content	Compaction — summarised instead of cut
Output tokens per game	Baseline	~6× fewer

The first row is the one that matters. ARC-AGI-3 is not a set of static puzzles — it is interactive, and the model plays a game over many turns. If its chain of thought is thrown away after each action, the model keeps a record of what it did but loses why it did it, and has to re-derive the rules of the game from scratch on every move. Compaction addresses the same problem from the other end: rather than silently dropping the oldest history once the context fills, it summarises it.

The detail that the official harness applied rolling truncation past roughly 175,000 characters comes from a single secondary report of OpenAI's post; we could not verify the exact threshold against a primary document.

ARC Prize: standardised means standardised

ARC Prize's position is procedural rather than a rebuttal of the result. Its official scores, it says, "use a standardized approach without provider-specific settings to ensure fair comparisons." ARC-AGI-3 is meant to measure the model, and letting each vendor supply its own configuration turns it into a measurement of model-plus-scaffolding.

Co-founder François Chollet drew a line that is more permissive than the headlines suggest. A custom-made harness built for the benchmark is out. General-purpose API settings that any developer can switch on are acceptable — provided they are clearly reported. He also conceded the obvious consequence: allowing different providers to run with different settings creates, in his words, a potential parity issue.

So OpenAI has not been accused of cheating. It has been told that its number is a legitimate result of a different configuration, and that a leaderboard whose entries were produced under different configurations is not a ranking.

Where OpenAI has a real point

It would be easy to write this off as a vendor gaming its own scorecard, and that would be too quick. Discarding a model's reasoning between moves in a multi-turn interactive benchmark is a defensible standardisation choice, but it is also a decision that penalises exactly the capability the benchmark claims to probe. A model asked to solve a novel environment, and then denied any memory of its own hypotheses each turn, is being tested on something narrower than fluid intelligence.

The token figure strengthens the argument rather than weakening it. If retained reasoning had simply thrown more compute at the problem, the score and the token count would have risen together. Scoring higher on roughly a sixth of the output tokens suggests the official configuration was burning tokens on rediscovery — the model repeatedly rebuilding context it had already worked out. That looks like a harness artefact, not a capability gain.

And the settings are not bespoke. If any developer using the Responses API gets this behaviour by default-adjacent configuration, then a "standardised" harness that turns it off is arguably measuring a setup nobody actually deploys.

Where the claim does not hold

None of that makes 38.3% comparable to 30.2%. Anthropic has its own analogues of both features — extended and interleaved thinking, and context management that summarises rather than truncates — and nobody has published an Opus 5 run on ARC-AGI-3 with them enabled. Until someone does, the honest statement is that GPT-5.6 Sol scored 38.3% under one configuration and Opus 5 scored 30.2% under another. "Beats Opus 5" is not established; it is assumed.

There is a second mismatch that predates this dispute and applies to the official table above. Sol's 7.8% was recorded at Max reasoning effort while Opus 5's 30.2% was recorded at High, so even ARC Prize's standardised numbers are not comparing identical settings. That is a live problem for every benchmark that publishes one figure per model, and it is the reason we treat single-number comparisons cautiously in our own head-to-head of Opus 5 and the GPT-5.6 family and in the four-way Claude comparison.

Worth noting too that Anthropic's ARC-AGI-3 result was independently run and verified by ARC Prize, whereas OpenAI's 38.3% is a self-reported figure from its own post. Both can be accurate; they carry different evidentiary weight.

What this means if you read benchmark scores

The practical lesson is narrower and more useful than "benchmarks are broken". It is that for agentic, multi-turn benchmarks, the harness is now a first-class variable — on the same order as the model. A 3× swing from two configuration flags is larger than the gap between most competing frontier models, which means a score quoted without its configuration is close to meaningless.

For anyone building on these models, that cuts the other way and is genuinely good news: if reasoning retention and compaction produce that kind of swing on a hard reasoning benchmark, they are worth checking in your own agent loops. The finding may be contested as a leaderboard claim while still being a real engineering result. Our model leaderboard uses a composite index precisely to avoid resting on any single benchmark, and stories like this are why. For background on the model at the centre of it, see our GPT-5.6 explainer.