Same Model, 13.3% to 38.3%

OpenAI changed two API settings on ARC-AGI-3 and GPT-5.6 Sol went from 13.3% to 38.3% while spending six times fewer output tokens. Same model, same weights, same benchmark. The official harness had been throwing away the model's reasoning after every move. Which means the score was never a measurement of the model, and neither is most of what we compare.

Two API settings. Same model. Same benchmark. Same task set.

13.3% to 38.3%, using one sixth the output tokens.

OpenAI published that result about GPT-5.6 Sol on ARC-AGI-3, and it is the cleanest natural experiment the field has produced on a question I have been arguing from first principles for a year. Nothing about the model changed. Everything that changed was around it.

The score is Relative Human Action Efficiency, not a pass rate, and OpenAI estimates the average human tester at 48% on the same set. So the model went from 34.7 RHAE points behind a human to 9.7 behind, and the entire move came from configuration. By my arithmetic that is about 72% of a gap people had been attributing to the model.

There is a blunter version of the same fact. On the public leaderboard for one of these games, no frontier model gets past the first level. With the reconfigured harness, GPT-5.6 Sol solves all six.

Everything that moved is in the right-hand column, and none of it is the model:

What the official harness was doing

This is the part worth sitting with, because it is not exotic. It is the most natural way to write an agent loop.

The benchmark’s harness discarded the model’s private reasoning after every move, and dropped earlier actions as the context filled up.

The truncation had a specific number attached: once the conversation exceeded 175,000 characters, the oldest messages were dropped.

Both of those are the default behaviour of almost every agent framework I have read. Reasoning tokens are expensive, they are not the answer, and the API returns them as a separate thing you have to deliberately keep. Dropping them feels like hygiene. Truncating the oldest turns when context fills is the two-line version of context management and it is what you write first.

The consequence, in OpenAI’s framing, is that the model had to work out the game from scratch on every turn.

Think about what that does to a task where the whole point is learning rules from interaction. The model plays a move, forms a theory about the rules, and the theory is deleted. Next turn it sees the board and its own past actions, but not why it took them. So it re-derives. It burns output tokens rebuilding a theory it already had, arrives somewhere slightly different, and acts on that.

That is not a model that cannot learn rules. That is a model with anterograde amnesia, being scored on rule learning.

flowchart TB
  subgraph OFF["official harness: 13.3%, 6x the output tokens"]
    direction LR
    o1["see the frame"] --> o2["re-derive the rules
from scratch"] --> o3["act"] --> o4["discard the reasoning"] --> o1
  end
  subgraph NEW["retained reasoning plus compaction: 38.3%"]
    direction LR
    n1["see the frame"] --> n2["extend the theory
it already has"] --> n3["act"] --> n4["keep the reasoning"] --> n1
  end
  OFF ~~~ NEW

The two loops differ by one arrow. The expensive one is the loop that has to rebuild its own premise on every pass.

ARC had a reason for the generic harness, and it is a good one. A simple harness makes model shortcomings more visible, and it makes comparisons between models fairer. Commercial developers tune a harness to each model’s features and quirks, which is exactly the thing a benchmark is trying to factor out.

That reasoning is sound and it still produced a broken measurement. Neutrality is not the absence of assumptions. A harness that discards reasoning has taken a position on whether reasoning should persist, and it happens to be the position that penalises models trained to think across turns.

The two settings

Retain the reasoning across turns. Compact the history instead of truncating it.

Neither is clever. Neither required a new model, a new prompt, or a new scaffold. They are settings, and OpenAI’s point in publishing was that they are the same settings already running in ChatGPT and Codex. The production harness had them on. The benchmark harness did not.

The token result is the one I keep returning to. Six times fewer output tokens, while nearly tripling the score.

We have all internalised a trade here: more careful reasoning costs more tokens, and quality is something you buy. This inverts it. The expensive configuration was expensive because it was worse. Every discarded chain of reasoning had to be regenerated, and regeneration is output tokens at output prices. The waste and the failure were the same event.

OpenAI describes the mechanism plainly: with reasoning retained, the model spent less time thinking before each action, because it no longer had to interpret the game from scratch every turn. Less thinking, better play. That only sounds paradoxical if you were counting the re-derivation as thinking.

I made the general version of this argument in Your AI Bill Is a Distributed Systems Problem: cost in agent systems is dominated by repeated work, not by unit price. This is that thesis with a controlled experiment attached, run by the model vendor, on a public benchmark.

Why the score was never about the model

OpenAI said the quiet part in their own writeup: an evaluation does not measure the model in isolation. It also measures the API configuration, the harness design, and how the prompt was rendered.

Their exact words: benchmarks rarely measure AI models in isolation, they also measure less visible choices about API settings, harness design, and prompting.

Then the line I keep coming back to:

This isn’t the first time we’ve been surprised by low scores on a public benchmark and then discovered that the eval runner was using a generic harness that dropped reasoning messages.

Not the first time. So there are published numbers, still in circulation, still being cited, that measure a harness defect and are read as a fact about a model. Nobody is going back to correct those.

That is an unusual thing for a lab to publish, and it is correct.

Every agentic benchmark number you have read is a measurement of a triple. Model, harness, configuration. The number is real. The attribution is where it goes wrong, and the attribution is the entire reason anyone reads the number.

This is the exact failure I wrote about in The Log Printed Exactly What I Wanted. A true signal, correctly produced, borrowed for a claim it does not support. 13.3% was a true fact about a system. It was read as a fact about a model. Those are different objects, and the difference here is 25 points.

It is also the wrong ruler at scale. The benchmark ran. It produced a number. The number was reproducible. None of that made it a measurement of the thing everyone thought was being measured, and the fact that it ran cleanly is exactly why nobody checked.

The uncomfortable follow-on

If a generic harness cost one model 25 points, then every cross-model comparison run through a single fixed harness is suspect in a specific way.

A harness is not neutral. It makes assumptions: about whether reasoning persists, about how history is trimmed, about tool-call formatting, about how much of the transcript survives. Those assumptions suit some models and penalise others. Holding the harness constant across models feels like the rigorous choice. It is actually a choice to measure each model’s fit to one particular set of assumptions.

So when a leaderboard says model A beats model B on an agentic task, the defensible reading is that the pair (A, harness) beat the pair (B, harness). Sometimes that is what you want to know, because you are going to run that harness. Usually it is presented as something else.

I do not think this makes benchmarks useless. I think it makes the harness a required disclosure, the way a benchmark suite already discloses temperature and prompt.

What this says about the three harnesses

I have been reading Codex and Pi against Claude Code in this series, and this result reframes the thing they disagree about most.

Retained reasoning is a state ownership question. Between two turns, somebody has to hold the model’s intermediate work. If the harness drops it, the model regenerates it. If the harness keeps it, the harness now owns a piece of state it has to version, serialise, compact and restore.

That is not a small commitment, which is why the cheap harness drops it. It is also, on this evidence, 25 points.

Compaction versus truncation is the same shape. Truncation is free and lossy in the worst possible way, since it deletes the oldest context, which is usually where the task was defined. Compaction costs an extra model call and a summarisation strategy, and it keeps the definition.

Both settings are cases of the harness paying a real engineering cost to avoid making the model redo work. That is the whole thesis of The 90% Problem, and it is why the three harnesses all spend between 794 and 1,729 lines on a five-step loop. The lines are where the redoing gets prevented.

It also lands squarely on agent memory as a cache coherence problem. Discarded reasoning is a cache with a hit rate of zero. The model recomputes on every access, correctly, and pays for it every time. Nothing is broken. It is just cold, forever.

What to check in your own stack

Two questions, and both have answers you can find today.

Does your framework persist the model’s reasoning between turns, or drop it? Many drop it, few say so, and it is usually one field. If you are running a multi-turn task where the model builds understanding over time, this is the first thing to look at.

Does your context management compact or truncate? If the answer is that you slice off the oldest messages when you approach the limit, you are running the configuration that scored 13.3%.

Neither question is about your model. That is the point. The most expensive variable in your agent system is one nobody is benchmarking, and it is sitting in your config file.

Figures and quotations are from OpenAI’s own writeup, “How enabling two settings tripled our scores on the ARC-AGI-3 benchmark”: GPT-5.6 Sol on the public task set, 13.3% under the official harness against 38.3% with retained reasoning and compaction, output tokens cut 6x, rolling truncation at 175,000 characters, estimated human tester average 48%, scores measured as Relative Human Action Efficiency. The 72% figure is my own arithmetic on the gap to that human baseline and is not OpenAI’s claim.

I occasionally advise small teams on backend reliability, Go performance, and production AI systems. Learn more: /services