Details
- OpenAI said GPT-5.6 Sol had already solved open mathematics problems, but it underperformed on ARC-AGI-3 because the benchmark harness prevented it from preserving what it learned between moves.
- The company said the standard setup discarded the model’s reasoning after each action and also dropped earlier actions as the context window filled, forcing repeated restarts.
- OpenAI reran the benchmark using the Responses API and enabled retained reasoning plus context compaction.
- On the public set, that change increased GPT-5.6 Sol’s score by 188% and cut output token usage by 6x.
- OpenAI’s takeaway is that benchmark results reflect both the model and the evaluation harness, not just raw model capability.
- The post frames the result as especially relevant for long-running agents, where preserving reasoning state can help the system build on prior steps instead of restarting each turn.
Impact
The result underscores how agent benchmarks can be shaped as much by API configuration as by model quality, especially for multi-step tasks that depend on memory across turns. It also gives OpenAI a practical signal that stateful tooling such as retained reasoning and compaction can materially improve long-horizon performance while reducing token costs. For developers, that likely raises the bar for evaluation discipline and makes the Responses API more central to agent workflows.
