Details
- OpenAI says an internal audit found SWE-Bench Pro no longer reliably measures frontier coding capability.
- The company says about 30% of SWE-Bench Pro tasks are broken and is retracting its earlier recommendation that the research community use it.
- The audit found that some correct solutions fail because of hidden requirements, contradictory instructions, overly strict tests, or incomplete grading criteria.
- OpenAI said it used model-based investigator agents plus reviews from five experienced software engineers to examine the benchmark at scale.
- The company argues that as coding models improve, benchmarks must become harder, fairer, and more trustworthy to track real progress.
- The announcement effectively questions a benchmark that had been positioned as a successor for evaluating advanced coding systems.
Impact
The move raises the bar for coding-model evaluation at a time when major labs are pushing newer frontier benchmarks and agents. By withdrawing support for SWE-Bench Pro, OpenAI is signaling that benchmark quality itself is now a competitive issue, not just model performance. It also reinforces a broader trend in AI evals: teams are moving away from easier or contaminated tests toward more controlled, human-verified measures that better reflect real engineering work.
