OpenAI audit of SWE-Bench Pro finds about 30% of evaluation tasks are flawed
Original title:Separating signal from noise in coding evaluations
AI summary
OpenAI conducted a detailed audit of the coding evaluation benchmark SWE-Bench Pro and found that about 30% of tasks are flawed. In the public subset of 731 tasks, the pass rate of frontier models rose from 23.3% to 80.3% in eight months, but data quality checks showed that many tasks had issues such as overly strict tests, insufficient prompt descriptions, incomplete test coverage, or misleading prompts. OpenAI recommends that model developers carefully scrutinize evaluation results and notes the growing usefulness of AI agents in large-scale data quality checks.
Key Takeaways
- OpenAI conducted a detailed audit of the coding evaluation benchmark SWE-Bench Pro and found that…
- In the public subset of 731 tasks,the pass rate of frontier models rose from 23.
- 3% in eight months,but data quality checks showed that many tasks had issues such as overly strict…
- OpenAI recommends that model developers carefully scrutinize evaluation results and notes the…
Quick Facts
- Published
- Updated
- Source
- OpenAI:官网动态(RSS · 排除企业/客户案例)
- Category
- Papers & Research
- AI score
- 70
OpenAI 自己审计了 SWE-Bench Pro,发现三成任务有缺陷,这个基准给出来的分数可能要打问号,做模型评测和选型的人该认真看看。
Related entities
Source & copyright
This article is based on curated titles and AI summaries from the AIHOT public API, with attribution and links back. Original copyright remains with the original publishers and authors; this site does not mirror full texts.
How to Cite
When citing this article, keep the title, date, and permalink, and credit Orange News as the source.
Orange News: OpenAI audit of SWE-Bench Pro finds about 30% of evaluation tasks are flawed. July 8, 2026. https://new.nat6.net/en/news/cmrcigf5k01bqihx54i8oph9u/