Why SWE-bench Verified no longer measures frontier coding capability
AI summary
OpenAI announced it will stop using the SWE-bench Verified benchmark to evaluate frontier coding capability. Built on historical GitHub issues, the benchmark's task distribution can no longer accurately reflect the types of real-world problems current AI coding assistants must solve. As model performance improves, the benchmark test set becomes saturated and less discriminative, with existing models approaching human-level performance. The team will therefore shift to new evaluation methods that are more challenging and reflect real-world complexity.
Key Takeaways
- OpenAI announced it will stop using the SWE-bench Verified benchmark to evaluate frontier coding…
- Built on historical GitHub issues,the benchmark's task distribution can no longer accurately…
- As model performance improves,the benchmark test set becomes saturated and less…
- The team will therefore shift to new evaluation methods that are more challenging and reflect…
Quick Facts
- Published
- Updated
- Source
- OpenAI:官网动态(RSS · 排除企业/客户案例)
- Category
- Opinion
- AI score
- 71
OpenAI 亲自给 SWE-bench Verified 判了死刑,这比任何第三方评测都有说服力。做 coding agent 选型的人该认真想想,你的 benchmark 体系是不是也该换了。
Related entities
Source & copyright
This article is based on curated titles and AI summaries from the AIHOT public API, with attribution and links back. Original copyright remains with the original publishers and authors; this site does not mirror full texts.
How to Cite
When citing this article, keep the title, date, and permalink, and credit Orange News as the source.
Orange News: Why SWE-bench Verified no longer measures frontier coding capability. April 27, 2026. https://new.nat6.net/en/news/cmog2im0800vbslr37vxebh5g/