Papers & Research

OpenAI audit of SWE-Bench Pro finds about 30% of evaluation tasks are flawed

Original title:Separating signal from noise in coding evaluations

AI summary

OpenAI conducted a detailed audit of the coding evaluation benchmark SWE-Bench Pro and found that about 30% of tasks are flawed. In the public subset of 731 tasks, the pass rate of frontier models rose from 23.3% to 80.3% in eight months, but data quality checks showed that many tasks had issues such as overly strict tests, insufficient prompt descriptions, incomplete test coverage, or misleading prompts. OpenAI recommends that model developers carefully scrutinize evaluation results and notes the growing usefulness of AI agents in large-scale data quality checks.

Key Takeaways

  • OpenAI conducted a detailed audit of the coding evaluation benchmark SWE-Bench Pro and found that…
  • In the public subset of 731 tasks,the pass rate of frontier models rose from 23.
  • 3% in eight months,but data quality checks showed that many tasks had issues such as overly strict…
  • OpenAI recommends that model developers carefully scrutinize evaluation results and notes the…

Quick Facts

Published
Updated
Source
OpenAI:官网动态(RSS · 排除企业/客户案例)
Category
Papers & Research
AI score
70
Read the original Read the full article at the original source
Why it matters

OpenAI 自己审计了 SWE-Bench Pro,发现三成任务有缺陷,这个基准给出来的分数可能要打问号,做模型评测和选型的人该认真看看。

Related entities

Source & copyright

This article is based on curated titles and AI summaries from the AIHOT public API, with attribution and links back. Original copyright remains with the original publishers and authors; this site does not mirror full texts.

Original article · AIHOT entry

How to Cite

When citing this article, keep the title, date, and permalink, and credit Orange News as the source.

Orange News: OpenAI audit of SWE-Bench Pro finds about 30% of evaluation tasks are flawed. July 8, 2026. https://new.nat6.net/en/news/cmrcigf5k01bqihx54i8oph9u/

View all →