METR study: about half of AI PRs passing SWE-bench Verified would not be merged by maintainers
Original title:Many SWE-bench-Passing PRs Would Not Be Merged into Main
AI summary
METR had four maintainers of scikit-learn, Sphinx, and pytest blindly review 296 AI-generated PRs and found that about half of the patches passing SWE-bench Verified automated scoring would not be merged into the main branch; after normalization against a gold-standard baseline, maintainer merge rates averaged 24.2 percentage points lower than automated scores. The study stresses that AI patches did not receive iterative feedback as human developers do, so this should not be taken as a capability ceiling, but inferring an agent's practical usefulness directly from benchmark scores may overestimate it.
Key Takeaways
- METR had four maintainers of scikit-learn,Sphinx,and pytest blindly review 296 AI-generated PRs…
- after normalization against a gold-standard baseline,maintainer merge rates averaged 24.
- 2 percentage points lower than automated scores.
- The study stresses that AI patches did not receive iterative feedback as human developers do,so…
Quick Facts
- Published
- Updated
- Source
- METR:Notes(网页)
- Category
- Papers & Research
- AI score
- 62
研究用真实仓库维护者盲审 AI 补丁,量化了 SWE-bench 分数与实际可合并之间的差距,为解读编码基准提供了参照。
Source & copyright
This article is based on curated titles and AI summaries from the AIHOT public API, with attribution and links back. Original copyright remains with the original publishers and authors; this site does not mirror full texts.
How to Cite
When citing this article, keep the title, date, and permalink, and credit Orange News as the source.
Orange News: METR study: about half of AI PRs passing SWE-bench Verified would not be merged by maintainers. March 10, 2026. https://new.nat6.net/en/news/cmtym8yln0009rolseat7ohr1/