METR study update: algorithmic scoring overestimates AI agents' real software engineering capabilities
Original title:Research Update: Algorithmic vs. Holistic Evaluation
AI summary
METR evaluated Claude 3.7 Sonnet (Inspect ReAct agent) on 18 real issues in the stdlib-js and hypothesis repositories. Algorithmic scoring based on maintainer test cases showed a 38% (±19%) success rate, but none of the 15 manually reviewed PRs could be merged directly.
Key Takeaways
- METR evaluated Claude 3.
- 7 Sonnet (Inspect ReAct agent) on 18 real issues in the stdlib-js and hypothesis repositories.
- Algorithmic scoring based on maintainer test cases showed a 38% (±19%) success rate,but none of…
Quick Facts
- Published
- Updated
- Source
- METR:Research(网页)
- Category
- Papers & Research
- AI score
- 61
METR 用人工评审对照算法评分,给出量化证据说明测试类基准为何高估智能体的真实软件工程能力。
Related entities
Source & copyright
This article is based on curated titles and AI summaries from the AIHOT public API, with attribution and links back. Original copyright remains with the original publishers and authors; this site does not mirror full texts.
How to Cite
When citing this article, keep the title, date, and permalink, and credit Orange News as the source.
Orange News: METR study update: algorithmic scoring overestimates AI agents' real software engineering capabilities. August 13, 2025. https://new.nat6.net/en/news/cmtyo7nb703vbrog0wobc9ksx/