Papers & Research

METR study update: algorithmic scoring overestimates AI agents' real software engineering capabilities

Original title:Research Update: Algorithmic vs. Holistic Evaluation

AI summary

METR evaluated Claude 3.7 Sonnet (Inspect ReAct agent) on 18 real issues in the stdlib-js and hypothesis repositories. Algorithmic scoring based on maintainer test cases showed a 38% (±19%) success rate, but none of the 15 manually reviewed PRs could be merged directly.

Key Takeaways

  • METR evaluated Claude 3.
  • 7 Sonnet (Inspect ReAct agent) on 18 real issues in the stdlib-js and hypothesis repositories.
  • Algorithmic scoring based on maintainer test cases showed a 38% (±19%) success rate,but none of…

Quick Facts

Published
Updated
Source
METR:Research(网页)
Category
Papers & Research
AI score
61
Read the original Read the full article at the original source
Why it matters

METR 用人工评审对照算法评分,给出量化证据说明测试类基准为何高估智能体的真实软件工程能力。

Related entities

Source & copyright

This article is based on curated titles and AI summaries from the AIHOT public API, with attribution and links back. Original copyright remains with the original publishers and authors; this site does not mirror full texts.

Original article · AIHOT entry

How to Cite

When citing this article, keep the title, date, and permalink, and credit Orange News as the source.

Orange News: METR study update: algorithmic scoring overestimates AI agents' real software engineering capabilities. August 13, 2025. https://new.nat6.net/en/news/cmtyo7nb703vbrog0wobc9ksx/

View all →