Papers & Research

Quantifying infrastructure interference in agentic coding evaluations

Original title:Quantifying infrastructure noise in agentic coding evals

AI summary

Research found that in agentic coding benchmarks such as SWE-bench, differences in infrastructure configuration can affect model scores more than the small score gaps between top models on leaderboards. Internal experiments showed that on Terminal-Bench 2.0, success rates differed by 6% between the strictest and most lenient resource settings. Strict resource limits caused nearly 6% of tasks to fail due to unexpected container termination, while under lenient configurations such error rates could drop to 0.5%. When resource headroom exceeded benchmark specifications by 3x, agents could even use extra resources to successfully complete tasks that were otherwise unsolvable. This indicates that the evaluation environment not only affects test stability but also changes the capability dimensions the benchmark actually measures.

Key Takeaways

  • Research found that in agentic coding benchmarks such as SWE-bench,differences in infrastructure…
  • Internal experiments showed that on Terminal-Bench 2.
  • 0,success rates differed by 6% between the strictest and most lenient resource settings.
  • Strict resource limits caused nearly 6% of tasks to fail due to unexpected container…
  • When resource headroom exceeded benchmark specifications by 3x,agents could even use extra…

Quick Facts

Published
Updated
Source
Anthropic:Engineering(事故复盘 + 工程实践 · 网页)
Category
Papers & Research
AI score
74
Read the original Read the full article at the original source
Why it matters

Anthropic 用自家数据证明,agentic coding benchmark 的排行榜差距可能只是硬件配置差异而非模型能力差距,3 个百分点以内的领先都该打问号。做模型选型的人别再迷信那几个百分点了。

Source & copyright

This article is based on curated titles and AI summaries from the AIHOT public API, with attribution and links back. Original copyright remains with the original publishers and authors; this site does not mirror full texts.

Original article · AIHOT entry

How to Cite

When citing this article, keep the title, date, and permalink, and credit Orange News as the source.

Orange News: Quantifying infrastructure interference in agentic coding evaluations. February 5, 2026. https://new.nat6.net/en/news/cmorb7ml0008bslhfiqfwufv7/

View all →