Papers & Research

ITBench-AA: Frontier models all score below 50% on first agentic enterprise IT task benchmark

Original title:ITBench-AA: Frontier Models Score Below 50% on the First Benchmark for Agentic Enterprise IT Tasks — by Artificial Analysis and IBM

AI summary

The ITBench-AA SRE benchmark from Artificial Analysis and IBM shows that no frontier model scored above 50%. Claude Opus 4.7 (adaptive reasoning, max effort) led at 47%, while GPT-5.5 (xhigh) and Qwen3.7 Max scored 46% and 42% respectively. The test comprises 59 agentic tasks that require investigating Kubernetes incident snapshots via shell commands and submitting a root-cause diagnosis. A key finding is that reasoning turns varied by nearly 3x, but longer trajectories did not translate into higher accuracy, and models that over-investigated were penalized for submitting false positives. On cost, open-source model Gemma 4 31B (Reasoning) scored 37% at $0.14 per task, outperforming closed-source models that cost more and scored lower.

Key Takeaways

  • The ITBench-AA SRE benchmark from Artificial Analysis and IBM shows that no frontier model scored…
  • Claude Opus 4.
  • 7 (adaptive reasoning,max effort) led at 47%,while GPT-5.
  • 5 (xhigh) and Qwen3.
  • 7 Max scored 46% and 42% respectively.

Quick Facts

Published
Updated
Source
Hugging Face 社区博客(混合发现)
Category
Papers & Research
AI score
70
Read the original Read the full article at the original source
Why it matters

IT运维这事儿,AI还是新手。ITBench-AA这份基准把Claude Opus 4.7逼到47%,开源模型GLM-5.1却用五分之一成本拿到40%,企业场景性价比可能不在闭源那边。

Related entities

Source & copyright

This article is based on curated titles and AI summaries from the AIHOT public API, with attribution and links back. Original copyright remains with the original publishers and authors; this site does not mirror full texts.

Original article · AIHOT entry

How to Cite

When citing this article, keep the title, date, and permalink, and credit Orange News as the source.

Orange News: ITBench-AA: Frontier models all score below 50% on first agentic enterprise IT task benchmark. May 28, 2026. https://new.nat6.net/en/news/cmpobyo1b04qeslv4jzefv5eu/

View all →