Papers & Research

Introducing ARFBench: a time-series QA benchmark built on real incidents

Original title:Introducing ARFBench: A time series question-answering benchmark based on real incidents

AI summary

System failures cost more than a trillion dollars a year, and engineers must quickly pinpoint problems by analyzing time-series data. Time-series question answering (TSQA) is a key operations task and a challenge for AI models. The research team therefore introduced ARFBench, built on Datadog's real internal incidents and telemetry data. Tests show that leading large language models, vision-language models and time-series foundation models all have substantial room for improvement on ARFBench. The team proposed a hybrid TSFM-VLM model whose overall performance approaches the frontier, offering a new evaluation framework and improvement direction for TSQA.

Key Takeaways

  • System failures cost more than a trillion dollars a year,and engineers must quickly pinpoint…
  • Time-series question answering (TSQA) is a key operations task and a challenge for AI models.
  • The research team therefore introduced ARFBench,built on Datadog's real internal incidents and…
  • Tests show that leading large language models,vision-language models and time-series foundation…
  • The team proposed a hybrid TSFM-VLM model whose overall performance approaches the…

Quick Facts

Published
Updated
Source
CMU:Machine Learning Blog
Category
Papers & Research
AI score
58
Read the original Read the full article at the original source
Why it matters

CMU 和 Datadog 联手搞了个基于真实事故的时序问答基准,结论很诚实,现有模型全拉胯。做 SRE Agent 的团队该看看,这比合成数据的 benchmark 有说服力得多。

Source & copyright

This article is based on curated titles and AI summaries from the AIHOT public API, with attribution and links back. Original copyright remains with the original publishers and authors; this site does not mirror full texts.

Original article · AIHOT entry

How to Cite

When citing this article, keep the title, date, and permalink, and credit Orange News as the source.

Orange News: Introducing ARFBench: a time-series QA benchmark built on real incidents. April 28, 2026. https://new.nat6.net/en/news/cmoi4t62r00q0sle9i02ni14q/

View all →