Introducing ARFBench: a time-series QA benchmark built on real incidents
Original title:Introducing ARFBench: A time series question-answering benchmark based on real incidents
AI summary
System failures cost more than a trillion dollars a year, and engineers must quickly pinpoint problems by analyzing time-series data. Time-series question answering (TSQA) is a key operations task and a challenge for AI models. The research team therefore introduced ARFBench, built on Datadog's real internal incidents and telemetry data. Tests show that leading large language models, vision-language models and time-series foundation models all have substantial room for improvement on ARFBench. The team proposed a hybrid TSFM-VLM model whose overall performance approaches the frontier, offering a new evaluation framework and improvement direction for TSQA.
Key Takeaways
- System failures cost more than a trillion dollars a year,and engineers must quickly pinpoint…
- Time-series question answering (TSQA) is a key operations task and a challenge for AI models.
- The research team therefore introduced ARFBench,built on Datadog's real internal incidents and…
- Tests show that leading large language models,vision-language models and time-series foundation…
- The team proposed a hybrid TSFM-VLM model whose overall performance approaches the…
Quick Facts
- Published
- Updated
- Source
- CMU:Machine Learning Blog
- Category
- Papers & Research
- AI score
- 58
CMU 和 Datadog 联手搞了个基于真实事故的时序问答基准,结论很诚实,现有模型全拉胯。做 SRE Agent 的团队该看看,这比合成数据的 benchmark 有说服力得多。
Source & copyright
This article is based on curated titles and AI summaries from the AIHOT public API, with attribution and links back. Original copyright remains with the original publishers and authors; this site does not mirror full texts.
How to Cite
When citing this article, keep the title, date, and permalink, and credit Orange News as the source.
Orange News: Introducing ARFBench: a time-series QA benchmark built on real incidents. April 28, 2026. https://new.nat6.net/en/news/cmoi4t62r00q0sle9i02ni14q/