Tutorials & Tips

ARC Prize tests OpenAI o1 on ARC-AGI-Pub

Original title:OpenAI o1 Results on ARC-AGI-Pub

AI summary

ARC Prize used a unified testing framework to measure o1-preview at 21.2% on the public ARC-AGI evaluation, comparable to Claude 3.5 Sonnet's 21%, but averaging 4.2 minutes per task, about 10 times longer. The article argues that o1 realizes the CoT paradigm at training and inference time, shifting from memorizing answers to memorizing reasoning, but remains limited to the pretraining data distribution; although more test-time compute can improve accuracy logarithmically, achieving AGI still requires new ideas.

Key Takeaways

  • ARC Prize used a unified testing framework to measure o1-preview at 21.
  • 2% on the public ARC-AGI evaluation,comparable to Claude 3.
  • 5 Sonnet's 21%,but averaging 4.
  • 2 minutes per task,about 10 times longer.
  • The article argues that o1 realizes the CoT paradigm at training and inference time,shifting from…

Quick Facts

Published
Updated
Source
ARC Prize:官方博客
Category
Tutorials & Tips
AI score
71
Read the original Read the full article at the original source
Why it matters

ARC Prize 用同一测试框架实测 o1 并给出测试时算力与效率分析,读者可借此理解 o1 在 ARC-AGI 上成绩有限的原因。

Related entities

Source & copyright

This article is based on curated titles and AI summaries from the AIHOT public API, with attribution and links back. Original copyright remains with the original publishers and authors; this site does not mirror full texts.

Original article · AIHOT entry

How to Cite

When citing this article, keep the title, date, and permalink, and credit Orange News as the source.

Orange News: ARC Prize tests OpenAI o1 on ARC-AGI-Pub. September 13, 2024. https://new.nat6.net/en/news/cmtyo2p2t03glrog0f2a4owcg/

View all →