ARC Prize tests OpenAI o1 on ARC-AGI-Pub
Original title:OpenAI o1 Results on ARC-AGI-Pub
AI summary
ARC Prize used a unified testing framework to measure o1-preview at 21.2% on the public ARC-AGI evaluation, comparable to Claude 3.5 Sonnet's 21%, but averaging 4.2 minutes per task, about 10 times longer. The article argues that o1 realizes the CoT paradigm at training and inference time, shifting from memorizing answers to memorizing reasoning, but remains limited to the pretraining data distribution; although more test-time compute can improve accuracy logarithmically, achieving AGI still requires new ideas.
Key Takeaways
- ARC Prize used a unified testing framework to measure o1-preview at 21.
- 2% on the public ARC-AGI evaluation,comparable to Claude 3.
- 5 Sonnet's 21%,but averaging 4.
- 2 minutes per task,about 10 times longer.
- The article argues that o1 realizes the CoT paradigm at training and inference time,shifting from…
Quick Facts
- Published
- Updated
- Source
- ARC Prize:官方博客
- Category
- Tutorials & Tips
- AI score
- 71
ARC Prize 用同一测试框架实测 o1 并给出测试时算力与效率分析,读者可借此理解 o1 在 ARC-AGI 上成绩有限的原因。
Related entities
Source & copyright
This article is based on curated titles and AI summaries from the AIHOT public API, with attribution and links back. Original copyright remains with the original publishers and authors; this site does not mirror full texts.
How to Cite
When citing this article, keep the title, date, and permalink, and credit Orange News as the source.
Orange News: ARC Prize tests OpenAI o1 on ARC-AGI-Pub. September 13, 2024. https://new.nat6.net/en/news/cmtyo2p2t03glrog0f2a4owcg/