Dwarkesh Patel study finds pretraining gains mainly from data improvements
Original title:Pretraining progress is mostly coming from data
AI summary
Dwarkesh Patel released an experimental analysis training representative model recipes and data corpora from 2019 to 2025 under compute budgets of up to 1e19 FLOPs, finding data improvements delivered a 12.0x compute efficiency gain versus 3.7x from model improvements, with data contributing about 3.24 times as much as models.
Key Takeaways
- Dwarkesh Patel released an experimental analysis training representative model recipes and data…
- 0x compute efficiency gain versus 3.
- 7x from model improvements,with data contributing about 3.
- 24 times as much as models.
Quick Facts
- Published
- Updated
- Category
- Papers & Research
- AI score
- 60
文章用小规模对照实验量化了 2019 到 2025 年预训练进步中数据与模型改进各自的算力效率贡献,并讨论结论对更大规模模型的适用边界。
Source & copyright
This article is based on curated titles and AI summaries from the AIHOT public API, with attribution and links back. Original copyright remains with the original publishers and authors; this site does not mirror full texts.
How to Cite
When citing this article, keep the title, date, and permalink, and credit Orange News as the source.
Orange News: Dwarkesh Patel study finds pretraining gains mainly from data improvements. September 9, 2026. https://new.nat6.net/en/news/cmtsx0wm4041jrob5liwjypsq/