Papers & Research

Dwarkesh Patel study finds pretraining gains mainly from data improvements

Original title:Pretraining progress is mostly coming from data

AI summary

Dwarkesh Patel released an experimental analysis training representative model recipes and data corpora from 2019 to 2025 under compute budgets of up to 1e19 FLOPs, finding data improvements delivered a 12.0x compute efficiency gain versus 3.7x from model improvements, with data contributing about 3.24 times as much as models.

Key Takeaways

  • Dwarkesh Patel released an experimental analysis training representative model recipes and data…
  • 0x compute efficiency gain versus 3.
  • 7x from model improvements,with data contributing about 3.
  • 24 times as much as models.

Quick Facts

Published
Updated
Source
Dwarkesh Patel:Podcast & Blog(RSS)
Category
Papers & Research
AI score
60
Read the original Read the full article at the original source
Why it matters

文章用小规模对照实验量化了 2019 到 2025 年预训练进步中数据与模型改进各自的算力效率贡献,并讨论结论对更大规模模型的适用边界。

Source & copyright

This article is based on curated titles and AI summaries from the AIHOT public API, with attribution and links back. Original copyright remains with the original publishers and authors; this site does not mirror full texts.

Original article · AIHOT entry

How to Cite

When citing this article, keep the title, date, and permalink, and credit Orange News as the source.

Orange News: Dwarkesh Patel study finds pretraining gains mainly from data improvements. September 9, 2026. https://new.nat6.net/en/news/cmtsx0wm4041jrob5liwjypsq/

View all →