Task-seeded synthetic QA generation for Nemotron pretraining
Original title:Task-Seeded Synthetic Q&A Generation for Nemotron Pretraining
AI summary
In a 100B-token continued-training experiment on the Nemotron-3 Nano model, Task-Seeded SDG improved MMLU-Pro by 1.8 points, average code by 1.9 points, commonsense understanding by 1.6 points and GPQA by 11.1 points, while math scores remained stable. The pipeline uses the training sets of about 70 public tasks (roughly 700 subtasks) in lm-eval-harness as seeds, generates new examples augmented with reasoning and context, and after format validation, deduplication and answer verification produces a curated synthetic dataset used for Nemotron Ultra and Super training.
Key Takeaways
- In a 100B-token continued-training experiment on the Nemotron-3 Nano model,Task-Seeded SDG…
- 8 points,average code by 1.
- 9 points,commonsense understanding by 1.
- 6 points and GPQA by 11.
- 1 points,while math scores remained stable.
Quick Facts
- Published
- Updated
- Source
- Hugging Face 社区博客(混合发现)
- Category
- Papers & Research
- AI score
- 68
NVIDIA 公开了 Nemotron 训练中造合成数据的详细方法论,用 70 个公开任务训练集做种子生成新题目,100B token 实验把 GPQA 拉高 11 个点,做预训练的人值得看看他们怎么造的数据。
Source & copyright
This article is based on curated titles and AI summaries from the AIHOT public API, with attribution and links back. Original copyright remains with the original publishers and authors; this site does not mirror full texts.
How to Cite
When citing this article, keep the title, date, and permalink, and credit Orange News as the source.
Orange News: Task-seeded synthetic QA generation for Nemotron pretraining. June 4, 2026. https://new.nat6.net/en/news/cmpzfevp50305slkpy89xsy0v/