KPop: Using Symmetric Binary KL Divergence to Solve Training-Inference Mismatch in Large Model RL
Original title:解决大模型的"人格分裂":KPop 如何应对强化学习的训练推理不一致问题
AI summary
The Ant Ling team proposed KPop to address training collapse caused by inconsistent probability distributions between training and inference engines in large model reinforcement learning. KPop replaces IcePop's fixed ratio threshold with symmetric binary KL divergence, requiring only one hyperparameter, being adaptively tolerant of rare tokens and strictly constraining high-frequency tokens. On Ring-flash-2.0 (100B total parameters), it supported 800+ steps of stable RL training, improving SWE-bench Verified from 70.8% to 76.28%.
Key Takeaways
- The Ant Ling team proposed KPop to address training collapse caused by inconsistent probability…
- KPop replaces IcePop's fixed ratio threshold with symmetric binary KL divergence,requiring only…
- On Ring-flash-2.
- 0 (100B total parameters),it supported 800+ steps of stable RL training,improving SWE-bench…
Quick Facts
- Published
- Updated
- Source
- 公众号:蚂蚁百灵(Ling)
- Category
- Papers & Research
- AI score
- 68
解释固定屏蔽比为何误杀稀有token,并给出用对称二元KL散度自适应调整的方法,有助于理解RL训练不稳定并指导稳定训练实践。
Source & copyright
This article is based on curated titles and AI summaries from the AIHOT public API, with attribution and links back. Original copyright remains with the original publishers and authors; this site does not mirror full texts.
How to Cite
When citing this article, keep the title, date, and permalink, and credit Orange News as the source.
Orange News: KPop: Using Symmetric Binary KL Divergence to Solve Training-Inference Mismatch in Large Model RL. June 2, 2026. https://new.nat6.net/en/news/cmsdu2qga010irofz1h15420q/