Papers & Research

KPop: Using Symmetric Binary KL Divergence to Solve Training-Inference Mismatch in Large Model RL

Original title:解决大模型的"人格分裂":KPop 如何应对强化学习的训练推理不一致问题

AI summary

The Ant Ling team proposed KPop to address training collapse caused by inconsistent probability distributions between training and inference engines in large model reinforcement learning. KPop replaces IcePop's fixed ratio threshold with symmetric binary KL divergence, requiring only one hyperparameter, being adaptively tolerant of rare tokens and strictly constraining high-frequency tokens. On Ring-flash-2.0 (100B total parameters), it supported 800+ steps of stable RL training, improving SWE-bench Verified from 70.8% to 76.28%.

Key Takeaways

  • The Ant Ling team proposed KPop to address training collapse caused by inconsistent probability…
  • KPop replaces IcePop's fixed ratio threshold with symmetric binary KL divergence,requiring only…
  • On Ring-flash-2.
  • 0 (100B total parameters),it supported 800+ steps of stable RL training,improving SWE-bench…

Quick Facts

Published
Updated
Source
公众号:蚂蚁百灵(Ling)
Category
Papers & Research
AI score
68
Read the original Read the full article at the original source
Why it matters

解释固定屏蔽比为何误杀稀有token,并给出用对称二元KL散度自适应调整的方法,有助于理解RL训练不稳定并指导稳定训练实践。

Source & copyright

This article is based on curated titles and AI summaries from the AIHOT public API, with attribution and links back. Original copyright remains with the original publishers and authors; this site does not mirror full texts.

Original article · AIHOT entry

How to Cite

When citing this article, keep the title, date, and permalink, and credit Orange News as the source.

Orange News: KPop: Using Symmetric Binary KL Divergence to Solve Training-Inference Mismatch in Large Model RL. June 2, 2026. https://new.nat6.net/en/news/cmsdu2qga010irofz1h15420q/

View all →