TRL adds Delta Weight Sync: weight deltas sent via Hub Bucket, per-step transfer cut from 1.2 GB to 20–35 MB
Original title:Shipping a Trillion Parameters With a Hub Bucket: Delta Weight Sync in TRL
AI summary
In asynchronous reinforcement learning, the trainer must send full model weights (e.g., a 1T-parameter checkpoint of about 1 TB) to the inference engine at every step. A new TRL PR exploits the fact that about 99% of bf16 weight bits are identical between adjacent RL optimization steps, encoding only the changed weights as sparse safetensors files, uploading them to a Hugging Face Bucket and notifying vLLM to fetch them. On Qwen3-0.6B, per-step transfer dropped from 1.2 GB to 20–35 MB. The experiment also demonstrated a fully disaggregated training setup: the trainer, vLLM and the Wordle environment on different machines and in separate Hugging Face Spaces, with weights flowing through a single Hub bucket and no shared cluster, RDMA or VPN required.
Key Takeaways
- In asynchronous reinforcement learning,the trainer must send full model weights (e.
- ,a 1T-parameter checkpoint of about 1 TB) to the inference engine at every step.
- A new TRL PR exploits the fact that about 99% of bf16 weight bits are identical between adjacent…
- On Qwen3-0.
- 6B,per-step transfer dropped from 1.
Quick Facts
- Published
- Updated
- Source
- Hugging Face 社区博客(混合发现)
- Category
- Tutorials & Tips
- AI score
- 61
异步RL训练中权重同步的瓶颈被HuggingFace用稀疏增量方案解决了,带宽直接省了两个数量级,还给了可运行的TRL分支,做RL训练的可以直接上手试。
Related entities
Source & copyright
This article is based on curated titles and AI summaries from the AIHOT public API, with attribution and links back. Original copyright remains with the original publishers and authors; this site does not mirror full texts.
How to Cite
When citing this article, keep the title, date, and permalink, and credit Orange News as the source.
Orange News: TRL adds Delta Weight Sync: weight deltas sent via Hub Bucket, per-step transfer cut from 1.2 GB to 20–35 MB. May 27, 2026. https://new.nat6.net/en/news/cmpo5j5lm034mslv4qv9h13v8/