Tutorials & Tips

TRL adds Delta Weight Sync: weight deltas sent via Hub Bucket, per-step transfer cut from 1.2 GB to 20–35 MB

Original title:Shipping a Trillion Parameters With a Hub Bucket: Delta Weight Sync in TRL

AI summary

In asynchronous reinforcement learning, the trainer must send full model weights (e.g., a 1T-parameter checkpoint of about 1 TB) to the inference engine at every step. A new TRL PR exploits the fact that about 99% of bf16 weight bits are identical between adjacent RL optimization steps, encoding only the changed weights as sparse safetensors files, uploading them to a Hugging Face Bucket and notifying vLLM to fetch them. On Qwen3-0.6B, per-step transfer dropped from 1.2 GB to 20–35 MB. The experiment also demonstrated a fully disaggregated training setup: the trainer, vLLM and the Wordle environment on different machines and in separate Hugging Face Spaces, with weights flowing through a single Hub bucket and no shared cluster, RDMA or VPN required.

Key Takeaways

  • In asynchronous reinforcement learning,the trainer must send full model weights (e.
  • ,a 1T-parameter checkpoint of about 1 TB) to the inference engine at every step.
  • A new TRL PR exploits the fact that about 99% of bf16 weight bits are identical between adjacent…
  • On Qwen3-0.
  • 6B,per-step transfer dropped from 1.

Quick Facts

Published
Updated
Source
Hugging Face 社区博客(混合发现)
Category
Tutorials & Tips
AI score
61
Read the original Read the full article at the original source
Why it matters

异步RL训练中权重同步的瓶颈被HuggingFace用稀疏增量方案解决了,带宽直接省了两个数量级,还给了可运行的TRL分支,做RL训练的可以直接上手试。

Related entities

Source & copyright

This article is based on curated titles and AI summaries from the AIHOT public API, with attribution and links back. Original copyright remains with the original publishers and authors; this site does not mirror full texts.

Original article · AIHOT entry

How to Cite

When citing this article, keep the title, date, and permalink, and credit Orange News as the source.

Orange News: TRL adds Delta Weight Sync: weight deltas sent via Hub Bucket, per-step transfer cut from 1.2 GB to 20–35 MB. May 27, 2026. https://new.nat6.net/en/news/cmpo5j5lm034mslv4qv9h13v8/

View all →