60x faster cold start: treating peer GPUs as weight servers
Original title:60x Faster Cold Starts: Treating Peer GPUs as Weight Servers
AI summary
The Runway platform team developed the NCCLBack system, which shortens model cold-start time from minutes to seconds through P2P weight transfer. Its core innovation is to let newly started GPU inference nodes obtain model parameters directly from peer GPUs in the cluster that have already loaded the weights, rather than repeatedly downloading from cloud storage. The system uses GPU interconnects such as InfiniBand and NVLink with bandwidth of up to 200-400 Gbps, an order-of-magnitude improvement over traditional storage downloads at 2-10 Gbps. Through Redis coordination and NCCL broadcast primitives, NCCLBack ensures efficiency and correctness of data transfer, so that when deploying new models in large-scale clusters, cold-start time does not grow linearly with the number of nodes and remains essentially constant.
Key Takeaways
- The Runway platform team developed the NCCLBack system,which shortens model cold-start time from…
- Its core innovation is to let newly started GPU inference nodes obtain model parameters directly…
- The system uses GPU interconnects such as InfiniBand and NVLink with bandwidth of up to 200-400…
- Through Redis coordination and NCCL broadcast primitives,NCCLBack ensures efficiency and…
Quick Facts
- Published
- Updated
- Source
- Runway:News(网页)
- Category
- Tutorials & Tips
- AI score
- 55
Runway 工程师把 GPU 冷启动从分钟压到秒级,原理是让已加载权重的 GPU 直接「喂」给新同伴,而不是各自从存储下载。做大规模推理部署的团队值得细读。
Related entities
Source & copyright
This article is based on curated titles and AI summaries from the AIHOT public API, with attribution and links back. Original copyright remains with the original publishers and authors; this site does not mirror full texts.
How to Cite
When citing this article, keep the title, date, and permalink, and credit Orange News as the source.
Orange News: 60x faster cold start: treating peer GPUs as weight servers. May 4, 2026. https://new.nat6.net/en/news/cmosrd1ac00v8slplayzv4fhm/