SGLang runs DeepSeek-V4-Flash and Kimi-K3 on consumer hardware with SSD Expert Pack
Original title:Blog Running DeepSeek-V4-Flash and Kimi-K3 on Consumer Hardware with SSD Expert Pack SGLang brings the core idea of SSD-LLaMA to MoE inference: keep routed experts that do not fit in VRAM and host RAM on an NVMe SSD, load only the experts selected by the router, and use Expert Pack la... WiCi AI Team, SGLang Team
AI summary
SGLang brings the SSD-LLaMA approach to MoE inference, placing routed experts that do not fit in VRAM or memory on NVMe SSDs and loading only the experts selected by the router, using contiguous Expert Pack layout, O_DIRECT direct I/O and a budgeted GPU LFU/LRU cache.
Key Takeaways
- SGLang brings the SSD-LLaMA approach to MoE inference
- placing routed experts that do not fit in VRAM or memory on NVMe SSDs and loading only the experts…
- using contiguous Expert Pack layout,O_DIRECT direct I/O and a budgeted GPU LFU/LRU cache.
Quick Facts
- Published
- Updated
- Source
- LMSYS Blog
- Category
- Tutorials & Tips
- AI score
- 66
原文给出完整的硬件配置、性能数字和复现命令,读者可以据此评估消费级 SSD 路线运行超大 MoE 模型的可行性与代价。
Related entities
Source & copyright
This article is based on curated titles and AI summaries from the AIHOT public API, with attribution and links back. Original copyright remains with the original publishers and authors; this site does not mirror full texts.
How to Cite
When citing this article, keep the title, date, and permalink, and credit Orange News as the source.
Orange News: SGLang runs DeepSeek-V4-Flash and Kimi-K3 on consumer hardware with SSD Expert Pack. August 29, 2026. https://new.nat6.net/en/news/cmu79ssfu0z2irowkbdofbmeq/