SGLang team and AMD partner to make large-scale DeepSeek-R1 disaggregated inference on AMD Instinct™ MI355X GPUs competitive on total cost of ownership
Original title:Blog Win on TCO: How AMD Instinct™ MI355X Achieves Cost-Competitive Distributed Inference Through SGLang with MoRI The SGLang and AMD team has worked closely to unlock competitive Total Cost of Ownership (TCO) for large-scale DeepSeek-R1 disaggregated inference on AMD Instinct™ MI355X GPUs. Building on SGLang's se... AMD & SGLang Team
AI summary
The SGLang and AMD teams collaborated on a series of full-stack optimizations that enable AMD Instinct™ MI355X GPUs to achieve highly competitive total cost of ownership when running DeepSeek-R1 large model inference. At an interactive latency of 129 tok/s/user, cost is $0.169 per million tokens, 5% lower than the NVIDIA B200 (Dynamo TRT-LLM) solution and 40% lower than the B200 (SGLang) solution. On throughput, 24 AMD GPUs reach 2,436 tok/s/GPU, 1.25x higher per-GPU throughput than the B200 SGLang solution using 48 GPUs. Core optimizations include: MoRI mixed FP4/FP8 quantized all-to-all communication, the MoRI-IO KV Cache backend, two-batch overlap with SDMA, Specv2 MTP on ROCm, and CPU streaming optimizations.
Key Takeaways
- The SGLang and AMD teams collaborated on a series of full-stack optimizations that enable AMD…
- At an interactive latency of 129 tok/s/user,cost is $0.
- 169 per million tokens,5% lower than the NVIDIA B200 (Dynamo TRT-LLM) solution and 40% lower than…
- On throughput,24 AMD GPUs reach 2,436 tok/s/GPU,1.
- 25x higher per-GPU throughput than the B200 SGLang solution using 48 GPUs.
Quick Facts
- Published
- Updated
- Source
- LMSYS Blog
- Category
- Papers & Research
- AI score
- 69
AMD MI355X跑DeepSeek-R1的TCO比NVIDIA B200低5%,吞吐还高出1.25倍,这是开源框架SGLang对闭源生态的一次真实挑战,做推理部署的应该点开看看完整的全栈优化。
Related entities
Source & copyright
This article is based on curated titles and AI summaries from the AIHOT public API, with attribution and links back. Original copyright remains with the original publishers and authors; this site does not mirror full texts.
How to Cite
When citing this article, keep the title, date, and permalink, and credit Orange News as the source.
Orange News: SGLang team and AMD partner to make large-scale DeepSeek-R1 disaggregated inference on AMD Instinct™ MI355X GPUs competitive on total cost of ownership. May 28, 2026. https://new.nat6.net/en/news/cmppprn04028nslvyqxgosjyn/