Papers & Research

SGLang team and AMD partner to make large-scale DeepSeek-R1 disaggregated inference on AMD Instinct™ MI355X GPUs competitive on total cost of ownership

Original title:Blog Win on TCO: How AMD Instinct™ MI355X Achieves Cost-Competitive Distributed Inference Through SGLang with MoRI The SGLang and AMD team has worked closely to unlock competitive Total Cost of Ownership (TCO) for large-scale DeepSeek-R1 disaggregated inference on AMD Instinct™ MI355X GPUs. Building on SGLang's se... AMD & SGLang Team

AI summary

The SGLang and AMD teams collaborated on a series of full-stack optimizations that enable AMD Instinct™ MI355X GPUs to achieve highly competitive total cost of ownership when running DeepSeek-R1 large model inference. At an interactive latency of 129 tok/s/user, cost is $0.169 per million tokens, 5% lower than the NVIDIA B200 (Dynamo TRT-LLM) solution and 40% lower than the B200 (SGLang) solution. On throughput, 24 AMD GPUs reach 2,436 tok/s/GPU, 1.25x higher per-GPU throughput than the B200 SGLang solution using 48 GPUs. Core optimizations include: MoRI mixed FP4/FP8 quantized all-to-all communication, the MoRI-IO KV Cache backend, two-batch overlap with SDMA, Specv2 MTP on ROCm, and CPU streaming optimizations.

Key Takeaways

  • The SGLang and AMD teams collaborated on a series of full-stack optimizations that enable AMD…
  • At an interactive latency of 129 tok/s/user,cost is $0.
  • 169 per million tokens,5% lower than the NVIDIA B200 (Dynamo TRT-LLM) solution and 40% lower than…
  • On throughput,24 AMD GPUs reach 2,436 tok/s/GPU,1.
  • 25x higher per-GPU throughput than the B200 SGLang solution using 48 GPUs.

Quick Facts

Published
Updated
Source
LMSYS Blog
Category
Papers & Research
AI score
69
Read the original Read the full article at the original source
Why it matters

AMD MI355X跑DeepSeek-R1的TCO比NVIDIA B200低5%,吞吐还高出1.25倍,这是开源框架SGLang对闭源生态的一次真实挑战,做推理部署的应该点开看看完整的全栈优化。

Related entities

Source & copyright

This article is based on curated titles and AI summaries from the AIHOT public API, with attribution and links back. Original copyright remains with the original publishers and authors; this site does not mirror full texts.

Original article · AIHOT entry

How to Cite

When citing this article, keep the title, date, and permalink, and credit Orange News as the source.

Orange News: SGLang team and AMD partner to make large-scale DeepSeek-R1 disaggregated inference on AMD Instinct™ MI355X GPUs competitive on total cost of ownership. May 28, 2026. https://new.nat6.net/en/news/cmppprn04028nslvyqxgosjyn/

View all →