LMSYS and Intel boost vision-language model serving with heterogeneous CPU+GPU EPD architecture
Original title:Blog Heterogeneous CPU + GPU EPD Disaggregation to Boost VLM Serving TL;DR We enabled heterogeneous Encode-Prefill-Decode (EPD) disaggregation via Dynamo and SGLang for Vision-Language Models (VLMs). By offloading vision encoding tasks to CPUs (the easiest-getting CPU... Intel & SGLang Team
AI summary
The LMSYS team (Intel and SGLang) enabled a heterogeneous encode-prefill-disaggregated (EPD) architecture for vision-language models (VLMs) through the Dynamo and SGLang frameworks. The approach offloads vision encoding tasks from GPUs to CPUs (such as Intel Xeon 6747P), working in tandem with GPUs. In tests with the Qwen3-VL-8B-Instruct model, a configuration using 4 CPUs + 1 GPU as encoders and 4 GPUs as prefill-decode (capability ratio R=12) achieved roughly 1.2x to 1.3x improvements in P99 TTFT and request throughput under an ISL/OSL 128/256 workload with eight 1080p images, while reducing P99 TPOT by about 1.3x to 30x.
Key Takeaways
- The LMSYS team (Intel and SGLang) enabled a heterogeneous encode-prefill-disaggregated (EPD)…
- The approach offloads vision encoding tasks from GPUs to CPUs (such as Intel Xeon 6747P),working…
- In tests with the Qwen3-VL-8B-Instruct model,a configuration using 4 CPUs + 1 GPU as encoders and…
- 3x improvements in P99 TTFT and request throughput under an ISL/OSL 128/256 workload with eight…
- 3x to 30x.
Quick Facts
- Published
- Updated
- Source
- LMSYS Blog
- Category
- Tutorials & Tips
- AI score
- 61
做VLM服务部署的可以认真看一下,用CPU头节点做异构EPD分离,几乎零成本换来了TTFT和TPOT的显著提升,有完整脚本和benchmark,能直接上手试。
Source & copyright
This article is based on curated titles and AI summaries from the AIHOT public API, with attribution and links back. Original copyright remains with the original publishers and authors; this site does not mirror full texts.
How to Cite
When citing this article, keep the title, date, and permalink, and credit Orange News as the source.
Orange News: LMSYS and Intel boost vision-language model serving with heterogeneous CPU+GPU EPD architecture. May 29, 2026. https://new.nat6.net/en/news/cmpvk42bo0015slfpngd3kyca/