Tutorials & Tips

LMSYS and Intel boost vision-language model serving with heterogeneous CPU+GPU EPD architecture

Original title:Blog Heterogeneous CPU + GPU EPD Disaggregation to Boost VLM Serving TL;DR We enabled heterogeneous Encode-Prefill-Decode (EPD) disaggregation via Dynamo and SGLang for Vision-Language Models (VLMs). By offloading vision encoding tasks to CPUs (the easiest-getting CPU... Intel & SGLang Team

AI summary

The LMSYS team (Intel and SGLang) enabled a heterogeneous encode-prefill-disaggregated (EPD) architecture for vision-language models (VLMs) through the Dynamo and SGLang frameworks. The approach offloads vision encoding tasks from GPUs to CPUs (such as Intel Xeon 6747P), working in tandem with GPUs. In tests with the Qwen3-VL-8B-Instruct model, a configuration using 4 CPUs + 1 GPU as encoders and 4 GPUs as prefill-decode (capability ratio R=12) achieved roughly 1.2x to 1.3x improvements in P99 TTFT and request throughput under an ISL/OSL 128/256 workload with eight 1080p images, while reducing P99 TPOT by about 1.3x to 30x.

Key Takeaways

  • The LMSYS team (Intel and SGLang) enabled a heterogeneous encode-prefill-disaggregated (EPD)…
  • The approach offloads vision encoding tasks from GPUs to CPUs (such as Intel Xeon 6747P),working…
  • In tests with the Qwen3-VL-8B-Instruct model,a configuration using 4 CPUs + 1 GPU as encoders and…
  • 3x improvements in P99 TTFT and request throughput under an ISL/OSL 128/256 workload with eight…
  • 3x to 30x.

Quick Facts

Published
Updated
Source
LMSYS Blog
Category
Tutorials & Tips
AI score
61
Read the original Read the full article at the original source
Why it matters

做VLM服务部署的可以认真看一下,用CPU头节点做异构EPD分离,几乎零成本换来了TTFT和TPOT的显著提升,有完整脚本和benchmark,能直接上手试。

Source & copyright

This article is based on curated titles and AI summaries from the AIHOT public API, with attribution and links back. Original copyright remains with the original publishers and authors; this site does not mirror full texts.

Original article · AIHOT entry

How to Cite

When citing this article, keep the title, date, and permalink, and credit Orange News as the source.

Orange News: LMSYS and Intel boost vision-language model serving with heterogeneous CPU+GPU EPD architecture. May 29, 2026. https://new.nat6.net/en/news/cmpvk42bo0015slfpngd3kyca/

View all →