Optimizing Ling-2.6-1T on TPU with SGLang-JAX: One Pallas Kernel Hides MoE Data Movement in Compute
Original title:Blog Optimizing Ling-2.6-1T on TPU with SGLang-JAX: Hiding MoE Data Movement Behind Compute with One Pallas Kernel SGLang-JAX now supports efficient serving of inclusionAI's Ling-2.6-1T on TPU v7x. With a working baseline in place, profiling pointed to the Mixture-of-Experts (MoE) path as the main bottleneck: each... Prayer, JamesBrianD, Haolin Fu, Haoguang Cai, Qinghan Chen
AI summary
SGLang-JAX now supports efficient inference of inclusionAI's Ling-2.6-1T (a 1T sparse MoE with 63B activated parameters, 256 routed experts, top-8 routing plus shared experts) on TPU v7x. The team developed Fused MoE V2, a Pallas kernel that fuses scatter, expert FFN and gather, hiding MoE data movement in compute and reducing MoE prefill latency from 5.16ms to 2.42ms (a 53% reduction) and decode kernel latency from 0.249ms to 0.211ms (about 15%). Replacing only the MoE kernel improved prefill throughput by 24.8% and decode throughput by 18.5%–35.3%. In SGLang decode benchmarks, 16 TPU v7x chips reached 1.29x (mc=128) to 1.77x (mc=512) the output throughput of 16 H200 GPUs. The full launch also includes hybrid KV/recurrent memory pools, GLA linear attention and single-controller data parallelism support.
Key Takeaways
- SGLang-JAX now supports efficient inference of inclusionAI's Ling-2.
- 6-1T (a 1T sparse MoE with 63B activated parameters,256 routed experts,top-8 routing plus shared…
- The team developed Fused MoE V2,a Pallas kernel that fuses scatter,expert FFN and gather,hiding…
- 16ms to 2.
- 42ms (a 53% reduction) and decode kernel latency from 0.
Quick Facts
- Published
- Updated
- Source
- LMSYS Blog
- Category
- Papers & Research
- AI score
- 69
这是针对TPU上MoE推理的硬核优化复盘,用成本模型定位瓶颈,通过单内核融合把延迟砍半,对做大规模推理工程的团队是高质量参考。
Source & copyright
This article is based on curated titles and AI summaries from the AIHOT public API, with attribution and links back. Original copyright remains with the original publishers and authors; this site does not mirror full texts.
How to Cite
When citing this article, keep the title, date, and permalink, and credit Orange News as the source.
Orange News: Optimizing Ling-2.6-1T on TPU with SGLang-JAX: One Pallas Kernel Hides MoE Data Movement in Compute. June 17, 2026. https://new.nat6.net/en/news/cmqiddvsx07xcslf0wcefjhm6/