Papers & Research

Optimizing Ling-2.6-1T on TPU with SGLang-JAX: One Pallas Kernel Hides MoE Data Movement in Compute

Original title:Blog Optimizing Ling-2.6-1T on TPU with SGLang-JAX: Hiding MoE Data Movement Behind Compute with One Pallas Kernel SGLang-JAX now supports efficient serving of inclusionAI's Ling-2.6-1T on TPU v7x. With a working baseline in place, profiling pointed to the Mixture-of-Experts (MoE) path as the main bottleneck: each... Prayer, JamesBrianD, Haolin Fu, Haoguang Cai, Qinghan Chen

AI summary

SGLang-JAX now supports efficient inference of inclusionAI's Ling-2.6-1T (a 1T sparse MoE with 63B activated parameters, 256 routed experts, top-8 routing plus shared experts) on TPU v7x. The team developed Fused MoE V2, a Pallas kernel that fuses scatter, expert FFN and gather, hiding MoE data movement in compute and reducing MoE prefill latency from 5.16ms to 2.42ms (a 53% reduction) and decode kernel latency from 0.249ms to 0.211ms (about 15%). Replacing only the MoE kernel improved prefill throughput by 24.8% and decode throughput by 18.5%–35.3%. In SGLang decode benchmarks, 16 TPU v7x chips reached 1.29x (mc=128) to 1.77x (mc=512) the output throughput of 16 H200 GPUs. The full launch also includes hybrid KV/recurrent memory pools, GLA linear attention and single-controller data parallelism support.

Key Takeaways

  • SGLang-JAX now supports efficient inference of inclusionAI's Ling-2.
  • 6-1T (a 1T sparse MoE with 63B activated parameters,256 routed experts,top-8 routing plus shared…
  • The team developed Fused MoE V2,a Pallas kernel that fuses scatter,expert FFN and gather,hiding…
  • 16ms to 2.
  • 42ms (a 53% reduction) and decode kernel latency from 0.

Quick Facts

Published
Updated
Source
LMSYS Blog
Category
Papers & Research
AI score
69
Read the original Read the full article at the original source
Why it matters

这是针对TPU上MoE推理的硬核优化复盘,用成本模型定位瓶颈,通过单内核融合把延迟砍半,对做大规模推理工程的团队是高质量参考。

Source & copyright

This article is based on curated titles and AI summaries from the AIHOT public API, with attribution and links back. Original copyright remains with the original publishers and authors; this site does not mirror full texts.

Original article · AIHOT entry

How to Cite

When citing this article, keep the title, date, and permalink, and credit Orange News as the source.

Orange News: Optimizing Ling-2.6-1T on TPU with SGLang-JAX: One Pallas Kernel Hides MoE Data Movement in Compute. June 17, 2026. https://new.nat6.net/en/news/cmqiddvsx07xcslf0wcefjhm6/

View all →