Other

Ling 2.5 Lightning Attention+MLA Hybrid Linear Architecture Retrofit Practice

AI summary

To improve computational efficiency under ultra-long context, the Ling 2.5 architecture changes Ling 2.0's GQA to a 1:7 hybrid linear attention of Lightning Attention and MLA. This aims to use Lightning Attention to improve long-sequence throughput and MLA to extremely compress KV Cache. To ensure no performance loss after the retrofit, the team designed a fine-grained smooth migration training strategy, including multi-stage continued training such as weight conversion, QK Norm fusion, and Partial RoPE modification. Scaling Law experiments determined 1:7 as the optimal hybrid ratio, ultimately enabling the trillion-parameter model to achieve lower compute cost and higher long-text reasoning efficiency.

Key Takeaways

  • To improve computational efficiency under ultra-long context,the Ling 2.
  • 5 architecture changes Ling 2.
  • 0's GQA to a 1:7 hybrid linear attention of Lightning Attention and MLA.
  • This aims to use Lightning Attention to improve long-sequence throughput and MLA to extremely…
  • To ensure no performance loss after the retrofit,the team designed a fine-grained smooth migration…

Quick Facts

Published
Updated
Source
蚂蚁百灵:Developer Blog(网页)
Category
Other
AI score
86
Read the original Read the full article at the original source
Why it matters

为大模型长上下文优化提供可落地的工程实践参考。

Source & copyright

This article is based on curated titles and AI summaries from the AIHOT public API, with attribution and links back. Original copyright remains with the original publishers and authors; this site does not mirror full texts.

Original article · AIHOT entry

How to Cite

When citing this article, keep the title, date, and permalink, and credit Orange News as the source.

Orange News: Ling 2.5 Lightning Attention+MLA Hybrid Linear Architecture Retrofit Practice. February 26, 2026. https://new.nat6.net/en/news/cmoecv1p6003usl1973hu5t5o/

View all →