Ling 2.5 Lightning Attention+MLA Hybrid Linear Architecture Retrofit Practice
AI summary
To improve computational efficiency under ultra-long context, the Ling 2.5 architecture changes Ling 2.0's GQA to a 1:7 hybrid linear attention of Lightning Attention and MLA. This aims to use Lightning Attention to improve long-sequence throughput and MLA to extremely compress KV Cache. To ensure no performance loss after the retrofit, the team designed a fine-grained smooth migration training strategy, including multi-stage continued training such as weight conversion, QK Norm fusion, and Partial RoPE modification. Scaling Law experiments determined 1:7 as the optimal hybrid ratio, ultimately enabling the trillion-parameter model to achieve lower compute cost and higher long-text reasoning efficiency.
Key Takeaways
- To improve computational efficiency under ultra-long context,the Ling 2.
- 5 architecture changes Ling 2.
- 0's GQA to a 1:7 hybrid linear attention of Lightning Attention and MLA.
- This aims to use Lightning Attention to improve long-sequence throughput and MLA to extremely…
- To ensure no performance loss after the retrofit,the team designed a fine-grained smooth migration…
Quick Facts
- Published
- Updated
- Source
- 蚂蚁百灵:Developer Blog(网页)
- Category
- Other
- AI score
- 86
为大模型长上下文优化提供可落地的工程实践参考。
Source & copyright
This article is based on curated titles and AI summaries from the AIHOT public API, with attribution and links back. Original copyright remains with the original publishers and authors; this site does not mirror full texts.
How to Cite
When citing this article, keep the title, date, and permalink, and credit Orange News as the source.
Orange News: Ling 2.5 Lightning Attention+MLA Hybrid Linear Architecture Retrofit Practice. February 26, 2026. https://new.nat6.net/en/news/cmoecv1p6003usl1973hu5t5o/