Differential Transformer V2
Original title:Differential Transformer V2
AI summary
Differential Transformer V2 has been released, centered on an improved differential attention mechanism. Compared with V1, V2 doubles the number of query heads while keeping key-value heads unchanged, and reduces dimensions after the differential operation, achieving decoding speed comparable to a standard Transformer without custom kernels. The new version also addresses numerical instability in V1 caused by uniform attention distributions, particularly eliminating the large scaling factors required by RMSNorm layers (such as under long sequences), thereby avoiding gradient explosion and aiming for more stable large-scale pretraining.
Key Takeaways
- Differential Transformer V2 has been released,centered on an improved differential attention…
- Compared with V1,V2 doubles the number of query heads while keeping key-value heads unchanged,and…
- The new version also addresses numerical instability in V1 caused by uniform attention…
Quick Facts
- Published
- Updated
- Source
- Hugging Face 社区博客(混合发现)
- Category
- Other
- AI score
- 83
新注意力架构提升解码效率与训练稳定性,开发者可低成本集成优化模型。
Source & copyright
This article is based on curated titles and AI summaries from the AIHOT public API, with attribution and links back. Original copyright remains with the original publishers and authors; this site does not mirror full texts.
How to Cite
When citing this article, keep the title, date, and permalink, and credit Orange News as the source.
Orange News: Differential Transformer V2. January 20, 2026. https://new.nat6.net/en/news/cmoegbhak009uslxxazhwpcva/