Other

Differential Transformer V2

Original title:Differential Transformer V2

AI summary

Differential Transformer V2 has been released, centered on an improved differential attention mechanism. Compared with V1, V2 doubles the number of query heads while keeping key-value heads unchanged, and reduces dimensions after the differential operation, achieving decoding speed comparable to a standard Transformer without custom kernels. The new version also addresses numerical instability in V1 caused by uniform attention distributions, particularly eliminating the large scaling factors required by RMSNorm layers (such as under long sequences), thereby avoiding gradient explosion and aiming for more stable large-scale pretraining.

Key Takeaways

  • Differential Transformer V2 has been released,centered on an improved differential attention…
  • Compared with V1,V2 doubles the number of query heads while keeping key-value heads unchanged,and…
  • The new version also addresses numerical instability in V1 caused by uniform attention…

Quick Facts

Published
Updated
Source
Hugging Face 社区博客(混合发现)
Category
Other
AI score
83
Read the original Read the full article at the original source
Why it matters

新注意力架构提升解码效率与训练稳定性,开发者可低成本集成优化模型。

Source & copyright

This article is based on curated titles and AI summaries from the AIHOT public API, with attribution and links back. Original copyright remains with the original publishers and authors; this site does not mirror full texts.

Original article · AIHOT entry

How to Cite

When citing this article, keep the title, date, and permalink, and credit Orange News as the source.

Orange News: Differential Transformer V2. January 20, 2026. https://new.nat6.net/en/news/cmoegbhak009uslxxazhwpcva/

View all →