NSA: a new hardware-aligned sparse attention mechanism
Original title:🚀 Introducing NSA: A Hardware-Aligned and Natively Trainable Sparse Attention mechanism for ultra-f...
AI summary
NSA is a hardware-aligned and natively trainable sparse attention mechanism designed for ultra-fast long-context training and inference. Its core adopts a dynamic hierarchical sparse strategy, combining coarse-grained token compression with fine-grained token selection. Through optimizations for modern hardware, NSA accelerates inference and reduces pretraining costs without sacrificing performance, matching or surpassing Full Attention models on general benchmarks, long-context tasks, and instruction reasoning.
Key Takeaways
- NSA is a hardware-aligned and natively trainable sparse attention mechanism designed for…
- Its core adopts a dynamic hierarchical sparse strategy,combining coarse-grained token compression…
- Through optimizations for modern hardware,NSA accelerates inference and reduces pretraining costs…
Quick Facts
- Published
- Updated
- Source
- X:DeepSeek (@deepseek_ai)
- Category
- Other
DeepSeek 推出硬件对齐稀疏注意力 NSA,长上下文训练推理双提速,预训练成本显著降低
Source & copyright
This article is based on curated titles and AI summaries from the AIHOT public API, with attribution and links back. Original copyright remains with the original publishers and authors; this site does not mirror full texts.
How to Cite
When citing this article, keep the title, date, and permalink, and credit Orange News as the source.
Orange News: NSA: a new hardware-aligned sparse attention mechanism. February 18, 2025. https://new.nat6.net/en/news/cmo77611u003hslhea6kr5k19/