Other

NSA: a new hardware-aligned sparse attention mechanism

Original title:🚀 Introducing NSA: A Hardware-Aligned and Natively Trainable Sparse Attention mechanism for ultra-f...

AI summary

NSA is a hardware-aligned and natively trainable sparse attention mechanism designed for ultra-fast long-context training and inference. Its core adopts a dynamic hierarchical sparse strategy, combining coarse-grained token compression with fine-grained token selection. Through optimizations for modern hardware, NSA accelerates inference and reduces pretraining costs without sacrificing performance, matching or surpassing Full Attention models on general benchmarks, long-context tasks, and instruction reasoning.

Key Takeaways

  • NSA is a hardware-aligned and natively trainable sparse attention mechanism designed for…
  • Its core adopts a dynamic hierarchical sparse strategy,combining coarse-grained token compression…
  • Through optimizations for modern hardware,NSA accelerates inference and reduces pretraining costs…

Quick Facts

Published
Updated
Source
X:DeepSeek (@deepseek_ai)
Category
Other
Read the original Read the full article at the original source
Why it matters

DeepSeek 推出硬件对齐稀疏注意力 NSA,长上下文训练推理双提速,预训练成本显著降低

Source & copyright

This article is based on curated titles and AI summaries from the AIHOT public API, with attribution and links back. Original copyright remains with the original publishers and authors; this site does not mirror full texts.

Original article · AIHOT entry

How to Cite

When citing this article, keep the title, date, and permalink, and credit Orange News as the source.

Orange News: NSA: a new hardware-aligned sparse attention mechanism. February 18, 2025. https://new.nat6.net/en/news/cmo77611u003hslhea6kr5k19/

View all →