MiniMax releases MSA sparse attention method, open-sources inference kernel, and launches MiniMax-M3 model
Original title:MiniMax Sparse Attention (MSA): a Two-Branch Block-Sparse Attention Trained on a 109B-Parameter MoE With a 3T-Token Budget
AI summary
MiniMax released MSA (MiniMax Sparse Attention), a sparse attention method built on Grouped Query Attention. It decomposes attention into an index branch and a main branch: the index branch selects 16 token blocks per GQA group at block granularity (128 tokens by default), with a fixed budget of 2,048 key-value tokens, and the main branch performs exact softmax attention only on these blocks. MSA was trained on a 109B-parameter MoE model; MiniMax open-sourced the inference kernel `fmha_sm100` for NVIDIA SM100 GPUs (MIT license, supporting BF16/FP8/NVFP4/FP4) and released the production model MiniMax-M3. MSA-PT scores 67.2, 77.7, 64.0, 84.2, and 77.5 on MMLU, GSM8K, HumanEval, RULER-8K, and RULER-32K respectively, on par with the full-attention baseline. At 128K context, its exp-free Top-k selection is 5.1x faster than `torch.topk`.
Key Takeaways
- MiniMax released MSA (MiniMax Sparse Attention),a sparse attention method built on Grouped Query…
- It decomposes attention into an index branch and a main branch: the index branch selects 16 token…
- MSA was trained on a 109B-parameter MoE model
- MiniMax open-sourced the inference kernel `fmha_sm100` for NVIDIA SM100 GPUs (MIT…
- MSA-PT scores 67.
Quick Facts
- Published
- Updated
- Source
- MarkTechPost(RSS)
- Category
- Papers & Research
- AI score
- 70
MiniMax 把长上下文注意力从 O(N) 压到固定每查询 2048 token,还同时开源高效内核与生产模型,对做长上下文 agent 的团队是即时可用的方法,遗憾是只限 SM100 GPU。
Related entities
Source & copyright
This article is based on curated titles and AI summaries from the AIHOT public API, with attribution and links back. Original copyright remains with the original publishers and authors; this site does not mirror full texts.
How to Cite
When citing this article, keep the title, date, and permalink, and credit Orange News as the source.
Orange News: MiniMax releases MSA sparse attention method, open-sources inference kernel, and launches MiniMax-M3 model. June 17, 2026. https://new.nat6.net/en/news/cmqhsdiv401zcslf08vuq50zc/