Papers & Research

MiniMax releases MSA sparse attention method, open-sources inference kernel, and launches MiniMax-M3 model

Original title:MiniMax Sparse Attention (MSA): a Two-Branch Block-Sparse Attention Trained on a 109B-Parameter MoE With a 3T-Token Budget

AI summary

MiniMax released MSA (MiniMax Sparse Attention), a sparse attention method built on Grouped Query Attention. It decomposes attention into an index branch and a main branch: the index branch selects 16 token blocks per GQA group at block granularity (128 tokens by default), with a fixed budget of 2,048 key-value tokens, and the main branch performs exact softmax attention only on these blocks. MSA was trained on a 109B-parameter MoE model; MiniMax open-sourced the inference kernel `fmha_sm100` for NVIDIA SM100 GPUs (MIT license, supporting BF16/FP8/NVFP4/FP4) and released the production model MiniMax-M3. MSA-PT scores 67.2, 77.7, 64.0, 84.2, and 77.5 on MMLU, GSM8K, HumanEval, RULER-8K, and RULER-32K respectively, on par with the full-attention baseline. At 128K context, its exp-free Top-k selection is 5.1x faster than `torch.topk`.

Key Takeaways

  • MiniMax released MSA (MiniMax Sparse Attention),a sparse attention method built on Grouped Query…
  • It decomposes attention into an index branch and a main branch: the index branch selects 16 token…
  • MSA was trained on a 109B-parameter MoE model
  • MiniMax open-sourced the inference kernel `fmha_sm100` for NVIDIA SM100 GPUs (MIT…
  • MSA-PT scores 67.

Quick Facts

Published
Updated
Source
MarkTechPost(RSS)
Category
Papers & Research
AI score
70
Read the original Read the full article at the original source
Why it matters

MiniMax 把长上下文注意力从 O(N) 压到固定每查询 2048 token,还同时开源高效内核与生产模型,对做长上下文 agent 的团队是即时可用的方法,遗憾是只限 SM100 GPU。

Related entities

Source & copyright

This article is based on curated titles and AI summaries from the AIHOT public API, with attribution and links back. Original copyright remains with the original publishers and authors; this site does not mirror full texts.

Original article · AIHOT entry

How to Cite

When citing this article, keep the title, date, and permalink, and credit Orange News as the source.

Orange News: MiniMax releases MSA sparse attention method, open-sources inference kernel, and launches MiniMax-M3 model. June 17, 2026. https://new.nat6.net/en/news/cmqhsdiv401zcslf08vuq50zc/

View all →