EMO: A mixture-of-experts model pretrained for emergent modularity
Original title:EMO: Pretraining mixture of experts for emergent modularity
AI summary
EMO is a new mixture-of-experts model that, through end-to-end pretraining, makes modular structure emerge directly from data without relying on human-defined priors. The model allows using only a 12.5% expert subset on specific tasks (i.e., part of the 8 active experts) while maintaining performance close to the full model; when all 128 experts are used together, it remains a strong general model. EMO has 1B active parameters and 14B total parameters, trained on 1 trillion tokens. Compared with standard MoE, EMO uses document-level routing constraints to encourage experts to form domain-specialized groups, supporting selective use without severe performance degradation, enabling a composable architecture and optimizing the memory-accuracy tradeoff of large sparse MoE.
Key Takeaways
- EMO is a new mixture-of-experts model that,through end-to-end pretraining,makes modular structure…
- The model allows using only a 12.
- 5% expert subset on specific tasks (i.
- ,part of the 8 active experts) while maintaining performance close to the full model
- when all 128 experts are used together,it remains a strong general model.
Quick Facts
- Published
- Updated
- Source
- Hugging Face 社区博客(混合发现)
- Category
- Model Releases
- AI score
- 72
EMO 让 MoE 专家从按词法分散进化到按语义域自然模块化,仅用 12.5% 专家就能接近全模型性能,对需要按需加载的大模型部署是真正的突破。
Source & copyright
This article is based on curated titles and AI summaries from the AIHOT public API, with attribution and links back. Original copyright remains with the original publishers and authors; this site does not mirror full texts.
How to Cite
When citing this article, keep the title, date, and permalink, and credit Orange News as the source.
Orange News: EMO: A mixture-of-experts model pretrained for emergent modularity. May 9, 2026. https://new.nat6.net/en/news/cmox4xh0p006lslguuhqaou8t/