Model Releases

EMO: A mixture-of-experts model pretrained for emergent modularity

Original title:EMO: Pretraining mixture of experts for emergent modularity

AI summary

EMO is a new mixture-of-experts model that, through end-to-end pretraining, makes modular structure emerge directly from data without relying on human-defined priors. The model allows using only a 12.5% expert subset on specific tasks (i.e., part of the 8 active experts) while maintaining performance close to the full model; when all 128 experts are used together, it remains a strong general model. EMO has 1B active parameters and 14B total parameters, trained on 1 trillion tokens. Compared with standard MoE, EMO uses document-level routing constraints to encourage experts to form domain-specialized groups, supporting selective use without severe performance degradation, enabling a composable architecture and optimizing the memory-accuracy tradeoff of large sparse MoE.

Key Takeaways

  • EMO is a new mixture-of-experts model that,through end-to-end pretraining,makes modular structure…
  • The model allows using only a 12.
  • 5% expert subset on specific tasks (i.
  • ,part of the 8 active experts) while maintaining performance close to the full model
  • when all 128 experts are used together,it remains a strong general model.

Quick Facts

Published
Updated
Source
Hugging Face 社区博客(混合发现)
Category
Model Releases
AI score
72
Read the original Read the full article at the original source
Why it matters

EMO 让 MoE 专家从按词法分散进化到按语义域自然模块化,仅用 12.5% 专家就能接近全模型性能,对需要按需加载的大模型部署是真正的突破。

Source & copyright

This article is based on curated titles and AI summaries from the AIHOT public API, with attribution and links back. Original copyright remains with the original publishers and authors; this site does not mirror full texts.

Original article · AIHOT entry

How to Cite

When citing this article, keep the title, date, and permalink, and credit Orange News as the source.

Orange News: EMO: A mixture-of-experts model pretrained for emergent modularity. May 9, 2026. https://new.nat6.net/en/news/cmox4xh0p006lslguuhqaou8t/

View all →