Answer.AI releases ModernBERT: a next-generation encoder model to replace BERT
Original title:Finally, a Replacement for BERT: Introducing ModernBERT
AI summary
Answer.AI and LightOn jointly released ModernBERT, a series of SOTA encoder models intended to replace BERT. The models apply recent LLM architectural improvements such as Flash Attention and RoPE to encoders, support an 8192-token context length (far beyond BERT's 512), and outperform older-generation models in both speed and downstream task performance. Available in base (149M) and large (395M) parameter sizes, ModernBERT is integrated into Hugging Face Transformers v4.48.0 and can serve as a drop-in replacement for BERT-class models in RAG, classification and code retrieval scenarios.
Key Takeaways
- Answer.
- AI and LightOn jointly released ModernBERT,a series of SOTA encoder models intended to replace BERT.
- The models apply recent LLM architectural improvements such as Flash Attention and RoPE to…
- Available in base (149M) and large (395M) parameter sizes,ModernBERT is integrated into Hugging…
- 0 and can serve as a drop-in replacement for BERT-class models in RAG,classification and code…
Quick Facts
- Published
- Updated
- Source
- Answer.AI 官方研发博客(RSS)
- Category
- Model Releases
- AI score
- 82
这是六年来首个真正意义上对 BERT 架构的重大升级,解决了长上下文限制并提升了效率,对依赖编码器的 RAG 和搜索系统开发者具有直接的生产力价值。
Related entities
Source & copyright
This article is based on curated titles and AI summaries from the AIHOT public API, with attribution and links back. Original copyright remains with the original publishers and authors; this site does not mirror full texts.
How to Cite
When citing this article, keep the title, date, and permalink, and credit Orange News as the source.
Orange News: Answer.AI releases ModernBERT: a next-generation encoder model to replace BERT. December 19, 2024. https://new.nat6.net/en/news/f9k3aucnsfxp6dv9j3qfq6pd3/