Model Releases

Answer.AI releases ModernBERT: a next-generation encoder model to replace BERT

Original title:Finally, a Replacement for BERT: Introducing ModernBERT

AI summary

Answer.AI and LightOn jointly released ModernBERT, a series of SOTA encoder models intended to replace BERT. The models apply recent LLM architectural improvements such as Flash Attention and RoPE to encoders, support an 8192-token context length (far beyond BERT's 512), and outperform older-generation models in both speed and downstream task performance. Available in base (149M) and large (395M) parameter sizes, ModernBERT is integrated into Hugging Face Transformers v4.48.0 and can serve as a drop-in replacement for BERT-class models in RAG, classification and code retrieval scenarios.

Key Takeaways

  • Answer.
  • AI and LightOn jointly released ModernBERT,a series of SOTA encoder models intended to replace BERT.
  • The models apply recent LLM architectural improvements such as Flash Attention and RoPE to…
  • Available in base (149M) and large (395M) parameter sizes,ModernBERT is integrated into Hugging…
  • 0 and can serve as a drop-in replacement for BERT-class models in RAG,classification and code…

Quick Facts

Published
Updated
Source
Answer.AI 官方研发博客(RSS)
Category
Model Releases
AI score
82
Read the original Read the full article at the original source
Why it matters

这是六年来首个真正意义上对 BERT 架构的重大升级,解决了长上下文限制并提升了效率,对依赖编码器的 RAG 和搜索系统开发者具有直接的生产力价值。

Related entities

Source & copyright

This article is based on curated titles and AI summaries from the AIHOT public API, with attribution and links back. Original copyright remains with the original publishers and authors; this site does not mirror full texts.

Original article · AIHOT entry

How to Cite

When citing this article, keep the title, date, and permalink, and credit Orange News as the source.

Orange News: Answer.AI releases ModernBERT: a next-generation encoder model to replace BERT. December 19, 2024. https://new.nat6.net/en/news/f9k3aucnsfxp6dv9j3qfq6pd3/

View all →