Cohere details North Mini Code megakernel inference engine, 1.25–1.41x faster than vLLM on a single H100
Original title:Inside the megakernel serving engine for North Mini Code
AI summary
Cohere released an inference engine built around a decode megakernel for North Mini Code, achieving 1.25–1.41x faster end-to-end decode throughput than vLLM on a single H100 in BF16, reaching 292 tok/s at batch size 1 (62% of SoL). The code is available on GitHub.
Key Takeaways
- Cohere released an inference engine built around a decode megakernel for North Mini Code,achieving…
- 41x faster end-to-end decode throughput than vLLM on a single H100 in BF16,reaching 292 tok/s at…
- The code is available on GitHub.
Quick Facts
- Published
- Updated
- Source
- Cohere Labs:官方研究博客
- Category
- Tutorials & Tips
- AI score
- 68
原文给出手写 decode megakernel 的完整实现细节与真实吞吐数据,读者可以了解如何把现有内核组装成单个持久内核并接入真实服务。
Source & copyright
This article is based on curated titles and AI summaries from the AIHOT public API, with attribution and links back. Original copyright remains with the original publishers and authors; this site does not mirror full texts.
How to Cite
When citing this article, keep the title, date, and permalink, and credit Orange News as the source.
Orange News: Cohere details North Mini Code megakernel inference engine, 1.25–1.41x faster than vLLM on a single H100. September 8, 2026. https://new.nat6.net/en/news/gpw1qeypdu4lyt3u0ijp5m2s5/