Tutorials & Tips

Cohere details North Mini Code megakernel inference engine, 1.25–1.41x faster than vLLM on a single H100

Original title:Inside the megakernel serving engine for North Mini Code

AI summary

Cohere released an inference engine built around a decode megakernel for North Mini Code, achieving 1.25–1.41x faster end-to-end decode throughput than vLLM on a single H100 in BF16, reaching 292 tok/s at batch size 1 (62% of SoL). The code is available on GitHub.

Key Takeaways

  • Cohere released an inference engine built around a decode megakernel for North Mini Code,achieving…
  • 41x faster end-to-end decode throughput than vLLM on a single H100 in BF16,reaching 292 tok/s at…
  • The code is available on GitHub.

Quick Facts

Published
Updated
Source
Cohere Labs:官方研究博客
Category
Tutorials & Tips
AI score
68
Read the original Read the full article at the original source
Why it matters

原文给出手写 decode megakernel 的完整实现细节与真实吞吐数据,读者可以了解如何把现有内核组装成单个持久内核并接入真实服务。

Source & copyright

This article is based on curated titles and AI summaries from the AIHOT public API, with attribution and links back. Original copyright remains with the original publishers and authors; this site does not mirror full texts.

Original article · AIHOT entry

How to Cite

When citing this article, keep the title, date, and permalink, and credit Orange News as the source.

Orange News: Cohere details North Mini Code megakernel inference engine, 1.25–1.41x faster than vLLM on a single H100. September 8, 2026. https://new.nat6.net/en/news/gpw1qeypdu4lyt3u0ijp5m2s5/

View all →