Efficient offline inference framework Flood: significantly leading throughput, supports multimodal and quantization
Original title:inclusionAI/flood
AI summary
Flood is an efficient large language model inference framework for offline applications. It uses pipeline parallelism to reduce communication overhead and improves continuity through segmented KV cache management. The framework supports continuous batching, chunked prefill, FP8/INT8 quantization, and multimodal model inference. Performance tests show that its throughput on various models and hardware can reach up to 2.4 times that of vLLM. Its dedicated kernel SegmentAttention improves decoding speed by up to 3.16 times compared with FlashAttention when processing long sequences. The project was open-sourced in March 2025 and has iterated rapidly, now supporting new features such as lookahead decoding.
Key Takeaways
- Flood is an efficient large language model inference framework for offline applications.
- It uses pipeline parallelism to reduce communication overhead and improves continuity through…
- The framework supports continuous batching,chunked prefill,FP8/INT8 quantization,and multimodal…
- Performance tests show that its throughput on various models and hardware can reach up to 2.
- 4 times that of vLLM.
Quick Facts
- Published
- Updated
- Source
- 蚂蚁 inclusionAI:GitHub 新仓库
- Category
- Products & Apps
- AI score
- 61
蚂蚁的 FLOOD 框架用流水线并行替代张量并行来压通信开销,实测吞吐比 vLLM 高 1.4 到 2.4 倍,做离线推理部署的团队值得花半小时跑一下 benchmark 看看自家场景能不能吃这个红利。
Source & copyright
This article is based on curated titles and AI summaries from the AIHOT public API, with attribution and links back. Original copyright remains with the original publishers and authors; this site does not mirror full texts.
How to Cite
When citing this article, keep the title, date, and permalink, and credit Orange News as the source.
Orange News: Efficient offline inference framework Flood: significantly leading throughput, supports multimodal and quantization. January 30, 2026. https://new.nat6.net/en/news/cmorb7ik3006sslhfoasz9t4c/