Never miss a token: understanding Token-In-Token-Out (TITO) in Miles
Original title:Blog No Token Left Behind: Demystifying Token-In-Token-Out in Miles In agentic RL, a rollout is not a single generation. It is a chain of model calls, tool outputs, harness messages, and resumed generations. Token-In-Token-Out (TITO) is a design principle that address... Miles Team: Jiajun Li, Yanbin Jiang, Mao Cheng, Shi Dong, Yusheng Su, Yueming Yuan, Zhichen Zeng, Banghua Zhu
AI summary
The Miles framework proposes the Token-In-Token-Out (TITO) principle to address the training-inference mismatch in agent reinforcement learning: ensuring the token sequence in the rollout process is bit-for-bit identical to the sequence evaluated by the trainer. TITO treats a multi-turn trajectory as one continuous sequence (one sample per task), saving an order of magnitude in compute overhead while maintaining on-policyness. Three failure scenarios: detokenize-retokenize mismatch, chat template trimming of reasoning content, and lossy template re-rendering. Miles implements this through an inference session server, three-level append-only guarantees, a pluggable TITO tokenizer and a sequence comparator. Typical tasks such as SWE-Bench contain 30-50 turns per trajectory.
Key Takeaways
- The Miles framework proposes the Token-In-Token-Out (TITO) principle to address the…
- TITO treats a multi-turn trajectory as one continuous sequence (one sample per task),saving an…
- Three failure scenarios: detokenize-retokenize mismatch,chat template trimming of reasoning…
- Miles implements this through an inference session server,three-level append-only guarantees,a…
- Typical tasks such as SWE-Bench contain 30-50 turns per trajectory.
Quick Facts
- Published
- Updated
- Source
- LMSYS Blog
- Category
- Papers & Research
- AI score
- 62
LMSYS团队把agent RL里最隐秘的训练-推理不一致问题解释透了,TITO原则直接告诉你为什么之前训练不稳,做agent训练的都该看看这篇。
Source & copyright
This article is based on curated titles and AI summaries from the AIHOT public API, with attribution and links back. Original copyright remains with the original publishers and authors; this site does not mirror full texts.
How to Cite
When citing this article, keep the title, date, and permalink, and credit Orange News as the source.
Orange News: Never miss a token: understanding Token-In-Token-Out (TITO) in Miles. June 5, 2026. https://new.nat6.net/en/news/cmq6w1tl0001jslbh8gv57k90/