Papers & Research

Never miss a token: understanding Token-In-Token-Out (TITO) in Miles

Original title:Blog No Token Left Behind: Demystifying Token-In-Token-Out in Miles In agentic RL, a rollout is not a single generation. It is a chain of model calls, tool outputs, harness messages, and resumed generations. Token-In-Token-Out (TITO) is a design principle that address... Miles Team: Jiajun Li, Yanbin Jiang, Mao Cheng, Shi Dong, Yusheng Su, Yueming Yuan, Zhichen Zeng, Banghua Zhu

AI summary

The Miles framework proposes the Token-In-Token-Out (TITO) principle to address the training-inference mismatch in agent reinforcement learning: ensuring the token sequence in the rollout process is bit-for-bit identical to the sequence evaluated by the trainer. TITO treats a multi-turn trajectory as one continuous sequence (one sample per task), saving an order of magnitude in compute overhead while maintaining on-policyness. Three failure scenarios: detokenize-retokenize mismatch, chat template trimming of reasoning content, and lossy template re-rendering. Miles implements this through an inference session server, three-level append-only guarantees, a pluggable TITO tokenizer and a sequence comparator. Typical tasks such as SWE-Bench contain 30-50 turns per trajectory.

Key Takeaways

  • The Miles framework proposes the Token-In-Token-Out (TITO) principle to address the…
  • TITO treats a multi-turn trajectory as one continuous sequence (one sample per task),saving an…
  • Three failure scenarios: detokenize-retokenize mismatch,chat template trimming of reasoning…
  • Miles implements this through an inference session server,three-level append-only guarantees,a…
  • Typical tasks such as SWE-Bench contain 30-50 turns per trajectory.

Quick Facts

Published
Updated
Source
LMSYS Blog
Category
Papers & Research
AI score
62
Read the original Read the full article at the original source
Why it matters

LMSYS团队把agent RL里最隐秘的训练-推理不一致问题解释透了,TITO原则直接告诉你为什么之前训练不稳,做agent训练的都该看看这篇。

Source & copyright

This article is based on curated titles and AI summaries from the AIHOT public API, with attribution and links back. Original copyright remains with the original publishers and authors; this site does not mirror full texts.

Original article · AIHOT entry

How to Cite

When citing this article, keep the title, date, and permalink, and credit Orange News as the source.

Orange News: Never miss a token: understanding Token-In-Token-Out (TITO) in Miles. June 5, 2026. https://new.nat6.net/en/news/cmq6w1tl0001jslbh8gv57k90/

View all →