OpenAI trains and deploys activation classifiers as safety monitoring layer for frontier models
Original title:Training activation classifiers for safety and alignment
AI summary
Since GPT-5.6, OpenAI has deployed activation classifiers (probes) on internal and external traffic for its most capable models, using internal model activations to score each token in real time. They can block policy-violating responses and tool calls mid-generation and refer them to a reasoning model for review. Experiments show probes are more robust than larger text classifiers against tested jailbreak attacks, with accuracy improving as model capability increases, and they transfer well across post-training checkpoints. However, they must be retrained for new base models and can only detect known violation types that can be explicitly trained for.
Key Takeaways
- Since GPT-5.
- 6,OpenAI has deployed activation classifiers (probes) on internal and external traffic for its…
- They can block policy-violating responses and tool calls mid-generation and refer them to a…
- Experiments show probes are more robust than larger text classifiers against tested jailbreak…
- However,they must be retrained for new base models and can only detect known violation types that…
Quick Facts
- Published
- Updated
- Source
- OpenAI:Alignment 研究博客(RSS)
- Category
- Papers & Research
- AI score
- 63
OpenAI 官方披露激活分类器的架构、训练与部署细节,并给出其相对文本分类器在越狱鲁棒性上的实验对比。
Related entities
Source & copyright
This article is based on curated titles and AI summaries from the AIHOT public API, with attribution and links back. Original copyright remains with the original publishers and authors; this site does not mirror full texts.
How to Cite
When citing this article, keep the title, date, and permalink, and credit Orange News as the source.
Orange News: OpenAI trains and deploys activation classifiers as safety monitoring layer for frontier models. November 3, 2026. https://new.nat6.net/en/news/wlmiqh13694tqabyvu6nzvlrs/