RVPO: Risk-sensitive alignment via variance regularization
Original title:RVPO: Risk-Sensitive Alignment via Variance Regularization
AI summary
Existing critic-free RLHF methods aggregate multi-objective rewards via arithmetic mean, which can easily lead to constraint neglect: a high score on one objective may mask severe failures on other key objectives (such as safety or format), thereby hiding low-performance bottleneck rewards that affect reliable alignment. This study proposes Reward Variance Policy Optimization (RVPO), a risk-sensitive framework that penalizes inter-reward variance in advantage aggregation, shifting the optimization objective from 'maximize sum' to 'maximize consistency.' Analysis shows that RVPO can effectively identify and improve the contribution of bottleneck rewards, achieving more balanced policy optimization in multi-objective alignment tasks such as safety and format adherence.
Key Takeaways
- Existing critic-free RLHF methods aggregate multi-objective rewards via arithmetic mean,which can…
- This study proposes Reward Variance Policy Optimization (RVPO),a risk-sensitive framework that…
- ' Analysis shows that RVPO can effectively identify and improve the contribution of bottleneck…
Quick Facts
- Published
- Updated
- Category
- Papers & Research
- AI score
- 68
当多数RLHF在‘求总分’,这篇Apple论文告诉你得分方差也致命,做安全对齐的人会看到新的损失函数怎么把一致性也纳入训练目标。
Source & copyright
This article is based on curated titles and AI summaries from the AIHOT public API, with attribution and links back. Original copyright remains with the original publishers and authors; this site does not mirror full texts.
How to Cite
When citing this article, keep the title, date, and permalink, and credit Orange News as the source.
Orange News: RVPO: Risk-sensitive alignment via variance regularization. May 8, 2026. https://new.nat6.net/en/news/cmox787mj007fslfoi53rqpks/