Papers & Research

RVPO: Risk-sensitive alignment via variance regularization

Original title:RVPO: Risk-Sensitive Alignment via Variance Regularization

AI summary

Existing critic-free RLHF methods aggregate multi-objective rewards via arithmetic mean, which can easily lead to constraint neglect: a high score on one objective may mask severe failures on other key objectives (such as safety or format), thereby hiding low-performance bottleneck rewards that affect reliable alignment. This study proposes Reward Variance Policy Optimization (RVPO), a risk-sensitive framework that penalizes inter-reward variance in advantage aggregation, shifting the optimization objective from 'maximize sum' to 'maximize consistency.' Analysis shows that RVPO can effectively identify and improve the contribution of bottleneck rewards, achieving more balanced policy optimization in multi-objective alignment tasks such as safety and format adherence.

Key Takeaways

  • Existing critic-free RLHF methods aggregate multi-objective rewards via arithmetic mean,which can…
  • This study proposes Reward Variance Policy Optimization (RVPO),a risk-sensitive framework that…
  • ' Analysis shows that RVPO can effectively identify and improve the contribution of bottleneck…

Quick Facts

Published
Updated
Source
Apple Machine Learning Research(RSS)
Category
Papers & Research
AI score
68
Read the original Read the full article at the original source
Why it matters

当多数RLHF在‘求总分’,这篇Apple论文告诉你得分方差也致命,做安全对齐的人会看到新的损失函数怎么把一致性也纳入训练目标。

Source & copyright

This article is based on curated titles and AI summaries from the AIHOT public API, with attribution and links back. Original copyright remains with the original publishers and authors; this site does not mirror full texts.

Original article · AIHOT entry

How to Cite

When citing this article, keep the title, date, and permalink, and credit Orange News as the source.

Orange News: RVPO: Risk-sensitive alignment via variance regularization. May 8, 2026. https://new.nat6.net/en/news/cmox787mj007fslfoi53rqpks/

View all →