Study on robustness and chain-of-thought consistency of RL-fine-tuned VLMs
Original title:On Robustness and Chain-of-Thought Consistency of RL-Finetuned VLMs
AI summary
Reinforcement learning (RL) fine-tuning has been extended to vision-language models (VLMs). The study finds that simple textual perturbations — misleading captions or incorrect chains of thought (CoT) — significantly reduce model robustness and confidence, with open-source models degrading more noticeably. Closed-source models show similar failure modes but greater robustness and reasoning consistency. Further analysis reveals a trade-off between accuracy and faithfulness: fine-tuning improves benchmark accuracy but erodes CoT reliability and robustness to context changes; adversarial enhancement improves robustness but cannot prevent faithfulness drift. Introducing a faithfulness-aware reward restores alignment between answers and reasoning, but combined with enhancement, training tends to collapse into shortcut policies. These findings highlight the need to jointly consider correctness, robustness, and faithfulness in visual reasoning.
Key Takeaways
- Reinforcement learning (RL) fine-tuning has been extended to vision-language models (VLMs).
- The study finds that simple textual perturbations — misleading captions or incorrect chains of…
- Closed-source models show similar failure modes but greater robustness and reasoning consistency.
- Further analysis reveals a trade-off between accuracy and faithfulness: fine-tuning improves…
- adversarial enhancement improves robustness but cannot prevent faithfulness drift.
Quick Facts
- Published
- Updated
- Category
- Papers & Research
- AI score
- 62
RL微调让VLM基准分变好看,却可能让它的推理链变得靠不住,这个反直觉的诊断对正在用RL打磨多模态模型的团队是个警醒。
Source & copyright
This article is based on curated titles and AI summaries from the AIHOT public API, with attribution and links back. Original copyright remains with the original publishers and authors; this site does not mirror full texts.
How to Cite
When citing this article, keep the title, date, and permalink, and credit Orange News as the source.
Orange News: Study on robustness and chain-of-thought consistency of RL-fine-tuned VLMs. July 2, 2026. https://new.nat6.net/en/news/cmr3rd0l901bwsl7ld54ejrgb/