Nine judges, two effective votes: correlation errors weaken LLM review panels
Original title:Nine Judges, Two Effective Votes: Correlated Errors Undermine LLM Evaluation Panels
AI summary
Apple's machine learning research team found that LLM-as-a-judge panels are severely limited by high correlation among models. Testing nine frontier large language models from seven model families on three natural language inference datasets showed that nine judges actually provide only about two independent votes' worth of information; panel accuracy is 8–22 percentage points below the ideal of independent voting, and the best single model matches or exceeds the entire panel. Adding judges or improving aggregation algorithms yields little benefit; even allowing the algorithm access to the correct answer closes at most 11% of the gap. The conclusion holds across multiple prompt variants, temperature settings, and preference tasks; the bottleneck is correlation among judges, not the aggregation algorithm.
Key Takeaways
- Apple's machine learning research team found that LLM-as-a-judge panels are severely limited by…
- Testing nine frontier large language models from seven model families on three natural language…
- panel accuracy is 8–22 percentage points below the ideal of independent voting,and the best single…
- Adding judges or improving aggregation algorithms yields little benefit
- even allowing the algorithm access to the correct answer closes at most 11% of the gap.
Quick Facts
- Published
- Updated
- Category
- Papers & Research
- AI score
- 68
这篇Apple论文揭示了一个反直觉的事实:在LLM评估面板中,9个法官实际上只提供约2个独立票的信息,因为模型会犯相似错误。这解释了为何简单聚合面板往往不如最佳单模型,做评估的团队必须重视法官相关性。
Related entities
Source & copyright
This article is based on curated titles and AI summaries from the AIHOT public API, with attribution and links back. Original copyright remains with the original publishers and authors; this site does not mirror full texts.
How to Cite
When citing this article, keep the title, date, and permalink, and credit Orange News as the source.
Orange News: Nine judges, two effective votes: correlation errors weaken LLM review panels. June 23, 2026. https://new.nat6.net/en/news/cmqr02j580f8oslp5i6wajucu/