Papers & Research

Nine judges, two effective votes: correlation errors weaken LLM review panels

Original title:Nine Judges, Two Effective Votes: Correlated Errors Undermine LLM Evaluation Panels

AI summary

Apple's machine learning research team found that LLM-as-a-judge panels are severely limited by high correlation among models. Testing nine frontier large language models from seven model families on three natural language inference datasets showed that nine judges actually provide only about two independent votes' worth of information; panel accuracy is 8–22 percentage points below the ideal of independent voting, and the best single model matches or exceeds the entire panel. Adding judges or improving aggregation algorithms yields little benefit; even allowing the algorithm access to the correct answer closes at most 11% of the gap. The conclusion holds across multiple prompt variants, temperature settings, and preference tasks; the bottleneck is correlation among judges, not the aggregation algorithm.

Key Takeaways

  • Apple's machine learning research team found that LLM-as-a-judge panels are severely limited by…
  • Testing nine frontier large language models from seven model families on three natural language…
  • panel accuracy is 8–22 percentage points below the ideal of independent voting,and the best single…
  • Adding judges or improving aggregation algorithms yields little benefit
  • even allowing the algorithm access to the correct answer closes at most 11% of the gap.

Quick Facts

Published
Updated
Source
Apple Machine Learning Research(RSS)
Category
Papers & Research
AI score
68
Read the original Read the full article at the original source
Why it matters

这篇Apple论文揭示了一个反直觉的事实:在LLM评估面板中,9个法官实际上只提供约2个独立票的信息,因为模型会犯相似错误。这解释了为何简单聚合面板往往不如最佳单模型,做评估的团队必须重视法官相关性。

Related entities

Source & copyright

This article is based on curated titles and AI summaries from the AIHOT public API, with attribution and links back. Original copyright remains with the original publishers and authors; this site does not mirror full texts.

Original article · AIHOT entry

How to Cite

When citing this article, keep the title, date, and permalink, and credit Orange News as the source.

Orange News: Nine judges, two effective votes: correlation errors weaken LLM review panels. June 23, 2026. https://new.nat6.net/en/news/cmqr02j580f8oslp5i6wajucu/

View all →