Arena study: LLM judges favor their own answers about 70% more often than humans
Original title:LLM judges prefer their own responses 70% more often than humans do
AI summary
Arena used 12 models to produce 34,580 verdicts on 1,460 real Text Arena battles and found that models favor their own answers about 70% more often than humans on average; GPT-6 Astra chose itself in 88% of battles. Three OpenAI judges were 37 points more lenient toward OpenAI models than humans; AI judges agreed with each other 79.4% of the time but with human voters only 56.9%. Models rarely declared ties, and Sol forced a winner in 96% of battles.
Key Takeaways
- Arena used 12 models to produce 34,580 verdicts on 1,460 real Text Arena battles and found that…
- GPT-6 Astra chose itself in 88% of battles.
- Three OpenAI judges were 37 points more lenient toward OpenAI models than humans
- AI judges agreed with each other 79.
- 4% of the time but with human voters only 56.
Quick Facts
- Published
- Updated
- Source
- X:Arena (@arena)
- Category
- Papers & Research
- AI score
- 67
基于 1,460 场真实对战和 34,580 条裁决的实测数据,读者可以据此校准对 LLM-as-a-judge 评测结果的信任程度。
Related entities
Source & copyright
This article is based on curated titles and AI summaries from the AIHOT public API, with attribution and links back. Original copyright remains with the original publishers and authors; this site does not mirror full texts.
How to Cite
When citing this article, keep the title, date, and permalink, and credit Orange News as the source.
Orange News: Arena study: LLM judges favor their own answers about 70% more often than humans. September 30, 2026. https://new.nat6.net/en/news/u6afu3icor0mjvuoie6d2fd2w/