Tutorials & Tips

Claude 3.5 Sonnet sets new record on SWE-bench Verified

Original title:Raising the bar on SWE-bench Verified with Claude 3.5 Sonnet

AI summary

The upgraded Claude 3.5 Sonnet achieved a 49% resolution rate on the software engineering benchmark SWE-bench Verified, surpassing the previous best model's 45%. The benchmark tests AI models' ability to complete software engineering tasks via real GitHub issues, requiring models to understand, modify, and test code in a given environment, with final validation by the original unit tests. The Claude team's agent design is simple, containing only a prompt, a Bash tool, and an edit tool, giving the model ample room for autonomous judgment to solve problems in flexible steps. No model has yet exceeded a 50% resolution rate on the benchmark.

Key Takeaways

  • The upgraded Claude 3.
  • 5 Sonnet achieved a 49% resolution rate on the software engineering benchmark SWE-bench…
  • The benchmark tests AI models' ability to complete software engineering tasks via real GitHub…
  • The Claude team's agent design is simple,containing only a prompt,a Bash tool,and an edit…
  • No model has yet exceeded a 50% resolution rate on the benchmark.

Quick Facts

Published
Updated
Source
Anthropic:Engineering(事故复盘 + 工程实践 · 网页)
Category
Tutorials & Tips
AI score
72
Read the original Read the full article at the original source
Why it matters

Anthropic 把自家 SWE-bench agent 的 prompt、工具设计和踩坑经验全公开了,做 coding agent 的人可以直接抄作业,比看十篇二手解读都管用。

Related entities

Source & copyright

This article is based on curated titles and AI summaries from the AIHOT public API, with attribution and links back. Original copyright remains with the original publishers and authors; this site does not mirror full texts.

Original article · AIHOT entry

How to Cite

When citing this article, keep the title, date, and permalink, and credit Orange News as the source.

Orange News: Claude 3.5 Sonnet sets new record on SWE-bench Verified. January 6, 2025. https://new.nat6.net/en/news/cmorb7ml1008rslhfl2itv1d3/

View all →