Cognition releases Devin SWE-bench technical report
Original title:SWE-bench technical report
AI summary
Cognition released a technical report on Devin's performance on SWE-bench. Devin solved 79 of 570 randomly sampled test cases, a 13.86% success rate, above the previous strongest assisted baseline of Claude 2 at 4.80% and the unassisted setting at 1.96%.
Key Takeaways
- Cognition released a technical report on Devin's performance on SWE-bench.
- Devin solved 79 of 570 randomly sampled test cases,a 13.
- 86% success rate,above the previous strongest assisted baseline of Claude 2 at 4.
- 80% and the unassisted setting at 1.
Quick Facts
- Published
- Updated
- Source
- Cognition 模型 / Devin 博客(网页)
- Category
- Papers & Research
- AI score
- 68
Cognition 公开了评测方法和 Devin 的代码编辑结果,读者可以对照复现并理解智能体在真实仓库上解题的边界。
Related entities
Source & copyright
This article is based on curated titles and AI summaries from the AIHOT public API, with attribution and links back. Original copyright remains with the original publishers and authors; this site does not mirror full texts.
How to Cite
When citing this article, keep the title, date, and permalink, and credit Orange News as the source.
Orange News: Cognition releases Devin SWE-bench technical report. March 15, 2024. https://new.nat6.net/en/news/s3ow9hpx0t1eg58icse78hr4f/