SWE-Bench Pro Public

SWE-Bench Pro is a software engineering benchmark created to address the limitations of existing benchmarks, such as data contamination, lack of task diversity, overly simple problems, and non-reproducible tests. Of its 1,865 tasks across 41 repositories, a public set of 731 is used for the public leaderboard, and to reduce contamination, tasks were taken from repositories under strong copyleft licenses such as GPL. Professional engineers build Docker-based reproducible environments, and humans augment the problem descriptions and requirements. The ranking metric is the Resolve Rate, the percentage of tasks that satisfy both conditions: passing the new tests and keeping existing tests passing.

• 731 tasks in the public set (1,865 in total, 41 repositories)
• Limits data contamination by using copyleft (GPL) repositories
• Docker-based reproducible environments
• Human-augmented problem descriptions and requirements
• Resolve Rate metric (resolve rate without regressions)
• Separate leaderboard for the startup private code set

Free (public benchmark/leaderboard)

Mostly free leaderboards/datasets released for academic and community purposes

Estimate based on AI knowledge - please verify on the official site

• One of 27 AI tools in the Benchmarks & Leaderboards field.
• Connected to 5 other tools — #11 in its field.