🔍 Read the full analysis: FrontierCode: What Happens When AI Submits A Pull Request? on ThorstenMeyerAI.com
Prime made for students and young adults
- Fast, free delivery for dorm and study essentials
- Prime Video and Amazon Music included
- Member-only deals
TL;DR
Artificial Analysis evaluated 23 models on FrontierCode, a benchmark that tests whether AI-generated code changes meet repository-specific checks and maintainers’ review criteria. Claude Opus 5.5 led with a 52.2% score and passed all blocking checks in 57.7% of attempts, leaving substantial room for human review.
Artificial Analysis evaluated 23 AI models on FrontierCode, a coding benchmark that asks whether a proposed code change would meet maintainers’ standards for acceptance. Claude Opus 5.5 led with a 52.2% score, but cleared every blocking check in 57.7% of attempts, suggesting that even the top result does not support relying on an AI agent to submit merge-ready changes without review.
Artificial Analysis added FrontierCode to its evaluations on October 12, 2026. It tested each model using the same open-source agent harness, Stirrup, with a limit of 250 turns per attempt and up to three attempts per task. The review stage used GPT-6 Luna (medium) as the judging model. The published results cover the benchmark’s 100-task Main set.
GPT-6.1 Sol (max) ranked second with a 49.1% score and a 54.7% pass rate. Claude Fable 5.1 (max with fallback) scored 49.0%, followed by GPT-6 Astra at 47.0% and Claude Sonnet 5.5 at 45.8%. Nine models from six labs scored between 43% and 52%; the results then fell off, with Grok 4.7 at 33.8% and Nemotron 3 Ultra at 10.5%.
The score and pass rate measure different things. A task attempt that fails any blocking check scores zero; attempts that clear all blocking checks receive a score from 70% to 100%, adjusted for quality checks. Artificial Analysis says nine of the 100 tasks were not solved by any model on any attempt. The available results do not establish how the models would perform across all software repositories or in day-to-day production work.
Infographic 01 · Would the maintainer merge it?
The best coding model clears the bar 58% of the time
FrontierCode asks whether an open-source maintainer would actually merge an agent’s change. A patch that fails any blocking check scores zero, even if the tests pass. 23 models, same harness, three attempts per task.
FrontierCode score, 23 models
Score in %, with the share of attempts that cleared every blocking check below it. Reasoning effort as tested.
Not on the board yet: Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode result as of 13 Oct 2026. Nemotron 3 Ultra has no published pass rate.
Source: Artificial Analysis, FrontierCode launch analysis (12 Oct 2026) and FrontierCode leaderboard, read 13 Oct 2026. FrontierCode 1.1 by Cognition, Main set: 100 private tasks, three attempts each, Stirrup harness, 250-turn cap.
Powered by Thorsten Meyer AI
Passing Tests Is Not Enough
FrontierCode targets a gap between code that runs and code a maintainer can reasonably accept. It examines correctness, regressions, the quality of an agent’s tests, the size and location of code changes, and an LLM-based review against maintainer criteria. That makes its results relevant to teams considering AI coding agents: a green test suite alone does not show that a patch is appropriately scoped or ready to merge.
The reported pass rates put a limit on claims that agents can handle pull requests without oversight. The leading model failed at least one blocking check in 42.3% of attempts, based on the 57.7% pass rate reported by Artificial Analysis. That does not mean every failed attempt was unusable, or that results transfer directly to a company’s own codebase. It does show that, under this benchmark’s rules, a human review step remains material.
Cost also varies sharply across the leading options. Artificial Analysis reports a cost of $10.15 per task for Opus and $0.81 for GPT-6.1 Sol, with Sol scoring 3.1 percentage points lower. These are benchmark cost estimates, not a full measure of deployment expense, and teams would need to weigh quality, model access, review time and their own task mix.
As an affiliate, we earn on qualifying purchases.
How FrontierCode Grades Changes
FrontierCode was created by Cognition with open-source maintainers and announced in June 2026. Version 1.1 followed in July with rules against searching online for task solutions. The full benchmark contains 150 tasks drawn from 36 repositories, including Celery, Budibase, uppy and Mattermost. More than 20 maintainers wrote tasks, spending over 40 hours on each, according to the source report. The tasks are private to reduce the chance that models have encountered them during training.
Its grading combines hidden tests, command checks for regressions and cleanliness, and tests designed to check whether an agent’s own tests catch the original bug. It also includes adaptive tests for complex behavior, limits on files touched and diff size, and an LLM code review based on maintainers’ criteria. A trajectory scanner can give an attempt a zero if it flags the agent for looking up a solution. Oversized diffs, edits outside relevant files and unhelpful tests can reduce quality scores.
That approach distinguishes FrontierCode from evaluations focused mainly on whether code executes or tests pass. Tomer Nosrati, a Celery maintainer, described the benchmark in Cognition’s announcement as grading “like a tech lead.” The phrase captures the project’s stated aim, though the results still reflect a designed benchmark and an LLM judge, not a final decision by every repository maintainer.
“Where others grade like a CI, FrontierCode grades like a tech lead.”
— Tomer Nosrati, Celery maintainer, in Cognition’s announcement
As an affiliate, we earn on qualifying purchases.
Limits of the Published Results
The scores do not show how often models produce acceptable changes across all real-world coding work. Artificial Analysis tested a defined 100-task set, under a particular agent harness, attempt limit and review process. How well those results generalize to different repositories, languages, prompts and human review practices remains unclear.
The benchmark’s review component uses GPT-6 Luna as a judge, so its scores reflect that evaluation setup as well as the tested models. The source report also says results for Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview were unavailable as of October 13. Their eventual scores could change the relative standings. The source material does not provide confidence intervals or enough detail to determine how much small differences between nearby scores should be treated as meaningful.
As an affiliate, we earn on qualifying purchases.
More Models and Replications
Further results may fill gaps in the current comparison, including scores for models that had no FrontierCode result as of October 13. Future benchmark runs could also show whether model rankings hold as systems and agent harnesses change. For teams, the practical next step is to test candidate models on representative tasks from their own repositories and compare not only completed work, but also review effort, failure types and cost.
Artificial Analysis’ published evaluation establishes a point-in-time comparison, not a forecast of when AI-generated pull requests can be merged without human checks. The benchmark’s results will become more useful as additional models are tested and independent teams assess how closely its tasks and grading match their own maintenance work.
As an affiliate, we earn on qualifying purchases.
Key Questions
What did FrontierCode measure?
FrontierCode measured whether AI-generated code changes met a set of repository-specific acceptance checks, including tests, change scope and a review against maintainer criteria. It was designed to assess more than whether code passed ordinary tests.
Which model ranked highest?
Claude Opus 5.5 (max with fallback) ranked first in Artificial Analysis’ results with a 52.2% FrontierCode score. It cleared every blocking check in 57.7% of attempts.
Does the top score mean AI can submit pull requests without review?
No. The results show the top model passed all blocking checks in 57.7% of benchmark attempts. The benchmark does not establish that AI pull requests are ready to merge without human review, either on these tasks or across production repositories.
How many tasks and models were included?
Artificial Analysis tested 23 models on the 100-task Main set. The full FrontierCode benchmark contains 150 tasks, according to the source report.
What remains uncertain about the results?
It is not yet clear how well the results generalize to different repositories and real-world coding workflows. Results for several models were also unavailable as of October 13, 2026, and the published material does not provide confidence intervals for closely ranked models.
Source: ThorstenMeyerAI.com
Evergreen bestsellers Picks
bestsellers
As an affiliate, we earn on qualifying purchases.
