Third-party AI index moves to v5, and the open-closed gap widens from 3 points to 12

Artificial Analysis, a third-party AI evaluation firm, rebuilt its composite Intelligence Index as v5.0 on October 27. The main change is the addition of tasks whose contents are kept private. The aim is to avoid "contamination," where test tasks end up in training data. As a result, the gap between open-weight and closed models widened from 3 points on v4.3 to 12 points.

On October 23, it was reported that the gap between open and closed models had narrowed to 3 points. Just four days later, the same firm's figure had quadrupled. The models had not changed at all. The yardstick had. The "3-point gap" of two weeks ago partly showed how a yardstick built on public tasks behaves, not only how far apart the models really were.

What changed: private coding tasks were added

According to Artificial Analysis's announcement, v5 adds two evaluations. One is Terminal-Bench Science. The other is a coding benchmark whose tests are not disclosed. The announcement describes the latter as follows.

Cognition, which built the benchmark, develops the coding agent Devin. The previous version, FrontierCode 1.1 (October 12), had 100 tasks created together with the maintainers of 36 open-source repositories, and GPT-6 Luna did the grading. Version 2.0 grades on whether a maintainer would merge the change, and keeps the tasks themselves private. It is a practical yardstick: the question is not whether code passes the tests, but whether it is good enough to be merged into a real repository.

The announcement explains briefly why the tasks are kept private.

When tasks are public, they eventually get into training data, and models learn how to solve them. Scores may go up without any gain in the ability to solve problems a model has never seen. Keeping tasks private closes that loophole. The trade-off is that outsiders cannot check what the tasks contain. Faced with that choice, Artificial Analysis chose to avoid contamination.

The gap grows from 3 points to 12

When the yardstick changed, open-weight models fell sharply. The announcement says:

On v4.3, as of October 23, Kimi K3.1 Pro scored 55 points and ranked fourth overall. The only model above it was Claude Opus 5.5 (58 points), a gap of 3 points. On v5, Kimi K3.1 Pro fell to 46 points while Opus 5.5 stayed at 58. The gap opened to 12 points.

Closed models did not all move the same way under the new yardstick, however. GPT-6 Astra, which scored 53 on v4, rose to 57 on v5. Gemini 4 Argon, also at 53, fell to 50. Some models gained from the new tasks and some lost. Among them, the top open-weight model fell by 9 points, the largest drop.

This "top open-weight model" has also not yet released its weights. Moonshot has promised to release them by November 4, but the public version will reportedly have its cyber training data removed. The score of the model people can actually download may end up different from 46.

A new leader: Carbon's 59 passes Opus 5.5's 58 a day later

When v5 came out on October 27, Claude Opus 5.5 led with 58 points. The next day, October 28, Google DeepMind released Gemini 4 Carbon. In an evaluation article, Artificial Analysis named Carbon the new No. 1.

On v5, Gemini 4 Carbon scores 59, Claude Opus 5.5 58 and GPT-6 Astra 57, putting the top three within 2 points of each other. Gemini 4 Argon, released two weeks ago, scored 50. Carbon is 9 points higher than Argon. The top open-weight model, at 46, is 13 points behind Carbon.

This week, Carbon is available only to some participants in the "Fairwind Program" for cyber defense. Its model card and Frontier Safety Framework report have not yet been published. That means the top-ranked model is one that ordinary developers cannot use yet.

What was the "3-point gap" measuring?

The difference between v4.3 and v5 can be read as a difference in how the yardsticks work. On a yardstick heavy with public tasks, the gap between open and closed models looked small. Once private, practical tasks were added, the gap looked large. If contamination is one of the causes, some models that seemed to have caught up on public benchmarks had not yet caught up on tasks they had never seen.

Other yardsticks point the same way. On October 16, Epoch AI estimated that open-weight models trail closed models by 6–7 months in cyber capabilities. In the third round of First Proof, which poses 10 unpublished research-level math problems, Kimi K3, the best open-weight model, solved 0. The gap appears larger where tasks are not public or have been newly created. The 12-point figure fits well with these results.

Still, contamination cannot be said to be the only reason. The tasks added in v5 also measure a particular kind of skill: writing code as an agent over long sessions and producing changes that satisfy maintainers. If some models are strong at this and others are not, rankings would shift even without contamination. GPT-6 Astra's rise and Gemini 4 Argon's fall may reflect such strengths and weaknesses.

Caveats

First, v4.x and v5 scores cannot be compared. GPT-6.1 Terra's 48 points and 25th place out of 221 models on October 26, and Kimi K3.1 Pro's 55 points and fourth place on October 23, are both numbers on the old yardstick. Opus 5.5 scoring 58 on both the old and new versions does not mean it showed the same ability.

Second, we could not find a dedicated changelog page from Artificial Analysis. It is unclear how much weight each new task carries in the index, or whether any tasks were removed. From the outside, it is not possible to separate how much of the 12-point gap comes from removing contamination, how much from changing the types of tasks, and how much from changing the weightings.

Third, outsiders cannot check private tasks. Hiding tasks to avoid contamination means relying more on trust in the evaluation firm and in the company that built the tasks. Cognition, which built FrontierCode 2.0, itself sells a coding agent. Coverage by GIGAZINE and others has followed Artificial Analysis's own explanation, and no third party has yet examined the tasks independently.

Even so, one thing can be said about this rebuild. Anyone stating the gap between open and closed models in points must also say which yardstick was used. The "3 points" of two weeks ago and the "12 points" of this week are both correct figures. What differs is the yardstick.