Artificial Analysis v5 とベンチマーク作成者のジレンマ

Artificial Analysis V5 and the Benchmarker's Dilemma was published on Wednesday, October 28, 2026. A sort of review of the benchmarking industry and what's missing. Hill climbing is easy. Knowing what to hill climb is hard. Making scores go up is expensive, but we have a much clearer playbook for it than we do for creating meaningful benchmarks at the true frontier of AI today. Independent evaluation of language models is underappreciated in how hard of a task it is. These groups are not just taking current models and certifying their scores, which is a technically involved task. This is their job if AI progress is slow. When AI progress is as fast as it is, the technical details of the job are the routine part of trying to put a single number on what makes for a good AI model today. The goal is to have a single number that is representative of the current models, matching vibe checks, community sentiment, and scientific analysis, while being forward-looking enough so that it is not outdated in three months. Everyone building a benchmark faces a sort of paradox.

You need your benchmark to reduce to one single number, so that it is easy to discuss, use in reports, and compare models. But every method of aggregating scores comes with meaningful trade-offs. The best scientists need to do this, everyone does. It's a matter of creating aggregating metrics that you think are most fair, knowing there will always be some imperfections. With the amount of money in the AI ecosystem today, the weight of independent evaluation is even higher. You dictate the winners and losers of a trillion-dollar market. You also influence some of the models themselves, steering the shape of progress. This is a lot of pressure. My prompt for this article is Artificial Analysis' recent update to their intelligence index, V5. The AA index is the most prominent independent intelligence metric for the latest frontier models, followed in a distant second by Epoch AI's Capability Index, ECI. To date, all of these have been incredibly useful for the community and massively flawed.

Language models are capable of so much today, and this scope of capabilities is growing quickly. Our era of singular intelligence indices should be phased out soon, but it will take a long time. A summary of AAV5 is below from their blog post, where they describe it as, quote, reflecting real-world usage. Quoting the Index blog post. Intelligence Index V5 weights. Knowledge Work 25 AA Briefcase 15 Automation Bench AA 5 And GDP 5 Coding 25 Frontier Code 2 15 And Terminal Bench 4 10 Science 25 Terminal Bench Science 15 and CritPoint V2 Hard 10 General 25 AI Omniscience V1 15 and HLE Diamond 10%. End quote. Here's the ranking of the new models. Click through to see an image where the top model is called Opus 5.5, followed by GBT6 Astra, and the next highest models are Gemini 4

Argonne and Muse. Note, after the release, Gemini has announced Gemini 4 Carbon, which is the highest score on this index. The problem with trying to benchmark real-world usage today is that it is extremely jagged, just like the models. The most proven use case is coding, where open models are far more competitive and likely to keep up. In knowledge work, I'm less confident in open models. At first glance, in aggregate, this ranking definitely looks reasonable. Still, it shows signs of strain and confusion, as any benchmark will, with GPT 6.1 Terra beating Claude Fable 5.1 on AAV5. Today, some benchmarks measure the absolute frontier of capabilities, playing nicely in open-ended tasks through your favorite harness, which often does take a large model. Some other benchmarks measure certain skills that will be needed of a model in repetitive agentic tasks, I get the feeling that AA is tuning their benchmark more towards the former, a general agentic collaborator, but all of these behaviors overlap.

This takes us to the plot going around that prompted me to write this, where they showed the balance of open vs. closed models on the new benchmark. This plot showed 8 closed models beating the best 2B open weight model Kimi K3.1 Pro, which is in color blue. This benchmark is taking the opinion of trying to measure where Frontier models are moving into, which grows the gap to open models. This is an okay decision to make, but over time, as existing workflows of open models become key factors of the economy, it could become a benchmark that misses where the core economic engine of the industry is. Click through to see image comparing the Artificial Analysis V5 Index with open and closed models. Open models' demise from this seem greatly exaggerated, but I do think the previous index was flattering the open models too strongly. Consider this plot from the Kimi K3.1 Pro launch. Yes, the open models are strong and useful in some cases, but the best open AI and entropic models have been a major step ahead in the flexibility and reliability. Click through to see image where Kimi K3.1 Pro

was the fourth overall model, which I think was flattering them too much. Some of the evaluations in the AAV5 that open models are most behind on are science evals like Terminal Bench Science and CripPTV2 Open models are behind but the value proposition was never to compete truly at the frontier. The open model economy will be about, one, reducing domains with high market demand for inference to a commodity price of hardware. Two, allowing ownership of intelligence, specializing it to internal data, and controlling internal agents across many companies in the economy. Both of these take a very long time and are partially represented in benchmarks. The supply of compute is currently so low that open model providers are also raking in solid margins. When this shrinks, we will know the system is starting to work. The same goes for diffusion into enterprises. It's not clear what the exact ceiling on it is because of the difficult ML expertise that is needed to build a valuation to do post-training today.

At the next major model cycles, say in 2-3 months with GLM-6 or KIMI 3.5, I expect the leading open model labs to make substantial steps on this index, again surprising many observers. This is a cycle we have been on for years now. I've written many times about how open models keep up with the frontier, and the summary is that, 1. The best models do benchmarks a little. The worst open models benchmarks a lot, but don't get a lot of attention anyways. 2. The best open models are a bit narrower in which capabilities they cover, as seen above with their gaps on scientific domains. The open model labs are working to address this, but I think it could come to be a strategic error to not focus. There is a more nuanced version of explanations of the performance gap today, where so many of the gains in training are coming from having RL environments with distributions close to the evaluations. Keeping up with rapid progress across multiple domains requires a sophisticated RL stack that not all the open labs may possess. Some may be reliant on buying data that has a moderate lead times as well.

Another part of this announcement is shifting to, quote, private benchmarks, where the prompts and environments are now public, as a sign that Open Labs are benchmarking. This sort of analysis has been done before and fits with my above theory on how Open Labs benchmarks. It is easier to create training data to support an evaluation when you know the shape of it, but the task isn't impossible with hidden data. All in, this is a change that directionally I think is defensible, but it's not unanimous. For example, you're putting complete trust in the private benchmark builder, as you no longer get the benefit of the community finding issues in the benchmark, and pushing the benchmark to improve itself, which happens a lot. There is enough money on the line where I'd rather see transparency in benchmarking and enabling many eyes to find bugs The final point I need to make which does flatter the closed labs is that the likes of Anthropic and OpenAI moved on from caring about these entities months or even years ago Their focus is to a large majority on supporting internal use cases like AI R automation, and the use cases of customers who are spending billions of dollars. Yes, they have

team members hill climbing and other capabilities to unlock new markets, but these behaviors are to a minority extent driven by public benchmarks. At the same time, it is existential for small to medium-sized labs to appear solid on these intelligence indices as their branding is fully built around catching up to the frontier. It's a very silly cat-and-mouse game at times, but one that will be lasting. The only solution to our current equilibrium, with one or two indices dominating the discourse on where the frontier is, is to have more intelligence indices with other biases, for example, just for coding. In some ways, this is forming with the likes of VALS AI and intelligence.ai growing, but the mindshare is very lopsided towards artificial analysis. Every benchmark creator has their worldview. The dilemma is that you can always shift the numbers to match your narrative. This has always been a challenge of doing good science. The most recurring version of this question has been, how much effort should you put into tuning your training baseline. More tuning makes your results more trustworthy, but often reduces

the apparent success of your method. This is doubly true in the era of agents, where much of the analysis work is done through the lens of LLMs. In evaluation, this comes down to which benchmarks you choose to include in your index. The benchmarking industry comes down to trust. I have a solid amount of trust for artificial analysis, but not the most I do for anyone in the ecosystem. Artificial analysis is a business model that makes money from labs to verify their results. They make a lot of money doing this, which is fine. Other benchmarking companies have a similar problem, such as Arena, which does pre-release testing for labs to pay the bills. The entities I trust most to be fair on evaluations are nonprofits, academics, and government institutions like Casey. Today, the institutions I trust to be accurate on the frontier evaluation are startups like artificial analysis and Arena. So we're all stuck with the trade-offs and dilemmas. No number is perfect, and there is a large amount of wiggle room in the numbers for people to take analysis in different directions. This is a second dilemma, one that is sort of my job, the dilemma of a benchmarker of benchmarkers. It is more sociological than scientific. Thanks for listening to the end.

I wanted to add a note of my appreciation to my audience. Much of Interconnect's growth relies on word of mouth from readers, so if you have a friend you think would like this, please share it. I also published a comprehensive introductory lecture on open models on my YouTube channel, NATO Lambert, which you may enjoy. Have a great day.