When a company releases a new AI model, it usually publishes a list of scores alongside it. Those scores come from benchmarks. A benchmark is a fixed set of questions or tasks, with known correct answers, that every model can be given in the same way. The score is the share the model gets right.

The idea is the same as a school exam. If everyone sits the same paper, the results can be compared. This article explains what the best-known benchmarks measure, and why a high score tells you less than it seems to.

What a benchmark measures

Each benchmark tests one kind of ability. Three examples show the range.

  • General knowledge. MMLU, published in 2020, covers 57 subjects, from elementary mathematics to US history and law. When it came out, most models scored close to what random guessing would achieve.
  • Practical work. SWE-bench, published in 2023, contains 2,294 real software problems taken from 12 widely used open-source projects. The model has to change the code so the problem is fixed. A later subset, SWE-bench Verified, keeps 500 of those problems that people have checked by hand.
  • Expert questions. Humanity’s Last Exam, published in 2025, has 2,500 questions across dozens of subjects, including mathematics, the humanities and the natural sciences. It was written to be hard for the best models of its day.

A different approach skips the answer key. In Chatbot Arena, people are shown answers from two models and vote for the one they prefer. Many such votes are combined into a rating for each model, in the way chess players are ranked.

Tests are being beaten faster than they can be written

The main problem is called saturation. A benchmark is saturated when the best models score so close to the maximum that it can no longer tell them apart.

Stanford University’s AI Index, an annual report that tracks the field, has followed this closely. Its 2025 edition described MMLU and two other older tests as saturated. The authors of Humanity’s Last Exam gave the same reason for writing it, noting that models now score over 90% on MMLU.

Newer tests have not lasted long either. The 2026 AI Index says tests meant to stay difficult for years are now “saturated in months”.

BenchmarkEarlier scoreLater scoreSource
MMLUNear random chance for most models (2020)Over 90% (2025)MMLU and Humanity’s Last Exam papers
SWE-bench1.96% for the best model (2023)71.7% (2024)SWE-bench paper; AI Index 2025
SWE-bench Verified60%Near 100% one year laterAI Index 2026
Humanity’s Last Exam8.80% for the top system30 percentage points higher a year laterAI Index 2025 and 2026
How quickly scores have risen on four benchmarks

These figures come from different reports and different versions of the tests, so they show the direction of travel and should not be read as one continuous series. The SWE-bench Verified figure is from the summary of the 2026 AI Index.

Saturation also squeezes the leaders together. The 2026 report found that, as of March 2026, four companies’ top models sat within 25 rating points of each other on the Arena leaderboard. When the gaps are that small, being ranked first means little.

A benchmark is saturated when the best models score so close to the maximum that it can no longer tell them apart.”

Scores on OSWorld, a test of AI agents using a computer% of tasks completed
Scores on OSWorld, a test of AI agents using a computer
Item% of tasks completed
Earlier top AI scoreAbout 12%
Latest top AI score66.3%
Human baselineAbout 72%

Source: Stanford AI Index 2026. Chart: The Sidelight.

Models may have seen the questions already

The second problem is contamination. AI models learn from very large amounts of text collected from the internet. Benchmark questions are usually published online, so they can end up in that training material. A model that has seen the questions before may partly be remembering answers, which is a different thing from working them out.

One team tested this directly. They wrote a new set of school-level maths problems, called GSM1k, designed to match the style and difficulty of an established test called GSM8k. Because the questions were new, no model could have seen them. Some models scored up to 8% lower on the new set.

The result cuts both ways. A drop of that size suggests some scores on the original test were inflated. But the same study found that models were, broadly, still able to solve maths problems they could not have seen in training. Contamination distorts scores without making them meaningless.

Why a score can still mislead

Even a fresh, unsaturated benchmark has limits.

  • The test itself can be wrong. The 2026 AI Index reports that the share of invalid questions in the benchmarks it reviewed ranged from 2% in the maths section of MMLU to 42% in GSM8K. A model cannot fairly be marked on a question that has no correct answer.
  • Leaderboards can be gamed. A 2025 paper, The Leaderboard Illusion, argued that a few companies could test many unreleased versions of a model privately on Chatbot Arena and publish only the best result. It identified 27 private versions tested by Meta before one launch. This is the authors’ analysis, and it concerns how the platform was run at the time.
  • Tests are narrower than real work. The 2026 AI Index notes that the top model reads an ordinary clock face correctly only about half the time, far below people. The report calls this unevenness “jagged intelligence”. Strong scores on hard exams do not guarantee competence at simple tasks.
  • Success rates hide failures. On OSWorld, a test for AI agents, which are systems that carry out multi-step tasks on their own, scores rose from about 12% to 66.3%. That is close to the human score of roughly 72%. It also means the agents failed about one attempt in three.

There is also a gap in what gets reported. The 2026 AI Index says nearly all leading developers publish results on tests of capability, while reporting on tests of safety and responsible use “remains spotty”.

How to read a benchmark claim

A few questions help when a company announces a record score.

  1. How old is the test? The older and more public it is, the more likely it is saturated or contaminated.
  2. How big is the gap? A lead of a point or two on a test where everyone scores highly is not meaningful.
  3. Who ran it? Scores reported by the company that built the model carry less weight than independent results.
  4. Does the test resemble your task? A coding score says little about how well a model summarises a contract.

Benchmarks remain the best shared measure available, and the steep rise in scores reflects real progress. They are a rough guide to what a model can do under test conditions, and a poor guide to how reliably it will do it anywhere else.