The PlaybookAI Benchmarks

The AI Benchmark Index.

Every model release since ChatGPT launched in November 2022, and the GDPval‑AA, SWE‑bench, AA Index and ARC‑AGI‑3 scores.

4 benchmarks

GDPval

The benchmark that scores models on real deliverables from 44 occupations, graded by working professionals. The one to quote if you are buying AI for knowledge work rather than code.

SWE-bench

Real GitHub issues, real repositories, a passing test suite or nothing. The coding benchmark, and the reason its Verified split had to be retired.

Artificial Analysis Intelligence Index

The composite that most launch posts cite without naming. What goes into it, and what happened when it was rebuilt.

ARC-AGI-3

Interactive games no model has seen, played with no instructions and no rules given. ARC Prize runs every model on one standard harness.

A new frontier model every 9 days

It used to be~10 weeksbetween new Anthropic and OpenAI models
Now~9 daysbetween new Anthropic and OpenAI models
Since September 1, 2026, on this page8Anthropic and OpenAI releases in 74 days
Six months to October 2026127launches across Anthropic, Google and OpenAI

A ten-week procurement cycle now spans 7 model releases, which is why the board below changes leader almost every month. September was the peak, with 6 dated releases on this page. October had 3.

Every update, month by month

Every monthly update since November 2025, and the releases and scores before them back to ChatGPT's launch.

Bars count the frontier releases listed for each year. Releases in bold took the lead.

  1. GDPval-AA
    GPT-6 Astra Is 3rd on the Index and 43rd on Real Work.

    Removed from the Intelligence Index on 27 October and still running. Claude Opus 5.5 leads at 1866, Kimi K3.1 Pro is second, and OpenAI’s best entry is 37th.

    ARC-AGI-3
    Claude Fable 5.2 81.2%, a new high (97.7% on the Provider Adapter; xhigh effort)
    Gemini 4 Carbon 65.1% (99.1% on the Provider Adapter)
    Released
  2. Reset: Index v5
    Index
    Fable 5.1 Falls From First to Eighth.

    The index went to v5 on 27 October and v5.0.1 eight days later. Fable lost four points and Astra gained four, neither having shipped a thing. FrontierCode 2.0 moved inside the composite.

    ARC-AGI-3
    Gemini 4 Argon 49.8% (97.5% on the Provider Adapter; 42 of 55 games scored)
    Released
  3. Index
    Fable 5.1 and GPT-6 Astra Tie at 53.

    Identical scores at identical list prices, and a 2.3x gap in cost per task. GLM 5.3 Flash ran the whole index for 25 cents.

    ARC-AGI-3
    GPT-6 Astra 62.7%, a new high (99.9% on the Provider Adapter)
    Claude Sonnet 5.5 24.3% (72.4% on the Provider Adapter)
    GPT-6.1 Sol 52.7% (96.4% on the Provider Adapter)
    Released
  4. Reset: GDPval-AA v2
    Index
    Grok 4.6 Joins the Frontier.

    Opus 5 at 63, Fable 5 at 62, Grok 4.6 and Sol at 61 on the AA index, with prices spanning 8x across four models separated by two points.

    GDPval-AA
    Opus 5 Leads at 1849 Elo.

    GDPval-AA moved to v2 and reset every June number. Grok 4.6 sits at 1746, and xAI's number-one claim does not hold.

    Released
  5. Index
    Opus 5 Matches Fable at Half.

    The busiest benchmark month of the year, and the arrival of the priced leaderboard. Opus 5 hit 66.7 against Fable's 66.5 on CursorBench at half the cost.

    ARC-AGI-3
    GPT-5.6 Sol 7.8%, a new high
    Claude Opus 5 30.2%, a new high
    Released
  6. GDPval-AA
    The Benchmark That Actually Matters.

    Claude Fable 5 leads at 1932 Elo. What expert parity on real deliverables means, and the catch in the fine print.

    SWE-bench
    Fable 5 Tops a Dying Leaderboard.

    80.3% on SWE-bench Pro, OpenAI kills Verified for good, and FrontierCode resets the leaderboard underneath.

    ARC-AGI-3
    Claude Opus 4.8 1.5%, a new high
    Released
  7. GDPval-AA
    The Leaderboard Reshuffles.

    Opus 4.8 takes the lead at 1890 Elo, Grok 4.3 jumps 321 points, and Gemini 3.5 Flash beats Google's own Pro tier on real work.

    SWE-bench
    Opus 4.8 Takes the Lead.

    69.2% on Pro, and open-weights models close to within 6 points at 8x lower cost.

    Released
  8. Reset: 8 benchmarks broken
    GDPval-AA
    GPT-5.5 Sets the Bar.

    Launches at 84.9% expert parity, and economists start writing about AI eating analyst work.

    SWE-bench
    The Month the Benchmark Broke.

    Berkeley researchers break eight agent benchmarks, and Mythos Preview exposes the Verified-versus-Pro gap.

    ARC-AGI-3
    Claude Opus 4.6 (Max) 0.5%, a new high
    Released
  9. GDPval-AA
    GPT-5.4 Crowds the Top.

    1674 Elo, 41 points above Sonnet 4.6. Three labs within 70 points, and what that compression means for buyers.

    SWE-bench
    GPT-5.4 Takes Pro.

    59.1% on SEAL's standardized Pro split, with Opus 4.6 holding the commercial subset.

    Released
    • GPT-5.4GDPval-AA 1674, SWE-bench Pro 59.1%
  10. Reset: Verified deprecated
    GDPval-AA
    Anthropic Takes Both Top Slots.

    Sonnet 4.6 at 1633 Elo and Opus 4.6 at 1606, with Gemini 3.1 Pro last among the majors.

    SWE-bench
    The Month Verified Died.

    OpenAI deprecates Verified over contamination, and Pro takes over as the number that counts.

    Released
  11. Reset: Index rebuilt
    Index
    The Index Overhaul.

    Artificial Analysis rebuilt the Intelligence Index: ten work-shaped evals in, saturated exams out, top scores down 20 points overnight.

    Released

    No frontier release

What twelve months of scores actually show

45.9% to 80.3%
Five resets, and the line kept climbing
NDJFMAMJJASON45.9%80.3%5 resets

The benchmark broke five times. The models did not stop improving. The Intelligence Index was rebuilt in January and top scores fell 20 points overnight. Verified was deprecated in February over contamination. Berkeley broke eight agent benchmarks in April. GDPval-AA moved to v2 in August and reset every June number. The Intelligence Index went to v5 in October and dropped GDPval-AA. Through all of it, SWE-bench Pro went from 45.9% in November 2025 to 80.3% in June 2026. The instrument kept failing and the thing being measured kept moving.

8 leaders
in eleven months on GDPval
DecemberGPT-5.2FebruarySonnet 4.6MarchGPT-5.4AprilGPT-5.5MayOpus 4.8JuneFable 5JulyOpus 5NovemberOpus 5.5OpenAIAnthropic

No model held the lead for two consecutive updates. GPT-5.2, then Sonnet 4.6, then GPT-5.4, then GPT-5.5, then Opus 4.8, then Fable 5, then Opus 5, then Opus 5.5. Eight leaders in eleven months. If you picked a model because it was top of a leaderboard, you have been wrong roughly every six weeks, and the cost of being wrong has been near zero. That is the argument for routing rather than standardising.

2 points, 8x
Score converged, price did not
Score gap, top four on the index2 pointsPrice gap, same four models8xAugust 2026. In March, three labs sat within 70 Elo.

The top of the table compressed until price became the tiebreak. In March, three labs sat within 70 Elo points. By August, four models sat within two points on the AA index with prices spanning 8x. A two-point capability gap and an 8x price gap is not a capability decision. It is a procurement decision, and that is why the July update started publishing cost next to score.

6 points
Open weights, at an eighth of the cost
SWE-bench Pro scoreFrontierOpen weights, within 6 pointsCostFrontierOpen weights8x lower

Open weights stopped being a separate conversation. By May 2026 open-weights models were within 6 points of the frontier on SWE-bench Pro at 8x lower cost. If you have not priced that option since then, you are working from a stale number. Start with What Are Open Weights?

The caveat that belongs on every number here

Every figure in this index is a published score, and published scores are marketing before they are measurement. Labs choose which benchmark to lead with, contamination is real and was the stated reason Verified died, and a reset means last month's number is not comparable to this month's even when both are on the page.

Read The Benchmarks Are Getting Gamed before you put any of these numbers in a deck. The short version: use benchmarks to eliminate models, not to pick one. The picking is done on your own work, with your own evals, which is the only test that cannot be trained against.

Updated monthly.

Portrait of John Tan
John Tan ↗

Founder of nativefirst.ai. Runs everything on agents and writes down what works.

Sources
1Artificial Analysis, GDPval-AA leaderboard and Intelligence Index, November 2025 through August 2026. GDPval-AA moved to v2 in August 2026, resetting prior Elo figures. artificialanalysis.ai
2OpenAI, GDPval, covering real deliverables across 44 occupations graded by working professionals. openai.com
3SWE-bench (Princeton) and SWE-bench Pro, including OpenAI's deprecation of the Verified split in February 2026 over contamination and its retirement in June 2026. openai.com · scale.com
4Scale AI SEAL leaderboard, standardized SWE-bench Pro results, March 2026. labs.scale.com
5UC Berkeley agent-benchmark analysis covering eight agent benchmarks, April 2026. moogician.github.io
6All figures are published scores from the labs and independent leaderboards named in each linked update, recorded on the date of that update. A benchmark reset means figures either side of it are not directly comparable.
7ARC Prize, ARC-AGI-3 leaderboard and verified results pages, April to November 2026. arcprize.org
8ARC Prize Foundation, ARC-AGI-3: A New Challenge for Frontier Agentic Intelligence, April 22, 2026. arcprize.org
9@jfonsecarivera on X, September 29, 2026: the interval between new Anthropic and OpenAI models.
10a16z, Top 100 Consumer AI Apps, October 5, 2026: 127 launches across Anthropic, Google and OpenAI in six months.

AI is moving really fast. Keep up.

Read the Playbook