---
title: "What Is ARC-AGI?"
url: https://nativefirst.ai/blog/what-is-arc-agi/
description: "The benchmark that measures learning rather than knowledge, its three generations, and why the same model scored 62.7% and 99.9% on the same test."
author: John Tan
category: "AI Benchmarks"
published: 2026-11-12
updated: 2026-11-12
publisher: nativefirst.ai
format: Markdown copy of the web page at the url above, same text.
---

# What Is ARC-AGI?

The benchmark that measures learning rather than knowledge, its three generations, and why the same model scored 62.7% and 99.9% on the same test.

John Tan · November 12, 2026 · 5 min read

Most AI benchmarks ask whether a model knows something. ARC-AGI asks whether it can work something out that it has never seen, and it has been asking that question since 2019.

It is the benchmark people reach for when they want to argue about whether a model is actually reasoning. It is also the one most often quoted wrongly, because a score means nothing until you know which version and which harness produced it.

## What it actually measures

**Fig. 1 What a task actually looks like · Three pictures, no instructions. Work out the rule.**

*Chart: A real ARC-AGI task. Two worked examples each show a three by three grid of orange and black squares turning into a nine by nine grid, where the original shape is stamped into itself wherever the original had a coloured square. A third three by three grid is shown with its answer left blank.*

*Chart: The same ARC-AGI task, stacked for a narrow screen.*

A real task from the public ARC-AGI-1 training set, drawn from its own data. The rule: stamp the shape into itself wherever the original has a coloured square. Figure by nativefirst.

[François Chollet](https://arcprize.org/about), who created the Keras deep-learning library and co-founded the ARC Prize Foundation, introduced ARC-AGI in 2019, in a paper called ["On the Measure of Intelligence"](https://arxiv.org/abs/1911.01547). Its central claim is a definition, and the whole benchmark follows from it: the intelligence of a system is its **"skill-acquisition efficiency over a scope of tasks"**.

Read that as a rejection of the usual way of scoring models. A system that has memorised a million examples of a task is not intelligent by this definition; it is prepared. What counts is how quickly it picks up something new.

So the tasks are built from what [ARC Prize](https://arcprize.org/arc-agi/1/) calls core knowledge priors. Objects, counting, symmetry, simple geometry. Nothing cultural, nothing you need an education or a language to have. The design rule is **"Easy for Humans, Hard for AI"**, and that gap is the measurement. When the gap closes, the benchmark has done its job and needs replacing. That has now happened twice.

## Three generations, and they are not the same test

This is where most misquoting starts. "It scored 90% on ARC-AGI" is not a statement until you say which one.

**Fig. 2 The three generations · Each version was built because the one before it stopped being hard.**

*Chart: Three generations of ARC-AGI. ARC-AGI-1, from 2019, is 800 grid puzzles where a solver infers a rule from about three worked examples; OpenAI's o3-preview reached 87.5 percent at high compute in December 2024. ARC-AGI-2, from 2025, has 1,000 training tasks and three 120-task evaluation sets, scored on two attempts, and every task has been solved by at least two people. ARC-AGI-3 is interactive: agents are dropped into novel game environments with no written instructions and must learn by playing.*

*Chart: The same three generations, stacked for a narrow screen.*

Figure by nativefirst, from ARC Prize's own pages for each version.

**ARC-AGI-1**, from 2019, is 800 grid puzzles. You get about three worked examples and have to infer the rule, then apply it to a new grid. It sat largely unbeaten for five years, which is why it mattered: it was the benchmark that first showed frontier test-time reasoning arriving, when OpenAI's o3-preview reached 75.7% at low compute and 87.5% at higher compute in December 2024.

**ARC-AGI-2**, from 2025, keeps the format and raises the difficulty: 1,000 training tasks and three evaluation sets of 120, scored on two attempts. Tasks that could be brute-forced were removed deliberately. It targets symbolic interpretation, compositional reasoning and contextual rule application, which in plain terms means tasks where the trick is noticing which rule applies rather than executing it.

The human check is the part worth knowing. ARC Prize ran a live study in San Diego in early 2025 with more than 400 members of the public, and kept only tasks ordinary people could solve. Their claim is precise: **"100% of tasks have been solved by at least 2 humans"** within two attempts. So when a model scores badly here, it is failing something every task in the set has been shown a person can do.

**ARC-AGI-3** is a different thing altogether. It is an **"interactive reasoning benchmark which challenges AI agents to explore novel environments"**. No puzzles, no written instructions. An agent is dropped into a game it has never seen and has to work out the rules by playing. It scores skill acquisition over time and long-horizon planning with sparse feedback, which is much closer to what an agent does inside a company than any grid puzzle.

## How to read a score without being fooled

One number, three questions. Which version, what did it cost, and **which harness**.

**LS20 · level 1 of 7** No title screen explanation, no rules, no text. You are the white cross. Everything else you have to work out by moving.

ARC Prize, ARC-AGI-3 public demo, task LS20, captured 13 November 2026. The site reports 776 human actions to complete the game. [Play it yourself](https://arcprize.org/tasks/ls20).

The third is new, and it is the one people miss. On ARC-AGI-3, GPT-6 Astra scored 62.7% under ARC Prize's neutral Standard harness and 99.9% under OpenAI's own Provider Adapter. [Same model, same tasks, same day](https://nativefirst.ai/blog/gpt-6-astra-day-1/). The adapter carries opaque reasoning state between calls and compacts long contexts, so the model reuses prior work instead of rebuilding it.

That is a real product capability and not a cheat. It is also not what the other models in the table were given. ARC Prize now publishes both columns for exactly this reason, and a headline score that does not say which one it came from is not a fact you can use.

For scale on the same benchmark, all on the Standard harness: GPT-5.6 Sol scored 7.8% and Claude Opus 5 scored 30.2% in July 2026, Astra reached 62.7% in September, [Gemini 4 Carbon](https://nativefirst.ai/blog/gemini-4-carbon-2-dollars-doubles/) 65.1% in November for $7.2K of compute, and **Claude Fable 5.2 leads at 81.2%** as of 9 November 2026, for $21.9K. Cost is the column nobody quotes: Carbon scored within 16 points of the leader for a third of the money. Fable's own Provider Adapter figure is 97.7%, which is lower than Astra's 99.9%. So the best Standard score and the best adapter score belong to different models, which is its own argument for reading the column label.

[The AI Benchmark Index](https://nativefirst.ai/blog/the-benchmark-index/) tracks every ARC-AGI-3 record on the Standard harness, month by month, next to GDPval, SWE-bench and the Intelligence Index.

## What a high score does not mean

ARC Prize is unusually disciplined about this, and it is worth quoting them against the hype rather than paraphrasing. On Astra's run they wrote that **"we are not claiming that it is AGI"**, and that saturating the benchmark would not be proof of achieving it.

Their reasoning is the useful part. ARC-AGI-3 has a tightly bounded scope, deterministic mechanics and closed-ended goals. It does not represent the open-endedness of real work. A model that beats it has demonstrated something narrow and real, which is not the same as general capability.

The second limit is structural. Every version so far has been retired by being solved, and each retirement took less time than the last. ARC-AGI-1 held for five years. ARC-AGI-2 did not. A benchmark whose entire purpose is to be hard for machines has a short shelf life by design, so any score you quote has a use-by date.

## What a CEO should take from this

Three things, none of which require you to care about grid puzzles.

**ARC-AGI-3 is the one to watch, because it is the one shaped like your problem.** A model that can be dropped into an unfamiliar environment, work out the rules and improve with experience is describing an agent joining your company, not a chatbot answering a question.

**Always ask which harness.** The 37-point gap between the same model's two scores is the clearest demonstration this year that the scaffolding around a model is worth as much as the model. If that is true for a frontier lab on a benchmark they have tuned, it is worth more inside your business, where nobody has tuned anything.

**Treat a saturated benchmark as news about the benchmark.** When a score jumps to the high nineties, the first question is whether the test got easier to game, not whether the machine got smarter. ARC Prize replaces its own benchmark when that happens, which is the behaviour to expect from a serious evaluator and a useful filter for the ones that are not.

**Ask which harness. Then the number.**

[John Tan ↗](https://www.linkedin.com/in/johntans/)

Founder of nativefirst.ai. Runs everything on agents and writes down what works.

Sources

1 ARC Prize, "What is ARC-AGI?" and the per-version pages for ARC-AGI-1, 2 and 3: intelligence defined as "skill-acquisition efficiency over a scope of tasks"; design rule "Easy for Humans, Hard for AI"; ARC-AGI-1 is 800 grid tasks; ARC-AGI-2 has 1,000 training tasks and three 120-task eval sets with "100% of tasks have been solved by at least 2 humans"; ARC-AGI-3 is an "interactive reasoning benchmark which challenges AI agents to explore novel environments". [arcprize.org](https://arcprize.org/arc-agi)

2 ARC Prize leaderboard, read 12 November 2026: ARC-AGI-3 Standard harness scores of 7.8% for GPT-5.6 Sol, 30.2% for Claude Opus 5, 62.7% for GPT-6 Astra, 65.1% for Gemini 4 Carbon at $7.2K, and 81.2% for Claude Fable 5.2 at $21.9K, with Provider Adapter figures of 99.9%, 99.1% and 97.7% respectively. [arcprize.org](https://arcprize.org/leaderboard)
