EPISODE TRANSCRIPT
ARC-AGI-3 and Time Horizons (ARC Prize and Thomas Kwa): What Do AI Tests Measure?
Automatically transcribed from the episode audio. Some words and names may be inaccurate.
Listen to the episodeDownload textDownload captions
we are looking at two highly influential cutting edge frameworks from early 2026. First, we have a technical report titled ARC -AGI -3, a new challenge for frontier agentic intelligence. That one was published on April 22nd, 2026 by the ARC Prize Foundation. And the authors include Francois Chalet and Mike Noop. Two very heavy hitters in this space. Definitely. And then second, we are analyzing a research note from Metter. This one is authored by Thomas Qua and it's titled Clarifying Limitations of Time Horizon. That was published January 22, 2026. To understand why Arc AGI 3 even exists, we really have to look at the crisis unfolding in AI evaluation right now. The memory crisis. Exactly. So going back a bit, the original Arc benchmark, which was Arc AGI 1, dropped back in 2019. It was basically a test of fluid intelligence using static grid -based puzzles. Right. You'd get these input -output examples? Yeah, like say a blue square turning into two red triangles. And you had to deduce the underlying geometric rule to solve a totally new unseen grid. And the foundational philosophy of ARC, going all the way back to Chalet's 2019 paper on the measure of intelligence, is that true intelligence isn't task -specific skill. Right. It isn't being a grandmaster at chess or writing a perfect sorting algorithm.
It is skill acquisition efficiency. How fast you can learn the thing. Exactly. If you were dropped into a novel situation with only basic core knowledge priors, and by that I mean things like object permanence, basic geometry, topology, how efficiently can you synthesize a working model of that specific environment? Which is hard for machines. It is. The original ARC was designed specifically to resist the kind of statistical pattern matching that deep learning excels at. It had very few examples per task. So it was incredibly hostile to standard pre -trained large language models. Totally hostile. And for about five years, it worked. The base LLMs that drove that whole pre -training scaling boom, they failed miserably at it. But then the paradigm shifted entirely. We got Large reasoning models. Ah, yes. The LRMs. Yeah. Models like OpenAI's O1 and O3 and NVIDIA's and VARC. Suddenly, the needle on ArcAGI actually started moving. The AI was solving these abstract grids. And the narrative immediately became, oh, the models are finally reasoning. Well, the underlying mechanism of how they worked changed. Instead of just predicting the next likely token based on pre -trained weights, these LRMs use test time compute. I think before they speak. Basically, yeah. They engage in a form of search during inference. When presented with an arc puzzle, the model might generate this long step -by -step chain of thought. Right.
It'll propose a hypothesis for the grid's transformation rule, maybe write a Python program to execute that rule, check if the output matches the examples, and if it fails, it can backtrack and try a totally different hypothesis. Which is wild to see happening in real time. It really is. And this test time adaptation allowed them to exhibit what the ARC authors themselves actually categorize as non -zero fluid intelligence. ARC -AGI3 is an interactive turn -based environment. It's basically a video game. It really is. The Foundation actually set up an in -house game studio just to develop this.
They moved from evaluating data -efficient modeling on a flat piece of paper to evaluating how an agent behaves when it's embedded in an active responsive world. OK, so let's break down the mechanics of this world, because reading the report, it is intentionally primitive. Extremely primitive. The environment is just a 64 by 64 grid. Each cell can be one of 16 colors, and the action space, like the actual verbs the AI can execute, is incredibly constrained. Yeah, what, do they have five actions? Yeah, there are only five key actions, an undo button and a coordinate selection click. That is the entire interface. The constraint is entirely the point.
There is zero language in this environment. There is zero cultural symbols. So no words, no signs. None. So the AI is dropped into this 64 by 64 pixel sandbox, and the technical report outlines four pillars of agentic intelligence that this specific environment is designed to test. They are exploration, modeling, goal setting, and planning and execution. Right. Walk me through how an AI actually executes these pillars in a simple turn -based grid. Sure. So, exploration is the first pillar. And it's a major departure from those old static tests. Think about the real world. You don't get all the information up front. Yeah, you have to figure things out.
Right. If you are placed in a dark room, you have to reach out and actually touch the walls. In Arc AGI 3, the agent has to physically probe the environment to reveal its hidden mechanics. Like pushing a button to see what it does. Exactly. It might... move a block to see if it slides infinitely or if it stops when it hits a wall. And that physical interaction feeds directly into the second pillar, which is modeling. Taking what it learned and building a theory? Precisely.
The agent must take the raw observations from its initial exploration and synthesize an internal, generalizable model of the physics specific to that exact puzzle. Okay, so if I push the blue block... and it bounces off the red block but destroys the yellow block, I have to update my mental model of how the rules in this specific universe work. Yes. But here is the part that completely breaks my brain. The third pillar is goal setting. This is the big one. Right.
The AI is never told what the objective actually is. There is no text prompt saying, destroy all the yellow blocks. That is easily the most aggressive design choice in the entire framework. It's crazy. the agent receives absolutely no explicit instructions. It must independently determine the win condition based solely on the environment's initial state and the feedback it gets from interacting with it. It literally has to infer what to target. But wait, if there are no instructions, how do we know the AI isn't just randomly clicking around? That's the first question everyone asks. Right.
Because if I have a 64 by 64 grid and I just start firing off actions as fast as I can. Won't I eventually stumble into the wind condition by accident? It really feels like we are inviting algorithmic brute force here. And the ARC engineering team anticipated that exact vulnerability. They had to. Yeah, because if a benchmark could be beaten by a random walk, it completely ceases to be a measure of intelligence. It's just a slot machine. And that leads us to the fourth pillar, planning and execution. OK, so how does that solve the random clicking? Well, once the agent infers the goal, it must strategically map an action path to get there, correcting its course based on intermediate feedback. So let's look at the data set architecture. It's split into 135 total environments, 25 public demonstration environments that literally anyone can go look at. Then there are 55 semi -private environments, which are used for testing models via an API without giving them the underlying code. Right.
Keep them honest. And finally, 55 fully private environments that are heavily guarded for their $2 million competition. But for any of these environments to make the final cut, they had to pass an exhaustive human calibration process. Yeah, I read about this. They actually set up a physical testing center in San Francisco. They did. They brought in 486 diverse participants who logged nearly 430 hours of continuous play. And the inclusion criteria was absolutely brutal. An environment was only accepted if it was deemed 100 percent human solvable. Now we need to define 100 percent human solvable really accurately here, because the phrasing alone can be a bit misleading. Right. It doesn't mean every single person beat it instantly. No, it does not imply that every single participant who sat down solved every single environment instantly on the first try. The specific rigorous metric is that at least two out of ten independent human testers had to successfully complete all levels of that environment on their very first attempt. Okay, two out of ten. Yes. And they had to do it entirely blind with zero prior training or instruction.
But getting back to my earlier concern, human solvability doesn't physically prevent an AI from just mashing buttons. How did the validation engine actually stop random luck from winning? They utilized an automated exploratory state space analysis. OK, big words. Basically, effectively modeling every single environment as an explicit directed graph. Let's unpack the graph concept for a second. We are talking about nodes and edges, right? Correct. In computer science, a directed graph maps relationships between different states. In this validation pipeline, every single unique state the 64 by 64 grid can possibly be in is represented as a node. A snapshot of the grid. Yes. And the actions a player takes, like moving a block left or clicking a specific coordinate, are the edges that connect one node to the next.
So it is mapping up the entire multiverse of the game. It's a great way to put it. If I start at node A, and I have five possible actions, those are five edges leading to five totally new universes, or nodes. And from each of those, five more edges. You're essentially building an enormous branching tree of every possible future in the game. And you can imagine the state space explosion is just astronomical. It has to be. A 64 by 64 grid with 16 colors has more possible configurations than there are atoms in the universe. Oh, wow. So the validation pipeline obviously doesn't map the entire mathematical space. It maps the reachable graph within finite budgets. Like putting a step limit on it. Exactly.
A maximum step limit. The acceptance threshold for an environment was that a random policy, meaning an agent just executing actions with completely uniform probability, basically guessing, should have an estimated success rate of less than 1 in 10 ,000 on the core evaluation levels. Less than 1 in 10 ,000. That's a solid buffer. It is. Now, editorially, we should note it is an empirical engineering estimate, not a universal mathematical impossibility theorem. Right. In fact, the report notes that on the intentionally easy tutorial levels, random agents will occasionally stumble into success. Which makes sense. Tutorials are usually just one or two steps. Exactly. And that is considered acceptable to help orient the player early on. But the evaluation doesn't stop at a simple pass or fail, does it? The way they score the AI is incredibly aggressive. Oh, it's brutal. The scoring metric is called RHAE.
R -H -A -E. Yes, which stands for relative human action efficiency. Under the ARC -AGI -3 philosophy, intelligence is not simply task completion. Just getting to the end isn't enough. No. Intelligence is defined strictly as efficiency. It is the ability to minimize the number of actions required to map the environment and reach the goal. Action efficiency. Yes. So they are literally measuring how much you had to flail around before the light bulb finally went off in your head? Basically, yes. The rationale is that a system that tries a hundred different incorrect hypotheses before finally stumbling on the right one is fundamentally displaying less fluid intelligence than a system that observes the environment, forms a correct mental model immediately, and executes a precise 10 -step plan. Which is very logical. To quantify this, the AI's action count is compared directly against a human baseline, and they don't just use the average human. No. They set the baseline at the upper -median best human on their very first run. The upper median. Yes. But the actual math of the score, this specific formula they use, is where it gets really punitive for the AI. Oh, yeah. Let's say the human baseline is 10 actions. The AI completes the level, but, you know, it takes 100 actions to get there.
The AI took 10 times as many moves. In a normal test, that's just 10 % efficiency, right? Yes. But under RHAE, they apply a power law formula. OK. What does that do? They take that raw efficiency ratio, so the human's 10 actions divided by the AI's 100 actions, which gives you 0 .1, and they square it. They square the ratio. Yes. So 0 .1 squared becomes 0 .01. The AI score for that level drops to exactly 1%. Squaring the ratio is absolutely brutal. A linear score gives the AI 10%. Squaring it literally vaporizes the score.
And they apply this brutal math on a per level basis, right, rather than looking at the total actions across the whole environment? They do, yes. Why slice it up by level like that? It's primarily a noise reduction technique. Okay. Think about it, an environment might consist of five levels. Level one might require only five actions, while a massive complex level four might require 80 actions. Right. If you aggregate the raw actions globally, A slight inefficiency on that huge level 4 would mathematically swamp the signal of perfect efficiency on the earlier smaller levels. Oh, I see. It would drown out the good data. Exactly. By calculating the squared efficiency per level and then averaging those percentages, you get a much clearer, high -resolution diagnostic map of exactly where the agent's reasoning capacity breaks down. And they also weight the levels, right? So an easy tutorial level contributes less to the final score than the hardest level.
Yes, weighted averaging. And they put a cap on the maximum score. But reading the technical report, I noticed a very strange mathematical contradiction regarding this specific cap. Ah, yes. The 115 % issue. Yeah. The prose in the original report explicitly states that the per -level score is capped at a maximum of 115%. However, if you look at the displayed mathematical formula in the paper itself, the 1 .15 value is placed inside the parentheses before the squaring function is actually applied. Which changes everything mathematically. Right. Because 1 .15 squared...
yields approximately 1 .32. So the text claims a 115 % cap, while the formula outputs a 132 % cap. That is a pretty significant typo for a rigorous technical paper. It is a discrepancy for sure, but the underlying intent of the authors remains completely clear here. What's the intent? The design allows AI models to receive a modest bonus for outperforming human efficiency while implementing a hard mathematical ceiling to prevent anomalous exploits from corrupting the aggregate metric. Right. So whether the cap is 115 or 132, the function is basically the same. It stops runaway exploit scores. Exactly. And honestly, the cap doesn't even matter right now because the frontier models are nowhere near it. No.
Not even close. Looking at the March 2026 leaderboard in the report, the scores were abysmal. On the official semi -private evaluation, Anthropix Opus 4 .6 scored 0 .50%. Half a percent. Google's Gemini 3 .1 Pro Preview scored 0 .40%. GPT 5 .4 High scored 0 .20%. They're virtually at zero. That is incredible to think about given how much hype surrounds these models. This is Thomas Kwa's research note from METER regarding the time horizon of AI models. Time horizon. Let's define this really carefully because it is one of the most widely misunderstood metrics in the entire industry.
Oh, definitely. When a CEO hears that an AI has a time horizon of five hours, they almost always assume it means continuous runtime. Right. They think of it like a battery life. Exactly. They think, oh, I can spin up this AI, let it run autonomously on a server for five hours, and then I need a human to check on it before it hallucinates or crashes. But that interpretation is fundamentally incorrect. Completely wrong. Time horizon has literally nothing to do with the continuous runtime of the GPU. It is a metric of human labor replacement. OK, say more about that. Specifically, time horizon is defined as the maximum length of serial human labor that an AI can successfully complete with a 50 % reliability rate. And it is calibrated strictly by the time human experts traditionally need for that task.
It anchors AI capability to the human economic baseline. And when Metter began testing this across different domains, they found a massive divergence in capability. So it's not universally good at everything. Where does it break down? Well, in domains strictly bounded by text and code, like software engineering or mathematical proofs, the time horizons are relatively consistent, and they are growing rapidly. Because it's all text. Right. However, when the tasks shift to visual computer use, The horizon completely collapses.
Meta observed a 40x to 100x drop in performance on tasks, requiring the AI to perceive and interact with graphical user interfaces. But even when we stick to pure software engineering, Kwan's note admits that measuring this time horizon is incredibly noisy. The statistical reality of measuring time horizon is incredibly messy right now. Yeah. In the report, METR states that Claude Opus 4 .5 has a 50 % time horizon of 4 hours and 49 minutes. But people read that exact number in a headline and completely ignore the error bars. The confidence interval is terrifying.
I looked at the chart. The 95 % confidence interval for that 4 hour 49 minute horizon spans from 1 hour 49 minutes all the way up to 20 hours 25 minutes. That massive variance. is an unavoidable consequence of their evaluation architecture. METR calculates the confidence interval using a statistical method called bootstrapping. Okay, I've heard that term, but how does bootstrapping actually generate that massive error range? Bootstrapping is essentially a resampling technique. Kwa explicitly states in the note that he cannot definitively say whether the true underlying horizon is 3 .5 hours or 6 .5 hours. The statistical resolution simply isn't there yet. with only 170 data points spanning such widely varying difficulties. And this statistical noise creates a massive hazard for businesses, right? Because they are trying to calculate ROI. If an operations manager reads that a model has a five -hour time horizon, their immediate instinct is to look at a five -hour task, deploy the AI, and just magically subtract five hours from their payroll budget. And that assumption violates what the industry calls the uplift equation. The uplift equation. Yes. Time horizon cannot be treated as a direct one -to -one subtraction of human labor. Calculating real productivity uplift requires modeling the entire workflow ecosystem. It's never just the task itself. No. You have to account for the human time spent engineering the prompt in the first place, the human time spent waiting for the multi -step generation to actually execute, and most critically, the human time required to verify the output. Therefore, a 50 % success rate on a five -hour task might actually generate a net negative productivity yield. The economics of automation are highly non -linear, and they are highly dependent on the cost of verification.
This brings up a critical methodological point that Qua highlights regarding how they calculate the human baseline itself. Ah, the geometric mean. Yes. When METRA has multiple human experts complete the same benchmark tasks, they don't just add up the times and find the standard average. They use a specific statistical calculation called the geometric mean instead of the normal arithmetic mean. But there is an even more complex statistical artifact in how MET models the time horizon curve that can create incredibly counterintuitive results.
We are getting into the deep statistical weeds here, and I love it. So Qua explains that they don't just look at the 50 % pass rate, right? Right. They are trying to map the entire capability curve of the model from the 80 % horizon, which are the easy tasks, the model almost always gets right, down to the 20 % horizon, which are the highly complex tasks the model usually fails. How do they actually model that curve? They fit a two -parameter logistic model to the pass -fail data across all the tasks. Logistic model. What does that look like? It generates an S -shaped curve. It starts flat at the bottom, accelerates steeply upward through the middle, and flattens out again at the top. The issue is that the entire shape of that S -curve is defined by only two mathematical parameters, essentially the midpoint and the slope. And because it's only two parameters, the top of the S -curve and the bottom of the S -curve are mathematically locked together.
They are coupled. Yes. Qua outlines the statistical requirements for measuring 99 % reliability. To have any real confidence in that quantile, you would need at least 300 highly diverse tasks concentrated in every single specific time bucket you want to measure. 300 tasks per time bucket. That's huge. That's huge. And furthermore, the label noise. The inherent error rate and how the benchmark verifies the answer must be strictly less than 1%. Label noise meaning the test itself is just wrong, like a unit test that is way too strict or a prompt that is somehow ambiguous.
Yes. If the benchmark's grading system has a 2 % error rate where it accidentally marks a correct AI solution as incorrect 2 % of the time, the absolute maximum measured capability of any model will mathematically saturate it 98%. Oh, wow. Even an omniscient, perfect AGI would evaluate as having a 99 % time horizon of exactly zero hours under those conditions. You simply cannot measure a signal cleaner than the noise floor of your instrument. The noise is large, yes, but the signal of capability improvement is significantly larger. And that leads to the 2030 extrapolation.
This is the part everyone talks about. If you track the historical progression of the 50 % time horizon across all the frontier models, What is the velocity? The long run trend indicates that the time horizon is doubling approximately every six to seven months. Doubling every six months. That's incredibly fast. It is. And if you apply a linear extrapolation to that exponential growth rate, it projects that a frontier AI model will reach a one month time horizon by the year 2030. A one month horizon. meaning the model could reliably, with a 50 % success rate, execute 167 contiguous hours of expert human labor. Right.
The METR benchmark tasks are currently entirely self -contained. The current METR suite evaluates absolutely none of that collaborative multi -turn organizational friction. The 2030 projection is highly, highly conditional. Neither metric alone provides a universal definitive ranking of AGI. You can mathematically aggregate them if you wish to create a composite score, sure, but doing so completely obscures the very real divergence in underlying capabilities. Yeah, that makes sense. A model can be an absolute virtuoso at one while remaining functionally incompetent at the other. And that divergence is exactly what I want to leave you, our listener, thinking about today because it's a wild thought experiment. Why might performance skyrocket on one measure while remaining dead in the water on the other? It's the ultimate paradox. What would that divergence reveal about the true nature of the intelligence we have built?