1 00:00:00,000 --> 00:00:02,459 we are looking at two highly influential cutting 2 00:00:02,459 --> 00:00:05,559 edge frameworks from early 2026. First, we have 3 00:00:05,559 --> 00:00:09,099 a technical report titled ARC -AGI -3, a new 4 00:00:09,099 --> 00:00:11,259 challenge for frontier agentic intelligence. 5 00:00:11,900 --> 00:00:14,679 That one was published on April 22nd, 2026 by 6 00:00:14,679 --> 00:00:17,239 the ARC Prize Foundation. And the authors include 7 00:00:17,239 --> 00:00:20,019 Francois Chalet and Mike Noop. Two very heavy 8 00:00:20,019 --> 00:00:22,820 hitters in this space. Definitely. And then second, 9 00:00:22,940 --> 00:00:25,539 we are analyzing a research note from Metter. 10 00:00:25,719 --> 00:00:28,539 This one is authored by Thomas Qua and it's titled 11 00:00:28,699 --> 00:00:31,320 Clarifying Limitations of Time Horizon. That 12 00:00:31,320 --> 00:00:34,579 was published January 22, 2026. To understand 13 00:00:34,579 --> 00:00:38,020 why Arc AGI 3 even exists, we really have to 14 00:00:38,020 --> 00:00:40,759 look at the crisis unfolding in AI evaluation 15 00:00:40,759 --> 00:00:43,880 right now. The memory crisis. Exactly. So going 16 00:00:43,880 --> 00:00:46,579 back a bit, the original Arc benchmark, which 17 00:00:46,579 --> 00:00:51,369 was Arc AGI 1, dropped back in 2019. It was basically 18 00:00:51,369 --> 00:00:54,390 a test of fluid intelligence using static grid 19 00:00:54,390 --> 00:00:57,090 -based puzzles. Right. You'd get these input 20 00:00:57,090 --> 00:01:00,090 -output examples? Yeah, like say a blue square 21 00:01:00,090 --> 00:01:03,109 turning into two red triangles. And you had to 22 00:01:03,109 --> 00:01:05,370 deduce the underlying geometric rule to solve 23 00:01:05,370 --> 00:01:08,230 a totally new unseen grid. And the foundational 24 00:01:08,230 --> 00:01:10,430 philosophy of ARC, going all the way back to 25 00:01:10,430 --> 00:01:12,969 Chalet's 2019 paper on the measure of intelligence, 26 00:01:13,590 --> 00:01:16,290 is that true intelligence isn't task -specific 27 00:01:16,290 --> 00:01:19,060 skill. Right. It isn't being a grandmaster at 28 00:01:19,060 --> 00:01:21,560 chess or writing a perfect sorting algorithm. 29 00:01:22,120 --> 00:01:25,159 It is skill acquisition efficiency. How fast 30 00:01:25,159 --> 00:01:27,859 you can learn the thing. Exactly. If you were 31 00:01:27,859 --> 00:01:30,420 dropped into a novel situation with only basic 32 00:01:30,420 --> 00:01:32,560 core knowledge priors, and by that I mean things 33 00:01:32,560 --> 00:01:35,560 like object permanence, basic geometry, topology, 34 00:01:36,120 --> 00:01:38,519 how efficiently can you synthesize a working 35 00:01:38,519 --> 00:01:41,239 model of that specific environment? Which is 36 00:01:41,239 --> 00:01:44,049 hard for machines. It is. The original ARC was 37 00:01:44,049 --> 00:01:46,590 designed specifically to resist the kind of statistical 38 00:01:46,590 --> 00:01:49,109 pattern matching that deep learning excels at. 39 00:01:49,510 --> 00:01:52,250 It had very few examples per task. So it was 40 00:01:52,250 --> 00:01:54,930 incredibly hostile to standard pre -trained large 41 00:01:54,930 --> 00:01:57,629 language models. Totally hostile. And for about 42 00:01:57,629 --> 00:02:00,790 five years, it worked. The base LLMs that drove 43 00:02:00,790 --> 00:02:03,930 that whole pre -training scaling boom, they failed 44 00:02:03,930 --> 00:02:06,909 miserably at it. But then the paradigm shifted 45 00:02:06,909 --> 00:02:09,840 entirely. We got Large reasoning models. Ah, 46 00:02:09,840 --> 00:02:13,580 yes. The LRMs. Yeah. Models like OpenAI's O1 47 00:02:13,580 --> 00:02:17,680 and O3 and NVIDIA's and VARC. Suddenly, the needle 48 00:02:17,680 --> 00:02:20,680 on ArcAGI actually started moving. The AI was 49 00:02:20,680 --> 00:02:23,520 solving these abstract grids. And the narrative 50 00:02:23,520 --> 00:02:26,080 immediately became, oh, the models are finally 51 00:02:26,080 --> 00:02:28,900 reasoning. Well, the underlying mechanism of 52 00:02:28,900 --> 00:02:32,000 how they worked changed. Instead of just predicting 53 00:02:32,000 --> 00:02:34,599 the next likely token based on pre -trained weights, 54 00:02:35,240 --> 00:02:39,000 these LRMs use test time compute. I think before 55 00:02:39,000 --> 00:02:41,580 they speak. Basically, yeah. They engage in a 56 00:02:41,580 --> 00:02:44,830 form of search during inference. When presented 57 00:02:44,830 --> 00:02:47,129 with an arc puzzle, the model might generate 58 00:02:47,129 --> 00:02:49,650 this long step -by -step chain of thought. Right. 59 00:02:49,849 --> 00:02:52,129 It'll propose a hypothesis for the grid's transformation 60 00:02:52,129 --> 00:02:54,689 rule, maybe write a Python program to execute 61 00:02:54,689 --> 00:02:57,150 that rule, check if the output matches the examples, 62 00:02:57,530 --> 00:03:00,509 and if it fails, it can backtrack and try a totally 63 00:03:00,509 --> 00:03:03,330 different hypothesis. Which is wild to see happening 64 00:03:03,330 --> 00:03:07,199 in real time. It really is. And this test time 65 00:03:07,199 --> 00:03:09,960 adaptation allowed them to exhibit what the ARC 66 00:03:09,960 --> 00:03:13,099 authors themselves actually categorize as non 67 00:03:13,099 --> 00:03:16,319 -zero fluid intelligence. ARC -AGI3 is an interactive 68 00:03:16,319 --> 00:03:18,860 turn -based environment. It's basically a video 69 00:03:18,860 --> 00:03:21,639 game. It really is. The Foundation actually set 70 00:03:21,639 --> 00:03:23,979 up an in -house game studio just to develop this. 71 00:03:24,620 --> 00:03:27,159 They moved from evaluating data -efficient modeling 72 00:03:27,159 --> 00:03:30,280 on a flat piece of paper to evaluating how an 73 00:03:30,280 --> 00:03:32,520 agent behaves when it's embedded in an active 74 00:03:32,520 --> 00:03:35,379 responsive world. OK, so let's break down the 75 00:03:35,379 --> 00:03:37,219 mechanics of this world, because reading the 76 00:03:37,219 --> 00:03:39,819 report, it is intentionally primitive. Extremely 77 00:03:39,819 --> 00:03:43,240 primitive. The environment is just a 64 by 64 78 00:03:43,240 --> 00:03:46,539 grid. Each cell can be one of 16 colors, and 79 00:03:46,539 --> 00:03:49,219 the action space, like the actual verbs the AI 80 00:03:49,219 --> 00:03:51,740 can execute, is incredibly constrained. Yeah, 81 00:03:51,740 --> 00:03:53,780 what, do they have five actions? Yeah, there 82 00:03:53,780 --> 00:03:56,639 are only five key actions, an undo button and 83 00:03:56,639 --> 00:03:59,219 a coordinate selection click. That is the entire 84 00:03:59,219 --> 00:04:02,139 interface. The constraint is entirely the point. 85 00:04:02,479 --> 00:04:05,000 There is zero language in this environment. There 86 00:04:05,000 --> 00:04:08,400 is zero cultural symbols. So no words, no signs. 87 00:04:08,699 --> 00:04:13,039 None. So the AI is dropped into this 64 by 64 88 00:04:13,039 --> 00:04:16,120 pixel sandbox, and the technical report outlines 89 00:04:16,120 --> 00:04:19,300 four pillars of agentic intelligence that this 90 00:04:19,300 --> 00:04:21,600 specific environment is designed to test. They 91 00:04:21,600 --> 00:04:25,779 are exploration, modeling, goal setting, and 92 00:04:25,779 --> 00:04:28,019 planning and execution. Right. Walk me through 93 00:04:28,019 --> 00:04:30,240 how an AI actually executes these pillars in 94 00:04:30,240 --> 00:04:34,730 a simple turn -based grid. Sure. So, exploration 95 00:04:34,730 --> 00:04:37,910 is the first pillar. And it's a major departure 96 00:04:37,910 --> 00:04:40,509 from those old static tests. Think about the 97 00:04:40,509 --> 00:04:42,649 real world. You don't get all the information 98 00:04:42,649 --> 00:04:44,879 up front. Yeah, you have to figure things out. 99 00:04:45,160 --> 00:04:48,279 Right. If you are placed in a dark room, you 100 00:04:48,279 --> 00:04:49,939 have to reach out and actually touch the walls. 101 00:04:50,680 --> 00:04:54,420 In Arc AGI 3, the agent has to physically probe 102 00:04:54,420 --> 00:04:56,480 the environment to reveal its hidden mechanics. 103 00:04:57,339 --> 00:05:00,439 Like pushing a button to see what it does. Exactly. 104 00:05:00,879 --> 00:05:03,819 It might... move a block to see if it slides 105 00:05:03,819 --> 00:05:06,459 infinitely or if it stops when it hits a wall. 106 00:05:06,879 --> 00:05:08,939 And that physical interaction feeds directly 107 00:05:08,939 --> 00:05:12,720 into the second pillar, which is modeling. Taking 108 00:05:12,720 --> 00:05:15,560 what it learned and building a theory? Precisely. 109 00:05:15,720 --> 00:05:17,920 The agent must take the raw observations from 110 00:05:17,920 --> 00:05:22,399 its initial exploration and synthesize an internal, 111 00:05:22,560 --> 00:05:25,860 generalizable model of the physics specific to 112 00:05:25,860 --> 00:05:29,100 that exact puzzle. Okay, so if I push the blue 113 00:05:29,100 --> 00:05:31,949 block... and it bounces off the red block but 114 00:05:31,949 --> 00:05:34,990 destroys the yellow block, I have to update my 115 00:05:34,990 --> 00:05:37,670 mental model of how the rules in this specific 116 00:05:37,670 --> 00:05:40,990 universe work. Yes. But here is the part that 117 00:05:40,990 --> 00:05:43,750 completely breaks my brain. The third pillar 118 00:05:43,750 --> 00:05:46,430 is goal setting. This is the big one. Right. 119 00:05:46,629 --> 00:05:49,110 The AI is never told what the objective actually 120 00:05:49,110 --> 00:05:52,250 is. There is no text prompt saying, destroy all 121 00:05:52,250 --> 00:05:54,949 the yellow blocks. That is easily the most aggressive 122 00:05:54,949 --> 00:05:57,009 design choice in the entire framework. It's crazy. 123 00:05:57,209 --> 00:06:00,209 the agent receives absolutely no explicit instructions. 124 00:06:00,810 --> 00:06:03,050 It must independently determine the win condition 125 00:06:03,050 --> 00:06:05,509 based solely on the environment's initial state 126 00:06:05,509 --> 00:06:07,750 and the feedback it gets from interacting with 127 00:06:07,750 --> 00:06:10,829 it. It literally has to infer what to target. 128 00:06:11,350 --> 00:06:13,709 But wait, if there are no instructions, how do 129 00:06:13,709 --> 00:06:16,509 we know the AI isn't just randomly clicking around? 130 00:06:17,589 --> 00:06:19,730 That's the first question everyone asks. Right. 131 00:06:20,189 --> 00:06:23,629 Because if I have a 64 by 64 grid and I just 132 00:06:23,629 --> 00:06:26,639 start firing off actions as fast as I can. Won't 133 00:06:26,639 --> 00:06:28,620 I eventually stumble into the wind condition 134 00:06:28,620 --> 00:06:32,060 by accident? It really feels like we are inviting 135 00:06:32,060 --> 00:06:34,639 algorithmic brute force here. And the ARC engineering 136 00:06:34,639 --> 00:06:37,519 team anticipated that exact vulnerability. They 137 00:06:37,519 --> 00:06:40,000 had to. Yeah, because if a benchmark could be 138 00:06:40,000 --> 00:06:42,879 beaten by a random walk, it completely ceases 139 00:06:42,879 --> 00:06:45,040 to be a measure of intelligence. It's just a 140 00:06:45,040 --> 00:06:47,279 slot machine. And that leads us to the fourth 141 00:06:47,279 --> 00:06:50,500 pillar, planning and execution. OK, so how does 142 00:06:50,500 --> 00:06:52,759 that solve the random clicking? Well, once the 143 00:06:52,759 --> 00:06:55,839 agent infers the goal, it must strategically 144 00:06:55,839 --> 00:06:58,680 map an action path to get there, correcting its 145 00:06:58,680 --> 00:07:01,500 course based on intermediate feedback. So let's 146 00:07:01,500 --> 00:07:03,579 look at the data set architecture. It's split 147 00:07:03,579 --> 00:07:07,600 into 135 total environments, 25 public demonstration 148 00:07:07,600 --> 00:07:09,939 environments that literally anyone can go look 149 00:07:09,939 --> 00:07:13,019 at. Then there are 55 semi -private environments, 150 00:07:13,519 --> 00:07:15,980 which are used for testing models via an API 151 00:07:15,980 --> 00:07:18,300 without giving them the underlying code. Right. 152 00:07:18,379 --> 00:07:21,300 Keep them honest. And finally, 55 fully private 153 00:07:21,300 --> 00:07:23,399 environments that are heavily guarded for their 154 00:07:23,399 --> 00:07:26,420 $2 million competition. But for any of these 155 00:07:26,420 --> 00:07:28,759 environments to make the final cut, they had 156 00:07:28,759 --> 00:07:31,480 to pass an exhaustive human calibration process. 157 00:07:31,819 --> 00:07:33,660 Yeah, I read about this. They actually set up 158 00:07:33,660 --> 00:07:36,040 a physical testing center in San Francisco. They 159 00:07:36,040 --> 00:07:39,040 did. They brought in 486 diverse participants 160 00:07:39,040 --> 00:07:42,139 who logged nearly 430 hours of continuous play. 161 00:07:42,600 --> 00:07:44,959 And the inclusion criteria was absolutely brutal. 162 00:07:45,220 --> 00:07:47,560 An environment was only accepted if it was deemed 163 00:07:47,560 --> 00:07:51,379 100 percent human solvable. Now we need to define 164 00:07:51,379 --> 00:07:54,120 100 percent human solvable really accurately 165 00:07:54,120 --> 00:07:56,060 here, because the phrasing alone can be a bit 166 00:07:56,060 --> 00:07:58,060 misleading. Right. It doesn't mean every single 167 00:07:58,060 --> 00:08:00,439 person beat it instantly. No, it does not imply 168 00:08:00,439 --> 00:08:02,660 that every single participant who sat down solved 169 00:08:02,660 --> 00:08:04,680 every single environment instantly on the first 170 00:08:04,680 --> 00:08:08,600 try. The specific rigorous metric is that at 171 00:08:08,600 --> 00:08:11,139 least two out of ten independent human testers 172 00:08:11,139 --> 00:08:13,779 had to successfully complete all levels of that 173 00:08:13,779 --> 00:08:15,959 environment on their very first attempt. Okay, 174 00:08:16,079 --> 00:08:18,980 two out of ten. Yes. And they had to do it entirely 175 00:08:18,980 --> 00:08:21,819 blind with zero prior training or instruction. 176 00:08:22,540 --> 00:08:25,040 But getting back to my earlier concern, human 177 00:08:25,040 --> 00:08:27,279 solvability doesn't physically prevent an AI 178 00:08:27,279 --> 00:08:29,740 from just mashing buttons. How did the validation 179 00:08:29,740 --> 00:08:31,959 engine actually stop random luck from winning? 180 00:08:32,190 --> 00:08:34,870 They utilized an automated exploratory state 181 00:08:34,870 --> 00:08:38,029 space analysis. OK, big words. Basically, effectively 182 00:08:38,029 --> 00:08:40,730 modeling every single environment as an explicit 183 00:08:40,730 --> 00:08:43,370 directed graph. Let's unpack the graph concept 184 00:08:43,370 --> 00:08:45,309 for a second. We are talking about nodes and 185 00:08:45,309 --> 00:08:48,750 edges, right? Correct. In computer science, a 186 00:08:48,750 --> 00:08:51,789 directed graph maps relationships between different 187 00:08:51,789 --> 00:08:54,909 states. In this validation pipeline, every single 188 00:08:54,909 --> 00:08:58,590 unique state the 64 by 64 grid can possibly be 189 00:08:58,590 --> 00:09:02,220 in is represented as a node. A snapshot of the 190 00:09:02,220 --> 00:09:05,419 grid. Yes. And the actions a player takes, like 191 00:09:05,419 --> 00:09:08,360 moving a block left or clicking a specific coordinate, 192 00:09:08,879 --> 00:09:11,679 are the edges that connect one node to the next. 193 00:09:12,179 --> 00:09:14,740 So it is mapping up the entire multiverse of 194 00:09:14,740 --> 00:09:17,500 the game. It's a great way to put it. If I start 195 00:09:17,500 --> 00:09:20,080 at node A, and I have five possible actions, 196 00:09:20,259 --> 00:09:22,500 those are five edges leading to five totally 197 00:09:22,500 --> 00:09:25,820 new universes, or nodes. And from each of those, 198 00:09:26,179 --> 00:09:28,750 five more edges. You're essentially building 199 00:09:28,750 --> 00:09:31,309 an enormous branching tree of every possible 200 00:09:31,309 --> 00:09:33,929 future in the game. And you can imagine the state 201 00:09:33,929 --> 00:09:36,649 space explosion is just astronomical. It has 202 00:09:36,649 --> 00:09:40,409 to be. A 64 by 64 grid with 16 colors has more 203 00:09:40,409 --> 00:09:42,990 possible configurations than there are atoms 204 00:09:42,990 --> 00:09:46,730 in the universe. Oh, wow. So the validation pipeline 205 00:09:46,730 --> 00:09:48,850 obviously doesn't map the entire mathematical 206 00:09:48,850 --> 00:09:52,190 space. It maps the reachable graph within finite 207 00:09:52,190 --> 00:09:54,730 budgets. Like putting a step limit on it. Exactly. 208 00:09:55,169 --> 00:09:57,909 A maximum step limit. The acceptance threshold 209 00:09:57,909 --> 00:10:00,649 for an environment was that a random policy, 210 00:10:00,769 --> 00:10:03,289 meaning an agent just executing actions with 211 00:10:03,289 --> 00:10:05,649 completely uniform probability, basically guessing, 212 00:10:06,429 --> 00:10:08,330 should have an estimated success rate of less 213 00:10:08,330 --> 00:10:11,850 than 1 in 10 ,000 on the core evaluation levels. 214 00:10:12,289 --> 00:10:14,529 Less than 1 in 10 ,000. That's a solid buffer. 215 00:10:14,809 --> 00:10:17,830 It is. Now, editorially, we should note it is 216 00:10:17,830 --> 00:10:21,070 an empirical engineering estimate, not a universal 217 00:10:21,070 --> 00:10:23,919 mathematical impossibility theorem. Right. In 218 00:10:23,919 --> 00:10:25,940 fact, the report notes that on the intentionally 219 00:10:25,940 --> 00:10:29,240 easy tutorial levels, random agents will occasionally 220 00:10:29,240 --> 00:10:32,019 stumble into success. Which makes sense. Tutorials 221 00:10:32,019 --> 00:10:35,200 are usually just one or two steps. Exactly. And 222 00:10:35,200 --> 00:10:37,720 that is considered acceptable to help orient 223 00:10:37,720 --> 00:10:40,480 the player early on. But the evaluation doesn't 224 00:10:40,480 --> 00:10:42,820 stop at a simple pass or fail, does it? The way 225 00:10:42,820 --> 00:10:46,240 they score the AI is incredibly aggressive. Oh, 226 00:10:46,259 --> 00:10:49,980 it's brutal. The scoring metric is called RHAE. 227 00:10:50,059 --> 00:10:53,600 R -H -A -E. Yes, which stands for relative human 228 00:10:53,600 --> 00:10:57,100 action efficiency. Under the ARC -AGI -3 philosophy, 229 00:10:57,519 --> 00:11:00,379 intelligence is not simply task completion. Just 230 00:11:00,379 --> 00:11:02,779 getting to the end isn't enough. No. Intelligence 231 00:11:02,779 --> 00:11:05,720 is defined strictly as efficiency. It is the 232 00:11:05,720 --> 00:11:07,940 ability to minimize the number of actions required 233 00:11:07,940 --> 00:11:11,179 to map the environment and reach the goal. Action 234 00:11:11,179 --> 00:11:14,679 efficiency. Yes. So they are literally measuring 235 00:11:14,679 --> 00:11:17,019 how much you had to flail around before the light 236 00:11:17,019 --> 00:11:18,919 bulb finally went off in your head? Basically, 237 00:11:19,080 --> 00:11:21,580 yes. The rationale is that a system that tries 238 00:11:21,580 --> 00:11:24,179 a hundred different incorrect hypotheses before 239 00:11:24,179 --> 00:11:27,120 finally stumbling on the right one is fundamentally 240 00:11:27,120 --> 00:11:29,740 displaying less fluid intelligence than a system 241 00:11:29,740 --> 00:11:32,399 that observes the environment, forms a correct 242 00:11:32,399 --> 00:11:35,419 mental model immediately, and executes a precise 243 00:11:35,419 --> 00:11:38,100 10 -step plan. Which is very logical. To quantify 244 00:11:38,100 --> 00:11:41,100 this, the AI's action count is compared directly 245 00:11:41,100 --> 00:11:43,299 against a human baseline, and they don't just 246 00:11:43,299 --> 00:11:45,740 use the average human. No. They set the baseline 247 00:11:45,740 --> 00:11:48,379 at the upper -median best human on their very 248 00:11:48,379 --> 00:11:52,409 first run. The upper median. Yes. But the actual 249 00:11:52,409 --> 00:11:54,509 math of the score, this specific formula they 250 00:11:54,509 --> 00:11:57,190 use, is where it gets really punitive for the 251 00:11:57,190 --> 00:11:59,370 AI. Oh, yeah. Let's say the human baseline is 252 00:11:59,370 --> 00:12:01,990 10 actions. The AI completes the level, but, 253 00:12:01,990 --> 00:12:03,750 you know, it takes 100 actions to get there. 254 00:12:04,230 --> 00:12:07,090 The AI took 10 times as many moves. In a normal 255 00:12:07,090 --> 00:12:09,990 test, that's just 10 % efficiency, right? Yes. 256 00:12:10,210 --> 00:12:13,450 But under RHAE, they apply a power law formula. 257 00:12:13,710 --> 00:12:16,009 OK. What does that do? They take that raw efficiency 258 00:12:16,009 --> 00:12:18,769 ratio, so the human's 10 actions divided by the 259 00:12:18,769 --> 00:12:22,330 AI's 100 actions, which gives you 0 .1, and they 260 00:12:22,330 --> 00:12:25,830 square it. They square the ratio. Yes. So 0 .1 261 00:12:25,830 --> 00:12:29,350 squared becomes 0 .01. The AI score for that 262 00:12:29,350 --> 00:12:32,350 level drops to exactly 1%. Squaring the ratio 263 00:12:32,350 --> 00:12:35,549 is absolutely brutal. A linear score gives the 264 00:12:35,549 --> 00:12:39,809 AI 10%. Squaring it literally vaporizes the score. 265 00:12:40,139 --> 00:12:42,820 And they apply this brutal math on a per level 266 00:12:42,820 --> 00:12:45,000 basis, right, rather than looking at the total 267 00:12:45,000 --> 00:12:47,580 actions across the whole environment? They do, 268 00:12:47,779 --> 00:12:50,740 yes. Why slice it up by level like that? It's 269 00:12:50,740 --> 00:12:52,879 primarily a noise reduction technique. Okay. 270 00:12:53,399 --> 00:12:55,220 Think about it, an environment might consist 271 00:12:55,220 --> 00:12:58,340 of five levels. Level one might require only 272 00:12:58,340 --> 00:13:01,340 five actions, while a massive complex level four 273 00:13:01,340 --> 00:13:04,519 might require 80 actions. Right. If you aggregate 274 00:13:04,519 --> 00:13:07,840 the raw actions globally, A slight inefficiency 275 00:13:07,840 --> 00:13:10,100 on that huge level 4 would mathematically swamp 276 00:13:10,100 --> 00:13:12,779 the signal of perfect efficiency on the earlier 277 00:13:12,779 --> 00:13:16,019 smaller levels. Oh, I see. It would drown out 278 00:13:16,019 --> 00:13:18,820 the good data. Exactly. By calculating the squared 279 00:13:18,820 --> 00:13:21,580 efficiency per level and then averaging those 280 00:13:21,580 --> 00:13:25,340 percentages, you get a much clearer, high -resolution 281 00:13:25,340 --> 00:13:28,440 diagnostic map of exactly where the agent's reasoning 282 00:13:28,440 --> 00:13:31,480 capacity breaks down. And they also weight the 283 00:13:31,480 --> 00:13:34,080 levels, right? So an easy tutorial level contributes 284 00:13:34,080 --> 00:13:36,279 less to the final score than the hardest level. 285 00:13:36,419 --> 00:13:39,059 Yes, weighted averaging. And they put a cap on 286 00:13:39,059 --> 00:13:41,360 the maximum score. But reading the technical 287 00:13:41,360 --> 00:13:44,440 report, I noticed a very strange mathematical 288 00:13:44,440 --> 00:13:46,879 contradiction regarding this specific cap. Ah, 289 00:13:46,940 --> 00:13:51,250 yes. The 115 % issue. Yeah. The prose in the 290 00:13:51,250 --> 00:13:53,870 original report explicitly states that the per 291 00:13:53,870 --> 00:13:56,970 -level score is capped at a maximum of 115%. 292 00:13:56,970 --> 00:13:59,210 However, if you look at the displayed mathematical 293 00:13:59,210 --> 00:14:02,830 formula in the paper itself, the 1 .15 value 294 00:14:02,830 --> 00:14:06,149 is placed inside the parentheses before the squaring 295 00:14:06,149 --> 00:14:09,029 function is actually applied. Which changes everything 296 00:14:09,029 --> 00:14:12,590 mathematically. Right. Because 1 .15 squared... 297 00:14:12,350 --> 00:14:15,990 yields approximately 1 .32. So the text claims 298 00:14:15,990 --> 00:14:20,690 a 115 % cap, while the formula outputs a 132 299 00:14:20,690 --> 00:14:23,710 % cap. That is a pretty significant typo for 300 00:14:23,710 --> 00:14:26,330 a rigorous technical paper. It is a discrepancy 301 00:14:26,330 --> 00:14:29,129 for sure, but the underlying intent of the authors 302 00:14:29,129 --> 00:14:31,850 remains completely clear here. What's the intent? 303 00:14:32,059 --> 00:14:35,179 The design allows AI models to receive a modest 304 00:14:35,179 --> 00:14:38,419 bonus for outperforming human efficiency while 305 00:14:38,419 --> 00:14:41,179 implementing a hard mathematical ceiling to prevent 306 00:14:41,179 --> 00:14:43,799 anomalous exploits from corrupting the aggregate 307 00:14:43,799 --> 00:14:47,759 metric. Right. So whether the cap is 115 or 132, 308 00:14:47,840 --> 00:14:50,240 the function is basically the same. It stops 309 00:14:50,240 --> 00:14:53,279 runaway exploit scores. Exactly. And honestly, 310 00:14:53,360 --> 00:14:55,279 the cap doesn't even matter right now because 311 00:14:55,279 --> 00:14:57,899 the frontier models are nowhere near it. No. 312 00:14:58,080 --> 00:15:01,000 Not even close. Looking at the March 2026 leaderboard 313 00:15:01,000 --> 00:15:04,080 in the report, the scores were abysmal. On the 314 00:15:04,080 --> 00:15:06,620 official semi -private evaluation, Anthropix 315 00:15:06,620 --> 00:15:12,419 Opus 4 .6 scored 0 .50%. Half a percent. Google's 316 00:15:12,419 --> 00:15:18,200 Gemini 3 .1 Pro Preview scored 0 .40%. GPT 5 317 00:15:18,200 --> 00:15:23,419 .4 High scored 0 .20%. They're virtually at zero. 318 00:15:23,570 --> 00:15:25,730 That is incredible to think about given how much 319 00:15:25,730 --> 00:15:28,429 hype surrounds these models. This is Thomas Kwa's 320 00:15:28,429 --> 00:15:31,470 research note from METER regarding the time horizon 321 00:15:31,470 --> 00:15:35,009 of AI models. Time horizon. Let's define this 322 00:15:35,009 --> 00:15:36,710 really carefully because it is one of the most 323 00:15:36,710 --> 00:15:39,370 widely misunderstood metrics in the entire industry. 324 00:15:39,710 --> 00:15:42,830 Oh, definitely. When a CEO hears that an AI has 325 00:15:42,830 --> 00:15:46,129 a time horizon of five hours, they almost always 326 00:15:46,129 --> 00:15:48,789 assume it means continuous runtime. Right. They 327 00:15:48,789 --> 00:15:50,629 think of it like a battery life. Exactly. They 328 00:15:50,629 --> 00:15:52,789 think, oh, I can spin up this AI, let it run 329 00:15:52,789 --> 00:15:54,809 autonomously on a server for five hours, and 330 00:15:54,809 --> 00:15:56,710 then I need a human to check on it before it 331 00:15:56,710 --> 00:15:59,389 hallucinates or crashes. But that interpretation 332 00:15:59,389 --> 00:16:01,610 is fundamentally incorrect. Completely wrong. 333 00:16:02,000 --> 00:16:04,659 Time horizon has literally nothing to do with 334 00:16:04,659 --> 00:16:07,500 the continuous runtime of the GPU. It is a metric 335 00:16:07,500 --> 00:16:10,320 of human labor replacement. OK, say more about 336 00:16:10,320 --> 00:16:13,019 that. Specifically, time horizon is defined as 337 00:16:13,019 --> 00:16:16,279 the maximum length of serial human labor that 338 00:16:16,279 --> 00:16:19,620 an AI can successfully complete with a 50 % reliability 339 00:16:19,620 --> 00:16:23,159 rate. And it is calibrated strictly by the time 340 00:16:23,159 --> 00:16:25,679 human experts traditionally need for that task. 341 00:16:25,960 --> 00:16:28,759 It anchors AI capability to the human economic 342 00:16:28,759 --> 00:16:31,490 baseline. And when Metter began testing this 343 00:16:31,490 --> 00:16:34,110 across different domains, they found a massive 344 00:16:34,110 --> 00:16:36,850 divergence in capability. So it's not universally 345 00:16:36,850 --> 00:16:38,669 good at everything. Where does it break down? 346 00:16:38,950 --> 00:16:41,549 Well, in domains strictly bounded by text and 347 00:16:41,549 --> 00:16:43,990 code, like software engineering or mathematical 348 00:16:43,990 --> 00:16:47,070 proofs, the time horizons are relatively consistent, 349 00:16:47,409 --> 00:16:50,190 and they are growing rapidly. Because it's all 350 00:16:50,190 --> 00:16:53,909 text. Right. However, when the tasks shift to 351 00:16:53,909 --> 00:16:57,230 visual computer use, The horizon completely collapses. 352 00:16:57,669 --> 00:17:00,929 Meta observed a 40x to 100x drop in performance 353 00:17:00,929 --> 00:17:04,089 on tasks, requiring the AI to perceive and interact 354 00:17:04,089 --> 00:17:07,130 with graphical user interfaces. But even when 355 00:17:07,130 --> 00:17:09,650 we stick to pure software engineering, Kwan's 356 00:17:09,650 --> 00:17:12,369 note admits that measuring this time horizon 357 00:17:12,369 --> 00:17:15,349 is incredibly noisy. The statistical reality 358 00:17:15,349 --> 00:17:18,049 of measuring time horizon is incredibly messy 359 00:17:18,049 --> 00:17:21,599 right now. Yeah. In the report, METR states that 360 00:17:21,599 --> 00:17:25,019 Claude Opus 4 .5 has a 50 % time horizon of 4 361 00:17:25,019 --> 00:17:27,980 hours and 49 minutes. But people read that exact 362 00:17:27,980 --> 00:17:30,960 number in a headline and completely ignore the 363 00:17:30,960 --> 00:17:33,900 error bars. The confidence interval is terrifying. 364 00:17:34,480 --> 00:17:37,559 I looked at the chart. The 95 % confidence interval 365 00:17:37,559 --> 00:17:41,039 for that 4 hour 49 minute horizon spans from 366 00:17:41,039 --> 00:17:44,619 1 hour 49 minutes all the way up to 20 hours 367 00:17:44,619 --> 00:17:48,819 25 minutes. That massive variance. is an unavoidable 368 00:17:48,819 --> 00:17:50,720 consequence of their evaluation architecture. 369 00:17:51,680 --> 00:17:53,960 METR calculates the confidence interval using 370 00:17:53,960 --> 00:17:57,039 a statistical method called bootstrapping. Okay, 371 00:17:57,059 --> 00:17:59,400 I've heard that term, but how does bootstrapping 372 00:17:59,400 --> 00:18:02,740 actually generate that massive error range? Bootstrapping 373 00:18:02,740 --> 00:18:05,900 is essentially a resampling technique. Kwa explicitly 374 00:18:05,900 --> 00:18:08,119 states in the note that he cannot definitively 375 00:18:08,119 --> 00:18:11,279 say whether the true underlying horizon is 3 376 00:18:11,279 --> 00:18:15,880 .5 hours or 6 .5 hours. The statistical resolution 377 00:18:15,880 --> 00:18:19,450 simply isn't there yet. with only 170 data points 378 00:18:19,450 --> 00:18:23,700 spanning such widely varying difficulties. And 379 00:18:23,700 --> 00:18:26,299 this statistical noise creates a massive hazard 380 00:18:26,299 --> 00:18:28,740 for businesses, right? Because they are trying 381 00:18:28,740 --> 00:18:31,900 to calculate ROI. If an operations manager reads 382 00:18:31,900 --> 00:18:35,019 that a model has a five -hour time horizon, their 383 00:18:35,019 --> 00:18:37,140 immediate instinct is to look at a five -hour 384 00:18:37,140 --> 00:18:40,400 task, deploy the AI, and just magically subtract 385 00:18:40,400 --> 00:18:43,019 five hours from their payroll budget. And that 386 00:18:43,019 --> 00:18:45,599 assumption violates what the industry calls the 387 00:18:45,599 --> 00:18:48,769 uplift equation. The uplift equation. Yes. Time 388 00:18:48,769 --> 00:18:51,269 horizon cannot be treated as a direct one -to 389 00:18:51,269 --> 00:18:53,829 -one subtraction of human labor. Calculating 390 00:18:53,829 --> 00:18:56,650 real productivity uplift requires modeling the 391 00:18:56,650 --> 00:18:59,369 entire workflow ecosystem. It's never just the 392 00:18:59,369 --> 00:19:01,670 task itself. No. You have to account for the 393 00:19:01,670 --> 00:19:03,589 human time spent engineering the prompt in the 394 00:19:03,589 --> 00:19:06,089 first place, the human time spent waiting for 395 00:19:06,089 --> 00:19:08,269 the multi -step generation to actually execute, 396 00:19:08,289 --> 00:19:11,289 and most critically, the human time required 397 00:19:11,289 --> 00:19:14,529 to verify the output. Therefore, a 50 % success 398 00:19:14,529 --> 00:19:17,190 rate on a five -hour task might actually generate 399 00:19:17,190 --> 00:19:20,529 a net negative productivity yield. The economics 400 00:19:20,529 --> 00:19:23,170 of automation are highly non -linear, and they 401 00:19:23,170 --> 00:19:25,329 are highly dependent on the cost of verification. 402 00:19:25,710 --> 00:19:28,269 This brings up a critical methodological point 403 00:19:28,269 --> 00:19:30,490 that Qua highlights regarding how they calculate 404 00:19:30,490 --> 00:19:33,089 the human baseline itself. Ah, the geometric 405 00:19:33,089 --> 00:19:36,630 mean. Yes. When METRA has multiple human experts 406 00:19:36,630 --> 00:19:38,730 complete the same benchmark tasks, they don't 407 00:19:38,730 --> 00:19:41,390 just add up the times and find the standard average. 408 00:19:41,529 --> 00:19:44,190 They use a specific statistical calculation called 409 00:19:44,190 --> 00:19:47,450 the geometric mean instead of the normal arithmetic 410 00:19:47,450 --> 00:19:50,670 mean. But there is an even more complex statistical 411 00:19:50,670 --> 00:19:54,009 artifact in how MET models the time horizon curve 412 00:19:54,009 --> 00:19:57,490 that can create incredibly counterintuitive results. 413 00:19:57,930 --> 00:20:00,470 We are getting into the deep statistical weeds 414 00:20:00,470 --> 00:20:03,170 here, and I love it. So Qua explains that they 415 00:20:03,170 --> 00:20:05,750 don't just look at the 50 % pass rate, right? 416 00:20:06,509 --> 00:20:08,630 Right. They are trying to map the entire capability 417 00:20:08,630 --> 00:20:11,490 curve of the model from the 80 % horizon, which 418 00:20:11,490 --> 00:20:13,869 are the easy tasks, the model almost always gets 419 00:20:13,869 --> 00:20:17,369 right, down to the 20 % horizon, which are the 420 00:20:17,369 --> 00:20:19,809 highly complex tasks the model usually fails. 421 00:20:20,430 --> 00:20:23,740 How do they actually model that curve? They fit 422 00:20:23,740 --> 00:20:26,700 a two -parameter logistic model to the pass -fail 423 00:20:26,700 --> 00:20:29,720 data across all the tasks. Logistic model. What 424 00:20:29,720 --> 00:20:32,279 does that look like? It generates an S -shaped 425 00:20:32,279 --> 00:20:35,680 curve. It starts flat at the bottom, accelerates 426 00:20:35,680 --> 00:20:37,980 steeply upward through the middle, and flattens 427 00:20:37,980 --> 00:20:40,900 out again at the top. The issue is that the entire 428 00:20:40,900 --> 00:20:44,359 shape of that S -curve is defined by only two 429 00:20:44,359 --> 00:20:46,859 mathematical parameters, essentially the midpoint 430 00:20:46,859 --> 00:20:49,640 and the slope. And because it's only two parameters, 431 00:20:49,799 --> 00:20:52,339 the top of the S -curve and the bottom of the 432 00:20:52,339 --> 00:20:54,319 S -curve are mathematically locked together. 433 00:20:54,480 --> 00:20:58,039 They are coupled. Yes. Qua outlines the statistical 434 00:20:58,039 --> 00:21:00,799 requirements for measuring 99 % reliability. 435 00:21:01,720 --> 00:21:03,680 To have any real confidence in that quantile, 436 00:21:03,720 --> 00:21:06,799 you would need at least 300 highly diverse tasks 437 00:21:06,799 --> 00:21:09,880 concentrated in every single specific time bucket 438 00:21:09,880 --> 00:21:14,109 you want to measure. 300 tasks per time bucket. 439 00:21:14,289 --> 00:21:17,029 That's huge. That's huge. And furthermore, the 440 00:21:17,029 --> 00:21:19,369 label noise. The inherent error rate and how 441 00:21:19,369 --> 00:21:22,150 the benchmark verifies the answer must be strictly 442 00:21:22,150 --> 00:21:25,730 less than 1%. Label noise meaning the test itself 443 00:21:25,730 --> 00:21:28,450 is just wrong, like a unit test that is way too 444 00:21:28,450 --> 00:21:30,650 strict or a prompt that is somehow ambiguous. 445 00:21:30,910 --> 00:21:33,950 Yes. If the benchmark's grading system has a 446 00:21:33,950 --> 00:21:36,390 2 % error rate where it accidentally marks a 447 00:21:36,390 --> 00:21:38,990 correct AI solution as incorrect 2 % of the time, 448 00:21:39,750 --> 00:21:42,170 the absolute maximum measured capability of any 449 00:21:42,170 --> 00:21:46,089 model will mathematically saturate it 98%. Oh, 450 00:21:46,089 --> 00:21:51,329 wow. Even an omniscient, perfect AGI would evaluate 451 00:21:51,329 --> 00:21:54,730 as having a 99 % time horizon of exactly zero 452 00:21:54,730 --> 00:21:57,549 hours under those conditions. You simply cannot 453 00:21:57,549 --> 00:22:00,150 measure a signal cleaner than the noise floor 454 00:22:00,150 --> 00:22:02,690 of your instrument. The noise is large, yes, 455 00:22:02,950 --> 00:22:05,789 but the signal of capability improvement is significantly 456 00:22:05,789 --> 00:22:09,190 larger. And that leads to the 2030 extrapolation. 457 00:22:09,289 --> 00:22:11,849 This is the part everyone talks about. If you 458 00:22:11,849 --> 00:22:14,450 track the historical progression of the 50 % 459 00:22:14,450 --> 00:22:16,789 time horizon across all the frontier models, 460 00:22:17,299 --> 00:22:20,359 What is the velocity? The long run trend indicates 461 00:22:20,359 --> 00:22:22,880 that the time horizon is doubling approximately 462 00:22:22,880 --> 00:22:26,140 every six to seven months. Doubling every six 463 00:22:26,140 --> 00:22:28,900 months. That's incredibly fast. It is. And if 464 00:22:28,900 --> 00:22:31,700 you apply a linear extrapolation to that exponential 465 00:22:31,700 --> 00:22:35,519 growth rate, it projects that a frontier AI model 466 00:22:35,519 --> 00:22:38,220 will reach a one month time horizon by the year 467 00:22:38,220 --> 00:22:42,369 2030. A one month horizon. meaning the model 468 00:22:42,369 --> 00:22:45,930 could reliably, with a 50 % success rate, execute 469 00:22:45,930 --> 00:22:50,569 167 contiguous hours of expert human labor. Right. 470 00:22:50,849 --> 00:22:53,869 The METR benchmark tasks are currently entirely 471 00:22:53,869 --> 00:22:57,250 self -contained. The current METR suite evaluates 472 00:22:57,250 --> 00:22:59,650 absolutely none of that collaborative multi -turn 473 00:22:59,650 --> 00:23:02,710 organizational friction. The 2030 projection 474 00:23:02,710 --> 00:23:05,809 is highly, highly conditional. Neither metric 475 00:23:05,809 --> 00:23:08,369 alone provides a universal definitive ranking 476 00:23:08,369 --> 00:23:10,940 of AGI. You can mathematically aggregate them 477 00:23:10,940 --> 00:23:12,960 if you wish to create a composite score, sure, 478 00:23:13,079 --> 00:23:16,019 but doing so completely obscures the very real 479 00:23:16,019 --> 00:23:17,940 divergence in underlying capabilities. Yeah, 480 00:23:18,059 --> 00:23:20,160 that makes sense. A model can be an absolute 481 00:23:20,160 --> 00:23:22,539 virtuoso at one while remaining functionally 482 00:23:22,539 --> 00:23:24,740 incompetent at the other. And that divergence 483 00:23:24,740 --> 00:23:26,980 is exactly what I want to leave you, our listener, 484 00:23:27,079 --> 00:23:29,220 thinking about today because it's a wild thought 485 00:23:29,220 --> 00:23:32,200 experiment. Why might performance skyrocket on 486 00:23:32,200 --> 00:23:34,599 one measure while remaining dead in the water 487 00:23:34,599 --> 00:23:37,380 on the other? It's the ultimate paradox. What 488 00:23:37,380 --> 00:23:39,599 would that divergence reveal about the true nature 489 00:23:39,599 --> 00:23:41,000 of the intelligence we have built?