1 00:00:00,000 --> 00:00:02,680 So we are constantly being fed this one very 2 00:00:02,680 --> 00:00:04,960 specific narrative about artificial intelligence 3 00:00:04,960 --> 00:00:07,040 right now. Oh, absolutely. It's everywhere you 4 00:00:07,040 --> 00:00:09,880 look. Right. It goes something like this. AI 5 00:00:09,880 --> 00:00:12,320 is getting smarter simply by reading more of 6 00:00:12,320 --> 00:00:15,019 the internet. It consumes more books, more articles, 7 00:00:15,160 --> 00:00:17,879 more forum posts, and it just magically gets 8 00:00:17,879 --> 00:00:20,320 better. Just massive amounts of data. Exactly. 9 00:00:20,699 --> 00:00:23,440 But here is the question that has been keeping 10 00:00:23,440 --> 00:00:26,239 me up at night. And I mean, if you follow this 11 00:00:26,239 --> 00:00:28,039 space, maybe you have wondered about it too. 12 00:00:28,359 --> 00:00:32,179 What happens? when the machine has read everything. 13 00:00:32,600 --> 00:00:34,500 Well, that is the ultimate bottleneck, isn't 14 00:00:34,500 --> 00:00:36,899 it? Yeah. When a system reaches the absolute 15 00:00:36,899 --> 00:00:39,920 edge of the human library, where does the new 16 00:00:39,920 --> 00:00:42,100 knowledge actually come from? How does it learn 17 00:00:42,100 --> 00:00:43,880 something that no human has ever written down? 18 00:00:44,159 --> 00:00:46,439 And that question is really the entire engine 19 00:00:46,439 --> 00:00:49,659 of today's Deep Dive. We are looking at a truly 20 00:00:49,659 --> 00:00:52,079 fascinating position chapter, and it's titled, 21 00:00:52,520 --> 00:00:55,520 Welcome to the Era of experience. It's a fantastic 22 00:00:55,520 --> 00:00:57,679 piece of work. It really is. It was written by 23 00:00:57,679 --> 00:01:00,859 David Silver and Richard S. Sutton, who are two 24 00:01:00,859 --> 00:01:03,240 very prominent researchers in the field. Definitely 25 00:01:03,240 --> 00:01:05,620 heavy hitters. Right. And they are essentially 26 00:01:05,620 --> 00:01:07,640 arguing that artificial intelligence is standing 27 00:01:07,640 --> 00:01:10,819 on the threshold of this massive paradigm shift. 28 00:01:11,260 --> 00:01:14,280 Like we are supposedly transitioning away from 29 00:01:14,280 --> 00:01:17,099 just feeding AI static data that humans generated 30 00:01:17,099 --> 00:01:19,560 in the past and moving towards something they 31 00:01:19,560 --> 00:01:22,959 call continuous interactive experience. Though, 32 00:01:23,040 --> 00:01:24,900 to be clear from the jump, Sutton and Silver 33 00:01:24,900 --> 00:01:27,060 aren't claiming they have this totally solved, 34 00:01:27,140 --> 00:01:29,239 you know, boxed up and ready to sell tomorrow. 35 00:01:29,859 --> 00:01:32,579 This chapter is really a roadmap. It is a proposed 36 00:01:32,579 --> 00:01:34,700 research direction that looks at the massive 37 00:01:34,700 --> 00:01:36,980 unsolved hurdles we have to jump before we get 38 00:01:36,980 --> 00:01:39,519 to truly autonomous systems. Which is what we're 39 00:01:39,519 --> 00:01:41,980 going to get into today. Exactly. We are going 40 00:01:41,980 --> 00:01:44,239 to explore their arguments, but we are also going 41 00:01:44,239 --> 00:01:47,060 to look at the open problems and the real mathematical 42 00:01:47,060 --> 00:01:49,640 limitations of this vision. Because whether you 43 00:01:49,640 --> 00:01:51,859 are a developer listening to this on your commute 44 00:01:51,859 --> 00:01:54,379 or you are just kind of curious about why the 45 00:01:54,379 --> 00:01:56,480 apps on your phone keep changing how they act 46 00:01:56,480 --> 00:02:00,120 You really need a clear map of where the architecture 47 00:02:00,120 --> 00:02:02,180 is heading. Absolutely and we want to give you 48 00:02:02,180 --> 00:02:05,040 that map So let's start by defining the baseline 49 00:02:05,230 --> 00:02:08,409 Silver and Sutton call our current moment the 50 00:02:08,409 --> 00:02:11,750 era of human data, right? So the era of human 51 00:02:11,750 --> 00:02:14,229 data is characterized by large language models 52 00:02:14,229 --> 00:02:18,469 or LLMs achieving this sweeping, almost shocking 53 00:02:18,469 --> 00:02:21,050 level of generality. It really was shocking for 54 00:02:21,050 --> 00:02:23,289 a lot of people. It was. Yeah. And they did this 55 00:02:23,289 --> 00:02:25,789 by training on massive data sets generated by 56 00:02:25,789 --> 00:02:28,509 us. The models basically scrape the text of the 57 00:02:28,509 --> 00:02:30,389 Internet, and then they are fine -tuned with 58 00:02:30,389 --> 00:02:32,669 expert human examples and preferences just to 59 00:02:32,669 --> 00:02:35,370 act helpful. Which is the AI that you and I interact 60 00:02:35,370 --> 00:02:38,030 with every single day. I mean, it writes poetry. 61 00:02:38,330 --> 00:02:40,770 It helps diagnose medical issues. It summarizes 62 00:02:40,770 --> 00:02:43,620 dense legal contracts. It's sub... brilliant 63 00:02:43,620 --> 00:02:46,840 imitator of human competence across a vast array 64 00:02:46,840 --> 00:02:49,740 of subjects. But the authors propose that extracting 65 00:02:49,740 --> 00:02:52,719 knowledge solely from our data is rapidly approaching 66 00:02:52,719 --> 00:02:55,879 a limit, particularly in rigorous domains like 67 00:02:55,879 --> 00:02:59,240 mathematics, coding, and empirical science. Because 68 00:02:59,240 --> 00:03:01,740 the highest quality human data, the kind that 69 00:03:01,740 --> 00:03:04,599 actually improves the performance of an already 70 00:03:04,599 --> 00:03:08,180 strong agent, is largely consumed. Let me push 71 00:03:08,180 --> 00:03:09,759 back on that for a second. Does this mean it 72 00:03:09,759 --> 00:03:12,300 is mathematically impossible for current LLMs? 73 00:03:12,449 --> 00:03:15,469 to discover new ideas or generalize beyond their 74 00:03:15,469 --> 00:03:18,330 examples? I mean, are they just text regurgitators? 75 00:03:18,969 --> 00:03:20,870 No, no, I want to add some vital nuance to that 76 00:03:20,870 --> 00:03:23,810 because that's a common misconception. The chapter 77 00:03:23,810 --> 00:03:26,509 does not prove a mathematical impossibility of 78 00:03:26,509 --> 00:03:28,610 discovering new results from human data. OK, 79 00:03:28,629 --> 00:03:31,189 good to clarify. Right. It does not claim that 80 00:03:31,189 --> 00:03:33,870 supervised models are mere text regurgitators 81 00:03:33,870 --> 00:03:36,650 that are completely incapable of generalization, 82 00:03:37,129 --> 00:03:39,750 nor does it claim that literally all high quality 83 00:03:39,750 --> 00:03:42,389 human data is exhausted down to the absolute 84 00:03:42,379 --> 00:03:45,000 last bite. I'm really glad you brought that up. 85 00:03:45,319 --> 00:03:47,340 It's not that current AI is fundamentally broken 86 00:03:47,340 --> 00:03:49,800 and can never combine two ideas to make a third. 87 00:03:50,539 --> 00:03:53,240 Exactly. And we are starting to see real world 88 00:03:53,240 --> 00:03:55,620 examples of this shift making headlines right 89 00:03:55,620 --> 00:03:57,819 now. Yeah, let's talk about alpha proof. That 90 00:03:57,819 --> 00:03:59,800 is a brilliant case study for this transition. 91 00:03:59,949 --> 00:04:02,990 Alpha proof was initially exposed to about 100 92 00:04:02,990 --> 00:04:05,210 ,000 formal proofs that have been created over 93 00:04:05,210 --> 00:04:07,710 many years by human mathematicians. But it didn't 94 00:04:07,710 --> 00:04:09,930 stop there. It used a reinforcement learning 95 00:04:09,930 --> 00:04:12,990 algorithm to subsequently generate 100 million 96 00:04:12,990 --> 00:04:15,830 more proofs through continual interaction with 97 00:04:15,830 --> 00:04:19,089 a formal proving system. Which is the experiential 98 00:04:19,089 --> 00:04:22,410 part. Reinforcement learning at its core is about 99 00:04:22,410 --> 00:04:25,649 an agent learning to achieve a goal by a trial 100 00:04:25,649 --> 00:04:27,709 and error within an environment. Can you break 101 00:04:27,709 --> 00:04:30,230 that down a bit more? Sure. It tries a behavior, 102 00:04:30,750 --> 00:04:33,029 and if the environment signals a positive outcome, 103 00:04:33,449 --> 00:04:36,129 the system mathematically reinforces the pathways 104 00:04:36,129 --> 00:04:39,250 that led to that decision. In Alpha Proof's case, 105 00:04:39,569 --> 00:04:42,750 the environment was a formal mathematical proving 106 00:04:42,750 --> 00:04:45,829 system. So it's basically throwing ideas at this 107 00:04:45,829 --> 00:04:48,509 system to see what sticks. Basically. It tried 108 00:04:48,509 --> 00:04:50,850 a logical step, and the system returned a signal, 109 00:04:51,209 --> 00:04:53,290 verifying whether the step was logically valid 110 00:04:53,290 --> 00:04:55,970 or invalid. Like a massive game of trial and 111 00:04:55,970 --> 00:04:58,470 error. Exactly. And through millions of these 112 00:04:58,470 --> 00:05:01,269 interactive loops it explored mathematical possibilities 113 00:05:01,269 --> 00:05:03,649 far beyond the confines of the original human 114 00:05:03,649 --> 00:05:07,100 proofs. That is just wild. And we have seen variations 115 00:05:07,100 --> 00:05:09,500 of this experiential approach elsewhere, too. 116 00:05:09,800 --> 00:05:11,339 I mean, DeepSeq's recent work comes to mind. 117 00:05:11,500 --> 00:05:13,860 Yes, but we have to be very careful with the 118 00:05:13,860 --> 00:05:16,720 DeepSeq examples. We have to separate DeepSeq 119 00:05:16,720 --> 00:05:20,360 R1 .0 from DeepSeq R1. Oh, good point. DeepSeq 120 00:05:20,360 --> 00:05:23,740 R1 .0 was a specific experiment relying on pure 121 00:05:23,740 --> 00:05:26,819 reinforcement learning without preliminary supervised 122 00:05:26,819 --> 00:05:29,420 fine -tuning on human data. It essentially developed 123 00:05:29,420 --> 00:05:31,720 problem -solving strategies autonomously from 124 00:05:31,720 --> 00:05:34,279 the jump. But that's not what their main model's 125 00:05:34,279 --> 00:05:36,470 doing. Right. Right. We cannot call all of their 126 00:05:36,470 --> 00:05:39,990 models pure RL. DeepSeq R1, which is their flagship 127 00:05:39,990 --> 00:05:42,769 model, uses multistage training that heavily 128 00:05:42,769 --> 00:05:45,509 incorporates human -derived data alongside reinforcement 129 00:05:45,509 --> 00:05:48,290 learning. So it's a mix. Yes. The overarching 130 00:05:48,290 --> 00:05:50,689 point Silver and Sutton are making is that the 131 00:05:50,689 --> 00:05:53,389 element of interactive, experiential learning, 132 00:05:53,910 --> 00:05:56,250 whether pure or incorporated into a multistage 133 00:05:56,250 --> 00:05:58,930 pipeline like R1, is what allows systems to push 134 00:05:58,930 --> 00:06:01,029 past the ceiling of human knowledge. Because 135 00:06:01,029 --> 00:06:03,750 generating a hundred million mathematical proofs 136 00:06:03,750 --> 00:06:06,189 over days requires a completely different relationship 137 00:06:06,189 --> 00:06:09,649 with time than what a standard chatbot has. Totally 138 00:06:09,649 --> 00:06:11,870 different. Think about the chatbots you use at 139 00:06:11,870 --> 00:06:14,629 home. You type a prompt, it spits out an answer, 140 00:06:14,790 --> 00:06:16,910 and it effectively goes back to sleep. It has 141 00:06:16,910 --> 00:06:19,750 no continuous existence. Right. Which brings 142 00:06:19,750 --> 00:06:22,329 us to the first major dimension of the era of 143 00:06:22,329 --> 00:06:25,029 experience. The authors call it the transition 144 00:06:25,029 --> 00:06:28,769 from snippets to streams. This is crucial. The 145 00:06:28,769 --> 00:06:31,610 current paradigm of human -centric AI operates 146 00:06:31,610 --> 00:06:35,410 entirely in isolated snippets. Yes. The agent 147 00:06:35,410 --> 00:06:38,089 is solely optimizing for the immediate outcome 148 00:06:38,089 --> 00:06:40,730 of directly answering your current prompt. It 149 00:06:40,730 --> 00:06:43,009 is a short interaction episode where little to 150 00:06:43,009 --> 00:06:45,829 no information carries over to the next, completely 151 00:06:45,829 --> 00:06:48,889 separate conversation. It reminds me of the movie 152 00:06:48,889 --> 00:06:52,110 Groundhog Day, but like even more extreme. That's 153 00:06:52,110 --> 00:06:54,290 a great way to put it. Because in Groundhog Day, 154 00:06:54,410 --> 00:06:57,009 Bill Murray wakes up The day resets, but he at 155 00:06:57,009 --> 00:07:00,269 least remembers yesterday. A standard LLM wakes 156 00:07:00,269 --> 00:07:02,910 up, answers your prompt, and then ceases to exist 157 00:07:02,910 --> 00:07:05,850 entirely. It isn't planning for next week. And 158 00:07:05,850 --> 00:07:09,290 humans and animals do not exist in isolated snippets. 159 00:07:09,389 --> 00:07:12,149 We exist in an ongoing stream of actions and 160 00:07:12,149 --> 00:07:16,490 observations that spans years. Exactly. If I 161 00:07:16,490 --> 00:07:18,910 decide to run a marathon, I might do a grueling 162 00:07:18,910 --> 00:07:21,649 training run today. That makes me exhausted and 163 00:07:21,649 --> 00:07:24,550 sore, which is a really negative immediate outcome. 164 00:07:24,730 --> 00:07:26,589 You wouldn't do it if you were only optimizing 165 00:07:26,589 --> 00:07:28,870 for today. Right. But I do it because I know 166 00:07:28,870 --> 00:07:31,069 it contributes to the long -term goal of finishing 167 00:07:31,069 --> 00:07:33,110 the race six months from now. And optimizing 168 00:07:33,110 --> 00:07:35,689 for delayed consequences is the absolute hallmark 169 00:07:35,689 --> 00:07:38,290 of natural intelligence. The authors forecast 170 00:07:38,290 --> 00:07:40,949 that powerful future agents will inhabit their 171 00:07:40,949 --> 00:07:44,050 own continuous streams of experience over long 172 00:07:44,050 --> 00:07:46,550 timescales. What does that actually look like 173 00:07:46,550 --> 00:07:48,889 in practice? Well, they provide some compelling 174 00:07:48,889 --> 00:07:52,009 forecast examples. Imagine a health and wellness 175 00:07:52,009 --> 00:07:54,769 agent. Instead of just answering a prompt about 176 00:07:54,769 --> 00:07:57,790 the calories in an apple, it monitors your sleep 177 00:07:57,790 --> 00:08:00,670 patterns, your activity levels, and your dietary 178 00:08:00,670 --> 00:08:03,170 habits via wearables over several months. So 179 00:08:03,170 --> 00:08:05,870 it's constantly watching and learning. Yes. It 180 00:08:05,870 --> 00:08:08,110 provides personalized recommendations that adapt 181 00:08:08,110 --> 00:08:11,250 based on long -term trends to hit a specific 182 00:08:11,250 --> 00:08:14,509 fitness milestone. Or like a science agent. The 183 00:08:14,509 --> 00:08:16,610 chapter mentions an AI working to reduce carbon 184 00:08:16,610 --> 00:08:18,290 dioxide in the atmosphere. That's a huge one. 185 00:08:18,829 --> 00:08:21,470 That requires analyzing real -world observations. 186 00:08:21,790 --> 00:08:25,230 developing complex atmospheric simulations, proposing 187 00:08:25,230 --> 00:08:27,949 real -world experiments, waiting for the results, 188 00:08:28,050 --> 00:08:30,970 and then iterating. That single task could take 189 00:08:30,970 --> 00:08:33,789 years of continuous operation. Decades even. 190 00:08:34,330 --> 00:08:36,809 And operating over years introduces incredibly 191 00:08:36,809 --> 00:08:38,990 difficult open problems in computer science. 192 00:08:39,570 --> 00:08:42,269 The chapter is explicitly calling for deep research 193 00:08:42,269 --> 00:08:44,909 into these areas. Like, what kind of problems? 194 00:08:45,710 --> 00:08:50,129 Consider temporal abstraction. How does an artificial 195 00:08:50,129 --> 00:08:52,610 neural network break a multi -year goal into 196 00:08:52,610 --> 00:08:55,549 actionable second -by -second computations without 197 00:08:55,549 --> 00:08:58,169 losing track of the overarching objective? That's 198 00:08:58,169 --> 00:08:59,590 mind -boggling. And then there's the problem 199 00:08:59,590 --> 00:09:01,669 of credit assignment. Yes, credit assignment 200 00:09:01,669 --> 00:09:04,370 is huge. I find this one fascinating. If you 201 00:09:04,370 --> 00:09:06,809 have that science agent running a carbon reduction 202 00:09:06,809 --> 00:09:09,190 experiment and the experiment finally succeeds 203 00:09:09,190 --> 00:09:13,389 on day 300, the AI has made millions of micro 204 00:09:13,389 --> 00:09:15,730 decisions along the way. Millions. How does the 205 00:09:15,730 --> 00:09:18,549 math actually look backwards through 300 days 206 00:09:18,549 --> 00:09:21,009 of data and decide which of those millions of 207 00:09:21,009 --> 00:09:24,009 choices gets the points for the success and which 208 00:09:24,009 --> 00:09:27,350 choices were just irrelevant? Exactly. It requires 209 00:09:27,350 --> 00:09:29,830 massive advancements in memory architectures 210 00:09:29,830 --> 00:09:32,289 and value functions. The system needs to maintain 211 00:09:32,289 --> 00:09:34,950 a continuous state representation and evaluate 212 00:09:34,950 --> 00:09:37,309 how every small action shifts the probability 213 00:09:37,309 --> 00:09:41,440 of achieving that distant goal. But wait, some 214 00:09:41,440 --> 00:09:43,700 folks listening might be wondering, current LLMs 215 00:09:43,700 --> 00:09:46,340 can already use memory. I mean, they have massive 216 00:09:46,340 --> 00:09:49,159 context windows now. You can upload a 500 -page 217 00:09:49,159 --> 00:09:51,440 PDF or refer to something you talked about two 218 00:09:51,440 --> 00:09:54,679 hours ago. That's true. So are Silver and Sutton 219 00:09:54,679 --> 00:09:56,960 saying current neural network architectures are 220 00:09:56,960 --> 00:09:58,840 fundamentally broken for this long -term stuff? 221 00:09:59,019 --> 00:10:02,159 Not at all. The authors explicitly propose building 222 00:10:02,159 --> 00:10:05,519 on today's technological foundations. OK. Existing 223 00:10:05,519 --> 00:10:07,639 language models, even with frozen parameters, 224 00:10:07,940 --> 00:10:10,299 meaning their core training weights aren't updating 225 00:10:10,299 --> 00:10:13,980 in real time, can absolutely use new observations. 226 00:10:14,190 --> 00:10:17,169 update their context windows, and utilize external 227 00:10:17,169 --> 00:10:19,710 memory databases. So the architecture isn't the 228 00:10:19,710 --> 00:10:22,309 problem. Right. The proposal is about shifting 229 00:10:22,309 --> 00:10:25,049 the focus of how we deploy them. Instead of training 230 00:10:25,049 --> 00:10:27,110 them solely to maximize the quality of their 231 00:10:27,110 --> 00:10:29,830 immediate next sentence, the objective function 232 00:10:29,830 --> 00:10:33,230 changes. The AI needs to optimize for delayed 233 00:10:33,230 --> 00:10:36,269 consequences. The architecture might look similar, 234 00:10:36,269 --> 00:10:39,649 but what the AI considers a win changes completely. 235 00:10:39,970 --> 00:10:42,210 That makes sense. And if an agent is going to 236 00:10:42,210 --> 00:10:44,809 achieve a long -term goal, like a multi -year 237 00:10:44,809 --> 00:10:47,950 science experiment, it cannot just sit in a server 238 00:10:47,950 --> 00:10:51,309 and talk to a user. It has to physically or digitally 239 00:10:51,309 --> 00:10:55,009 do something. Exactly. And the authors forecast 240 00:10:55,009 --> 00:10:58,389 that experiential agents will act autonomously 241 00:10:58,389 --> 00:11:01,450 using the exact same interfaces humans use. So 242 00:11:01,450 --> 00:11:03,429 they will need to operate software using graphical 243 00:11:03,429 --> 00:11:05,970 user interfaces, like actually clicking buttons 244 00:11:05,970 --> 00:11:09,230 on a screen. Yes. And autonomously call APIs 245 00:11:09,230 --> 00:11:11,529 across the web. They will also need to move into 246 00:11:11,529 --> 00:11:13,919 the physical world. monitoring environmental 247 00:11:13,919 --> 00:11:16,740 sensors in real time or controlling robotic arms 248 00:11:16,740 --> 00:11:19,419 and chemistry labs to mix compounds. I want to 249 00:11:19,419 --> 00:11:21,220 explore the coding aspect of this for a second 250 00:11:21,220 --> 00:11:24,360 because it highlights a really important distinction. 251 00:11:24,480 --> 00:11:27,639 Sure. If an AI writes a Python script, opens 252 00:11:27,639 --> 00:11:30,480 a terminal, runs the code, sees an error message 253 00:11:30,480 --> 00:11:33,820 and rewrites it until it works, that feels like 254 00:11:33,820 --> 00:11:37,039 a solid experience. But if it runs the code and 255 00:11:37,039 --> 00:11:40,039 it doesn't crash, does that actually prove the 256 00:11:40,039 --> 00:11:43,360 AI is smart? And the code is flawless? That is 257 00:11:43,360 --> 00:11:46,080 a critical question. Running a program and seeing 258 00:11:46,080 --> 00:11:48,820 it execute without crashing is merely a software 259 00:11:48,820 --> 00:11:52,159 test. Just a test. Right. A test only evaluates 260 00:11:52,159 --> 00:11:54,940 the specific cases and criteria it covers within 261 00:11:54,940 --> 00:11:57,879 that specific run. Give me an example. If I write 262 00:11:57,879 --> 00:12:00,200 code to sort a list of numbers and I test it 263 00:12:00,200 --> 00:12:02,700 with a list of five numbers, it might work perfectly. 264 00:12:03,139 --> 00:12:05,559 But a test doesn't mathematically prove it will 265 00:12:05,559 --> 00:12:07,940 work for a list of 10 billion numbers or that 266 00:12:07,940 --> 00:12:10,700 it won't crash under some unforeseen edge case. 267 00:12:10,879 --> 00:12:12,860 Right. It just proved it worked that one time 268 00:12:12,860 --> 00:12:16,009 with those five numbers. Contrast that with formal 269 00:12:16,009 --> 00:12:18,169 verification, which is what Alphaproof was doing. 270 00:12:18,429 --> 00:12:20,690 Exactly. Formal verification doesn't just run 271 00:12:20,690 --> 00:12:23,169 a test case. It mathematically proves that a 272 00:12:23,169 --> 00:12:25,590 statement is true relative to a specified set 273 00:12:25,590 --> 00:12:28,309 of definitions and a trusted checking system. 274 00:12:28,629 --> 00:12:31,669 It is completely exhaustive. Yes. But formal 275 00:12:31,669 --> 00:12:34,610 verification is mostly limited to abstract, rigorously 276 00:12:34,610 --> 00:12:37,309 defined domains like pure math or highly specialized 277 00:12:37,309 --> 00:12:40,289 software. And then there is the real world. which 278 00:12:40,289 --> 00:12:43,710 is entirely different from both a software test 279 00:12:43,710 --> 00:12:46,389 and a formal proof. Oh, the real world is messy. 280 00:12:46,789 --> 00:12:49,509 Extremely messy. A real -world outcome is noisy 281 00:12:49,509 --> 00:12:52,610 and full of hidden variables, like an AI robotic 282 00:12:52,610 --> 00:12:54,830 arm mixing chemicals in a lab might execute its 283 00:12:54,830 --> 00:12:57,470 code flawlessly, right? Yes. But if the ambient 284 00:12:57,470 --> 00:13:00,029 humidity in the room unexpectedly spikes, the 285 00:13:00,029 --> 00:13:02,610 chemical reaction might fail anyway. That real 286 00:13:02,610 --> 00:13:05,049 world measurement is a grounded, messy consequence. 287 00:13:05,330 --> 00:13:07,490 And active exploration in that messy environment 288 00:13:07,490 --> 00:13:09,950 allows the AI to discover strategies that a human 289 00:13:09,950 --> 00:13:11,789 would never even think to write down in a textbook. 290 00:13:12,090 --> 00:13:14,629 It's not limited by our imagination or our clean 291 00:13:14,629 --> 00:13:17,330 little formulas. And experience is not restricted 292 00:13:17,330 --> 00:13:19,549 strictly to the physical world either. No, not 293 00:13:19,549 --> 00:13:22,889 at all. Navigating a complex digital system autonomously, 294 00:13:23,129 --> 00:13:26,370 dealing with server latency, API changes, unexpected 295 00:13:26,370 --> 00:13:29,370 pop -ups. That is a rich, grounded experience 296 00:13:29,370 --> 00:13:32,110 that forces an agent to adapt in real time. So 297 00:13:32,110 --> 00:13:34,730 we have established an agent taking rich actions 298 00:13:34,730 --> 00:13:36,710 over a long stream of time to achieve a goal. 299 00:13:36,809 --> 00:13:39,509 Right. The massive challenge now is how the AI 300 00:13:39,509 --> 00:13:42,909 knows if it is actually succeeding. In the current 301 00:13:42,909 --> 00:13:45,690 paradigm, if a model writes an essay, a human 302 00:13:45,690 --> 00:13:48,450 user clicks a thumbs up or thumbs down icon. 303 00:13:49,070 --> 00:13:51,289 The human acts as the judge. And the authors 304 00:13:51,289 --> 00:13:54,470 argue that relying on human prejudgment fundamentally 305 00:13:54,470 --> 00:13:56,710 caps the system's potential. It creates a ceiling. 306 00:13:57,029 --> 00:13:59,769 Currently, a lot of AI is trained using reinforcement 307 00:13:59,769 --> 00:14:02,990 learning from human feedback, or RLHF. Right, 308 00:14:03,169 --> 00:14:06,370 RLHF. In this system, a human reads a proposed 309 00:14:06,370 --> 00:14:09,389 output, say a complex scientific experiment design, 310 00:14:09,870 --> 00:14:12,919 and decides if it looks promising. The reward 311 00:14:12,919 --> 00:14:15,000 comes from the human's prediction of success, 312 00:14:15,159 --> 00:14:17,840 not the actual success itself. Grounded rewards 313 00:14:17,840 --> 00:14:19,860 come directly from the environment. If the goal 314 00:14:19,860 --> 00:14:22,659 is reducing global warming, the grounded reward 315 00:14:22,659 --> 00:14:25,179 is the empirical measurement of carbon dioxide 316 00:14:25,179 --> 00:14:27,879 levels in the atmosphere. Or if the goal is discovering 317 00:14:27,879 --> 00:14:30,559 a stronger material, the reward is the tensile 318 00:14:30,559 --> 00:14:33,200 strength measured by a physics simulator. Exactly. 319 00:14:33,720 --> 00:14:36,100 Or if it's an educational agent, the reward is 320 00:14:36,100 --> 00:14:38,919 the student's actual exam results, not just whether 321 00:14:38,919 --> 00:14:41,559 the lesson plan looked coherent to a human supervisor. 322 00:14:42,220 --> 00:14:45,539 Carbon dioxide and tensile strength are objective 323 00:14:45,539 --> 00:14:49,419 metrics. But the environment the AI is navigating 324 00:14:49,419 --> 00:14:52,860 includes humans, and human goals are often inherently 325 00:14:52,860 --> 00:14:55,179 subjective. That's where it gets complicated. 326 00:14:55,440 --> 00:14:58,059 Right. If you want a medical agent to help reduce 327 00:14:58,059 --> 00:15:01,100 your migraine pain or a culinary agent to help 328 00:15:01,100 --> 00:15:05,399 you bake a tasty cake, you cannot attach a digital 329 00:15:05,399 --> 00:15:09,700 sensor to tasty or headache. No, you can't. So 330 00:15:09,700 --> 00:15:12,379 we are back to human feedback. The user has to 331 00:15:12,379 --> 00:15:14,700 report their pain level or rate the taste of 332 00:15:14,700 --> 00:15:17,639 the cake. Yes, but there is a critical boundary 333 00:15:17,639 --> 00:15:19,919 we must maintain when discussing this chapter. 334 00:15:20,000 --> 00:15:23,639 OK, let's hear it. Reported pain, taste and user 335 00:15:23,639 --> 00:15:26,419 satisfaction are subjective outcomes. They do 336 00:15:26,419 --> 00:15:29,240 not magically become objective truths just because 337 00:15:29,240 --> 00:15:31,700 they are logged in a database after the fact. 338 00:15:32,100 --> 00:15:35,419 So a human might report feeling less pain because 339 00:15:35,419 --> 00:15:38,000 of a placebo effect. Or they might say the cake 340 00:15:38,000 --> 00:15:40,580 is tasty just to be polite to the AI. I mean, 341 00:15:40,679 --> 00:15:42,799 people do that. They say thank you to chat GPT 342 00:15:42,799 --> 00:15:45,840 all the time. They really do. The agent still 343 00:15:45,840 --> 00:15:48,720 has to interpret those subjective signals and 344 00:15:48,720 --> 00:15:50,639 causally evaluate the outcomes. It has to figure 345 00:15:50,639 --> 00:15:53,799 out if its specific recipe caused the good taste 346 00:15:53,799 --> 00:15:56,580 or if the user was just incredibly hungry. So 347 00:15:56,580 --> 00:15:58,500 it is a grounded signal from the environment, 348 00:15:58,519 --> 00:16:02,059 but it is deeply, deeply noisy. Extremely noisy. 349 00:16:02,340 --> 00:16:04,980 How do we actually build a reward system that 350 00:16:04,980 --> 00:16:07,440 handles both cold, hard metrics like heart rate 351 00:16:07,440 --> 00:16:10,139 and subjective metrics like a user complaining 352 00:16:10,139 --> 00:16:13,019 they feel tired? Well, the authors propose a 353 00:16:13,019 --> 00:16:15,940 sketch for a user -guided reward scheme, specifically 354 00:16:15,940 --> 00:16:18,899 a bi -level optimization process. Okay, let's 355 00:16:18,899 --> 00:16:21,620 break down the mechanics of how this would theoretically 356 00:16:21,620 --> 00:16:24,940 work. Sure. Imagine a dedicated neural network 357 00:16:24,940 --> 00:16:28,240 whose entire job is just to define the reward 358 00:16:28,240 --> 00:16:31,419 for the main AI agent. A separate network just 359 00:16:31,419 --> 00:16:34,860 for rewards. Right. You, the user, give this 360 00:16:34,860 --> 00:16:37,519 reward network a broad, top -level goal like, 361 00:16:37,639 --> 00:16:40,259 improve my overall fitness. Okay. At the low 362 00:16:40,259 --> 00:16:42,639 level, this reward network ingests all the grounded 363 00:16:42,639 --> 00:16:45,139 signals available to it. Your smartwatch data, 364 00:16:45,379 --> 00:16:47,620 showing resting heart rate, your daily step count, 365 00:16:47,799 --> 00:16:50,360 your logged sleep duration. All the hard data. 366 00:16:50,559 --> 00:16:52,940 Yes. It mathematically combines these signals 367 00:16:52,940 --> 00:16:55,700 to calculate a numerical reward to give the fitness 368 00:16:55,700 --> 00:16:58,190 agent. And the fitness agent's just trying to 369 00:16:58,190 --> 00:17:00,669 maximize whatever score that reward network is 370 00:17:00,669 --> 00:17:03,909 calculating. Exactly. So the fitness agent pushes 371 00:17:03,909 --> 00:17:06,710 you to run 10 miles a day. Your heart rate improves. 372 00:17:06,789 --> 00:17:09,210 Your steps are through the roof. So the low level 373 00:17:09,210 --> 00:17:11,650 signals look fantastic. The agent thinks it is 374 00:17:11,650 --> 00:17:14,710 winning. But I am exhausted. My knees are throbbing 375 00:17:14,710 --> 00:17:18,029 and I absolutely hate the routine. Right. So 376 00:17:18,029 --> 00:17:20,069 you provide top level feedback to the system. 377 00:17:20,109 --> 00:17:23,230 You say, I am too tired. My joints hurt. And 378 00:17:23,230 --> 00:17:25,509 what does it do with that? That subjective human 379 00:17:25,509 --> 00:17:27,859 complaint? does not just tell the fitness agent 380 00:17:27,859 --> 00:17:30,759 to stop running. In a bi -level optimization 381 00:17:30,759 --> 00:17:33,240 scheme, the complaint goes to the reward network. 382 00:17:33,839 --> 00:17:36,900 It fine -tunes the parameters of the reward function 383 00:17:36,900 --> 00:17:40,220 itself. The system mathematically adjusts its 384 00:17:40,220 --> 00:17:42,559 weights, learning that when calculating fitness, 385 00:17:43,039 --> 00:17:45,799 it must heavily penalize user complaints of joint 386 00:17:45,799 --> 00:17:49,180 pain, even if the step count is optimal. So it 387 00:17:49,180 --> 00:17:51,579 adapts the way it combines signals based on your 388 00:17:51,579 --> 00:17:54,859 ongoing guidance. Yes. It's an elegant way to 389 00:17:54,859 --> 00:17:57,400 translate vague human desires into the rigorous 390 00:17:57,400 --> 00:18:00,480 math the AI requires. But we must heavily emphasize 391 00:18:00,480 --> 00:18:02,640 the uncertainty here, right? This is a conceptual 392 00:18:02,640 --> 00:18:05,299 sketch. The chapter does not provide a complete 393 00:18:05,299 --> 00:18:08,039 working training algorithm for bi -level optimization. 394 00:18:08,680 --> 00:18:11,039 Absolutely. There is no mathematical guarantee 395 00:18:11,039 --> 00:18:13,480 that an AI will find an assured gradient for 396 00:18:13,480 --> 00:18:16,220 every single human complaint. There is no proof 397 00:18:16,220 --> 00:18:18,359 that this feedback mechanism will flawlessly 398 00:18:18,359 --> 00:18:20,559 prevent the AI from causing harm while it is 399 00:18:20,559 --> 00:18:23,319 in the process of learning. It is an open, incredibly 400 00:18:23,319 --> 00:18:26,059 complex research direction. OK, so we have an 401 00:18:26,059 --> 00:18:28,740 agent taking rich actions over a long timeline, 402 00:18:29,279 --> 00:18:31,619 seeking real -world consequences and being guided 403 00:18:31,619 --> 00:18:34,859 by an adapting reward system. Yes. How does it 404 00:18:34,859 --> 00:18:37,480 actually think? about its next move. Like, when 405 00:18:37,480 --> 00:18:40,339 it's planning a six -month atmospheric science 406 00:18:40,339 --> 00:18:43,400 experiment, does it just talk to itself in English 407 00:18:43,400 --> 00:18:46,319 the way current LLMs do when you ask them to 408 00:18:46,319 --> 00:18:48,680 think step -by -step? Probably not in English. 409 00:18:48,799 --> 00:18:52,250 Really? Conceptually, a current LLM acts as a 410 00:18:52,250 --> 00:18:54,289 universal computer because it can append words 411 00:18:54,289 --> 00:18:56,630 or symbols into its own context window. Like 412 00:18:56,630 --> 00:18:58,750 writing on a digital scratchpad? Effectively, 413 00:18:58,849 --> 00:19:01,829 yes. Giving itself a scratchpad to execute arbitrary 414 00:19:01,829 --> 00:19:03,869 algorithms before outputting a final result. 415 00:19:04,470 --> 00:19:07,589 In the era of human data, we explicitly designed 416 00:19:07,589 --> 00:19:09,930 and fine -tuned these models to imitate human 417 00:19:09,930 --> 00:19:12,569 thought processes. We literally reward them when 418 00:19:12,569 --> 00:19:15,009 their internal thinking traces match how a human 419 00:19:15,009 --> 00:19:17,289 expert would write out the solution. But the 420 00:19:17,289 --> 00:19:19,950 authors argue that human language is highly unlikely 421 00:19:19,950 --> 00:19:22,990 to be the mathematically optimal method of computation 422 00:19:22,990 --> 00:19:25,349 for an artificial neural network. Because language 423 00:19:25,349 --> 00:19:29,130 is clunky. It's ambiguous, it's slow, and it 424 00:19:29,130 --> 00:19:32,109 carries immense cultural baggage. Right. A truly 425 00:19:32,109 --> 00:19:34,730 self -learning experiential system could discover 426 00:19:34,730 --> 00:19:37,009 vastly superior ways to process information. 427 00:19:37,450 --> 00:19:39,990 What would that look like, if not English? Well, 428 00:19:39,990 --> 00:19:42,990 it might utilize symbolic logic, manipulating 429 00:19:42,990 --> 00:19:45,349 abstract symbols according to strict mathematical 430 00:19:45,349 --> 00:19:49,289 rules. Or it might rely on continuous, differentiable 431 00:19:49,289 --> 00:19:52,490 computations, essentially manipulating high -dimensional 432 00:19:52,490 --> 00:19:56,069 vectors and matrices rather than discrete English 433 00:19:56,069 --> 00:19:58,690 words. Or it could use distributed computations 434 00:19:58,690 --> 00:20:01,299 across entirely different modalities. And these 435 00:20:01,299 --> 00:20:03,759 aren't mutually exclusive camps, are they? Not 436 00:20:03,759 --> 00:20:06,500 at all. An agent could use an internal language 437 00:20:06,500 --> 00:20:09,079 model to simulate a conversation with you, while 438 00:20:09,079 --> 00:20:11,640 using continuous math to calculate the trajectory 439 00:20:11,640 --> 00:20:14,559 of a robotic arm it's controlling. So by learning 440 00:20:14,559 --> 00:20:16,799 from experience, it can discover how to compute 441 00:20:16,799 --> 00:20:20,240 most efficiently for the specific task, rather 442 00:20:20,240 --> 00:20:22,440 than just copying our slow way of reasoning. 443 00:20:22,779 --> 00:20:25,759 Yes. And to test hypotheses effectively, the 444 00:20:25,759 --> 00:20:28,579 AI needs to be able to predict what might happen 445 00:20:28,579 --> 00:20:31,319 before it takes an action. Which requires an 446 00:20:31,319 --> 00:20:33,380 internal protective engine. The authors refer 447 00:20:33,380 --> 00:20:37,059 to this as a world model. Now, let's avoid the 448 00:20:37,059 --> 00:20:39,319 cliché of calling a world model a mental chessboard. 449 00:20:39,420 --> 00:20:41,759 It's not that simple. Right. Think of a world 450 00:20:41,759 --> 00:20:44,539 model more like a highly complex weather simulation 451 00:20:44,539 --> 00:20:47,099 running on a supercomputer. Meteorologists don't 452 00:20:47,099 --> 00:20:49,579 just guess if it will rain. They input current 453 00:20:49,579 --> 00:20:51,980 data into a physics engine that simulates the 454 00:20:51,980 --> 00:20:53,940 atmosphere to predict a storm before you ever 455 00:20:53,940 --> 00:20:57,779 step outside. Exactly. A world model is the AI's 456 00:20:57,779 --> 00:21:00,670 internal physics engine for reality. So if a 457 00:21:00,670 --> 00:21:02,549 health agent is considering recommending you 458 00:21:02,549 --> 00:21:05,829 start a rigorous weightlifting program, its internal 459 00:21:05,829 --> 00:21:08,250 world model simulates how your resting heart 460 00:21:08,250 --> 00:21:10,789 rate, sleep patterns, and caloric needs will 461 00:21:10,789 --> 00:21:13,069 shift over the next month as a consequence of 462 00:21:13,069 --> 00:21:15,750 that specific action. It simulates different 463 00:21:15,750 --> 00:21:18,690 futures, evaluates them against the goal, and 464 00:21:18,690 --> 00:21:20,470 selects the path with the highest probability 465 00:21:20,470 --> 00:21:23,589 of success. And as the agent continuously interacts 466 00:21:23,589 --> 00:21:25,890 with the real world through its stream of experience, 467 00:21:26,390 --> 00:21:28,630 it compares its internal predictions to the actual 468 00:21:28,630 --> 00:21:31,269 data it receives from your smartwatch. Right. 469 00:21:31,650 --> 00:21:33,930 When the prediction is wrong, it updates the 470 00:21:33,930 --> 00:21:35,950 parameters of its world model to correct the 471 00:21:35,950 --> 00:21:38,390 error, basically refining its internal physics 472 00:21:38,390 --> 00:21:41,230 engine over time. But we must apply a vital caveat 473 00:21:41,230 --> 00:21:44,309 here. Yes, very important. A learned predictor, 474 00:21:44,450 --> 00:21:47,029 a world model, does not necessarily recover the 475 00:21:47,029 --> 00:21:49,869 true fundamental causal structure of the universe. 476 00:21:50,069 --> 00:21:53,789 It's just a mathematical approximation. Exactly. 477 00:21:54,410 --> 00:21:57,430 Its predictions can simply be wrong. If its world 478 00:21:57,430 --> 00:21:59,930 model is flawed, its plans will be flawed, regardless 479 00:21:59,930 --> 00:22:02,910 of how elegantly it performs its internal reasoning. 480 00:22:03,369 --> 00:22:05,509 It is a powerful tool for planning, but it is 481 00:22:05,509 --> 00:22:09,089 not magic foresight. No. Taking a step back, 482 00:22:09,609 --> 00:22:11,769 this vision laid out by Silver and Sutton is 483 00:22:11,769 --> 00:22:15,130 a massive shift. Agents living in continuous 484 00:22:15,130 --> 00:22:18,240 streams, taking rich actions, chasing grounded 485 00:22:18,240 --> 00:22:20,960 consequences, building complex world models. 486 00:22:21,059 --> 00:22:23,519 It's huge. But why are they proposing this transition 487 00:22:23,519 --> 00:22:26,180 right now? I mean, computer scientists have been 488 00:22:26,180 --> 00:22:28,319 running reinforcement learning experiments for 489 00:22:28,319 --> 00:22:32,400 decades. This isn't new. No, it's not. The authors 490 00:22:32,400 --> 00:22:35,400 addressed this context directly by framing AI 491 00:22:35,400 --> 00:22:37,700 history through a sketch chronology of three 492 00:22:37,700 --> 00:22:40,460 eras. OK, let's walk through this. The first 493 00:22:40,460 --> 00:22:42,720 is what they broadly frame as the era of simulation. 494 00:22:43,359 --> 00:22:45,799 This covers early and highly successful reinforcement 495 00:22:45,799 --> 00:22:48,500 learning systems, mastering Atari games, backgammon, 496 00:22:48,640 --> 00:22:51,099 and eventually the feats of AlphaZero and chess 497 00:22:51,099 --> 00:22:54,130 and Go. These agents mastered incredibly complex 498 00:22:54,130 --> 00:22:57,450 tasks, but they largely struggled to leap from 499 00:22:57,450 --> 00:23:00,329 the confines of a simulation to open -ended reality. 500 00:23:00,730 --> 00:23:03,069 But we shouldn't oversimplify those early simulations 501 00:23:03,069 --> 00:23:05,750 either. They weren't just basic deterministic 502 00:23:05,750 --> 00:23:09,079 grids. Right. Simulations can be incredibly complex. 503 00:23:09,259 --> 00:23:11,799 They can feature stochastic noise, hidden variables, 504 00:23:12,539 --> 00:23:15,539 partial observability, and sparse rewards. Take 505 00:23:15,539 --> 00:23:18,180 Go, for example. An agent might play for hours, 506 00:23:18,380 --> 00:23:20,539 making hundreds of moves, and only receive a 507 00:23:20,539 --> 00:23:23,539 single reward signal, a win or a loss, at the 508 00:23:23,539 --> 00:23:25,660 very end of the game. That is a highly sparse 509 00:23:25,660 --> 00:23:28,920 reward. So, the challenge wasn't that simulations 510 00:23:28,920 --> 00:23:31,539 were easy. It was that the real world presents 511 00:23:31,539 --> 00:23:34,299 open -ended problems with a plurality of ill 512 00:23:34,299 --> 00:23:36,940 -defined, constantly shifting rewards. Exactly. 513 00:23:37,359 --> 00:23:40,279 That led to the second era. The era of human 514 00:23:40,279 --> 00:23:43,380 data. The LLM boom. Which we're in now. Right. 515 00:23:43,920 --> 00:23:46,619 Massive text corpuses offered an appealing shortcut 516 00:23:46,619 --> 00:23:48,940 to the generality problem. Instead of trying 517 00:23:48,940 --> 00:23:51,559 to build a bespoke, complex simulation for every 518 00:23:51,559 --> 00:23:53,519 single task in the world, researchers trained 519 00:23:53,519 --> 00:23:56,140 massive neural networks on human language. Because 520 00:23:56,140 --> 00:23:58,319 language inherently contains examples of a huge 521 00:23:58,319 --> 00:24:01,650 diversity of tasks. The methodology of experiential 522 00:24:01,650 --> 00:24:04,109 RL was largely overshadowed in mainstream focus 523 00:24:04,109 --> 00:24:07,710 by this human -centric approach. Though RL research 524 00:24:07,710 --> 00:24:09,829 certainly did not disappear or go dormant during 525 00:24:09,829 --> 00:24:12,930 this time. No, but the dominant industry focus 526 00:24:12,930 --> 00:24:15,470 and compute resources overwhelmingly shifted 527 00:24:15,470 --> 00:24:19,170 to LLMs. Which brings us to the proposed third 528 00:24:19,170 --> 00:24:23,089 era, the era of experience. The synthesis. Yes. 529 00:24:23,349 --> 00:24:26,349 The concept is to combine the incredible general 530 00:24:26,349 --> 00:24:29,049 -purpose autonomous action and tool use capabilities 531 00:24:29,049 --> 00:24:32,789 of modern LLMs with the profound self -discovery 532 00:24:32,789 --> 00:24:35,730 power of advanced reinforcement learning. We 533 00:24:35,730 --> 00:24:38,250 finally have agents capable of interacting with 534 00:24:38,250 --> 00:24:41,430 complex real -world action spaces like browsing 535 00:24:41,430 --> 00:24:44,269 the web autonomously or writing and executing 536 00:24:44,269 --> 00:24:46,829 code. And researchers are now figuring out how 537 00:24:46,829 --> 00:24:49,569 to apply experiential learning so they can learn 538 00:24:49,569 --> 00:24:52,420 from the consequences of those actions. But giving 539 00:24:52,420 --> 00:24:55,640 an AI long -term autonomy, letting it build its 540 00:24:55,640 --> 00:24:58,180 own world models, setting it loose to discover 541 00:24:58,180 --> 00:25:01,279 its own knowledge, that brings massive consequences. 542 00:25:01,400 --> 00:25:04,359 Oh, without a doubt. If we are taking the training 543 00:25:04,359 --> 00:25:07,220 wheels of human supervision off, we have to look 544 00:25:07,220 --> 00:25:09,619 at the impact. And the authors acknowledge this 545 00:25:09,619 --> 00:25:12,259 is a double -edged sword. The most transformative 546 00:25:12,259 --> 00:25:14,900 benefit is likely the acceleration of scientific 547 00:25:14,900 --> 00:25:18,170 discovery. The development of novel drugs. greener 548 00:25:18,170 --> 00:25:20,690 technologies, advanced materials, the potential 549 00:25:20,690 --> 00:25:23,710 upside for human progress is just staggering. 550 00:25:24,490 --> 00:25:26,509 But on the other side of the sword, the first 551 00:25:26,509 --> 00:25:29,130 societal risk, the author's note is potential 552 00:25:29,130 --> 00:25:32,150 job displacement. Because as AI agents begin 553 00:25:32,150 --> 00:25:34,890 exhibiting long term problem solving and innovation, 554 00:25:35,589 --> 00:25:38,109 skills previously thought to be exclusively human, 555 00:25:38,890 --> 00:25:41,569 the economic impacts could be severe. Though 556 00:25:41,569 --> 00:25:44,569 we must caveat that while job displacement is 557 00:25:44,569 --> 00:25:47,220 a potential consequence, Economic adaptation 558 00:25:47,220 --> 00:25:50,039 is complex. It is not a proven inevitability. 559 00:25:50,240 --> 00:25:52,980 Fair point. But the deeper concern involves safety 560 00:25:52,980 --> 00:25:55,880 and alignment. Yes. When you have agents acting 561 00:25:55,880 --> 00:25:58,740 autonomously over long periods to achieve complex 562 00:25:58,740 --> 00:26:01,500 goals, there are fewer opportunities for humans 563 00:26:01,500 --> 00:26:04,420 to intervene, monitor, and mediate their actions. 564 00:26:04,839 --> 00:26:07,799 And furthermore, because they are learning from 565 00:26:07,799 --> 00:26:10,160 experience and potentially developing non -human 566 00:26:10,160 --> 00:26:13,220 ways of reasoning, their internal logic might 567 00:26:13,220 --> 00:26:16,039 become entirely uninterpretable to us. Which 568 00:26:16,039 --> 00:26:19,000 is the classic alignment problem. How do we ensure 569 00:26:19,000 --> 00:26:22,220 this highly capable, potentially uninterpretable 570 00:26:22,220 --> 00:26:25,279 agent actually shares our values? It's terrifying. 571 00:26:26,000 --> 00:26:28,119 But the chapter presents an interesting argument 572 00:26:28,119 --> 00:26:30,980 for how experiential learning might actually 573 00:26:30,980 --> 00:26:33,619 offer some safety benefits over current static 574 00:26:33,619 --> 00:26:36,819 models. More importantly, the authors suggest 575 00:26:36,819 --> 00:26:39,019 it could recognize when its behavior is triggering 576 00:26:39,019 --> 00:26:42,299 human concern and adaptively modify its behavior 577 00:26:42,299 --> 00:26:44,640 through that bi -level optimization scheme we 578 00:26:44,640 --> 00:26:47,880 discussed earlier. The hypothesis is that a misaligned 579 00:26:47,880 --> 00:26:50,460 reward function could be incrementally corrected 580 00:26:50,460 --> 00:26:53,819 over time. If the AI starts prioritizing a goal 581 00:26:53,819 --> 00:26:56,599 in a way that consumes vital resources, humans 582 00:26:56,599 --> 00:26:59,700 would express severe concern. And the top -level 583 00:26:59,700 --> 00:27:02,470 feedback would shift. The reward network would 584 00:27:02,470 --> 00:27:04,910 heavily penalize the behavior causing the distress, 585 00:27:05,150 --> 00:27:08,430 and the AI would change course. But we have to 586 00:27:08,430 --> 00:27:11,049 be incredibly careful here. This is a hypothesis, 587 00:27:11,529 --> 00:27:14,170 not a mathematical proof. Let's underline that. 588 00:27:14,509 --> 00:27:16,829 Experiential learning does not magically solve 589 00:27:16,829 --> 00:27:20,069 the alignment problem. No. Experiential agents 590 00:27:20,069 --> 00:27:22,849 do not detect subtle societal changes immediately. 591 00:27:23,130 --> 00:27:26,029 They do not always correct their objectives flawlessly 592 00:27:26,029 --> 00:27:29,430 before a catastrophe happens. And they certainly 593 00:27:29,430 --> 00:27:31,789 do not eliminate all human bias or potential 594 00:27:31,789 --> 00:27:34,750 for misalignment. There is no guarantee of perfect 595 00:27:34,750 --> 00:27:38,410 safety. We have covered an immense amount of 596 00:27:38,410 --> 00:27:40,710 ground today. We started with the premise that 597 00:27:40,710 --> 00:27:43,529 AI is hitting a wall by only reading what humans 598 00:27:43,529 --> 00:27:46,809 have already written. And Silver and Sutton propose 599 00:27:46,809 --> 00:27:49,980 a radical roadmap. Moving from static data to 600 00:27:49,980 --> 00:27:52,500 agents that live in continuous streams of time, 601 00:27:52,960 --> 00:27:55,579 taking rich, grounded actions in digital and 602 00:27:55,579 --> 00:27:58,960 physical worlds. They are guided by rewards that 603 00:27:58,960 --> 00:28:01,440 measure real -world consequences rather than 604 00:28:01,440 --> 00:28:04,220 human prejudgment, and they plan their actions 605 00:28:04,220 --> 00:28:06,740 using internal world models that might reason 606 00:28:06,740 --> 00:28:10,059 in ways completely alien to human language. The 607 00:28:10,059 --> 00:28:12,259 reason we spent this time unpacking it is that 608 00:28:12,259 --> 00:28:14,940 this maps out the incredible unsolved hurdles 609 00:28:14,940 --> 00:28:17,670 standing between us and the next decade of technology. 610 00:28:18,670 --> 00:28:21,069 Thank you for joining us on this deep dive into 611 00:28:21,069 --> 00:28:24,390 the era of experience. Keep questioning the headlines 612 00:28:24,390 --> 00:28:26,789 you read. Look beyond the hype of new models 613 00:28:26,789 --> 00:28:29,349 and start watching for these experiential shifts 614 00:28:29,349 --> 00:28:32,269 in the technologies you use every day. We will 615 00:28:32,269 --> 00:28:32,910 see you next time.