1 00:00:00,000 --> 00:00:03,520 Imagine for a second that you are trying to learn 2 00:00:03,520 --> 00:00:08,019 a highly complex physical skill. Like, let's 3 00:00:08,019 --> 00:00:10,119 say you're trying to learn how to walk from scratch. 4 00:00:10,740 --> 00:00:12,480 Or, you know, maybe you're trying to figure out 5 00:00:12,480 --> 00:00:15,039 how to perfectly swing this heavy pendulum up 6 00:00:15,039 --> 00:00:17,539 into the air and balance it on its pivot point. 7 00:00:17,800 --> 00:00:20,460 Right, a classic control problem. Exactly. But 8 00:00:20,460 --> 00:00:24,440 here is the catch, and it is a huge one. You 9 00:00:24,440 --> 00:00:27,510 are not allowed to feel your own muscles. You 10 00:00:27,510 --> 00:00:30,109 are like entirely stripped of your sense of touch. 11 00:00:30,210 --> 00:00:32,409 Oh, wow. Okay Yeah, so you're not allowed to 12 00:00:32,409 --> 00:00:34,549 feel the ground under your feet the weight of 13 00:00:34,549 --> 00:00:36,950 that pendulum in your hand or even The shift 14 00:00:36,950 --> 00:00:39,429 of your own center of gravity. So it's a sensory 15 00:00:39,429 --> 00:00:41,490 deprivation nightmare. Basically complete nightmare 16 00:00:41,490 --> 00:00:43,670 Okay, the only way you can learn this complex 17 00:00:43,670 --> 00:00:46,250 physical skill is by looking at a stream of flat 18 00:00:46,250 --> 00:00:48,649 Two -dimensional third -person images on a computer 19 00:00:48,649 --> 00:00:51,189 screen like a video feed. Yeah, you're watching 20 00:00:51,189 --> 00:00:54,770 yourself from the outside in pixels and To make 21 00:00:54,770 --> 00:00:57,979 matters even more difficult You want to master 22 00:00:57,979 --> 00:01:00,579 this skill with as little trial and error as 23 00:01:00,579 --> 00:01:03,219 possible because failures are expensive, right? 24 00:01:03,280 --> 00:01:05,219 Exactly. Every time you fall over or drop that 25 00:01:05,219 --> 00:01:08,200 pendulum. It's incredibly costly Maybe the robot 26 00:01:08,200 --> 00:01:10,500 breaks or it just takes a painfully long time 27 00:01:10,500 --> 00:01:13,079 to reset the simulation So the goal is that you 28 00:01:13,079 --> 00:01:15,439 want to figure out how to do it mostly in your 29 00:01:15,439 --> 00:01:18,379 head before you even make a physical move Which 30 00:01:18,379 --> 00:01:21,420 I mean that sounds like an impossible setup for 31 00:01:21,420 --> 00:01:24,739 a biological organism. We rely so heavily on 32 00:01:24,739 --> 00:01:27,930 our proprioception that innate sense of where 33 00:01:27,930 --> 00:01:30,930 our limbs are in space. We rely on the physical 34 00:01:30,930 --> 00:01:33,290 feedback of the world pushing back against us. 35 00:01:33,769 --> 00:01:36,489 So to strip all of that away, to just mute the 36 00:01:36,489 --> 00:01:39,150 physical world entirely and ask for physical 37 00:01:39,150 --> 00:01:42,109 mastery based purely on a sequence of flat pictures, 38 00:01:42,329 --> 00:01:44,349 well, it's a staggering challenge. It really 39 00:01:44,349 --> 00:01:47,030 is. And yet, this is exactly the problem domain 40 00:01:47,030 --> 00:01:49,069 we are looking at today. We're exploring how 41 00:01:49,069 --> 00:01:51,709 an artificial intelligence agent can learn to 42 00:01:51,709 --> 00:01:54,109 control complex, simulated physical environments 43 00:01:54,109 --> 00:01:57,810 using only images, and how it can do so while 44 00:01:57,810 --> 00:01:59,950 drastically reducing the number of times it has 45 00:01:59,950 --> 00:02:02,430 to actually bumble around and interact with that 46 00:02:02,430 --> 00:02:05,609 environment. OK, so let's really unpack this, 47 00:02:05,750 --> 00:02:07,969 because the implications of getting this right 48 00:02:07,969 --> 00:02:10,389 are just massive for the field of robotics and 49 00:02:10,389 --> 00:02:13,419 AI. Today's Deep Dive is centered on a really 50 00:02:13,419 --> 00:02:16,219 fascinating paper from 2019. A very influential 51 00:02:16,219 --> 00:02:18,800 one, yeah. Yeah. It's titled Learning Latent 52 00:02:18,800 --> 00:02:22,080 Dynamics for Planning from Pixels. And it introduces 53 00:02:22,080 --> 00:02:25,500 an AI agent called Planet. The authors are Dan 54 00:02:25,500 --> 00:02:27,560 and Jar Hafner and a team of colleagues from 55 00:02:27,560 --> 00:02:29,919 Google Brain, DeepMind, and the University of 56 00:02:29,919 --> 00:02:32,599 Toronto. That's a heavy -hitting lineup. Definitely. 57 00:02:33,560 --> 00:02:37,080 This paper sits right at this brilliant, complicated 58 00:02:37,080 --> 00:02:40,360 intersection of two massive AI fields. First, 59 00:02:40,580 --> 00:02:42,960 perception, you know, taking in raw pixels from 60 00:02:42,960 --> 00:02:46,340 a camera, and second, planning, which is figuring 61 00:02:46,340 --> 00:02:48,879 out a sequence of actions to take to achieve 62 00:02:48,879 --> 00:02:51,919 a specific goal. Right. It's essentially about 63 00:02:51,919 --> 00:02:54,020 an agent learning how the physics of its world 64 00:02:54,020 --> 00:02:56,939 works just by watching it, and then using that 65 00:02:56,939 --> 00:02:59,419 internally learned model to plan its physical 66 00:02:59,419 --> 00:03:02,469 actions. But. And I want to be really clear here 67 00:03:02,469 --> 00:03:06,349 before we get swept away by the conceptual elegance 68 00:03:06,349 --> 00:03:09,550 of an AI imagining the future to solve problems. 69 00:03:10,310 --> 00:03:13,069 We are going to meticulously deconstruct the 70 00:03:13,069 --> 00:03:15,310 mechanics behind this architecture. Yeah, we 71 00:03:15,310 --> 00:03:18,389 have to stay grounded. Absolutely. crucial that 72 00:03:18,389 --> 00:03:21,210 we separate the actual mathematical contributions 73 00:03:21,210 --> 00:03:24,750 of this paper from the kind of science fiction 74 00:03:24,750 --> 00:03:26,789 hype that often surrounds anything involving 75 00:03:26,789 --> 00:03:29,729 AI and imagination. We need to look exactly at 76 00:03:29,729 --> 00:03:31,770 what the math is doing, what the constraints 77 00:03:31,770 --> 00:03:34,969 are, and why certain very specific architectural 78 00:03:34,969 --> 00:03:37,590 choices were made by the researchers. Right. 79 00:03:38,110 --> 00:03:41,509 So no crystal balls, no sentient robots waking 80 00:03:41,509 --> 00:03:44,110 up and understanding the universe, just rigorous, 81 00:03:44,490 --> 00:03:48,430 brilliant engineering. Exactly. To really appreciate 82 00:03:48,430 --> 00:03:50,870 what the planet agent is doing, we first have 83 00:03:50,870 --> 00:03:52,949 to understand the historical context it was born 84 00:03:52,949 --> 00:03:55,849 into. Because before we get into how this agent 85 00:03:55,849 --> 00:03:58,849 solves the problem of learning from pixels, we 86 00:03:58,849 --> 00:04:01,810 need to establish why learning control from images 87 00:04:01,810 --> 00:04:04,009 is such a monumental challenge in the first place. 88 00:04:04,349 --> 00:04:07,409 Right, because planning itself isn't a new concept. 89 00:04:07,710 --> 00:04:09,810 The paper points out that planning in AI has 90 00:04:09,810 --> 00:04:12,030 actually been highly successful for a very long 91 00:04:12,030 --> 00:04:14,159 time. Yeah, it's foundational. If you look at 92 00:04:14,159 --> 00:04:16,199 the history of artificial intelligence, planning 93 00:04:16,199 --> 00:04:18,459 algorithms have achieved incredible things. But, 94 00:04:18,660 --> 00:04:20,759 and this is the key constraint, they have historically 95 00:04:20,759 --> 00:04:22,839 been successful in environments where the rules 96 00:04:22,839 --> 00:04:24,839 of the world are perfectly known in advance. 97 00:04:25,000 --> 00:04:28,379 Oh, like chess. Exactly. We call these the dynamics 98 00:04:28,379 --> 00:04:30,740 of the environment. Think about classical board 99 00:04:30,740 --> 00:04:33,920 games like chess or Go. The AI agent knows exactly 100 00:04:33,920 --> 00:04:35,899 what a knight does. It knows that if it moves 101 00:04:35,899 --> 00:04:38,160 a piece to a specific square, the board will 102 00:04:38,160 --> 00:04:40,800 look exactly a certain way in the very next step. 103 00:04:41,019 --> 00:04:43,550 There's no mystery there. There is zero uncertainty 104 00:04:43,550 --> 00:04:45,709 about the rules of the transition from one state 105 00:04:45,709 --> 00:04:48,089 of the board to another. And because those dynamics 106 00:04:48,089 --> 00:04:50,569 are known perfectly, the agent can use sheer 107 00:04:50,569 --> 00:04:53,069 computational power to search through massive 108 00:04:53,069 --> 00:04:55,629 branching trees of possible future moves to find 109 00:04:55,629 --> 00:04:58,209 a winning strategy. So it basically has a perfect 110 00:04:58,209 --> 00:05:02,329 rulebook. But the real world, or even a complex 111 00:05:02,329 --> 00:05:04,930 physics simulation of a robot walking, that doesn't 112 00:05:04,930 --> 00:05:07,569 come with a neat little rulebook attached. The 113 00:05:07,569 --> 00:05:09,670 AI doesn't automatically know that gravity pulls 114 00:05:09,670 --> 00:05:13,069 at 9 .8 meters per second squared or that its 115 00:05:13,069 --> 00:05:16,110 left leg weighs exactly 15 pounds. That's the 116 00:05:16,110 --> 00:05:18,910 core of the issue right there. To plan in an 117 00:05:18,910 --> 00:05:21,290 unknown environment, the agent must first learn 118 00:05:21,290 --> 00:05:24,230 those rules, the dynamics, from its own experience. 119 00:05:24,490 --> 00:05:26,550 It has to interact with the world, observe what 120 00:05:26,550 --> 00:05:29,870 happens, and build an internal mathematical model 121 00:05:29,870 --> 00:05:32,670 of cause and effect. Which sounds hard enough 122 00:05:32,670 --> 00:05:35,250 on its own. It is. Doing this is already hard 123 00:05:35,250 --> 00:05:37,730 if you have clean data. like the exact coordinates 124 00:05:37,730 --> 00:05:40,509 of all your joints. But doing this when your 125 00:05:40,509 --> 00:05:43,670 only input is high -dimensional data, like thousands 126 00:05:43,670 --> 00:05:45,870 of raw pixels changing over time from a camera 127 00:05:45,870 --> 00:05:49,149 feed, that brings a cascade of massive technical 128 00:05:49,149 --> 00:05:51,410 challenges. Let's break down those roadblocks 129 00:05:51,410 --> 00:05:53,850 because the paper highlights a few specific ones 130 00:05:53,850 --> 00:05:55,930 that have historically made this pixel -based 131 00:05:55,930 --> 00:05:57,790 planning so difficult. The first one they mentioned 132 00:05:57,790 --> 00:06:01,069 is just, well, basic model inaccuracies. Yes. 133 00:06:01,230 --> 00:06:03,670 When you're trying to learn the laws of physics 134 00:06:03,670 --> 00:06:05,949 by looking at thousands of pixels changing frame 135 00:06:05,949 --> 00:06:08,790 by frame, your learned mathematical model is 136 00:06:08,790 --> 00:06:10,689 never going to be perfect. Right. There will 137 00:06:10,689 --> 00:06:12,990 always be slight errors in how you predict the 138 00:06:12,990 --> 00:06:15,610 very next frame. Which makes total sense. I mean, 139 00:06:15,689 --> 00:06:17,870 it's looking at a blurry image and trying to 140 00:06:17,870 --> 00:06:19,689 guess the next blurry image. It's not going to 141 00:06:19,689 --> 00:06:22,310 get every single pixel perfectly right. But the 142 00:06:22,310 --> 00:06:24,589 problem is that when you are planning an action, 143 00:06:24,730 --> 00:06:26,470 you need to predict not just one step ahead, 144 00:06:26,589 --> 00:06:29,389 but many steps ahead. And this leads to the second 145 00:06:29,389 --> 00:06:31,589 major challenge, which is accumulating errors 146 00:06:31,589 --> 00:06:33,870 in multi -step predictions. OK, let me make sure 147 00:06:33,870 --> 00:06:37,569 I'm visualizing this correctly. It's like if 148 00:06:37,569 --> 00:06:39,930 I try to draw a perfectly straight line freehand 149 00:06:39,930 --> 00:06:42,819 on a giant whiteboard. For the first inch, it 150 00:06:42,819 --> 00:06:44,879 looks pretty straight. Sure. But if I try to 151 00:06:44,879 --> 00:06:48,180 draw that same line for 10 feet, my slight initial 152 00:06:48,180 --> 00:06:50,360 angle error compounds. By the time I get to the 153 00:06:50,360 --> 00:06:52,620 end of the board, my line is way off course. 154 00:06:52,980 --> 00:06:55,379 That is a very apt way to visualize it. Yeah. 155 00:06:55,759 --> 00:06:57,920 If your first prediction of the world is slightly 156 00:06:57,920 --> 00:07:00,720 off and you were forced to base your second prediction 157 00:07:00,720 --> 00:07:04,019 on that flawed first prediction, the error compounds 158 00:07:04,019 --> 00:07:06,790 mathematically. It just... Snowballs. Exactly. 159 00:07:07,149 --> 00:07:10,810 By the time the AI has predicted 10 or 15 steps 160 00:07:10,810 --> 00:07:14,089 into the future, its internal imagination of 161 00:07:14,089 --> 00:07:16,870 where the robot is might have completely diverged 162 00:07:16,870 --> 00:07:19,189 from where the robot would actually be in physical 163 00:07:19,189 --> 00:07:22,509 reality. Any plan based on that diverging imagination 164 00:07:22,509 --> 00:07:26,829 is going to fail catastrophically. Okay, so accumulating 165 00:07:26,829 --> 00:07:29,670 errors is a massive hurdle. What else makes learning 166 00:07:29,670 --> 00:07:32,800 physics from pixels so hard? A third major issue 167 00:07:32,800 --> 00:07:36,100 is the failure to capture multiple possible futures. 168 00:07:36,600 --> 00:07:39,079 What do you mean by that? Well, in many environments, 169 00:07:39,360 --> 00:07:42,259 the outcome of an action isn't perfectly deterministic 170 00:07:42,259 --> 00:07:44,819 from the agent's limited point of view. Let's 171 00:07:44,819 --> 00:07:47,480 say you're controlling a robot outside. A gust 172 00:07:47,480 --> 00:07:50,360 of wind hits. Or maybe the agent's own camera 173 00:07:50,360 --> 00:07:52,959 sensors are noisy and glitch out for a millisecond. 174 00:07:53,360 --> 00:07:55,930 Oh, okay. A good internal model needs to understand 175 00:07:55,930 --> 00:07:58,029 that, given the current state of the world and 176 00:07:58,029 --> 00:08:00,529 a specific action, the outcome could actually 177 00:08:00,529 --> 00:08:03,610 be a range of probabilities, not just one single, 178 00:08:04,129 --> 00:08:06,889 rigid, guaranteed outcome. So it has to expect 179 00:08:06,889 --> 00:08:09,410 the unexpected, mathematically speaking. Right. 180 00:08:10,069 --> 00:08:12,110 And finally, the paper notes the challenge of 181 00:08:12,110 --> 00:08:14,470 overconfidence outside the training distribution. 182 00:08:14,569 --> 00:08:17,509 This is a classic machine learning problem. Like 183 00:08:17,509 --> 00:08:20,110 when you see something entirely new. Yes. If 184 00:08:20,110 --> 00:08:22,170 the agent finds itself in a situation it hasn't 185 00:08:22,170 --> 00:08:24,389 seen before during its training, maybe the robot 186 00:08:24,389 --> 00:08:26,930 is tilted at a weird angle it's never experienced, 187 00:08:27,470 --> 00:08:30,329 a poor model might confidently predict a totally 188 00:08:30,329 --> 00:08:32,649 physically impossible outcome. Right. You might 189 00:08:32,649 --> 00:08:34,389 think, oh, from this angle, if I push my leg, 190 00:08:34,509 --> 00:08:36,710 I will magically fly through the ceiling. Exactly. 191 00:08:37,210 --> 00:08:39,690 And that completely ruins any plan relying on 192 00:08:39,690 --> 00:08:43,460 that absurd prediction. So given all these immense 193 00:08:43,460 --> 00:08:46,169 hurdles, The compounding errors, the uncertainty, 194 00:08:46,450 --> 00:08:49,210 the overconfidence. The field of AI has generally 195 00:08:49,210 --> 00:08:51,769 been divided into two main camps for tackling 196 00:08:51,769 --> 00:08:55,049 unknown environments. Yes. We have model -free 197 00:08:55,049 --> 00:08:58,289 reinforcement learning and model -based reinforcement 198 00:08:58,289 --> 00:09:01,070 learning. And this planet paper is planting its 199 00:09:01,070 --> 00:09:03,950 flag firmly in the model -based camp. And I think 200 00:09:03,950 --> 00:09:06,210 we need to spend some time clarifying that distinction, 201 00:09:06,769 --> 00:09:08,409 because it's foundational to everything that 202 00:09:08,409 --> 00:09:10,029 follows. Let's start with model -free. How does 203 00:09:10,029 --> 00:09:12,799 that work? OK. So in model -free reinforcement 204 00:09:12,799 --> 00:09:15,620 learning, the agent does not explicitly attempt 205 00:09:15,620 --> 00:09:18,220 to learn the rules of the environment. It doesn't 206 00:09:18,220 --> 00:09:20,419 build a transition model to simulate what will 207 00:09:20,419 --> 00:09:23,039 happen in the future. So it doesn't even try 208 00:09:23,039 --> 00:09:25,700 to learn the physics. Then how does it do anything? 209 00:09:26,529 --> 00:09:28,830 Instead of predicting the future, it simply learns 210 00:09:28,830 --> 00:09:32,710 a policy. A policy is just a direct mapping from 211 00:09:32,710 --> 00:09:35,090 what it sees in the present moment to what action 212 00:09:35,090 --> 00:09:37,389 it should take right now. It learns this mapping 213 00:09:37,389 --> 00:09:39,710 based on a long history of trial and error and 214 00:09:39,710 --> 00:09:41,899 receiving rewards. Okay, I want to make sure 215 00:09:41,899 --> 00:09:45,220 I understand this. Let's avoid the usual maze 216 00:09:45,220 --> 00:09:47,659 analogies for a second. Think about playing a 217 00:09:47,659 --> 00:09:49,940 really difficult, fast -paced video game, like 218 00:09:49,940 --> 00:09:52,600 a fighting game or a platformer. Okay. If I'm 219 00:09:52,600 --> 00:09:55,080 using a model -free approach in my brain, I'm 220 00:09:55,080 --> 00:09:57,259 not calculating the physics engine of the game 221 00:09:57,259 --> 00:09:59,720 in real time. I'm just playing the level a thousand 222 00:09:59,720 --> 00:10:02,559 times. Eventually, muscle memory takes over. 223 00:10:03,000 --> 00:10:05,220 I know that when the screen looks exactly like 224 00:10:05,220 --> 00:10:08,240 this, I need to press the jump button to get 225 00:10:08,240 --> 00:10:10,940 the points and not die. I don't know why the 226 00:10:10,940 --> 00:10:13,720 jump works in terms of the game's code. I just 227 00:10:13,720 --> 00:10:17,220 know that in this state, this action equals reward. 228 00:10:18,519 --> 00:10:20,320 That is a helpful way to frame it, actually. 229 00:10:20,659 --> 00:10:23,340 In model -free methods, learning relies on propagating 230 00:10:23,340 --> 00:10:26,059 a reward signal backwards through time over many, 231 00:10:26,340 --> 00:10:28,399 many trials. In the literature, this is often 232 00:10:28,399 --> 00:10:31,460 done using what are called Bellman backups. Time 233 00:10:31,460 --> 00:10:34,480 out. Bellman backups. Treat me like I barely 234 00:10:34,480 --> 00:10:36,919 passed high school math here. What is a Bellman 235 00:10:36,919 --> 00:10:39,789 backup actually doing? in the context of our 236 00:10:39,789 --> 00:10:42,710 video game playing robot. I apologize. That is 237 00:10:42,710 --> 00:10:46,009 a dense term. Simply put, a Bellman backup is 238 00:10:46,009 --> 00:10:48,269 a mathematical equation used to estimate the 239 00:10:48,269 --> 00:10:51,190 value of being in a certain state. Okay. Imagine 240 00:10:51,190 --> 00:10:53,450 you are playing that video game and you finally 241 00:10:53,450 --> 00:10:55,610 find a treasure chest. You get a massive reward. 242 00:10:56,330 --> 00:10:58,669 A Bellman backup is the mathematical mechanism 243 00:10:58,669 --> 00:11:01,629 that says, okay, the state immediately before 244 00:11:01,629 --> 00:11:04,509 I opened the chest was also highly valuable because 245 00:11:04,509 --> 00:11:06,820 it led to the chest. And the state before that 246 00:11:06,820 --> 00:11:10,039 was pretty valuable too. It propagates the value 247 00:11:10,039 --> 00:11:12,740 of the reward backward through the sequence of 248 00:11:12,740 --> 00:11:16,120 actions that led to it. Oh, okay. Like leaving 249 00:11:16,120 --> 00:11:18,659 a trail of breadcrumbs leading back from the 250 00:11:18,659 --> 00:11:21,039 treasure so next time you know you're getting 251 00:11:21,039 --> 00:11:24,830 warmer. Yes. But the limitation is that it requires 252 00:11:24,830 --> 00:11:26,990 massive amounts of interaction with the environment 253 00:11:26,990 --> 00:11:30,409 to slowly adjust that policy. You have to stumble 254 00:11:30,409 --> 00:11:33,330 upon the treasure many, many times from many 255 00:11:33,330 --> 00:11:36,029 different paths to build a good, reliable trail 256 00:11:36,029 --> 00:11:38,830 of breadcrumbs. Which takes forever in simulation 257 00:11:38,830 --> 00:11:41,409 and is basically impossible for a real fragile 258 00:11:41,409 --> 00:11:44,450 robot. Exactly. Which brings us to the model 259 00:11:44,450 --> 00:11:47,110 -based approach which Planet uses. How is that 260 00:11:47,110 --> 00:11:49,549 different? With a model -based approach, the 261 00:11:49,549 --> 00:11:51,950 agent is actively trying to build an internal 262 00:11:51,950 --> 00:11:54,570 model of how the world works. It gets a much 263 00:11:54,570 --> 00:11:58,230 richer training signal. How so? Well, it is learning 264 00:11:58,230 --> 00:12:00,970 constantly by trying to predict the very next 265 00:12:00,970 --> 00:12:03,769 observation, regardless of whether it received 266 00:12:03,769 --> 00:12:06,690 a reward or a penalty. If it predicts the ball 267 00:12:06,690 --> 00:12:09,529 will fall down and it falls up, it updates its 268 00:12:09,529 --> 00:12:12,330 model. It doesn't need to wait for a treasure 269 00:12:12,330 --> 00:12:14,529 chest reward to learn something useful about 270 00:12:14,529 --> 00:12:17,820 the world. That makes a lot of sense. Furthermore, 271 00:12:18,220 --> 00:12:21,159 having a learned model carries a massive advantage. 272 00:12:21,539 --> 00:12:23,879 You can improve your performance simply by increasing 273 00:12:23,879 --> 00:12:26,600 your computational search budget. Meaning what, 274 00:12:26,720 --> 00:12:29,639 exactly? Meaning, if you have a good internal 275 00:12:29,639 --> 00:12:33,039 model of the physics, you can just pause, spend 276 00:12:33,039 --> 00:12:35,240 more computer processing time searching through 277 00:12:35,240 --> 00:12:38,419 possible futures in your imagination to find 278 00:12:38,419 --> 00:12:40,700 a better plan without needing to interact with 279 00:12:40,700 --> 00:12:43,480 the real world any further. You think before 280 00:12:43,480 --> 00:12:46,460 you act. Okay, so... If model -based learning 281 00:12:46,460 --> 00:12:49,240 is so intuitive, if building a mental map of 282 00:12:49,240 --> 00:12:52,559 how the world works is clearly so advantageous 283 00:12:52,559 --> 00:12:55,360 for adapting and being data efficient, why hasn't 284 00:12:55,360 --> 00:12:58,019 it always been used for pixel -based environments 285 00:12:58,019 --> 00:13:01,940 before this 2019 paper? Why were model -free 286 00:13:01,940 --> 00:13:07,019 algorithms like A3C or D4PG the absolute standard 287 00:13:07,019 --> 00:13:09,399 for so long in these complex visual tasks? It 288 00:13:09,399 --> 00:13:11,919 seems like model -based is obviously better. 289 00:13:12,149 --> 00:13:14,470 Let's be very careful there because that is a 290 00:13:14,470 --> 00:13:17,629 trap many people fall into. Ooh, really? Yes. 291 00:13:17,850 --> 00:13:20,490 We cannot mischaracterize model -free algorithms 292 00:13:20,490 --> 00:13:23,409 as primitive. It is essential to understand that 293 00:13:23,409 --> 00:13:26,429 model -free methods, despite not explicitly learning 294 00:13:26,429 --> 00:13:28,350 a predictive transition model of the future, 295 00:13:28,909 --> 00:13:31,850 are absolutely not just engaging in mindless, 296 00:13:32,009 --> 00:13:34,590 brittle memorization. So they aren't just blindly 297 00:13:34,590 --> 00:13:38,059 memorizing button presses? No, not at all. Modern 298 00:13:38,059 --> 00:13:40,440 model -free deep reinforcement learning algorithms 299 00:13:40,440 --> 00:13:43,919 can learn incredibly rich, complex, and generalized 300 00:13:43,919 --> 00:13:46,000 representations of the environment hidden within 301 00:13:46,000 --> 00:13:48,379 their neural networks. They can be incredibly 302 00:13:48,379 --> 00:13:51,019 robust and achieve astonishingly high performance 303 00:13:51,019 --> 00:13:54,720 on visual tasks. OK, so if they're so good, why 304 00:13:54,720 --> 00:13:57,350 the push for model -based? Because the historical 305 00:13:57,350 --> 00:13:59,909 reason model -free methods dominated pixel -based 306 00:13:59,909 --> 00:14:02,570 tasks wasn't because they were conceptually superior, 307 00:14:03,149 --> 00:14:05,850 it was because learning an accurate forward predictive 308 00:14:05,850 --> 00:14:08,789 model from high dimensional pixels was simply 309 00:14:08,789 --> 00:14:11,610 too mathematically difficult. Ah, because of 310 00:14:11,610 --> 00:14:13,590 the compounding errors we talked about earlier. 311 00:14:13,889 --> 00:14:16,870 Exactly. The accumulating errors would render 312 00:14:16,870 --> 00:14:19,509 the learned model useless for planning if your 313 00:14:19,509 --> 00:14:22,039 mental model of the world devolves into static 314 00:14:22,039 --> 00:14:24,139 after just three steps, you are better off just 315 00:14:24,139 --> 00:14:26,159 using a model -free policy based on immediate 316 00:14:26,159 --> 00:14:29,320 reactions. That makes total sense. So planet 317 00:14:29,320 --> 00:14:31,960 success is a major milestone because it demonstrates 318 00:14:31,960 --> 00:14:35,399 extreme data efficiency in a specific setup using 319 00:14:35,399 --> 00:14:38,039 a model -based approach. But, and I want to really 320 00:14:38,039 --> 00:14:40,539 underline this, it does not prove that model 321 00:14:40,539 --> 00:14:43,519 -based methods will fundamentally or always dominate 322 00:14:43,519 --> 00:14:46,539 model -free approaches in every scenario. They 323 00:14:46,539 --> 00:14:49,139 are different paradigms with different mathematical 324 00:14:49,139 --> 00:14:52,110 strengths. OK, that context is incredibly helpful. 325 00:14:52,769 --> 00:14:54,629 Planet didn't invent model -based planning, but 326 00:14:54,629 --> 00:14:56,649 it managed to overcome the historical hurdles 327 00:14:56,649 --> 00:14:59,330 that made it so hard to do from raw pixels. So 328 00:14:59,330 --> 00:15:01,129 let's look at how it actually achieves that. 329 00:15:02,710 --> 00:15:04,950 Let's step into the agent's shoes. What does 330 00:15:04,950 --> 00:15:07,950 it actually see? And how does it process that 331 00:15:07,950 --> 00:15:10,169 raw information? Because it doesn't get to look 332 00:15:10,169 --> 00:15:12,169 at the underlying code of the physics simulation. 333 00:15:12,490 --> 00:15:15,049 Right. The paper lays out the problems set up 334 00:15:15,049 --> 00:15:17,789 very clearly in section two. They formalize the 335 00:15:17,789 --> 00:15:20,309 environment as a partially observable Markov 336 00:15:20,309 --> 00:15:25,149 decision process, or POMDP. POMDP. That is quite 337 00:15:25,149 --> 00:15:27,950 a mouthful of an acronym. It sounds intimidating, 338 00:15:28,049 --> 00:15:30,610 but the core concept is quite grounded. The most 339 00:15:30,610 --> 00:15:33,259 important part for our discussion is the partially 340 00:15:33,259 --> 00:15:35,980 observable bit. It just means the agent is acting 341 00:15:35,980 --> 00:15:38,299 in a world where its current observation, the 342 00:15:38,299 --> 00:15:40,620 picture it gets in any single moment, doesn't 343 00:15:40,620 --> 00:15:42,580 tell it everything it needs to know about the 344 00:15:42,580 --> 00:15:45,639 true underlying state of the world. Let me try 345 00:15:45,639 --> 00:15:48,480 to give a relatable example of partial observability 346 00:15:48,480 --> 00:15:52,120 for you listening. If I show you a single, perfectly 347 00:15:52,120 --> 00:15:54,460 still photograph of a baseball hovering in midair 348 00:15:54,460 --> 00:15:57,179 above a grassy field, you know where the baseball 349 00:15:57,179 --> 00:16:00,110 is. You know its position. Right. But you don't 350 00:16:00,110 --> 00:16:02,269 know how fast it's moving. You don't know if 351 00:16:02,269 --> 00:16:04,269 it's flying toward the camera at 90 miles an 352 00:16:04,269 --> 00:16:07,110 hour or if it was tossed gently straight up and 353 00:16:07,110 --> 00:16:09,529 is currently hanging exactly at the apex of its 354 00:16:09,529 --> 00:16:13,029 arc. A single frame doesn't capture velocity 355 00:16:13,029 --> 00:16:15,629 or acceleration. That is an excellent way to 356 00:16:15,629 --> 00:16:17,950 illustrate it. So the agent cannot just look 357 00:16:17,950 --> 00:16:20,409 at one image and know exactly what is going on. 358 00:16:20,669 --> 00:16:23,190 And to make it harder, the images the agent receives 359 00:16:23,190 --> 00:16:26,629 at every discrete time step are quite low resolution. 360 00:16:26,990 --> 00:16:29,659 How low are we talking? They are downscaled to 361 00:16:29,659 --> 00:16:33,200 64 by 64 pixels with three color channels. It 362 00:16:33,200 --> 00:16:35,940 is a very blurry, low -information view of the 363 00:16:35,940 --> 00:16:39,539 world. Oh wow, 64 by 64? Yeah. That is tiny. 364 00:16:39,919 --> 00:16:43,149 Yes. And based on that blurry view, it has to 365 00:16:43,149 --> 00:16:46,110 output what they call a continuous action vector. 366 00:16:46,210 --> 00:16:48,389 Meaning? Which means it's not just pressing a 367 00:16:48,389 --> 00:16:51,549 digital D -pad up or down. It can apply varying 368 00:16:51,549 --> 00:16:54,769 degrees of force to its robotic joints, like 369 00:16:54,769 --> 00:16:57,470 slightly twisting a wrist or applying 30 % power 370 00:16:57,470 --> 00:17:00,250 to a knee actuator. And at each step, it receives 371 00:17:00,250 --> 00:17:03,049 a scalar reward, a single number telling it how 372 00:17:03,049 --> 00:17:06,140 well it's doing. OK. So, given that a single 373 00:17:06,140 --> 00:17:09,740 64 by 64 image is completely insufficient to 374 00:17:09,740 --> 00:17:11,579 understand the state of the world, it has no 375 00:17:11,579 --> 00:17:15,160 velocity data, how does the agent proceed? Because 376 00:17:15,160 --> 00:17:16,920 it can't plan if it doesn't know how fast the 377 00:17:16,920 --> 00:17:19,849 baseball's moving. This is where the paper introduces 378 00:17:19,849 --> 00:17:23,170 two vital intertwined components, the encoder 379 00:17:23,170 --> 00:17:26,299 and the observation model. Because an individual 380 00:17:26,299 --> 00:17:29,220 image lacks velocity and context, the agent needs 381 00:17:29,220 --> 00:17:31,740 to compress the entire history of what it has 382 00:17:31,740 --> 00:17:36,160 seen and what actions it has taken into a compact 383 00:17:36,160 --> 00:17:38,359 mathematical representation. A summary of its 384 00:17:38,359 --> 00:17:42,059 life so far. Yes. The encoder is a neural network. 385 00:17:42,460 --> 00:17:45,180 Its job is to take the history of past observations 386 00:17:45,180 --> 00:17:49,180 and past actions and infer an approximate posterior 387 00:17:49,180 --> 00:17:51,880 belief over the current hidden state of the environment. 388 00:17:52,140 --> 00:17:54,359 Okay, I have to throw the flag on the play again. 389 00:17:55,039 --> 00:17:57,700 Approximate posterior belief. Let's translate 390 00:17:57,700 --> 00:18:00,859 that. Posterior in statistics usually refers 391 00:18:00,859 --> 00:18:03,279 to what we believe after we've seen some evidence, 392 00:18:03,579 --> 00:18:06,400 right? Correct. The prior is your belief before 393 00:18:06,400 --> 00:18:09,240 you see the evidence. The posterior is your updated 394 00:18:09,240 --> 00:18:11,220 belief after you look at the evidence, which 395 00:18:11,220 --> 00:18:13,440 in this case is the sequence of blurry images. 396 00:18:13,819 --> 00:18:15,960 So the encoder is looking at the history of pixels 397 00:18:15,960 --> 00:18:18,480 and compressing them into a smaller set of numbers, 398 00:18:18,799 --> 00:18:21,609 a latent state. And that latent state is essentially 399 00:18:21,609 --> 00:18:23,970 its statistical belief about everything important 400 00:18:23,970 --> 00:18:26,269 happening in the world right now. Exactly. It 401 00:18:26,269 --> 00:18:28,930 represents the velocity of the baseball, even 402 00:18:28,930 --> 00:18:31,829 if it can't explicitly name it as velocity. But 403 00:18:31,829 --> 00:18:34,289 how does the AI know if its compressed latent 404 00:18:34,289 --> 00:18:36,930 state is actually accurate? How does it check 405 00:18:36,930 --> 00:18:39,509 its work? That is the job of the observation 406 00:18:39,509 --> 00:18:42,349 model. The observation model takes that compact 407 00:18:42,349 --> 00:18:44,950 internal latent state, that array of numbers, 408 00:18:45,289 --> 00:18:48,410 and tries to run it in reverse. Run it in reverse? 409 00:18:48,720 --> 00:18:52,140 Yes, it tries to generate or reconstruct a 64 410 00:18:52,140 --> 00:18:55,740 by 64 image that matches the actual real observation 411 00:18:55,740 --> 00:18:57,940 it just received from the environment. Oh, I 412 00:18:57,940 --> 00:19:01,099 see. It tries to draw a picture of what it thinks 413 00:19:01,099 --> 00:19:03,980 it's seeing based on its internal summary. Yes. 414 00:19:04,680 --> 00:19:07,519 And the mathematical error, the pixel -by -pixel 415 00:19:07,519 --> 00:19:09,740 difference between the image the observation 416 00:19:09,740 --> 00:19:12,400 model generated and the real image provides a 417 00:19:12,400 --> 00:19:15,039 massive training signal. By constantly trying 418 00:19:15,039 --> 00:19:18,000 to minimize that reconstruction error, the encoder 419 00:19:18,000 --> 00:19:20,240 learns to pack the most relevant, physically 420 00:19:20,240 --> 00:19:22,859 accurate information into that compact latent 421 00:19:22,859 --> 00:19:24,779 state. Okay, let me pause you right there because 422 00:19:24,779 --> 00:19:26,559 this is where I think a lot of people, including 423 00:19:26,559 --> 00:19:30,460 myself, get tempted to jump to a massive conclusion. 424 00:19:30,779 --> 00:19:33,299 Go ahead. If the model gets really, really good 425 00:19:33,299 --> 00:19:35,680 at drawing a picture of what happens next, if 426 00:19:35,680 --> 00:19:37,900 its observation model can perfectly reconstruct 427 00:19:37,900 --> 00:19:39,880 the pixels of that baseball flying through the 428 00:19:39,880 --> 00:19:42,220 air, complete with motion blur and lighting, 429 00:19:43,240 --> 00:19:45,160 doesn't that inherently mean the neural network 430 00:19:45,160 --> 00:19:47,900 has developed a true fundamental understanding 431 00:19:47,900 --> 00:19:50,480 of the physics of the world? I have to push back 432 00:19:50,480 --> 00:19:53,619 on that formulation firmly because it is perhaps 433 00:19:53,619 --> 00:19:56,119 the most common misconception in deep learning. 434 00:19:56,299 --> 00:19:59,180 we must rigorously emphasize that the observation 435 00:19:59,180 --> 00:20:01,880 model simply provides a mathematical training 436 00:20:01,880 --> 00:20:04,900 signal. The successful reconstruction of pixels 437 00:20:04,900 --> 00:20:07,539 on a screen does not prove that the learned dynamics 438 00:20:07,539 --> 00:20:10,720 inside the latent space are fundamentally correct. 439 00:20:11,119 --> 00:20:13,700 Really? But if it looks perfectly white, doesn't 440 00:20:13,700 --> 00:20:16,440 it mean it understands it? No. And it definitely 441 00:20:16,440 --> 00:20:19,640 does not mean they represent guaranteed objective 442 00:20:19,640 --> 00:20:21,980 physical laws like Newton's laws of motion. Why 443 00:20:21,980 --> 00:20:24,799 not? The loss function, the mathematical goal 444 00:20:24,799 --> 00:20:27,400 the AI is trying to achieve, simply tells the 445 00:20:27,400 --> 00:20:30,460 network to make the output pixels match the input 446 00:20:30,460 --> 00:20:33,279 pixels. The neural network will find whatever 447 00:20:33,279 --> 00:20:35,960 mathematical shortcut, whatever twisted geometry 448 00:20:35,960 --> 00:20:39,079 it can in the latent space to minimize that specific 449 00:20:39,079 --> 00:20:41,759 pixel error. It doesn't care about the elegance 450 00:20:41,759 --> 00:20:44,339 of physics. It just cares about the math checking 451 00:20:44,339 --> 00:20:48,640 out. So it might be doing completely wild, unphysical 452 00:20:48,640 --> 00:20:51,240 math behind the scenes as long as the final picture 453 00:20:51,240 --> 00:20:54,299 looks right. Exactly. Furthermore, the encoder 454 00:20:54,299 --> 00:20:57,700 produces approximate statistical beliefs. These 455 00:20:57,700 --> 00:21:00,240 are probability distributions over vectors of 456 00:21:00,240 --> 00:21:04,140 numbers. They are not verified ground truth physical 457 00:21:04,140 --> 00:21:06,700 states of the universe. So it's not actually 458 00:21:06,700 --> 00:21:09,680 thinking, like, the mass is x and the velocity 459 00:21:09,680 --> 00:21:13,660 is y? No, not at all. We must avoid anthropomorphizing 460 00:21:13,660 --> 00:21:16,599 these latent representations as literal thoughts 461 00:21:16,599 --> 00:21:19,880 or true comprehension of reality. The agent is 462 00:21:19,880 --> 00:21:22,319 optimizing a mathematical objective in a high 463 00:21:22,319 --> 00:21:25,779 dimensional space. It is a highly useful, incredibly 464 00:21:25,779 --> 00:21:28,119 effective statistical proxy for reality, but 465 00:21:28,119 --> 00:21:31,960 a proxy is all it is. That is a phenomenal clarification. 466 00:21:32,279 --> 00:21:34,759 It really keeps us grounded. It's a statistical 467 00:21:34,759 --> 00:21:38,299 proxy. So, okay, we have an encoder that gives 468 00:21:38,299 --> 00:21:40,519 us a statistical belief about the present moment 469 00:21:40,519 --> 00:21:44,839 based on past pixels. But to plan, as we established, 470 00:21:45,299 --> 00:21:47,900 the agent must predict the future. Right. And 471 00:21:47,900 --> 00:21:50,359 it obviously can't just keep feeding actual future 472 00:21:50,359 --> 00:21:53,079 images into its encoder because those images 473 00:21:53,079 --> 00:21:56,519 haven't happened yet. Which brings us to the 474 00:21:56,519 --> 00:21:59,140 absolute structural heart of the planet paper. 475 00:21:59,220 --> 00:22:03,119 This is the engine of imagination. the recurrent 476 00:22:03,119 --> 00:22:06,779 state space model or RSSM. This is where the 477 00:22:06,779 --> 00:22:09,640 engineering gets remarkably elegant. The thing 478 00:22:09,640 --> 00:22:12,259 about the computational burden here, to plan 479 00:22:12,259 --> 00:22:14,880 an optimal sequence of movements for a robotic 480 00:22:14,880 --> 00:22:17,900 arm, you need to evaluate thousands of potential 481 00:22:17,900 --> 00:22:20,349 action sequences in your head. Sure. If you had 482 00:22:20,349 --> 00:22:23,029 to use your observation model to generate a full 483 00:22:23,029 --> 00:22:26,809 64 by 64 pixel image for every single microstep 484 00:22:26,809 --> 00:22:29,650 of every single imagined future, the computational 485 00:22:29,650 --> 00:22:31,910 costs would be astronomical. Yeah, you'd spend 486 00:22:31,910 --> 00:22:34,150 all your computing power just rendering the graphics 487 00:22:34,150 --> 00:22:37,509 of your own imagination. Precisely. It would 488 00:22:37,509 --> 00:22:40,549 take a supercomputer hours to plan a three -second 489 00:22:40,549 --> 00:22:43,509 movement. It would be entirely unfeasible for 490 00:22:43,509 --> 00:22:46,859 real -time robotic control. So the transition 491 00:22:46,859 --> 00:22:50,359 model architecture and planet, the RSSM, is designed 492 00:22:50,359 --> 00:22:52,880 to allow the planner to operate purely in the 493 00:22:52,880 --> 00:22:56,920 compact latent space. OK. It predicts the next 494 00:22:56,920 --> 00:22:58,859 latent state mathematically from the current 495 00:22:58,859 --> 00:23:01,559 latent state and the chosen action without ever 496 00:23:01,559 --> 00:23:04,180 needing to generate the expensive pixels in between. 497 00:23:04,700 --> 00:23:07,779 So it's imagining the future as shifting vectors 498 00:23:07,779 --> 00:23:11,799 of numbers, not as a mental movie. It only reconstructs 499 00:23:11,799 --> 00:23:14,480 pixels during the training phase to refine its 500 00:23:14,480 --> 00:23:16,380 model, but during the actual planning phase, 501 00:23:16,720 --> 00:23:19,200 when it's trying to act quickly, it stays entirely 502 00:23:19,200 --> 00:23:22,079 in the math of the latent space. Yes. And let's 503 00:23:22,079 --> 00:23:23,940 talk about the structure of that transition model, 504 00:23:23,980 --> 00:23:26,740 because the researchers made a major architectural 505 00:23:26,740 --> 00:23:29,539 finding here. What did they find? They discovered 506 00:23:29,539 --> 00:23:32,019 that to make this work, it was absolutely necessary 507 00:23:32,019 --> 00:23:34,359 to split the latent state into two different 508 00:23:34,359 --> 00:23:37,640 complementary paths, a deterministic path and 509 00:23:37,640 --> 00:23:39,990 a stochastic path. Two paths. Let's break those 510 00:23:39,990 --> 00:23:42,690 down. What is the deterministic path doing? The 511 00:23:42,690 --> 00:23:45,210 deterministic path uses a recurrent neural network, 512 00:23:45,549 --> 00:23:48,210 or RNN. The primary role of this deterministic 513 00:23:48,210 --> 00:23:51,009 path is to allow the model to robustly remember 514 00:23:51,009 --> 00:23:53,890 information over many, many time steps. OK, so 515 00:23:53,890 --> 00:23:57,309 it's the memory. Yes. In a standard RNN, the 516 00:23:57,309 --> 00:23:59,450 hidden state updates deterministically based 517 00:23:59,450 --> 00:24:01,730 on the previous state and the new input. There 518 00:24:01,730 --> 00:24:04,529 is no randomness. This is fantastic for holding 519 00:24:04,529 --> 00:24:06,910 onto a rigid memory. For example, remembering 520 00:24:06,910 --> 00:24:09,509 that a block is sitting behind you, even if you've 521 00:24:09,509 --> 00:24:11,109 turned the camera and can't currently see it. 522 00:24:11,289 --> 00:24:14,769 Right. It's the rigid memory. But if your entire 523 00:24:14,769 --> 00:24:17,549 prediction of the future is purely deterministic, 524 00:24:17,670 --> 00:24:19,470 you run into one of those roadblocks we talked 525 00:24:19,470 --> 00:24:21,930 about in the very beginning, failing to capture 526 00:24:21,930 --> 00:24:25,450 multiple possible futures. Exactly. If the transition 527 00:24:25,450 --> 00:24:27,930 model is entirely deterministic, it has a very 528 00:24:27,930 --> 00:24:30,190 hard time dealing with the inherent uncertainty 529 00:24:30,190 --> 00:24:33,029 of the environment. If the robot's initial starting 530 00:24:33,029 --> 00:24:35,369 position is slightly ambiguous because the first 531 00:24:35,369 --> 00:24:38,309 image was blurry, a deterministic model is forced 532 00:24:38,309 --> 00:24:41,930 to pick a single average future trajectory. And 533 00:24:41,930 --> 00:24:44,349 what's wrong with that? Mathematically, averaging 534 00:24:44,349 --> 00:24:47,069 two possible physical outcomes often results 535 00:24:47,069 --> 00:24:49,309 in a prediction that is physically impossible 536 00:24:49,309 --> 00:24:52,789 in reality. Imagine averaging going around the 537 00:24:52,789 --> 00:24:55,130 tree to the left and going around the tree to 538 00:24:55,130 --> 00:24:57,109 the right. You end up predicting you will walk 539 00:24:57,109 --> 00:24:59,529 straight through the solid trunk. Yeah, that's 540 00:24:59,529 --> 00:25:02,440 a problem. So to fix that, PLANET introduces 541 00:25:02,440 --> 00:25:05,500 the second path, a stochastic state -space model, 542 00:25:06,000 --> 00:25:08,680 alongside the deterministic RNN. Stochastic just 543 00:25:08,680 --> 00:25:11,059 means involving a random variable, right? How 544 00:25:11,059 --> 00:25:13,980 does that work? At every time step in the model, 545 00:25:14,079 --> 00:25:16,619 a stochastic latent variable is mathematically 546 00:25:16,619 --> 00:25:19,740 sampled from a distribution. And crucially, this 547 00:25:19,740 --> 00:25:22,619 distribution is parameterized, meaning its shape 548 00:25:22,619 --> 00:25:25,779 is determined by the deterministic state. OK. 549 00:25:26,440 --> 00:25:28,779 This stochastic path allows the model to explicitly 550 00:25:28,779 --> 00:25:32,220 represent uncertainty. It allows it to say, based 551 00:25:32,220 --> 00:25:35,140 on what I know deterministically, the next state 552 00:25:35,140 --> 00:25:37,099 could be within this range of possibilities. 553 00:25:37,920 --> 00:25:40,759 OK, I want to try an analogy to see if I'm grasping 554 00:25:40,759 --> 00:25:43,039 how these two paths work together to make a plan. 555 00:25:43,700 --> 00:25:47,690 Tell me if I'm way off base. Go for it. The deterministic 556 00:25:47,690 --> 00:25:50,690 part is like remembering the constant, unchanging 557 00:25:50,690 --> 00:25:53,049 rules of the world. Remembering that the ground 558 00:25:53,049 --> 00:25:55,950 is solid, that gravity pulls down, that your 559 00:25:55,950 --> 00:25:58,670 robotic arm is exactly two feet long. Those things 560 00:25:58,670 --> 00:26:00,730 don't randomly change from second to second. 561 00:26:00,750 --> 00:26:04,089 I follow you. And the stochastic part is acknowledging 562 00:26:04,089 --> 00:26:06,410 the chaos, acknowledging that a gust of wind 563 00:26:06,410 --> 00:26:08,690 might blow, or that the block you're trying to 564 00:26:08,690 --> 00:26:11,269 pick up might be a millimeter to the left of 565 00:26:11,269 --> 00:26:13,809 where your blurry vision thinks it is, or that 566 00:26:13,809 --> 00:26:15,950 there's just electrical noise in your own motors. 567 00:26:16,569 --> 00:26:19,250 We need both the rigid memory of the rules, the 568 00:26:19,250 --> 00:26:21,690 deterministic, and the flexible acknowledgement 569 00:26:21,690 --> 00:26:24,809 of uncertainty, the stochastic, to make a plan 570 00:26:24,809 --> 00:26:26,869 that won't immediately fail when reality isn't 571 00:26:26,869 --> 00:26:29,730 perfect. That analogy provides a very good intuitive 572 00:26:29,730 --> 00:26:32,390 grasp of why both elements are practically necessary 573 00:26:32,390 --> 00:26:34,849 for robust planning in these messy environments. 574 00:26:35,549 --> 00:26:38,549 However, I need to add a fairly technical distinction 575 00:26:38,549 --> 00:26:40,710 here, because it's a point of confusion in the 576 00:26:40,710 --> 00:26:44,369 machine learning community. We must be very careful 577 00:26:44,369 --> 00:26:46,829 not to imply that recurrent neural networks, 578 00:26:47,029 --> 00:26:50,769 the deterministic part, are inherently mathematically 579 00:26:50,769 --> 00:26:52,769 incapable of dealing with uncertainty on their 580 00:26:52,769 --> 00:26:56,039 own. Oh, really? I thought... Deterministic meant 581 00:26:56,039 --> 00:26:59,119 zero uncertainty, like by definition. A deterministic 582 00:26:59,119 --> 00:27:01,539 neural network can absolutely be used to output 583 00:27:01,539 --> 00:27:03,900 the parameters of a probability distribution. 584 00:27:04,779 --> 00:27:07,180 A deterministic network can output two numbers 585 00:27:07,180 --> 00:27:10,059 representing a mean and a variance, thereby describing 586 00:27:10,059 --> 00:27:12,559 uncertainty. So if it can do that, why do they 587 00:27:12,559 --> 00:27:15,849 need the explicit stochastic path? The specific 588 00:27:15,849 --> 00:27:18,630 reason this dual architecture splitting them 589 00:27:18,630 --> 00:27:21,490 explicitly is so critical in planet is actually 590 00:27:21,490 --> 00:27:24,190 more structural than philosophical. Structural 591 00:27:24,190 --> 00:27:28,000 how? In this specific studied architecture, the 592 00:27:28,000 --> 00:27:30,000 researchers set it up so that all information 593 00:27:30,000 --> 00:27:32,519 about the visual observations must physically 594 00:27:32,519 --> 00:27:34,819 pass through the sampling step of the stochastic 595 00:27:34,819 --> 00:27:37,599 encoder. This creates what we call a stochastic 596 00:27:37,599 --> 00:27:41,039 bottleneck. A bottleneck. So it's forcing the 597 00:27:41,039 --> 00:27:44,619 data through a narrow passage. Exactly. It specifically 598 00:27:44,619 --> 00:27:47,779 prevents the neural network from finding a deterministic 599 00:27:47,779 --> 00:27:50,460 shortcut directly from the image inputs to the 600 00:27:50,460 --> 00:27:52,690 image reconstructions. Right, the shortcut. If 601 00:27:52,690 --> 00:27:54,650 it had that shortcut, it might just act like 602 00:27:54,650 --> 00:27:56,890 a Xerox machine copying the input to the output 603 00:27:56,890 --> 00:27:59,029 without actually bothering to build a model of 604 00:27:59,029 --> 00:28:02,009 the underlying physics. By forcing the information 605 00:28:02,009 --> 00:28:04,849 through a sampled stochastic variable, the model 606 00:28:04,849 --> 00:28:08,089 is compelled to learn a more robust, meaningful 607 00:28:08,089 --> 00:28:10,769 representation of the state. That is incredibly 608 00:28:10,769 --> 00:28:14,160 clever. So it's not that RNNs can't handle uncertainty 609 00:28:14,160 --> 00:28:16,720 in general, it's that this specific bottlenecking 610 00:28:16,720 --> 00:28:19,359 structure forces the network to learn the hard 611 00:28:19,359 --> 00:28:21,680 way, solving a critical optimization problem 612 00:28:21,680 --> 00:28:24,700 in this specific setup. And importantly, it doesn't 613 00:28:24,700 --> 00:28:27,019 magically solve all possible memorization or 614 00:28:27,019 --> 00:28:29,420 model errors perfectly, but the ablations show 615 00:28:29,420 --> 00:28:32,259 it is highly effective here. Okay, that is fascinating. 616 00:28:33,119 --> 00:28:35,940 The structure of the math forces the AI to be 617 00:28:35,940 --> 00:28:39,450 honest and actually learn the dynamics. rather 618 00:28:39,450 --> 00:28:42,190 than taking a lazy mathematical shortcut. Okay, 619 00:28:42,509 --> 00:28:46,269 so now this is cool because now we have built 620 00:28:46,269 --> 00:28:49,710 our engine of imagination, okay? We have a robust 621 00:28:49,710 --> 00:28:52,309 model capable of predicting future latent states, 622 00:28:52,910 --> 00:28:55,910 predicting the rewards associated with those 623 00:28:55,910 --> 00:28:59,549 states, all while staying purely in this compact, 624 00:29:00,150 --> 00:29:05,730 fast mathematical space. Cool. But dreaming about 625 00:29:05,730 --> 00:29:07,890 the future isn't the same as acting in the present. 626 00:29:08,109 --> 00:29:13,890 Right? So how does the agent actually use this 627 00:29:13,890 --> 00:29:16,890 engine to decide what action, like physical action, 628 00:29:16,910 --> 00:29:19,009 to take right now in the simulation? This brings 629 00:29:19,009 --> 00:29:21,769 us to the actual planning algorithm. Planet uses 630 00:29:21,769 --> 00:29:24,009 a technique called the Cross -Entropy Method, 631 00:29:24,190 --> 00:29:27,569 or CEM. It is a population -based optimization 632 00:29:27,569 --> 00:29:30,250 algorithm. When the agent needs to make a decision, 633 00:29:30,390 --> 00:29:33,210 it essentially runs a massive high -speed internal 634 00:29:33,210 --> 00:29:35,950 simulation. Let's walk the listener through the 635 00:29:35,950 --> 00:29:38,710 exact mechanics of CEM. step by step, because 636 00:29:38,710 --> 00:29:41,329 it's a really cool, almost evolutionary concept. 637 00:29:42,009 --> 00:29:45,180 How does the brainstorming session start? It 638 00:29:45,180 --> 00:29:48,700 starts with a blank slate. Specifically, it initializes 639 00:29:48,700 --> 00:29:51,079 a belief over what the optimal sequence of future 640 00:29:51,079 --> 00:29:54,380 actions might look like. Mathematically, it initializes 641 00:29:54,380 --> 00:29:57,359 a time -dependent diagonal Gaussian belief over 642 00:29:57,359 --> 00:30:00,180 action sequences, starting with zero mean and 643 00:30:00,180 --> 00:30:04,670 unit variance. OK, pause. Diagonal Gaussian belief 644 00:30:04,670 --> 00:30:07,930 with zero mean and unit variance. Let's translate 645 00:30:07,930 --> 00:30:09,869 that into plain English for those of us who don't 646 00:30:09,869 --> 00:30:13,029 dream in math. A Gaussian distribution is just 647 00:30:13,029 --> 00:30:15,630 a bell curve. The mean is the center of the bell 648 00:30:15,630 --> 00:30:18,609 curve where we think the best action is. The 649 00:30:18,609 --> 00:30:21,269 variance is how wide the bell curve is, how unsure 650 00:30:21,269 --> 00:30:24,910 we are. By starting with zero mean and unit variance, 651 00:30:25,170 --> 00:30:27,690 the algorithm is essentially saying, at first 652 00:30:27,690 --> 00:30:30,130 I have absolutely no idea what actions are best. 653 00:30:30,369 --> 00:30:32,869 Any combination of pushing the arm left, right, 654 00:30:32,970 --> 00:30:35,430 up or down is equally likely to be the right 655 00:30:35,430 --> 00:30:38,029 answer. It's casting the widest possible net. 656 00:30:38,710 --> 00:30:41,230 So it's completely guessing at first. From this 657 00:30:41,230 --> 00:30:45,150 wide, random net, what happens next? From this 658 00:30:45,150 --> 00:30:47,710 initial broad distribution, the algorithm samples 659 00:30:47,710 --> 00:30:50,029 a large number of candidate action sequences. 660 00:30:50,150 --> 00:30:53,470 Let's call that number J. So it randomly brainstorms, 661 00:30:53,470 --> 00:30:56,089 say, a thousand completely different sequences 662 00:30:56,089 --> 00:30:58,269 of joystick movements. It could make over the 663 00:30:58,269 --> 00:31:02,819 next... say, 12 time steps. Yes. Then it evaluates 664 00:31:02,819 --> 00:31:04,940 every single one of those thousand sequences. 665 00:31:05,539 --> 00:31:08,420 It feeds each sequence of actions into its learned 666 00:31:08,420 --> 00:31:11,319 transition model, that RSSM we just talked about. 667 00:31:11,440 --> 00:31:14,000 Right. It simulates the future states purely 668 00:31:14,000 --> 00:31:16,920 in the fast latent space, and it uses its learned 669 00:31:16,920 --> 00:31:19,440 reward model to estimate the total score it would 670 00:31:19,440 --> 00:31:22,410 get for each sequence. Because this is all happening 671 00:31:22,410 --> 00:31:25,190 in the latent space, it can evaluate these thousands 672 00:31:25,190 --> 00:31:28,529 of sequences incredibly fast. Once they are all 673 00:31:28,529 --> 00:31:31,769 scored, it sorts them. It takes the top K sequences, 674 00:31:31,930 --> 00:31:33,549 the ones that produced the highest predicted 675 00:31:33,549 --> 00:31:35,710 reward. Okay, so it takes the best of the bunch, 676 00:31:35,990 --> 00:31:37,829 the movements that didn't immediately cause the 677 00:31:37,829 --> 00:31:40,789 robot to fall over. And then what? It doesn't 678 00:31:40,789 --> 00:31:43,450 just execute the best one right away, right? 679 00:31:43,650 --> 00:31:46,890 No, this is where the optimization happens. It 680 00:31:46,890 --> 00:31:49,509 uses those top -case sequences to update its 681 00:31:49,509 --> 00:31:52,710 belief. It refits its Gaussian distribution to 682 00:31:52,710 --> 00:31:55,089 match the mean and variance of those top -performing 683 00:31:55,089 --> 00:31:57,730 sequences. How does that look conceptually? It's 684 00:31:57,730 --> 00:32:01,549 like evolution or breeding. It looks at the top 685 00:32:01,549 --> 00:32:03,910 surviving ideas. Maybe all the top sequences 686 00:32:03,910 --> 00:32:06,230 involved moving the arm slightly to the left. 687 00:32:06,450 --> 00:32:09,250 So it shifts the center of its bell curve, the 688 00:32:09,250 --> 00:32:12,630 mean, towards move left. And it shrinks the width 689 00:32:12,630 --> 00:32:14,950 of the bell curve, the variance, saying, I'm 690 00:32:14,950 --> 00:32:16,829 more confident now that the answer is somewhere 691 00:32:16,829 --> 00:32:19,990 in this leftward direction. Ah, it's narrowing 692 00:32:19,990 --> 00:32:22,930 in on the solution. Exactly. Then it repeats 693 00:32:22,930 --> 00:32:25,410 this entire process for a set number of iterations. 694 00:32:25,650 --> 00:32:28,390 Let's call it i. It samples a brand new batch 695 00:32:28,390 --> 00:32:31,390 of j sequences. But this time, they aren't totally 696 00:32:31,390 --> 00:32:34,089 random. They are drawn from this new, refined, 697 00:32:34,650 --> 00:32:37,349 left -leaning bell curve. I see. It evaluates 698 00:32:37,349 --> 00:32:40,890 them, takes the top k again, and refits again. 699 00:32:41,349 --> 00:32:43,569 With each iteration, the population of action 700 00:32:43,569 --> 00:32:46,509 sequences converges closer and closer to an optimized, 701 00:32:46,769 --> 00:32:49,470 highly specific plan. After the final iteration, 702 00:32:49,849 --> 00:32:52,450 the planner returns the mean of its final, highly 703 00:32:52,450 --> 00:32:55,130 refined belief distribution as its chosen plan. 704 00:32:55,650 --> 00:32:57,529 Let me make sure I have this exactly right, because 705 00:32:57,529 --> 00:33:01,670 it's so dynamic. It's brainstorming thousands 706 00:33:01,670 --> 00:33:05,609 of random moves, scoring them, taking the survivors 707 00:33:05,609 --> 00:33:07,970 and using the DNA of those survivors to brainstorm 708 00:33:07,970 --> 00:33:11,650 an even better more focused set of moves, refining 709 00:33:11,650 --> 00:33:13,769 the ideas over and over until it has a master 710 00:33:13,769 --> 00:33:16,910 plan. That's a great summary. But when it's finally 711 00:33:16,910 --> 00:33:20,130 done and has this ultimate 12 -step master plan, 712 00:33:20,670 --> 00:33:22,569 it doesn't just close its eyes and execute the 713 00:33:22,569 --> 00:33:24,829 whole thing blindly, right? No, it does not. 714 00:33:25,150 --> 00:33:27,569 And that is vital for surviving in an environment 715 00:33:27,569 --> 00:33:30,049 where your internal model of physics isn't perfect. 716 00:33:30,529 --> 00:33:33,809 Even after this extensive CEM search, the agent 717 00:33:33,809 --> 00:33:36,369 uses what is called model predictive control, 718 00:33:36,630 --> 00:33:40,710 or MPC. MPC. What does that do? This means the 719 00:33:40,710 --> 00:33:43,349 agent only executes the very first action of 720 00:33:43,349 --> 00:33:45,750 that optimized plan in the actual environment. 721 00:33:46,309 --> 00:33:49,710 It takes one single step? Yes, it takes one step, 722 00:33:49,890 --> 00:33:52,549 receives a brand new updated image observation 723 00:33:52,549 --> 00:33:55,529 from the real world, and then replans the entire 724 00:33:55,529 --> 00:33:58,029 thing from scratch. Wait, when you say from scratch, 725 00:33:58,450 --> 00:34:00,950 you mean... It throws away the beautiful planet 726 00:34:00,950 --> 00:34:03,430 just spent all that computational energy refining? 727 00:34:03,809 --> 00:34:06,769 Yes. The belief over action sequences is completely 728 00:34:06,769 --> 00:34:09,349 reset to zero mean and unit variance for the 729 00:34:09,349 --> 00:34:11,869 new time step. Why on earth would it do that? 730 00:34:12,159 --> 00:34:13,960 Why not just use the rest of the planet already 731 00:34:13,960 --> 00:34:16,559 made? Because the world might have reacted slightly 732 00:34:16,559 --> 00:34:19,239 differently than the model predicted. Maybe your 733 00:34:19,239 --> 00:34:21,539 robotic foot slipped a fraction of an inch on 734 00:34:21,539 --> 00:34:24,679 the virtual floor. If you blindly follow steps 735 00:34:24,679 --> 00:34:27,579 two through 12 of a plan based on a faulty assumption 736 00:34:27,579 --> 00:34:30,500 of step one, you will fail. That makes sense. 737 00:34:31,059 --> 00:34:33,239 The agent must look at the new reality, accept 738 00:34:33,239 --> 00:34:36,280 it, and brainstorm a fresh plan based on the 739 00:34:36,280 --> 00:34:38,920 new actual current state to avoid getting stuck 740 00:34:38,920 --> 00:34:42,159 in a cascading error. OK, that's incredibly cautious. 741 00:34:42,880 --> 00:34:44,760 Now, there is one more mechanical detail in the 742 00:34:44,760 --> 00:34:47,139 planning section we have to clarify, which is 743 00:34:47,139 --> 00:34:50,019 action repeats. The paper mentions repeating 744 00:34:50,019 --> 00:34:53,559 an action R times. Why does it do that instead 745 00:34:53,559 --> 00:34:55,539 of deciding a new action every single frame? 746 00:34:56,280 --> 00:34:58,880 Action repeats are a very common algorithmic 747 00:34:58,880 --> 00:35:01,300 technique in reinforcement learning. Instead 748 00:35:01,300 --> 00:35:03,840 of making a new decision at every single microframe 749 00:35:03,840 --> 00:35:06,300 of a simulation, you choose an action and hold 750 00:35:06,300 --> 00:35:09,900 it constant for, say, two or four consecutive 751 00:35:09,900 --> 00:35:13,039 time steps. Why? Wouldn't faster decisions be 752 00:35:13,039 --> 00:35:16,079 better? Not necessarily. It effectively reduces 753 00:35:16,079 --> 00:35:18,280 the length of the planning horizon. Think about 754 00:35:18,280 --> 00:35:20,800 it mathematically. If you need to plan one second 755 00:35:20,800 --> 00:35:23,380 into the future and your simulation runs at 60 756 00:35:23,380 --> 00:35:26,000 frames per second, you don't want to force your 757 00:35:26,000 --> 00:35:29,199 CEM algorithm to plan a sequence of 60 distinct 758 00:35:29,199 --> 00:35:31,559 independent actions. Right, the search space 759 00:35:31,559 --> 00:35:34,460 would be huge. Exactly, the search space is too 760 00:35:34,460 --> 00:35:37,480 massive. If you repeat actions four times, you 761 00:35:37,480 --> 00:35:40,449 only need to plan a sequence of 15 actions to 762 00:35:40,449 --> 00:35:43,110 cover that same one second. It makes the search 763 00:35:43,110 --> 00:35:46,150 space much smaller and provides a clearer, less 764 00:35:46,150 --> 00:35:49,150 noisy learning signal over those slightly longer 765 00:35:49,150 --> 00:35:51,710 physical stretches. Now I want to clarify something 766 00:35:51,710 --> 00:35:55,070 critical here for you listening so we don't accidentally 767 00:35:55,070 --> 00:35:58,050 mislead anyone. When we talk about executing 768 00:35:58,050 --> 00:36:01,469 a sequence of actions and iterating and planning 769 00:36:01,469 --> 00:36:05,369 horizons, we are strictly talking about discrete 770 00:36:05,369 --> 00:36:08,690 algorithmic time steps in the simulation's logic. 771 00:36:08,769 --> 00:36:11,489 Yes, I am very glad you brought that up. We should 772 00:36:11,489 --> 00:36:13,949 absolutely not be speculating about real -world 773 00:36:13,949 --> 00:36:16,670 wall clock latency. Like, I just want to make 774 00:36:16,670 --> 00:36:18,570 sure people aren't picturing a physical robot 775 00:36:18,570 --> 00:36:20,670 sitting there frozen for 10 milliseconds while 776 00:36:20,670 --> 00:36:23,849 its brain whirls and then violently jerking forward. 777 00:36:24,510 --> 00:36:27,449 The paper defines these as discrete algorithmic 778 00:36:27,449 --> 00:36:30,110 steps in a simulation. That is exactly right. 779 00:36:30,570 --> 00:36:32,590 And further, we need to be really clear about 780 00:36:32,590 --> 00:36:35,369 the limits of this CEM search method. It's not 781 00:36:35,369 --> 00:36:39,210 magic. It is a heuristic. It is a very effective 782 00:36:39,210 --> 00:36:41,809 search heuristic, but it is still a heuristic. 783 00:36:42,230 --> 00:36:44,670 We must be clear, neither the learned transition 784 00:36:44,670 --> 00:36:47,510 model nor the CEM algorithm itself provides any 785 00:36:47,510 --> 00:36:50,250 mathematical guarantee of finding a globally 786 00:36:50,250 --> 00:36:52,869 optimal plan. It doesn't promise a win every 787 00:36:52,869 --> 00:36:55,329 time. No, it does not guarantee a plan that will 788 00:36:55,329 --> 00:36:57,949 definitely yield a high reward in the real environment. 789 00:36:58,210 --> 00:37:00,710 It is simply an informed search through a massive 790 00:37:00,710 --> 00:37:03,630 space of possibilities. It can absolutely still 791 00:37:03,630 --> 00:37:06,130 fail, especially if the underlying learned model 792 00:37:06,130 --> 00:37:08,860 is inaccurate. The agent is doing its best guess 793 00:37:08,860 --> 00:37:12,280 based on imperfect internal simulations. Okay. 794 00:37:12,639 --> 00:37:14,659 We've covered how the final planet agent actually 795 00:37:14,659 --> 00:37:19,159 works. The encoder, the RSSM, the CEM planner. 796 00:37:19,980 --> 00:37:23,079 But if we keep reading the paper, they take a 797 00:37:23,079 --> 00:37:25,380 bit of a theoretical detour. They do, in section 798 00:37:25,380 --> 00:37:27,980 four. Yeah, they dedicate a whole section to 799 00:37:27,980 --> 00:37:30,659 detailing a novel mathematical concept called 800 00:37:30,659 --> 00:37:33,699 latent overshooting. And the funny thing is, 801 00:37:33,960 --> 00:37:36,059 as they admit later in their own ablation studies, 802 00:37:36,340 --> 00:37:38,599 The final planet agent didn't actually end up 803 00:37:38,599 --> 00:37:41,179 needing latent overshooting to get its high scores. 804 00:37:41,380 --> 00:37:43,980 That's right. So why is it in the paper? What 805 00:37:43,980 --> 00:37:46,039 is it? It's an important part of the scientific 806 00:37:46,039 --> 00:37:48,840 contribution. To understand latent overshooting, 807 00:37:48,960 --> 00:37:51,260 we have to look back at the standard variational 808 00:37:51,260 --> 00:37:53,760 bound used to train these kinds of models. In 809 00:37:53,760 --> 00:37:56,179 the paper, this is equation three. Let me guess. 810 00:37:56,579 --> 00:37:59,440 Variational bound is another heavy math term 811 00:37:59,440 --> 00:38:02,460 we need to translate? Yes. In simple terms, a 812 00:38:02,460 --> 00:38:05,059 variational bound is a mathematical way of measuring 813 00:38:05,059 --> 00:38:07,920 the gap between our model's approximate guess 814 00:38:07,920 --> 00:38:11,440 and the true mathematically perfect probability 815 00:38:11,440 --> 00:38:13,639 distribution, which is usually impossible to 816 00:38:13,639 --> 00:38:16,059 calculate directly. We use this bound as our 817 00:38:16,059 --> 00:38:18,239 training objective. We try to minimize that gap. 818 00:38:18,559 --> 00:38:21,639 OK. Now, the standard objective function uses 819 00:38:21,639 --> 00:38:24,920 something called KL divergence to train the stochastic 820 00:38:24,920 --> 00:38:27,840 path of the transition model. OK. KL divergence. 821 00:38:28,239 --> 00:38:31,000 Treat me like a beginner again. What is KL divergence 822 00:38:31,000 --> 00:38:34,659 actually doing here? KL divergence is a statistical 823 00:38:34,659 --> 00:38:38,559 tool used to measure how one probability distribution 824 00:38:38,559 --> 00:38:41,840 differs from a second reference probability distribution. 825 00:38:42,260 --> 00:38:44,960 Imagine you have two bell curves. KL divergence 826 00:38:44,960 --> 00:38:47,659 measures how much you would have to stretch and 827 00:38:47,659 --> 00:38:50,579 shift the first bell curve to make it perfectly 828 00:38:50,579 --> 00:38:53,539 overlap the second one. In this model, it's used 829 00:38:53,539 --> 00:38:55,679 to make sure the model's prediction aligns with 830 00:38:55,679 --> 00:38:58,989 reality. Got it. So, what's the problem with 831 00:38:58,989 --> 00:39:01,530 the standard way they were doing it? The limitation 832 00:39:01,530 --> 00:39:03,809 of the standard bound in equation 3 is that it 833 00:39:03,809 --> 00:39:06,510 only trains the model for one -step predictions. 834 00:39:07,210 --> 00:39:09,610 The mathematical gradient, the learning signal, 835 00:39:10,050 --> 00:39:11,949 only flows from the prediction to the target 836 00:39:11,949 --> 00:39:15,030 for a single step in time. The model is never 837 00:39:15,030 --> 00:39:17,730 explicitly forced to practice a chain of multiple 838 00:39:17,730 --> 00:39:20,449 predictions during this specific part of the 839 00:39:20,449 --> 00:39:22,780 optimization. And as we discussed half an hour 840 00:39:22,780 --> 00:39:25,159 ago, the whole problem with planning is accumulating 841 00:39:25,159 --> 00:39:28,019 errors over multi -step predictions. If you only 842 00:39:28,019 --> 00:39:30,119 train it to be really good at predicting one 843 00:39:30,119 --> 00:39:33,000 step, it might still drift wildly off course 844 00:39:33,000 --> 00:39:35,619 when you ask it to predict 15 steps during the 845 00:39:35,619 --> 00:39:39,360 CEM planning phase. Precisely the issue. So in 846 00:39:39,360 --> 00:39:43,539 equation 7, the authors introduce latent overshooting. 847 00:39:44,000 --> 00:39:46,079 It is a generalized objective function designed 848 00:39:46,079 --> 00:39:49,019 to fix this. Instead of just penalizing one -step 849 00:39:49,019 --> 00:39:52,139 errors, Latent overshooting explicitly trains 850 00:39:52,139 --> 00:39:54,679 the model on multi -step predictions of all distances 851 00:39:54,679 --> 00:39:57,480 up to the planning horizon. Wait, how does it 852 00:39:57,480 --> 00:39:59,780 actually do that mathematically without generating 853 00:39:59,780 --> 00:40:01,860 a million images? We established earlier that 854 00:40:01,860 --> 00:40:03,780 generating images for multi -step predictions 855 00:40:03,780 --> 00:40:06,500 is way too computationally expensive. That is 856 00:40:06,500 --> 00:40:09,079 the beauty of staying in the latent space. Latent 857 00:40:09,079 --> 00:40:11,840 overshooting operates entirely within the compact 858 00:40:11,840 --> 00:40:14,940 mathematical representations. The KL divergence 859 00:40:14,940 --> 00:40:18,539 objective here acts as a regularizer. A regularizer. 860 00:40:18,670 --> 00:40:21,190 Meaning it keeps the math from getting too wild 861 00:40:21,190 --> 00:40:23,789 and complex. Yes, it guides the learning. It 862 00:40:23,789 --> 00:40:26,369 takes the model's multi -step priors, which are 863 00:40:26,369 --> 00:40:28,570 what the model blindly predicts will happen multiple 864 00:40:28,570 --> 00:40:30,650 steps in the future without seeing the intervening 865 00:40:30,650 --> 00:40:34,530 images, and compares them to the informed approximate 866 00:40:34,530 --> 00:40:37,090 posteriors, which are the latent states the model 867 00:40:37,090 --> 00:40:39,449 infers when it is allowed to see those intervening 868 00:40:39,449 --> 00:40:42,570 images during training. So it compares the blind 869 00:40:42,570 --> 00:40:45,750 guess to the informed guess. Exactly. It encourages 870 00:40:45,750 --> 00:40:48,690 consistency. It forces the long -term blind prediction 871 00:40:48,690 --> 00:40:51,150 to closely match the short -term informed prediction, 872 00:40:51,349 --> 00:40:53,389 and it does this efficiently because it's just 873 00:40:53,389 --> 00:40:55,469 calculating the distance between vectors of numbers, 874 00:40:55,510 --> 00:40:59,130 not rendering pixels. That's super smart. As 875 00:40:59,130 --> 00:41:01,030 a minor technical note for completeness, the 876 00:41:01,030 --> 00:41:03,130 mathematical formula includes beta -weighting 877 00:41:03,130 --> 00:41:04,889 values for the different prediction distances, 878 00:41:04,989 --> 00:41:07,750 but the paper explicitly notes that for distances 879 00:41:07,750 --> 00:41:10,369 greater than 1, all beta values are simply set 880 00:41:10,369 --> 00:41:12,789 to the exact same value for simplicity. Okay, 881 00:41:12,789 --> 00:41:15,130 so I have to ask the obvious question again. 882 00:41:15,769 --> 00:41:19,230 If the final RSSM agent with its dual deterministic 883 00:41:19,230 --> 00:41:22,010 and stochastic paths didn't even need this fancy 884 00:41:22,010 --> 00:41:24,050 latent overshooting math to get its high scores 885 00:41:24,050 --> 00:41:27,599 on the robotics tasks, why did the authors dedicate 886 00:41:27,599 --> 00:41:29,760 so much real estate to proposing it? Because 887 00:41:29,760 --> 00:41:32,380 proposing a generalized mathematical bound is 888 00:41:32,380 --> 00:41:34,780 how the field maps out the theoretical landscape. 889 00:41:35,900 --> 00:41:39,219 We must explicitly state that the final RSSM 890 00:41:39,219 --> 00:41:41,840 agent does not require latent overshooting to 891 00:41:41,840 --> 00:41:44,880 achieve the reported results. The RSSM architecture 892 00:41:44,880 --> 00:41:47,599 itself was structurally robust enough to handle 893 00:41:47,599 --> 00:41:49,679 the multi -step predictions on its own in these 894 00:41:49,679 --> 00:41:52,960 specific tasks. Right. However, the authors perform 895 00:41:52,960 --> 00:41:55,980 rigorous ablation studies, where they test different 896 00:41:55,980 --> 00:41:58,219 combinations of model components. Taking parts 897 00:41:58,219 --> 00:42:01,139 of the engine out to see what breaks. Yes. And 898 00:42:01,139 --> 00:42:03,300 they show that while the RSSM didn't need it, 899 00:42:03,369 --> 00:42:06,010 Other types of models like purely stochastic 900 00:42:06,010 --> 00:42:08,670 state space models absolutely benefited from 901 00:42:08,670 --> 00:42:11,010 latent overshooting. It provides a theoretical 902 00:42:11,010 --> 00:42:13,409 tool for future researchers who might be working 903 00:42:13,409 --> 00:42:16,090 with different, perhaps more complex architectures 904 00:42:16,090 --> 00:42:18,070 where latent overshooting could be the critical 905 00:42:18,070 --> 00:42:20,250 missing piece. So it's for the broader community. 906 00:42:20,909 --> 00:42:22,949 Yes. Furthermore, it's crucial to understand 907 00:42:22,949 --> 00:42:25,170 that while the KL objective in latent overshooting 908 00:42:25,170 --> 00:42:27,849 encourages consistency, it does not magically 909 00:42:27,849 --> 00:42:30,539 turn these predictions into true states. nor 910 00:42:30,539 --> 00:42:32,900 does it mathematically guarantee that all intermediate 911 00:42:32,900 --> 00:42:36,320 predictions are perfectly accurate. It is a regularizer, 912 00:42:36,760 --> 00:42:39,500 a guide towards better behavior, not a magic 913 00:42:39,500 --> 00:42:41,880 bullet for perfect physical simulation. That 914 00:42:41,880 --> 00:42:44,320 makes perfect sense. It's about providing a complete 915 00:42:44,320 --> 00:42:46,820 theoretical toolkit for the community, even if 916 00:42:46,820 --> 00:42:49,059 you didn't need every single tool to build this 917 00:42:49,059 --> 00:42:52,320 specific robot. Exactly. All right. We have waded 918 00:42:52,320 --> 00:42:54,320 through the deep, heavy waters of the architecture, 919 00:42:54,699 --> 00:42:59,019 the RSSM, the CEM, and the math. Let's see how 920 00:42:59,019 --> 00:43:01,420 this thing actually performs. Let's head to the 921 00:43:01,420 --> 00:43:04,530 proving grounds. The authors put Planet to the 922 00:43:04,530 --> 00:43:07,409 test on simulated tasks from the DeepMind control 923 00:43:07,409 --> 00:43:10,650 suite. Yes, they selected six diverse continuous 924 00:43:10,650 --> 00:43:13,730 control tasks, all driven solely by pixel observations. 925 00:43:14,309 --> 00:43:16,230 It is worth detailing the physical constraints 926 00:43:16,230 --> 00:43:18,909 of these tasks to understand the breadth of the 927 00:43:18,909 --> 00:43:21,489 challenge the algorithm is facing. Let's run 928 00:43:21,489 --> 00:43:23,789 through them. There is cart -pull swing -up. 929 00:43:23,929 --> 00:43:26,429 A cart must swing a dangling pole upright and 930 00:43:26,429 --> 00:43:28,909 balance it. This requires a very long planning 931 00:43:28,909 --> 00:43:31,110 horizon, and it involves dealing with the cart 932 00:43:31,110 --> 00:43:33,150 moving in and out of the camera's fixed view. 933 00:43:33,409 --> 00:43:36,769 OK. There is Reacher, where an articulated arm 934 00:43:36,769 --> 00:43:39,550 must move to a specific spatial target. This 935 00:43:39,550 --> 00:43:42,550 is difficult because the reward is very sparse. 936 00:43:42,730 --> 00:43:45,710 The agent gets no points for being close. It 937 00:43:45,710 --> 00:43:48,469 only gets a reward when it is precisely on target. 938 00:43:48,769 --> 00:43:51,050 That sounds frustrating to learn. Then there 939 00:43:51,050 --> 00:43:53,650 is Cheetah Run. This is a two -legged robot that 940 00:43:53,650 --> 00:43:56,610 looks a bit like a stick figure cheetah. It has 941 00:43:56,610 --> 00:43:59,849 to run forward. This involves complex contact 942 00:43:59,849 --> 00:44:02,590 dynamics with the ground, feet hitting and slipping, 943 00:44:03,070 --> 00:44:05,190 and a large action space with many joints to 944 00:44:05,190 --> 00:44:07,769 coordinate. And there's also finger spin, where 945 00:44:07,769 --> 00:44:10,690 a robotic finger has to constantly flick a geometric 946 00:44:10,690 --> 00:44:12,889 object to keep it spinning on a pivot, again 947 00:44:12,889 --> 00:44:15,510 involving complex continuous physical contact 948 00:44:15,510 --> 00:44:18,409 and friction. And cup catch, a digital version 949 00:44:18,409 --> 00:44:21,000 of the classic ball -in -a -cup game. This has 950 00:44:21,000 --> 00:44:23,860 incredibly sparse rewards. The agent gets absolutely 951 00:44:23,860 --> 00:44:25,940 no positive feedback until the precise moment 952 00:44:25,940 --> 00:44:28,900 the ball is actually resting in the cup. Finally, 953 00:44:29,079 --> 00:44:32,199 there is Walker Walk, a bipedal robot that starts 954 00:44:32,199 --> 00:44:34,480 lying flat on the floor. It has to figure out 955 00:44:34,480 --> 00:44:36,619 how to stand up, balance itself, and then walk 956 00:44:36,619 --> 00:44:39,420 forward. This involves highly unpredictable collisions 957 00:44:39,420 --> 00:44:41,659 with the floor during the early learning process. 958 00:44:42,239 --> 00:44:45,659 That is a brutal gauntlet of physics tests. So 959 00:44:45,659 --> 00:44:48,199 how did Planet actually do against the reigning 960 00:44:48,199 --> 00:44:51,010 model -free champions of the time? The performance 961 00:44:51,010 --> 00:44:53,750 results are reported very carefully in the abstract 962 00:44:53,750 --> 00:44:56,590 and the body of the paper. Across all the tasks, 963 00:44:57,010 --> 00:44:58,969 Planet was substantially better than the model 964 00:44:58,969 --> 00:45:02,070 -free A3C algorithm. Nice. And in some cases, 965 00:45:02,289 --> 00:45:05,090 notably the complex Cheetah running task, it 966 00:45:05,090 --> 00:45:07,849 achieved final performance higher than D4PG, 967 00:45:07,909 --> 00:45:10,010 which was a very strong model -free algorithm 968 00:45:10,010 --> 00:45:12,130 baseline at the time. But the real headline number, 969 00:45:12,409 --> 00:45:14,389 the one that makes everyone in robotics sit up 970 00:45:14,389 --> 00:45:17,940 and take notice, is the data efficiency. The 971 00:45:17,940 --> 00:45:20,300 paper claims Planet achieves this high performance 972 00:45:20,300 --> 00:45:24,360 with on average 200 times less environment interaction. 973 00:45:24,920 --> 00:45:28,380 That's a huge claim. That is a staggering improvement. 974 00:45:28,679 --> 00:45:30,739 It means the agent learns what it needs to learn 975 00:45:30,739 --> 00:45:32,860 in a fraction of the time spent actually messing 976 00:45:32,860 --> 00:45:35,800 around and falling over in the simulation. We 977 00:45:35,800 --> 00:45:39,099 need to contextualize that 200 times less claim 978 00:45:39,099 --> 00:45:42,599 very carefully to avoid over -promising. The 979 00:45:42,599 --> 00:45:45,519 paper notes this extreme data efficiency alongside 980 00:45:45,519 --> 00:45:48,619 similar computation times in the specific setup 981 00:45:48,619 --> 00:45:51,280 tested. The agent gets to the high score using 982 00:45:51,280 --> 00:45:54,000 far fewer episodes in the simulator. However, 983 00:45:54,300 --> 00:45:56,820 we must be absolutely rigorous here. We cannot 984 00:45:56,820 --> 00:45:59,320 extrapolate these simulator episode counts to 985 00:45:59,320 --> 00:46:01,579 guaranteed training times on physical robots 986 00:46:01,579 --> 00:46:03,960 in the real world. Right. I was just thinking, 987 00:46:04,260 --> 00:46:06,400 if it takes 200 days to train a real robot and 988 00:46:06,400 --> 00:46:08,599 now it takes one day, that changes the whole 989 00:46:08,599 --> 00:46:10,460 industry. But you're saying we can't make that 990 00:46:10,460 --> 00:46:14,119 job. No, we cannot. A simulation is still a pristine, 991 00:46:14,400 --> 00:46:16,380 controlled mathematical environment. There is 992 00:46:16,380 --> 00:46:19,400 no dust on the lens, no unexpected wear and tear 993 00:46:19,400 --> 00:46:23,099 on a joint, no unpredictable wind. The transfer 994 00:46:23,099 --> 00:46:25,539 to the messy reality of physical hardware and 995 00:46:25,539 --> 00:46:28,039 real -world physics is a different domain entirely. 996 00:46:28,559 --> 00:46:31,340 The 200x efficiency is a massive algorithmic 997 00:46:31,340 --> 00:46:34,059 achievement in simulation data efficiency, not 998 00:46:34,059 --> 00:46:35,739 a promise that you can take this code and teach 999 00:46:35,739 --> 00:46:37,900 a physical robot to run in your living room in 1000 00:46:37,900 --> 00:46:40,400 five minutes. That's an essential reality check. 1001 00:46:40,619 --> 00:46:42,739 Now, how do the researchers know their agent 1002 00:46:42,739 --> 00:46:45,179 is actually doing well, in an absolute sense? 1003 00:46:45,519 --> 00:46:47,579 They use a specific baseline for comparison to 1004 00:46:47,579 --> 00:46:50,539 see how close they got to the perfect score. 1005 00:46:51,079 --> 00:46:54,760 Yes. To understand the upper bound of what their 1006 00:46:54,760 --> 00:46:57,340 CEM planning algorithm could theoretically achieve, 1007 00:46:57,880 --> 00:47:00,639 they ran the CEM planner using the true simulator 1008 00:47:00,639 --> 00:47:03,199 dynamics instead of their learned neural network 1009 00:47:03,199 --> 00:47:07,000 model. Meaning they let the planner cheat. They 1010 00:47:07,000 --> 00:47:09,059 gave it the actual code of the physics engine 1011 00:47:09,059 --> 00:47:11,699 instead of making it guess the physics from pixels. 1012 00:47:12,380 --> 00:47:16,340 Exactly. This CEM true simulator comparison provides 1013 00:47:16,340 --> 00:47:19,320 a strong baseline for what is possible with that 1014 00:47:19,320 --> 00:47:22,559 specific planning heuristic. But again, a crucial 1015 00:47:22,559 --> 00:47:25,679 distinction. This is exactly that, a comparison 1016 00:47:25,679 --> 00:47:29,159 point for this specific experiment. It is not 1017 00:47:29,159 --> 00:47:31,619 a mathematical upper bound on all achievable 1018 00:47:31,619 --> 00:47:34,590 control in the universe. A different, more advanced 1019 00:47:34,590 --> 00:47:36,469 planning algorithm might achieve even higher 1020 00:47:36,469 --> 00:47:38,969 scores given the true simulator. And they also 1021 00:47:38,969 --> 00:47:41,030 did ablation studies, which we touched on earlier 1022 00:47:41,030 --> 00:47:43,369 with the latent overshooting. They stripped away 1023 00:47:43,369 --> 00:47:45,250 parts of their model to prove they were necessary. 1024 00:47:45,869 --> 00:47:48,530 Correct. They tested a purely deterministic model 1025 00:47:48,530 --> 00:47:51,190 using a standard GRU and a purely stochastic 1026 00:47:51,190 --> 00:47:54,429 model using an SSM. The ablations prove that 1027 00:47:54,429 --> 00:47:56,929 for this specific architecture on these specific 1028 00:47:56,929 --> 00:47:59,750 tasks, both the deterministic and stochastic 1029 00:47:59,750 --> 00:48:02,449 paths of the RSSM were vital for achieving high 1030 00:48:02,449 --> 00:48:05,469 performance. But remember, ablations show what 1031 00:48:05,469 --> 00:48:07,710 helps in these reported experiments. They do 1032 00:48:07,710 --> 00:48:10,130 not prove what is fundamentally impossible with 1033 00:48:10,130 --> 00:48:13,349 unlimited data or alternative algorithmic approaches 1034 00:48:13,349 --> 00:48:16,039 in the future. Now, here's an experiment they 1035 00:48:16,039 --> 00:48:18,639 ran that I think is completely mind -blowing. 1036 00:48:19,400 --> 00:48:22,659 The multitask experiment. They didn't just train 1037 00:48:22,659 --> 00:48:25,639 six different separate agents for the six different 1038 00:48:25,639 --> 00:48:30,480 tasks. They trained one single agent on all six 1039 00:48:30,480 --> 00:48:32,920 tasks simultaneously. It's an impressive feat. 1040 00:48:33,619 --> 00:48:35,739 They just dropped the agent into a simulation, 1041 00:48:35,739 --> 00:48:38,260 and it wasn't explicitly told, hey, you are playing 1042 00:48:38,260 --> 00:48:41,019 cart pole now, or hey, you are walking a robot 1043 00:48:41,019 --> 00:48:43,360 now. They just started getting a feed of pixels 1044 00:48:43,360 --> 00:48:46,219 and had to figure out what universe it was currently 1045 00:48:46,219 --> 00:48:48,380 inhabiting and act accordingly. And it worked? 1046 00:48:48,880 --> 00:48:51,940 It successfully solved all of them. But let's 1047 00:48:51,940 --> 00:48:54,320 be careful here. Does this mean the agent has 1048 00:48:54,320 --> 00:48:57,460 developed a generalized holistic understanding 1049 00:48:57,460 --> 00:49:00,920 of physics in a way that it could suddenly walk 1050 00:49:00,920 --> 00:49:04,019 a completely new unseen robot design? I must 1051 00:49:04,019 --> 00:49:07,539 push back on that idea firmly. No. It absolutely 1052 00:49:07,539 --> 00:49:10,619 does not. This is a very common trap when interpreting 1053 00:49:10,619 --> 00:49:13,519 multitask AI results. The multitask experiment 1054 00:49:13,519 --> 00:49:15,739 proves that the model has sufficient mathematical 1055 00:49:15,739 --> 00:49:18,980 capacity, enough neurons and parameters, to learn 1056 00:49:18,980 --> 00:49:21,639 and distinguish between six diverse visual environments 1057 00:49:21,639 --> 00:49:24,579 when explicitly trained on them. It proves the 1058 00:49:24,579 --> 00:49:27,260 architecture is flexible. So it's more of a memory 1059 00:49:27,260 --> 00:49:31,210 capacity test. Yes. It is not evidence of zero 1060 00:49:31,210 --> 00:49:33,889 -shot transfer to arbitrary environments. If 1061 00:49:33,889 --> 00:49:36,289 you dropped this planet agent into a simulation 1062 00:49:36,289 --> 00:49:38,829 of a completely different robot, say, a four 1063 00:49:38,829 --> 00:49:41,170 -legged spider bot, or asked it to drive a car, 1064 00:49:41,550 --> 00:49:44,150 it would fail catastrophically until it had collected 1065 00:49:44,150 --> 00:49:46,889 enough new data to train its model on the physics 1066 00:49:46,889 --> 00:49:50,929 and visuals of that new spider bot. OK. It has 1067 00:49:50,929 --> 00:49:53,550 not abstracted a universal understanding of physics. 1068 00:49:53,670 --> 00:49:56,409 It has learned six separate complex patterns 1069 00:49:56,409 --> 00:49:58,849 and how to identify which pattern it is currently 1070 00:49:58,849 --> 00:50:02,010 looking at. Okay. That is a very important boundary 1071 00:50:02,010 --> 00:50:05,329 to draw. It's a master of the six tasks it studied, 1072 00:50:05,849 --> 00:50:09,090 not a general artificial intelligence. Which 1073 00:50:09,090 --> 00:50:12,050 leads us perfectly into the final phase of our 1074 00:50:12,050 --> 00:50:15,210 deep dive. The boundaries and limitations of 1075 00:50:15,210 --> 00:50:18,030 the deep planning network. Having seen all its 1076 00:50:18,030 --> 00:50:20,750 successes in data efficiency, we have to critically 1077 00:50:20,750 --> 00:50:23,710 evaluate what planet doesn't solve. Where does 1078 00:50:23,710 --> 00:50:27,690 this approach break down? The paper is quite 1079 00:50:27,690 --> 00:50:29,590 honest and explicit about the limits of their 1080 00:50:29,590 --> 00:50:33,030 study. First, as we just touched upon, the tasks 1081 00:50:33,030 --> 00:50:36,389 are strictly restricted to simulated visual environments. 1082 00:50:36,909 --> 00:50:38,969 Furthermore, they are mostly fixed or highly 1083 00:50:38,969 --> 00:50:41,909 predictable third -person camera angles. So the 1084 00:50:41,909 --> 00:50:44,130 camera isn't shaking, the lighting isn't changing 1085 00:50:44,130 --> 00:50:46,889 from noon to dusk. Right. It's learning in a 1086 00:50:46,889 --> 00:50:49,389 clean room, relatively speaking. The agent doesn't 1087 00:50:49,389 --> 00:50:51,630 have to deal with a camera bouncing around wildly 1088 00:50:51,630 --> 00:50:54,409 or dynamic shadows or occlusions where someone 1089 00:50:54,409 --> 00:50:57,570 walks in front of the robot. The paper also acknowledges 1090 00:50:57,570 --> 00:50:59,730 some structural limitations of the algorithm 1091 00:50:59,730 --> 00:51:02,530 itself. They mention the need for future work 1092 00:51:02,530 --> 00:51:06,369 in temporal abstraction. Temporal abstraction. 1093 00:51:06,789 --> 00:51:08,750 Instead of relying on the fixed action repeats 1094 00:51:08,750 --> 00:51:10,929 we discussed earlier, where it holds a move for 1095 00:51:10,929 --> 00:51:14,369 four frames. Yes. Fixed action repeats are a 1096 00:51:14,400 --> 00:51:17,199 blunt instrument, a more sophisticated agent 1097 00:51:17,199 --> 00:51:20,380 would learn temporal abstraction. It would organically 1098 00:51:20,380 --> 00:51:22,920 learn when it needs to make a microsecond decision, 1099 00:51:23,460 --> 00:51:25,800 like during a sudden collision, and when it can 1100 00:51:25,800 --> 00:51:28,380 plan a single sweeping motion that lasts several 1101 00:51:28,380 --> 00:51:31,400 seconds without being locked into a rigid, predefined 1102 00:51:31,400 --> 00:51:33,880 grid of time steps. That makes a lot of sense. 1103 00:51:34,400 --> 00:51:36,420 Additionally, they point out the limitation of 1104 00:51:36,420 --> 00:51:39,420 relying entirely on the CEM planner over a finite 1105 00:51:39,420 --> 00:51:42,940 horizon. Yes. Because CEM only looks a certain 1106 00:51:42,940 --> 00:51:46,320 number of steps ahead, say 12 steps, it can't 1107 00:51:46,320 --> 00:51:48,619 optimize for a reward that might happen 50 steps 1108 00:51:48,619 --> 00:51:51,940 from now. It's short -sighted. So it might choose 1109 00:51:51,940 --> 00:51:54,099 a move that looks good for the next two seconds, 1110 00:51:54,380 --> 00:51:56,980 but causes it to fall over in three. Exactly. 1111 00:51:57,460 --> 00:52:00,139 The author suggests that future work should involve 1112 00:52:00,139 --> 00:52:02,579 learning value functions to estimate the long 1113 00:52:02,579 --> 00:52:04,800 -term rewards beyond the explicit planning window. 1114 00:52:05,400 --> 00:52:07,960 And finally, there is the ever -present problem 1115 00:52:07,960 --> 00:52:10,849 of model error. The compounding errors we started 1116 00:52:10,849 --> 00:52:13,369 this whole conversation with. Precisely. While 1117 00:52:13,369 --> 00:52:16,630 PLANET mitigates it brilliantly with the RSSM 1118 00:52:16,630 --> 00:52:18,869 and latent planning, it does not eliminate it. 1119 00:52:19,309 --> 00:52:21,690 Because the model isn't perfect, long -term predictions 1120 00:52:21,690 --> 00:52:24,829 will inherently degrade. The uncertainty estimations 1121 00:52:24,829 --> 00:52:27,750 provided by the stochastic path are also imperfect. 1122 00:52:28,170 --> 00:52:30,730 So we have to be very careful to prevent sensationalism 1123 00:52:30,730 --> 00:52:33,090 here. This is a crucial distinction we need to 1124 00:52:33,090 --> 00:52:35,960 make for you listening. You cannot extrapolate 1125 00:52:35,960 --> 00:52:38,239 the findings of this single paper into claims 1126 00:52:38,239 --> 00:52:40,500 that we can now perfectly predict complex systems 1127 00:52:40,500 --> 00:52:43,940 just because we have an RSSM. Absolutely. The 1128 00:52:43,940 --> 00:52:46,099 success of planet in a simulated physics engine 1129 00:52:46,099 --> 00:52:48,300 does not mean we are one step away from using 1130 00:52:48,300 --> 00:52:51,059 a similar architecture to perfectly predict financial 1131 00:52:51,059 --> 00:52:54,199 markets, model human societies, or immediately 1132 00:52:54,199 --> 00:52:56,420 solve real -world autonomous driving in blizzard 1133 00:52:56,420 --> 00:52:59,440 conditions. The transferability of these latent 1134 00:52:59,440 --> 00:53:01,900 planning models from clean, constrained simulations 1135 00:53:01,900 --> 00:53:05,360 to the messy, visually chaotic, dynamically unpredictable 1136 00:53:05,360 --> 00:53:08,760 real world remains a massive, unresolved hurdle 1137 00:53:08,760 --> 00:53:11,039 in the field. So what does this all mean for 1138 00:53:11,039 --> 00:53:13,260 you, the listener? Let's bring this all together. 1139 00:53:13,639 --> 00:53:16,000 It means that with Planet, researchers have found 1140 00:53:16,000 --> 00:53:19,340 an incredibly clever, elegant mathematical shortcut 1141 00:53:19,340 --> 00:53:22,519 for teaching an AI in a simulator. By encoding 1142 00:53:22,519 --> 00:53:25,380 raw pixels into statistical beliefs, predicting 1143 00:53:25,380 --> 00:53:28,659 the future using a dual deterministic and stochastic 1144 00:53:28,659 --> 00:53:31,579 recurrent state space model, and planning actions 1145 00:53:31,579 --> 00:53:34,659 purely in that compact mathematical space using 1146 00:53:34,659 --> 00:53:37,460 the cross entropy method, they achieved massive 1147 00:53:37,460 --> 00:53:40,420 data efficiency. It really is a feat. They taught 1148 00:53:40,420 --> 00:53:42,920 an agent to master complex continuous control 1149 00:53:42,920 --> 00:53:45,340 tasks without it having to physically bump into 1150 00:53:45,340 --> 00:53:48,139 a million virtual walls. It is a brilliant piece 1151 00:53:48,139 --> 00:53:50,280 of engineering that changed how we look at model 1152 00:53:50,280 --> 00:53:52,599 -based reinforcement learning. But it doesn't 1153 00:53:52,599 --> 00:53:55,130 mean we have a crystal ball. It is a profound 1154 00:53:55,130 --> 00:53:57,769 step forward. It demonstrates that learning accurate 1155 00:53:57,769 --> 00:54:00,389 multi -step dynamics in a latent space is not 1156 00:54:00,389 --> 00:54:03,050 just possible, but highly effective for planning, 1157 00:54:03,409 --> 00:54:05,670 bypassing the data inefficiencies of model -free 1158 00:54:05,670 --> 00:54:08,289 methods in these specific simulated environments. 1159 00:54:08,929 --> 00:54:11,590 The architectural choices, particularly the interplay 1160 00:54:11,590 --> 00:54:14,369 of deterministic memory and stochastic uncertainty, 1161 00:54:14,989 --> 00:54:17,190 provide a powerful blueprint for future dynamics 1162 00:54:17,190 --> 00:54:19,590 models. It's an amazing blueprint. But I want 1163 00:54:19,590 --> 00:54:21,949 to leave you with a lingering question, a provocative 1164 00:54:21,949 --> 00:54:24,909 thought to mull over as we wrap up. We spent 1165 00:54:24,909 --> 00:54:26,809 this whole time talking about how the agent builds 1166 00:54:26,809 --> 00:54:30,610 a latent space, an internal mathematical representation 1167 00:54:30,610 --> 00:54:33,710 of its world. Right. But remember, that entire 1168 00:54:33,710 --> 00:54:36,170 internal world is shaped solely by two mathematical 1169 00:54:36,170 --> 00:54:38,710 drives, the drive to reconstruct the pixels it 1170 00:54:38,710 --> 00:54:41,909 saw, and the drive to maximize a specific scalar 1171 00:54:41,909 --> 00:54:46,230 reward score. If that is the only way the internal 1172 00:54:46,230 --> 00:54:49,070 representation is formed, how do we evaluate 1173 00:54:49,070 --> 00:54:52,269 the quality or the truth of that representation 1174 00:54:52,269 --> 00:54:54,449 beyond just looking at the final high score on 1175 00:54:54,449 --> 00:54:57,369 the game? That's a deep question. If you trained 1176 00:54:57,369 --> 00:54:59,690 two completely different agents with slightly 1177 00:54:59,690 --> 00:55:02,170 different initial conditions and they both learned 1178 00:55:02,170 --> 00:55:05,710 to solve the walker task perfectly, do they imagine 1179 00:55:05,710 --> 00:55:07,989 the physical world in the exact same mathematical 1180 00:55:07,989 --> 00:55:11,269 way? Or have they constructed entirely different 1181 00:55:11,269 --> 00:55:13,489 alien physics engines in their latent space that 1182 00:55:13,489 --> 00:55:15,590 just happen to produce the exact same walking 1183 00:55:15,590 --> 00:55:19,150 behavior? That is the profound, perhaps unsolvable, 1184 00:55:19,150 --> 00:55:21,769 open question. of deep learning representations. 1185 00:55:22,530 --> 00:55:23,750 And it's something to think about next time you 1186 00:55:23,750 --> 00:55:26,530 see an AI effortlessly perform a complex task. 1187 00:55:27,170 --> 00:55:28,969 Thanks for joining us on this deep dive into 1188 00:55:28,969 --> 00:55:31,489 the source material. Keep questioning the mechanisms 1189 00:55:31,489 --> 00:55:32,130 behind the headlines.