1 00:00:00,000 --> 00:00:02,819 First up, we have a massive, newly -published 2 00:00:02,819 --> 00:00:05,700 nature article titled, Mastering Diverse Control 3 00:00:05,700 --> 00:00:08,820 Tasks Through World Models. Right. This is by 4 00:00:08,820 --> 00:00:12,199 Dani R. Hoffner, Jurgis Pasekonis, Jimmy Baugh, 5 00:00:12,400 --> 00:00:15,140 and Timothy Lilikrap. It's the paper that introduces 6 00:00:15,140 --> 00:00:17,519 the third generation of the Dreamer algorithm, 7 00:00:17,879 --> 00:00:21,670 aptly named Dreamer V3. It is a profoundly important 8 00:00:21,670 --> 00:00:23,929 piece of work for simulated environments and 9 00:00:23,929 --> 00:00:25,850 control tasks. Because, you know, it shows what 10 00:00:25,850 --> 00:00:27,649 happens when you push the concept of a world 11 00:00:27,649 --> 00:00:30,670 model to its absolute engineering limits. And 12 00:00:30,670 --> 00:00:32,210 then, because we can't just live in video games 13 00:00:32,210 --> 00:00:33,929 forever, we are going to look at a companion 14 00:00:33,929 --> 00:00:37,070 paper to give us a perspective on physical real 15 00:00:37,070 --> 00:00:39,369 -world applications. Essential for robotics. 16 00:00:39,590 --> 00:00:43,649 Exactly. That one is called VJEPA 2. Self -supervised 17 00:00:43,649 --> 00:00:45,909 video models enable understanding, prediction, 18 00:00:46,189 --> 00:00:48,920 and planning. by Mahmoud Asran and colleagues, 19 00:00:49,460 --> 00:00:51,799 with Nicolas Ballas and Michael Rabbit as the 20 00:00:51,799 --> 00:00:54,200 corresponding authors. These two papers offer 21 00:00:54,200 --> 00:00:57,960 such a fascinating contrast. They represent two 22 00:00:57,960 --> 00:01:00,619 cutting -edge, yet very distinct paths toward 23 00:01:00,619 --> 00:01:03,100 giving an AI an internal understanding of how 24 00:01:03,100 --> 00:01:06,379 the world works. But... Ah, yes, the caveat. 25 00:01:06,540 --> 00:01:08,840 Right. Before we dive into the math and the mechanisms, 26 00:01:09,040 --> 00:01:11,739 I want to set a very firm baseline for you, our 27 00:01:11,739 --> 00:01:14,739 listener. Yes, this is a crucial caveat. Because 28 00:01:14,739 --> 00:01:16,500 while the experiments we are going to unpack 29 00:01:16,500 --> 00:01:19,920 today show staggering feats of engineering, we 30 00:01:19,920 --> 00:01:22,040 are not talking about a universally competent 31 00:01:22,040 --> 00:01:24,879 super agent here. No, definitely not. When we 32 00:01:24,879 --> 00:01:29,519 say an AI understands physics or imagines a solution, 33 00:01:30,079 --> 00:01:32,959 we are just using human words to describe mathematical 34 00:01:32,959 --> 00:01:36,280 processes. Exactly. We aren't talking about mathematical 35 00:01:36,280 --> 00:01:38,560 guarantees of success in any situation. Right. 36 00:01:38,650 --> 00:01:41,069 Nor are we talking about an artificial general 37 00:01:41,069 --> 00:01:43,129 intelligence that can just wake up and solve 38 00:01:43,129 --> 00:01:46,689 everything. We are examining specific bounded 39 00:01:46,689 --> 00:01:49,670 experimental results. So specific algorithms 40 00:01:49,670 --> 00:01:52,150 tested under highly specific conditions. Right. 41 00:01:52,450 --> 00:01:54,689 And what makes them remarkable are the clever 42 00:01:54,689 --> 00:01:57,310 engineering mechanisms that allow them to succeed 43 00:01:57,310 --> 00:01:59,590 within those bounds. We're essentially looking 44 00:01:59,590 --> 00:02:02,109 at the engine of the car, how the pistons fire 45 00:02:02,109 --> 00:02:04,109 and why it can drive on so many different tracks. 46 00:02:04,329 --> 00:02:06,269 So let's pop the hood. I want to start with a 47 00:02:06,269 --> 00:02:09,300 foundational philosophy here. Why should an AI 48 00:02:09,300 --> 00:02:11,979 try to model the entire world just to solve a 49 00:02:11,979 --> 00:02:15,360 task? Because intuitively, if I want a robotic 50 00:02:15,360 --> 00:02:17,360 arm to pick up an apple on a table, it just needs 51 00:02:17,360 --> 00:02:19,900 to know where the apple is and how to close its 52 00:02:19,900 --> 00:02:22,060 grippers. Right, the direct mapping approach. 53 00:02:22,360 --> 00:02:24,840 Yeah. Why does it need a complex internal model 54 00:02:24,840 --> 00:02:27,319 of the room's physics? To answer that, we actually 55 00:02:27,319 --> 00:02:29,379 have to look backward into cognitive theory. 56 00:02:29,879 --> 00:02:32,400 If you look closely at the V. Jeppa II text, 57 00:02:32,580 --> 00:02:34,840 they draw a direct line to the work of Kenneth 58 00:02:34,840 --> 00:02:37,020 Craig. Kenneth Craig. Yeah, he was a philosopher 59 00:02:37,020 --> 00:02:38,840 and psychologist writing all the way back in 60 00:02:38,840 --> 00:02:42,840 1943. Oh, wow. Over 80 years ago. Exactly. Craig 61 00:02:42,840 --> 00:02:45,139 proposed that humans and biological creatures 62 00:02:45,139 --> 00:02:48,360 in general don't just react to raw sensory inputs. 63 00:02:48,979 --> 00:02:51,000 Instead, we learn internal models of the world 64 00:02:51,000 --> 00:02:53,259 by integrating all that low level sensory data. 65 00:02:53,449 --> 00:02:55,490 So we're building a simulation in our heads. 66 00:02:55,949 --> 00:02:58,530 Right. We use these internal models to represent 67 00:02:58,530 --> 00:03:01,349 reality and, most critically, to predict future 68 00:03:01,349 --> 00:03:05,009 states. OK, so it's about compression and prediction. 69 00:03:05,330 --> 00:03:08,009 Like, when I look at a room, I don't see a grid 70 00:03:08,009 --> 00:03:10,430 of a billion individual photons. No, your brain 71 00:03:10,430 --> 00:03:12,430 compresses that. Right. It compresses that into 72 00:03:12,430 --> 00:03:15,889 concepts like chair, table, apple. And because 73 00:03:15,889 --> 00:03:19,110 I have a model of what an apple is, I know that 74 00:03:19,110 --> 00:03:21,590 if I push it, it will roll. I don't have to literally 75 00:03:21,590 --> 00:03:24,800 test it every single time. Precisely. And our 76 00:03:24,800 --> 00:03:26,879 ability to predict the effects of our actions 77 00:03:26,879 --> 00:03:29,860 on future states is exactly what makes goal -oriented 78 00:03:29,860 --> 00:03:32,439 planning possible. Makes sense. If you want to 79 00:03:32,439 --> 00:03:34,419 build an artificial agent that can adapt like 80 00:03:34,419 --> 00:03:37,099 a human, the prevailing theory here is that it 81 00:03:37,099 --> 00:03:39,379 needs to understand the physical world, predict 82 00:03:39,379 --> 00:03:41,819 future states, and plan its actions based on 83 00:03:41,819 --> 00:03:44,319 those predictions. It basically needs its own 84 00:03:44,319 --> 00:03:46,860 internal representation of reality. I always 85 00:03:46,860 --> 00:03:48,840 picture it like playing chess. Oh, that's a good 86 00:03:48,840 --> 00:03:51,300 analogy. Yeah. A grandmaster doesn't physically 87 00:03:51,300 --> 00:03:54,580 pick up a knight. move it to a square, let go, 88 00:03:54,699 --> 00:03:57,000 wait to see the opponent's face, and then say, 89 00:03:57,060 --> 00:03:59,439 oh, wait, bad move, let me take that back. Right, 90 00:03:59,560 --> 00:04:01,960 they'd lose immediately. Exactly. They simulate 91 00:04:01,960 --> 00:04:04,419 the board state in their head. They imagine the 92 00:04:04,419 --> 00:04:06,439 future trajectories of the game based on the 93 00:04:06,439 --> 00:04:08,699 rules of reality, or, well, the rules of chess 94 00:04:08,699 --> 00:04:11,729 in this case. That is a perfect analogy. You 95 00:04:11,729 --> 00:04:14,610 are rolling forward an internal world model to 96 00:04:14,610 --> 00:04:17,069 evaluate the outcomes of potential actions before 97 00:04:17,069 --> 00:04:18,870 you ever commit to them in the physical world. 98 00:04:19,509 --> 00:04:21,629 And that brings us directly to the architecture 99 00:04:21,629 --> 00:04:24,850 of Dreamer V3. How does it actually build that 100 00:04:24,850 --> 00:04:27,750 map? Right. Let's deconstruct this thing. According 101 00:04:27,750 --> 00:04:30,689 to the Nature paper, Dreamer V3 isn't just one 102 00:04:30,689 --> 00:04:33,029 monolithic piece of code. No, it's highly structured. 103 00:04:33,290 --> 00:04:35,910 It consists of three main neural networks operating 104 00:04:35,910 --> 00:04:39,329 in tandem. I like to call it the neural triumvirate. 105 00:04:39,449 --> 00:04:42,250 You have the world model, the critic, and the 106 00:04:42,250 --> 00:04:44,949 actor. Let's take them one by one, starting with 107 00:04:44,949 --> 00:04:47,829 the world model. How does it actually represent 108 00:04:47,829 --> 00:04:50,189 the world? It learns compact representations 109 00:04:50,189 --> 00:04:53,569 of sensory inputs through autoencoding. Let's 110 00:04:53,569 --> 00:04:55,550 define that for anyone who might not be deep 111 00:04:55,550 --> 00:04:58,589 in machine learning. Autoencoding. Think of an 112 00:04:58,589 --> 00:05:01,209 autoencoder as a two -part system. An encoder. 113 00:05:01,370 --> 00:05:04,230 and a decoder. OK. The encoder takes in high 114 00:05:04,230 --> 00:05:07,370 dimensional sensory data, let's say a 64 by 64 115 00:05:07,370 --> 00:05:09,370 pixel image from a video game screen. That's 116 00:05:09,370 --> 00:05:12,269 over 12 ,000 individual color values. Which is 117 00:05:12,269 --> 00:05:14,470 a lot of numbers for a network to crunch. Exactly. 118 00:05:14,949 --> 00:05:17,550 So the encoder compresses that massive amount 119 00:05:17,550 --> 00:05:20,449 of data into a tiny dense vector of numbers called 120 00:05:20,449 --> 00:05:22,910 a latent representation. It throws away the noise 121 00:05:22,910 --> 00:05:24,910 and keeps only the most essential information. 122 00:05:25,670 --> 00:05:27,860 And the decoder. The decoder then takes that 123 00:05:27,860 --> 00:05:30,300 tiny vector and tries to reconstruct the original 124 00:05:30,300 --> 00:05:33,420 64 by 64 image. So it's basically playing a game 125 00:05:33,420 --> 00:05:36,040 of telephone with itself, forcing the image through 126 00:05:36,040 --> 00:05:38,420 a tiny bottleneck to see if it can still draw 127 00:05:38,420 --> 00:05:40,459 the picture on the other side. Right. And if 128 00:05:40,459 --> 00:05:42,740 it can, that means the tiny bottleneck vector 129 00:05:42,740 --> 00:05:45,620 must contain the essence of the image. That's 130 00:05:45,620 --> 00:05:48,560 it exactly. But Dreamer V3 doesn't just use a 131 00:05:48,560 --> 00:05:51,610 simple static autoencoder. It uses something 132 00:05:51,610 --> 00:05:54,250 much more sophisticated for its world model called 133 00:05:54,250 --> 00:05:58,610 a Recurrent State Space Model, or RSSM. Recurrent 134 00:05:58,610 --> 00:06:01,810 State Space Model. OK, unpack that for us. The 135 00:06:01,810 --> 00:06:04,350 RSSM is fascinating because it's dealing with 136 00:06:04,350 --> 00:06:06,990 sequences of time, not just isolated pictures. 137 00:06:07,269 --> 00:06:09,810 Because life isn't a single photograph. Exactly. 138 00:06:10,529 --> 00:06:12,810 If you are playing a game, the current frame 139 00:06:12,810 --> 00:06:15,000 doesn't tell you everything. If you see a picture 140 00:06:15,000 --> 00:06:16,500 of a ball in the air, you don't know if it's 141 00:06:16,500 --> 00:06:19,019 going up or falling down, unless you know what 142 00:06:19,019 --> 00:06:21,399 happened in the previous frames. Right. So the 143 00:06:21,399 --> 00:06:24,959 RSSM uses a hybrid latent state. For every time 144 00:06:24,959 --> 00:06:27,220 step, its internal model state is made up of 145 00:06:27,220 --> 00:06:29,620 two distinct parts. What's the first part? First, 146 00:06:29,639 --> 00:06:32,079 there is a continuous deterministic recurrent 147 00:06:32,079 --> 00:06:34,079 state, which the paper denotes as the letter 148 00:06:34,079 --> 00:06:37,779 H. Think of H as the model's memory. Okay, continuous 149 00:06:37,779 --> 00:06:40,399 memory. Right, it's a continuous vector that 150 00:06:40,399 --> 00:06:42,839 carries the historical context forward from one 151 00:06:42,839 --> 00:06:45,980 time step to the next. It's calculated deterministically 152 00:06:45,980 --> 00:06:48,379 based on the previous memory and the previous 153 00:06:48,379 --> 00:06:51,040 action the agent took. It remembers the momentum 154 00:06:51,040 --> 00:06:53,139 of the ball or the fact that a door was open 155 00:06:53,139 --> 00:06:55,779 three seconds ago. Okay, so H is the continuous 156 00:06:55,779 --> 00:06:58,160 history. What is the second part? The second 157 00:06:58,160 --> 00:07:01,379 part is a discrete stochastic state. denoted 158 00:07:01,379 --> 00:07:04,959 as Z. While H is the historical context, Z is 159 00:07:04,959 --> 00:07:06,839 the model's representation of what is happening 160 00:07:06,839 --> 00:07:09,860 right now in the current frame. The encoder maps 161 00:07:09,860 --> 00:07:12,360 the current sensory inputs, the pixels, into 162 00:07:12,360 --> 00:07:15,220 this discrete representation Z. Right, hold on. 163 00:07:15,379 --> 00:07:18,639 A discrete state? Yes. I know just enough about 164 00:07:18,639 --> 00:07:21,300 deep learning to know that continuous variables 165 00:07:21,300 --> 00:07:24,279 are usually the go -to. Neural networks learn 166 00:07:24,279 --> 00:07:26,620 using calculus, specifically back propagation 167 00:07:26,620 --> 00:07:28,699 calculating gradients. Right, the chain rule. 168 00:07:28,889 --> 00:07:31,269 And calculus requires smooth, continuous slopes 169 00:07:31,269 --> 00:07:32,930 to figure out which way to adjust the weights. 170 00:07:33,910 --> 00:07:36,490 If you have a discrete variable, like rolling 171 00:07:36,490 --> 00:07:38,569 a six -sided die where it's either a one, two, 172 00:07:38,569 --> 00:07:40,689 three, four, five, or six, with no smooth decimals 173 00:07:40,689 --> 00:07:43,829 in between, you create a mathematical cliff. 174 00:07:44,389 --> 00:07:46,949 You can't take a derivative of a cliff. No, you 175 00:07:46,949 --> 00:07:50,009 cannot. So how on earth are you using a discrete 176 00:07:50,009 --> 00:07:53,209 state z and still training a neural network? 177 00:07:53,730 --> 00:07:55,769 Does making it discrete magically solve some 178 00:07:55,769 --> 00:07:58,360 other problem? You've hit on a very complex and 179 00:07:58,360 --> 00:08:00,720 crucial piece of the architecture. First, to 180 00:08:00,720 --> 00:08:03,699 address your assumption, no, using discrete variables 181 00:08:03,699 --> 00:08:06,759 does not magically solve problems. Right. In 182 00:08:06,759 --> 00:08:09,300 fact, it introduces the exact calculus problem 183 00:08:09,300 --> 00:08:11,660 you just described. They use a clever technique 184 00:08:11,660 --> 00:08:14,500 called a straight -through estimator. I have 185 00:08:14,500 --> 00:08:16,819 never heard of that. It is essentially a mathematical 186 00:08:16,819 --> 00:08:20,939 hack. Oh. Let's use an analogy. Okay. Imagine 187 00:08:20,939 --> 00:08:23,540 you want to calculate how hard you threw a rubber 188 00:08:23,540 --> 00:08:26,160 ball by measuring how far it traveled. If you 189 00:08:26,160 --> 00:08:28,620 throw it on a smooth, continuous hill, you can 190 00:08:28,620 --> 00:08:30,819 use calculus to trace the exact curve of its 191 00:08:30,819 --> 00:08:33,100 path back to your hand. But what if you throw 192 00:08:33,100 --> 00:08:35,460 it down a jagged set of stairs? It's bouncing 193 00:08:35,460 --> 00:08:37,740 all over the place in sharp, discrete jumps. 194 00:08:38,080 --> 00:08:40,460 Exactly. Calculus breaks down on the stairs. 195 00:08:41,299 --> 00:08:43,179 The straight -through estimator solves this by 196 00:08:43,179 --> 00:08:45,960 doing one thing going forward and another thing 197 00:08:45,960 --> 00:08:48,789 going backward. Okay, break that down. During 198 00:08:48,789 --> 00:08:51,389 the forward pass, when the model's actually making 199 00:08:51,389 --> 00:08:53,730 predictions and playing the game, it uses the 200 00:08:53,730 --> 00:08:56,590 discrete jagged steps. It snaps the probabilities 201 00:08:56,590 --> 00:08:59,350 to hard ones and zeros. Right. But during the 202 00:08:59,350 --> 00:09:01,570 backward pass, when the network is calculating 203 00:09:01,570 --> 00:09:03,769 its errors and updating its weights using back 204 00:09:03,769 --> 00:09:06,210 propagation, it pretends the steps aren't there. 205 00:09:06,610 --> 00:09:09,289 It pretends the stairs are actually a smooth, 206 00:09:09,570 --> 00:09:12,070 continuous ramp and passes the gradient straight 207 00:09:12,070 --> 00:09:14,370 through the discrete sampling operation as if 208 00:09:14,370 --> 00:09:18,049 it were the identity function. So... It literally 209 00:09:18,049 --> 00:09:20,950 lies to the backpropagation algorithm. It says, 210 00:09:21,250 --> 00:09:23,009 don't worry about the cliff, just assume it was 211 00:09:23,009 --> 00:09:25,110 a smooth slope and keep passing the error signal 212 00:09:25,110 --> 00:09:27,629 backward. That's a very colloquial way to put 213 00:09:27,629 --> 00:09:30,330 it, but functionally, yes. It's an approximation, 214 00:09:30,590 --> 00:09:32,769 and it doesn't guarantee perfect stability on 215 00:09:32,769 --> 00:09:35,570 its own. But combined with the continuous history 216 00:09:35,570 --> 00:09:39,289 state H, this hybrid RSSM setup gives Dreamer 217 00:09:39,289 --> 00:09:41,850 a highly robust way to predict the sequence of 218 00:09:41,850 --> 00:09:44,149 future representations based on past actions. 219 00:09:44,690 --> 00:09:47,299 OK. So the world model has this hybrid state. 220 00:09:47,480 --> 00:09:50,639 It remembers history. It creates a discrete representation 221 00:09:50,639 --> 00:09:53,379 of the present. And its entire job is to predict 222 00:09:53,379 --> 00:09:56,080 what comes next. It tries to reconstruct the 223 00:09:56,080 --> 00:09:58,440 image of the game. It predicts the reward it's 224 00:09:58,440 --> 00:10:00,340 going to get. And it predicts whether the game 225 00:10:00,340 --> 00:10:02,820 is going to continue or if it's game over. Yeah, 226 00:10:02,919 --> 00:10:06,679 precisely. It minimizes prediction losses to 227 00:10:06,679 --> 00:10:09,059 force its internal representation to accurately 228 00:10:09,059 --> 00:10:11,500 mirror the dynamics of the actual environment. 229 00:10:11,899 --> 00:10:14,080 But here's where this gets absolutely wild to 230 00:10:14,080 --> 00:10:17,070 me. The paper is called Dreamer. It gets that 231 00:10:17,070 --> 00:10:19,110 name because the agent learns by imagining the 232 00:10:19,110 --> 00:10:21,769 future. But if its model of the world is based 233 00:10:21,769 --> 00:10:25,330 purely on what it has seen so far, that imagination 234 00:10:25,330 --> 00:10:27,889 has to be flawed, right? Oh, fundamentally flawed. 235 00:10:28,230 --> 00:10:29,929 Like, imagine the agent is playing a game with 236 00:10:29,929 --> 00:10:32,269 standard gravity. It drops a block, the block 237 00:10:32,269 --> 00:10:34,850 falls. It learns that. But what if the agent 238 00:10:34,850 --> 00:10:37,230 takes an action that pushes it into a weird edge 239 00:10:37,230 --> 00:10:39,990 case of the game, where gravity suddenly reverses? 240 00:10:40,250 --> 00:10:43,110 Because it hasn't seen that its internal world 241 00:10:43,110 --> 00:10:45,129 model will imagine the block falling normally 242 00:10:45,129 --> 00:10:48,610 Its dream will contain completely physically 243 00:10:48,610 --> 00:10:50,789 impossible physics compared to the real game 244 00:10:50,789 --> 00:10:53,740 if that is exactly what happens The learned world 245 00:10:53,740 --> 00:10:56,960 model is an imperfect predictor. It is fundamentally 246 00:10:56,960 --> 00:10:59,580 an approximation of reality based only on its 247 00:10:59,580 --> 00:11:02,120 past experience. Right. The reconstruction and 248 00:11:02,120 --> 00:11:04,200 prediction losses encourage it to agree with 249 00:11:04,200 --> 00:11:06,460 the past, but they absolutely do not prevent 250 00:11:06,460 --> 00:11:09,679 it from imagining inaccurate or physically impossible 251 00:11:09,679 --> 00:11:11,820 outcomes when it wanders into new territory. 252 00:11:12,080 --> 00:11:14,200 So it's hallucinating. It's dreaming up things 253 00:11:14,200 --> 00:11:17,059 that make no sense. And yet this is the environment 254 00:11:17,059 --> 00:11:19,799 the agent is supposed to learn in. Yes, and this 255 00:11:19,799 --> 00:11:21,919 creates a fascinating and somewhat dangerous 256 00:11:21,919 --> 00:11:24,360 dynamic for the learning process. This brings 257 00:11:24,360 --> 00:11:26,399 us to the second and third parts of our neural 258 00:11:26,399 --> 00:11:29,580 triumvirate, the actor and the critic. Okay, 259 00:11:29,679 --> 00:11:32,779 the actor. The actor's job is to choose actions 260 00:11:32,779 --> 00:11:35,879 that maximize future rewards. But here is the 261 00:11:35,879 --> 00:11:39,539 critical detail. The actor in Dreamer V3 never 262 00:11:39,539 --> 00:11:42,059 ever learns directly from interacting with the 263 00:11:42,059 --> 00:11:44,919 real world. Wait, really? It never looks at the 264 00:11:44,919 --> 00:11:48,590 actual screen to learn how to move? Never. The 265 00:11:48,590 --> 00:11:51,850 actor lives entirely inside the dream generated 266 00:11:51,850 --> 00:11:53,970 by the world model's latent representations. 267 00:11:54,370 --> 00:11:56,830 And this means the actor will ruthlessly exploit 268 00:11:56,830 --> 00:11:59,570 any flaws in that model's imagination. Oh, I 269 00:11:59,570 --> 00:12:02,129 see where this is going. If the world model has 270 00:12:02,129 --> 00:12:05,309 a flaw where it inaccurately imagines that walking 271 00:12:05,309 --> 00:12:07,870 through a solid wall will instantly give the 272 00:12:07,870 --> 00:12:10,110 agent a million points, the actor is going to 273 00:12:10,110 --> 00:12:12,889 say, great, I'm going to walk through walls all 274 00:12:12,889 --> 00:12:15,149 day long. Exactly. Because the actor does know 275 00:12:15,149 --> 00:12:17,309 it's a hallucination. It only knows the rules 276 00:12:17,309 --> 00:12:20,350 of the dream it's trapped in. We call this model 277 00:12:20,350 --> 00:12:23,149 exploitation. The actor optimizes its policy 278 00:12:23,149 --> 00:12:25,210 to take advantage of the world model's errors. 279 00:12:25,710 --> 00:12:27,750 That sounds like a complete disaster. If the 280 00:12:27,750 --> 00:12:30,789 actor is optimizing for a hallucinated world, 281 00:12:31,350 --> 00:12:33,289 how does it ever manage to play the real game? 282 00:12:33,850 --> 00:12:36,889 Because it is an online iterative loop. The actor 283 00:12:36,889 --> 00:12:39,549 learns a policy in the dream and then that policy 284 00:12:39,549 --> 00:12:42,309 is used to control the agent in the actual environment. 285 00:12:43,070 --> 00:12:44,929 And when it tries to walk through the wall in 286 00:12:44,929 --> 00:12:47,210 reality, it just bumps its head and gets zero 287 00:12:47,210 --> 00:12:50,850 points. Exactly. It gathers real, observed experience 288 00:12:50,850 --> 00:12:53,149 from that failure. And that new data is added 289 00:12:53,149 --> 00:12:56,049 to a replay buffer. The world model then pulls 290 00:12:56,049 --> 00:12:58,090 that real data from the buffer and trains on 291 00:12:58,090 --> 00:13:01,309 it. It updates itself. It is a continuous, alternating 292 00:13:01,309 --> 00:13:03,610 loop of interaction, dreaming, and correction. 293 00:13:04,070 --> 00:13:07,129 That is incredibly elegant. So we have the world 294 00:13:07,129 --> 00:13:09,269 model generating the dream, the actor trying 295 00:13:09,269 --> 00:13:12,389 to win inside the dream. What about the critic? 296 00:13:12,539 --> 00:13:15,299 What is its role? The critic evaluates how good 297 00:13:15,299 --> 00:13:18,220 a particular state is. While the actor is choosing 298 00:13:18,220 --> 00:13:20,360 actions, the critic is looking at the states 299 00:13:20,360 --> 00:13:23,399 the actor ends up in and predicting the expected 300 00:13:23,399 --> 00:13:25,600 distribution of returns from that point forward. 301 00:13:26,120 --> 00:13:29,580 You use that word returns. In reinforcement learning, 302 00:13:29,740 --> 00:13:32,500 a reward is the points you get right now, but 303 00:13:32,500 --> 00:13:35,139 a return is the sum of all the rewards you expect 304 00:13:35,139 --> 00:13:38,389 to get in the future, right? Correct. The critic 305 00:13:38,389 --> 00:13:40,649 is trying to estimate the total future value, 306 00:13:41,210 --> 00:13:43,710 but in Dreamer v3, the critic doesn't just predict 307 00:13:43,710 --> 00:13:46,470 a simple sum. It predicts what are called bootstrapped 308 00:13:46,470 --> 00:13:49,730 lambda returns. Okay, we definitely need to unpack 309 00:13:49,730 --> 00:13:53,830 that. Bootstrapped lambda returns. I know bootstrapping 310 00:13:53,830 --> 00:13:55,909 generally means pulling yourself up by your own 311 00:13:55,909 --> 00:13:58,409 laces, relying on yourself. What does it mean 312 00:13:58,409 --> 00:14:02,009 for a neural network to bootstrap? In reinforcement 313 00:14:02,009 --> 00:14:04,090 learning, bootstrapping means updating a guess 314 00:14:04,090 --> 00:14:06,529 based on another guess. Okay. Imagine you are 315 00:14:06,529 --> 00:14:08,450 trying to predict if it will rain next Tuesday. 316 00:14:08,909 --> 00:14:10,769 You could wait until next Tuesday to see what 317 00:14:10,769 --> 00:14:12,509 happens and then update your prediction model. 318 00:14:13,029 --> 00:14:14,990 That's not bootstrapping. That's learning from 319 00:14:14,990 --> 00:14:17,350 the final outcome. Right. But what if on Friday 320 00:14:17,350 --> 00:14:19,769 you see dark clouds gathering? You might update 321 00:14:19,769 --> 00:14:21,970 your prediction for Tuesday based on your prediction 322 00:14:21,970 --> 00:14:24,769 for Saturday. You're using your current estimate 323 00:14:24,769 --> 00:14:26,769 of a future state to update your estimate of 324 00:14:26,769 --> 00:14:28,870 an earlier state without waiting for the final 325 00:14:28,870 --> 00:14:31,169 truth. That makes perfect sense. That's bootstrapping. 326 00:14:31,399 --> 00:14:33,620 The critic network estimates the value of state 327 00:14:33,620 --> 00:14:36,340 A by looking at the reward it gets plus its own 328 00:14:36,340 --> 00:14:39,460 guess for the value of state B. So it's a network 329 00:14:39,460 --> 00:14:42,320 feeding on its own predictions. But what about 330 00:14:42,320 --> 00:14:45,379 the lambda part? What is a lambda return? Lambda 331 00:14:45,379 --> 00:14:47,700 is a parameter that dictates how far into the 332 00:14:47,700 --> 00:14:50,379 imagined future the critic looks before it relies 333 00:14:50,379 --> 00:14:53,179 on its own guess. Exactly. The lambda parameter 334 00:14:53,179 --> 00:14:55,659 mixes different time horizons together to balance 335 00:14:55,659 --> 00:14:59,059 that bias and variance. It creates a smooth target 336 00:14:59,059 --> 00:15:01,419 for the critic to learn from. Does the critic 337 00:15:01,419 --> 00:15:03,940 only learn from the imagination, just like the 338 00:15:03,940 --> 00:15:07,399 actor? Actually, no. And this is a vital nuance 339 00:15:07,399 --> 00:15:09,740 in the published implementation of Dreamer V3. 340 00:15:10,100 --> 00:15:12,700 While the actor learns purely from imagination, 341 00:15:13,120 --> 00:15:15,539 the critic is trained on both the imagined trajectories 342 00:15:15,539 --> 00:15:18,860 and the replay trajectories. Replay trajectories 343 00:15:18,860 --> 00:15:21,019 being the actual real experiences stored in the 344 00:15:21,019 --> 00:15:24,200 buffer. Yes. Why give the critic access to reality 345 00:15:24,200 --> 00:15:26,330 if the actor is stuck in the matrix? Because 346 00:15:26,330 --> 00:15:28,990 the critic's job is extremely difficult. It has 347 00:15:28,990 --> 00:15:31,929 to predict long -term value. In many environments, 348 00:15:32,149 --> 00:15:34,309 rewards are sparse and incredibly challenging 349 00:15:34,309 --> 00:15:36,970 to predict accurately based purely on a hallucinated 350 00:15:36,970 --> 00:15:40,049 model. By training the critic on real replay 351 00:15:40,049 --> 00:15:42,649 data as well, even with a smaller loss weight, 352 00:15:43,190 --> 00:15:45,730 it provides a grounding anchor. It ensures the 353 00:15:45,730 --> 00:15:48,529 value annotations remain somewhat tied to reality, 354 00:15:48,629 --> 00:15:51,250 which stabilizes the actor's learning. So the 355 00:15:51,250 --> 00:15:52,889 critic has one foot in the dream and one foot 356 00:15:52,889 --> 00:15:55,070 in the real world, keeping things grounded. Yes. 357 00:15:55,309 --> 00:15:57,649 And to further stabilize learning, the critic 358 00:15:57,649 --> 00:16:00,049 is regularized towards an exponentially moving 359 00:16:00,049 --> 00:16:02,870 average of its own parameters. Meaning it's aiming 360 00:16:02,870 --> 00:16:06,490 at a slowly moving target of itself. Right. And 361 00:16:06,490 --> 00:16:08,990 I want to reiterate for you listening, this entire 362 00:16:08,990 --> 00:16:12,029 setup, the RSSM world model, the dream -bound 363 00:16:12,029 --> 00:16:15,309 actor, the grounded critic, is an online interaction 364 00:16:15,309 --> 00:16:17,750 and learning loop. The agent is constantly exploring 365 00:16:17,750 --> 00:16:20,309 the environment, updating its model, generating 366 00:16:20,309 --> 00:16:23,029 new dreams, and refining its policy. So if we 367 00:16:23,029 --> 00:16:25,629 pull back, And look at the big picture of this 368 00:16:25,629 --> 00:16:29,049 triumvirate. We have a world model that is fundamentally 369 00:16:29,049 --> 00:16:31,409 imperfect and prone to hallucinating physically 370 00:16:31,409 --> 00:16:35,110 impossible scenarios. Yes. We have an actor that 371 00:16:35,110 --> 00:16:37,549 will ruthlessly exploit any of those hallucinations 372 00:16:37,549 --> 00:16:40,629 to get virtual points. And we have a critic trying 373 00:16:40,629 --> 00:16:42,789 to rein it all in by guessing future values based 374 00:16:42,789 --> 00:16:45,730 on its own previous guesses. That sums it up. 375 00:16:45,929 --> 00:16:48,350 That sounds inherently chaotic. It sounds like 376 00:16:48,350 --> 00:16:51,669 a house of cards. Yet... The Nature paper claims 377 00:16:51,669 --> 00:16:54,690 that Dreamer V3 mastered over 150 diverse tasks 378 00:16:54,690 --> 00:16:58,210 with fixed hyperparameters. Which is astounding. 379 00:16:58,730 --> 00:17:01,230 It is. And it brings me to a massive sticking 380 00:17:01,230 --> 00:17:04,109 point. If I try to tune a simple neural network 381 00:17:04,109 --> 00:17:06,829 to play Pong, and then I use those exact same 382 00:17:06,829 --> 00:17:10,170 settings to control a 3D robot, it fails. Why? 383 00:17:10,390 --> 00:17:13,230 Because in Pong, the score goes up by one. In 384 00:17:13,230 --> 00:17:15,410 a robot simulation, the roared signal might be 385 00:17:15,410 --> 00:17:17,490 based on the angle of a joint, fluctuating by 386 00:17:17,490 --> 00:17:20,150 10 ,000 points a second. The scales are completely 387 00:17:20,150 --> 00:17:22,990 mismatched. Right. If I don't manually tune the 388 00:17:22,990 --> 00:17:25,450 learning rates and the reward scales, the network's 389 00:17:25,450 --> 00:17:28,950 math explodes. So how did Dreamer V3 achieve 390 00:17:28,950 --> 00:17:31,049 this robustness without a human reconfiguring 391 00:17:31,049 --> 00:17:33,630 it for every game? This leads us perfectly into 392 00:17:33,630 --> 00:17:36,069 the engineering marvels of the paper. The paper 393 00:17:36,069 --> 00:17:39,210 introduces several incredibly elegant mathematical 394 00:17:39,210 --> 00:17:42,150 solutions to solve this exact problem. If you 395 00:17:42,150 --> 00:17:45,150 want a single algorithm to succeed across 150 396 00:17:45,150 --> 00:17:47,609 wildly different domains without human tweaking, 397 00:17:48,190 --> 00:17:50,150 all three components, the world model, the actor, 398 00:17:50,150 --> 00:17:52,369 and the critic, need to be able to accommodate 399 00:17:52,369 --> 00:17:55,069 extreme variations in signal magnitudes without 400 00:17:55,069 --> 00:17:57,329 breaking. So what's the first tool in their toolbox? 401 00:17:57,650 --> 00:17:59,829 The first major tool they use is called the SimLog 402 00:17:59,829 --> 00:18:02,690 transformation. SimLog. I'm assuming that stands 403 00:18:02,690 --> 00:18:05,690 for symmetric logarithm? Yes, a bi -symmetric 404 00:18:05,690 --> 00:18:08,470 logarithmic transformation. Okay. Let's think 405 00:18:08,470 --> 00:18:11,190 about how a neural network learns. It makes a 406 00:18:11,190 --> 00:18:14,049 prediction, compares it to the true target, calculates 407 00:18:14,049 --> 00:18:16,029 the difference, which is the error, and updates 408 00:18:16,029 --> 00:18:18,670 its weights. Standard back prop. Right. If you 409 00:18:18,670 --> 00:18:20,990 are using standard squared error, predicting 410 00:18:20,990 --> 00:18:23,990 large targets can be disastrous. If the true 411 00:18:23,990 --> 00:18:26,670 reward in a game is 10 ,000 and the network guesses 412 00:18:26,670 --> 00:18:30,430 zero, the error is 10 ,000. When you square that 413 00:18:30,430 --> 00:18:33,289 for the loss function, it becomes 100 million. 414 00:18:33,369 --> 00:18:36,529 That is a massive number. The simlog function 415 00:18:36,529 --> 00:18:38,470 solves this by compressing the magnitudes of 416 00:18:38,470 --> 00:18:40,970 both large positive and large negative values. 417 00:18:41,450 --> 00:18:44,349 So it squashes them. A massive reward of 10 ,000 418 00:18:44,349 --> 00:18:46,930 is mathematically squashed down to a much smaller, 419 00:18:47,109 --> 00:18:50,089 manageable logarithmic scale. Okay, squashing 420 00:18:50,089 --> 00:18:52,490 big numbers makes sense. But what about a game 421 00:18:52,490 --> 00:18:55,059 where the rewards are tiny? What if a good move 422 00:18:55,059 --> 00:18:59,259 gives you a reward of 0 .01? If SimLog squashes 423 00:18:59,259 --> 00:19:01,839 everything, does that tiny reward just get crushed 424 00:19:01,839 --> 00:19:05,000 into absolute zero and disappear? No, and that 425 00:19:05,000 --> 00:19:07,619 is the brilliant part of the bi -symmetric logarithm. 426 00:19:07,980 --> 00:19:10,859 It is designed to approximate the identity function 427 00:19:10,859 --> 00:19:14,470 around the origin. Meaning? That means for small 428 00:19:14,470 --> 00:19:17,130 values near zero, say between negative 1 and 429 00:19:17,130 --> 00:19:20,390 1, the function acts linearly. It leaves them 430 00:19:20,390 --> 00:19:23,170 mostly untouched. It only applies heavy compression 431 00:19:23,170 --> 00:19:26,529 to the massive outliers. Oh, that's clever. This 432 00:19:26,529 --> 00:19:29,049 allows the optimization process to quickly move 433 00:19:29,049 --> 00:19:31,869 predictions to large values when needed without 434 00:19:31,869 --> 00:19:34,430 unleashing that tidal wave of gradients that 435 00:19:34,430 --> 00:19:36,829 destroys the network. That handles the inputs 436 00:19:36,829 --> 00:19:39,710 and the targets nicely, but the paper also details 437 00:19:39,710 --> 00:19:42,150 how the network actually outputs its predictions 438 00:19:42,150 --> 00:19:45,630 for noisy values, like rewards and those bootstrap 439 00:19:45,630 --> 00:19:47,990 returns we discussed. They use something called 440 00:19:47,990 --> 00:19:52,000 the SimExp Two -hot loss. Yes, the two -hot loss. 441 00:19:52,299 --> 00:19:54,759 This sounds deeply complicated. What is a two 442 00:19:54,759 --> 00:19:57,539 -hot encoding? Let's start with a concept you 443 00:19:57,539 --> 00:20:00,440 might know. One -hot encoding. If you want a 444 00:20:00,440 --> 00:20:02,940 neural network to classify an image as either 445 00:20:02,940 --> 00:20:05,839 a cat, a dog, or a bird, you don't have it output 446 00:20:05,839 --> 00:20:08,660 a single number like one, two, or three. Right, 447 00:20:08,799 --> 00:20:11,579 because a bird isn't three times a cat. Exactly. 448 00:20:11,720 --> 00:20:14,059 You give it three separate output buckets. If 449 00:20:14,059 --> 00:20:17,259 it's a dog, the target is zero for cat, one for 450 00:20:17,259 --> 00:20:20,339 dog, and zero for bird. That's one hot. A single 451 00:20:20,339 --> 00:20:23,720 bucket is hot. Makes sense for categories. The 452 00:20:23,720 --> 00:20:25,700 too -hot encoding takes this concept and applies 453 00:20:25,700 --> 00:20:28,559 it to continuous numbers. Instead of the network 454 00:20:28,559 --> 00:20:31,220 outputting a single continuous number for a predictive 455 00:20:31,220 --> 00:20:34,359 reward, it outputs probabilities across a set 456 00:20:34,359 --> 00:20:37,299 of exponentially spaced bins. Bins. Think of 457 00:20:37,299 --> 00:20:41,039 it like tuning an old -school analog radio. Okay, 458 00:20:41,119 --> 00:20:43,420 I'm visualizing the radio dial with all the frequency 459 00:20:43,420 --> 00:20:45,839 markers. Imagine the markers on the dial are 460 00:20:45,839 --> 00:20:48,920 your bins. Let's say you have a marker at 4 and 461 00:20:48,920 --> 00:20:51,819 a marker at 6. But the true signal you are trying 462 00:20:51,819 --> 00:20:54,859 to predict is exactly 5. In a one -hot system, 463 00:20:54,940 --> 00:20:58,099 you'd have to round to 4 or 6. Which loses precision. 464 00:20:58,400 --> 00:21:01,420 Right. But in a two -hot system, the true continuous 465 00:21:01,420 --> 00:21:04,059 value is encoded by placing weights on the two 466 00:21:04,059 --> 00:21:07,079 closest bins. To represent a value of 5, the 467 00:21:07,079 --> 00:21:10,059 target places a 0 .5 weight on the 4 marker and 468 00:21:10,059 --> 00:21:12,859 a 0 .5 weight on the 6 marker. It interpolates 469 00:21:12,859 --> 00:21:15,460 perfectly between neighboring bins. So if the 470 00:21:15,460 --> 00:21:19,539 true value is 4 .5, it would put 75 % of the 471 00:21:19,539 --> 00:21:23,000 weight on the 4 bin and 25 % on the 6 bin? Exactly. 472 00:21:23,400 --> 00:21:25,940 It's a target encoding mechanism. The network 473 00:21:25,940 --> 00:21:28,539 is then trained to minimize the categorical cross 474 00:21:28,539 --> 00:21:31,160 entropy loss against these too hot, soft targets. 475 00:21:31,579 --> 00:21:34,880 Wait, if it's outputting 50 % on this bin and 476 00:21:34,880 --> 00:21:37,519 50 % on that bin, does that mean the network 477 00:21:37,519 --> 00:21:40,230 is expressing its subjective confidence? Like 478 00:21:40,230 --> 00:21:42,549 it's saying, I'm not sure, I'm 50 % confident 479 00:21:42,549 --> 00:21:46,109 it's a four and 50 % confident it's a six. That's 480 00:21:46,109 --> 00:21:49,470 a very common misconception. While it uses probabilities, 481 00:21:49,829 --> 00:21:52,109 the weights in the too -hot target interpolate 482 00:21:52,109 --> 00:21:55,670 as a strict mathematical encoding. It is not 483 00:21:55,670 --> 00:21:57,869 inherently a measure of the network's subjective 484 00:21:57,869 --> 00:21:59,950 uncertainty that the true number might be one 485 00:21:59,950 --> 00:22:02,950 of the endpoints. So it's just a representation. 486 00:22:03,750 --> 00:22:07,539 Right. The primary benefit here is entirely mathematical. 487 00:22:08,099 --> 00:22:10,859 By reframing a regression problem, predicting 488 00:22:10,859 --> 00:22:14,200 a continuous number into a classification problem, 489 00:22:14,380 --> 00:22:16,500 predicting the probability of discrete bins, 490 00:22:17,019 --> 00:22:20,160 the loss calculation changes completely. How 491 00:22:20,160 --> 00:22:23,210 so? The cross -entropy gradient with respect 492 00:22:23,210 --> 00:22:25,630 to the network's outputs only depends on the 493 00:22:25,630 --> 00:22:28,250 probabilities assigned to the bins. It fully 494 00:22:28,250 --> 00:22:30,569 decouples the size of the gradient from how massive 495 00:22:30,569 --> 00:22:33,690 the original reward scale was. It abstracts the 496 00:22:33,690 --> 00:22:36,750 magnitude away into probabilities. Yes. That 497 00:22:36,750 --> 00:22:39,549 is incredibly clever. It sounds like a mathematical 498 00:22:39,549 --> 00:22:42,200 shield. Like, it guarantees the network can never 499 00:22:42,200 --> 00:22:43,819 crash no matter what environment you throw it 500 00:22:43,819 --> 00:22:46,460 in. It is highly effective engineering, but we 501 00:22:46,460 --> 00:22:48,220 should definitely manage our language there. 502 00:22:48,400 --> 00:22:51,059 It is not an absolute mathematical guarantee 503 00:22:51,059 --> 00:22:54,019 that every network update is bounded and perfectly 504 00:22:54,019 --> 00:22:56,680 safe. OK, fair enough. Neural network draining 505 00:22:56,680 --> 00:22:59,960 remains a complex, highly non -linear optimization 506 00:22:59,960 --> 00:23:03,450 problem. Things can still go wrong. But SimLog 507 00:23:03,450 --> 00:23:05,829 and 2hot targets dramatically stabilize learning 508 00:23:05,829 --> 00:23:08,809 across diverse domains. OK, so we've stabilized 509 00:23:08,809 --> 00:23:12,089 the reward predictions. But what about the actor? 510 00:23:12,829 --> 00:23:15,230 The actor's job is to choose actions, but it 511 00:23:15,230 --> 00:23:18,869 also has to explore. It uses an entropy regularizer 512 00:23:18,869 --> 00:23:20,890 to force itself to try new things and not just 513 00:23:20,890 --> 00:23:23,920 get stuck in a rut. Right, exploration versus 514 00:23:23,920 --> 00:23:26,579 exploitation. But if the rewards are huge in 515 00:23:26,579 --> 00:23:29,140 one game and tiny in another, how does the actor 516 00:23:29,140 --> 00:23:32,440 know how much to prioritize exploring versus 517 00:23:32,440 --> 00:23:35,279 just chasing the known rewards? You've hit on 518 00:23:35,279 --> 00:23:38,079 another critical scaling problem. If a game gives 519 00:23:38,079 --> 00:23:41,119 you 10 ,000 points for a good action, the network 520 00:23:41,119 --> 00:23:43,279 will want to chase that immediately and stop 521 00:23:43,279 --> 00:23:46,460 exploring. And if a game gives you 0 .01 points? 522 00:23:46,630 --> 00:23:49,329 the entropy regularizer might overpower it, and 523 00:23:49,329 --> 00:23:51,230 the agent will just explore randomly forever. 524 00:23:51,869 --> 00:23:55,349 To use a fixed exploration scale across all 150 525 00:23:55,349 --> 00:23:58,049 tasks, you must normalize the returns, so they 526 00:23:58,049 --> 00:24:00,809 all sit on roughly the same scale. The Dreamer 527 00:24:00,809 --> 00:24:03,910 v3 paper accomplishes this using percentile return 528 00:24:03,910 --> 00:24:07,230 scaling. Okay, normalizing returns. Why not just 529 00:24:07,230 --> 00:24:09,589 look at the absolute maximum return the agent 530 00:24:09,589 --> 00:24:12,710 has ever seen, and the absolute minimum, and 531 00:24:12,710 --> 00:24:15,579 scale everything based on that gap? Because return 532 00:24:15,579 --> 00:24:17,640 distributions and reinforcement learning often 533 00:24:17,640 --> 00:24:20,880 have massive, rare outliers. Imagine an agent 534 00:24:20,880 --> 00:24:22,900 playing a game where it usually gets 10 points. 535 00:24:23,079 --> 00:24:25,880 But one time, purely by accident, it hits a glitch 536 00:24:25,880 --> 00:24:29,000 and gets a million points. A fluke. If you normalize 537 00:24:29,000 --> 00:24:31,559 the entire scale based on that single one million 538 00:24:31,559 --> 00:24:34,660 point outlier, all the normal 10 point rewards 539 00:24:34,660 --> 00:24:37,400 get scaled down to essentially zero. The agent 540 00:24:37,400 --> 00:24:39,140 will stop learning how to play the normal game. 541 00:24:39,670 --> 00:24:43,089 By using a smoothed 5th to 95th percentile range 542 00:24:43,089 --> 00:24:45,450 over a batch of data, the algorithm becomes robust 543 00:24:45,450 --> 00:24:47,970 to those extreme outliers. It ignores the crazy 544 00:24:47,970 --> 00:24:50,509 maximums and minimums. And does this percentile 545 00:24:50,509 --> 00:24:52,490 scaling just neatly clip every single reward 546 00:24:52,490 --> 00:24:56,269 perfectly into a clean 0 to 1 interval? It does 547 00:24:56,269 --> 00:24:58,390 not, and the mechanics of how it avoids doing 548 00:24:58,390 --> 00:25:00,829 that are vital for environments with sparse rewards. 549 00:25:01,650 --> 00:25:04,150 They calculate the range between the 5th and 550 00:25:04,150 --> 00:25:07,809 95th percentile, which they call S. They subtract 551 00:25:07,809 --> 00:25:10,890 an offset from the return and divide by s. But 552 00:25:10,890 --> 00:25:13,390 crucially, the denominator they actually use 553 00:25:13,390 --> 00:25:17,710 is the maximum of 1 and s. Okay. Max 1 comma 554 00:25:17,710 --> 00:25:20,869 s. Let's translate that into plain English for 555 00:25:20,869 --> 00:25:23,630 you listening. What does it actually do? It means 556 00:25:23,630 --> 00:25:26,430 that if the range of returns s is very large, 557 00:25:26,549 --> 00:25:29,950 say 10 ,000, the denominator is 10 ,000, and 558 00:25:29,950 --> 00:25:31,890 the large returns are scaled down to a manageable 559 00:25:31,890 --> 00:25:34,609 size. But if the range of returns is very small, 560 00:25:34,730 --> 00:25:37,349 less than 1, meaning the agent is barely getting 561 00:25:37,349 --> 00:25:39,910 any rewards at all, the denominator defaults 562 00:25:39,910 --> 00:25:43,450 to 1. Ah. So it divides by 1, it doesn't scale 563 00:25:43,450 --> 00:25:46,630 them up. Exactly. They scale down large return 564 00:25:46,630 --> 00:25:49,049 magnitudes, but they leave small returns untouched. 565 00:25:49,430 --> 00:25:51,430 If you have an environment where the agent wanders 566 00:25:51,430 --> 00:25:54,269 around for hours getting zero reward and occasionally 567 00:25:54,269 --> 00:25:58,349 gets 0 .001, the range is tiny. And if you divide 568 00:25:58,349 --> 00:26:00,880 it by that tiny range... You would amplify that 569 00:26:00,880 --> 00:26:03,559 tiny signal, which is often just noise or a very 570 00:26:03,559 --> 00:26:06,400 weak early signal, into a massive update. By 571 00:26:06,400 --> 00:26:09,799 using MAX 1 ,S, they prevent amplifying pure 572 00:26:09,799 --> 00:26:13,019 noise in sparse environments. That is brilliant. 573 00:26:13,960 --> 00:26:16,680 It tames the huge outliers, so they don't dominate 574 00:26:16,680 --> 00:26:19,660 the math, but it refuses to inflate tiny sparse 575 00:26:19,660 --> 00:26:22,680 signals into artificial earthquakes. Well said. 576 00:26:23,579 --> 00:26:25,740 There is one more major engineering trick mentioned 577 00:26:25,740 --> 00:26:27,990 in the paper. And this one is about training 578 00:26:27,990 --> 00:26:31,180 the dynamics of the world model itself. It's 579 00:26:31,180 --> 00:26:35,140 called KL balancing and free bits. Yes. This 580 00:26:35,140 --> 00:26:37,920 brings us back to the RSSM, the sequence model 581 00:26:37,920 --> 00:26:40,420 predicting the future. The model learns to predict 582 00:26:40,420 --> 00:26:43,039 the next discrete representation by minimizing 583 00:26:43,039 --> 00:26:46,980 the Kuhlbach -Libler or KL divergence. For those 584 00:26:46,980 --> 00:26:49,759 who aren't deep into probability, KL divergence 585 00:26:49,759 --> 00:26:52,779 is basically a ruler that measures how different 586 00:26:52,779 --> 00:26:55,619 two probability distributions are. If they are 587 00:26:55,619 --> 00:26:58,539 exactly the same, the KL divergence is zero. 588 00:26:59,119 --> 00:27:01,660 Correct. The model is trying to make its predicted 589 00:27:01,660 --> 00:27:04,160 distribution match the actual representation 590 00:27:04,160 --> 00:27:06,819 distribution produced by the encoder. At the 591 00:27:06,819 --> 00:27:09,539 same time, the encoder is being trained by a 592 00:27:09,539 --> 00:27:11,660 representation loss to produce distributions 593 00:27:11,660 --> 00:27:14,039 that are easier for the sequence model to predict. 594 00:27:14,420 --> 00:27:17,200 So they are dancy with each other. Exactly. They 595 00:27:17,200 --> 00:27:19,660 co -adapt. But if they are both adjusting to 596 00:27:19,660 --> 00:27:22,440 make the distance zero, what stops the encoder 597 00:27:22,440 --> 00:27:25,019 from just giving up and saying, hey, I'll just 598 00:27:25,019 --> 00:27:27,279 output the exact same blank representation for 599 00:27:27,279 --> 00:27:30,059 every single image I see. A blank wall. Then 600 00:27:30,059 --> 00:27:31,819 the sequence model can predict it perfectly, 601 00:27:32,099 --> 00:27:34,799 and the distance is zero. That's a degenerate 602 00:27:34,799 --> 00:27:37,480 solution. It achieves the mathematical goal, 603 00:27:37,619 --> 00:27:39,700 but it learns absolutely nothing about the game. 604 00:27:40,240 --> 00:27:43,019 That degenerate solution, often called representation 605 00:27:43,019 --> 00:27:46,279 collapse, is a massive problem. To avoid it, 606 00:27:46,420 --> 00:27:49,019 Dreamer v3 employs a technique called free bits. 607 00:27:49,319 --> 00:27:52,180 They clip the KL lost from below at a value of 608 00:27:52,180 --> 00:27:55,779 1 NAT. Okay, two things. First, what is a NAT? 609 00:27:56,299 --> 00:27:58,200 I've heard of bits in computing, but not a NAT. 610 00:27:58,400 --> 00:28:00,680 They're essentially the same concept of information 611 00:28:00,680 --> 00:28:03,059 theory, just using a different mathematical base. 612 00:28:03,599 --> 00:28:07,119 A bit uses base two logarithms, think binary, 613 00:28:07,480 --> 00:28:11,859 zeros and ones. A NAT uses base e logarithms, 614 00:28:11,940 --> 00:28:14,759 the natural logarithm. One NAT is equivalent 615 00:28:14,759 --> 00:28:18,519 to roughly 1 .44 bits of information. Got it. 616 00:28:18,900 --> 00:28:21,819 So they clip the loss from below at one NAT. 617 00:28:22,160 --> 00:28:24,640 What does clipping it from below actually achieve 618 00:28:24,640 --> 00:28:27,299 in preventing that degenerate blank wall? It 619 00:28:27,299 --> 00:28:29,700 removes the optimization pressure to reduce an 620 00:28:29,700 --> 00:28:32,559 already small KL divergence. Think of it like 621 00:28:32,559 --> 00:28:34,940 this. If the difference between the distributions 622 00:28:34,940 --> 00:28:38,339 is larger than 1 NAT, the network is penalized 623 00:28:38,339 --> 00:28:40,680 and tries to push them closer together. And if 624 00:28:40,680 --> 00:28:42,539 it gets smaller? Once the difference shrinks 625 00:28:42,539 --> 00:28:45,400 below 1 NAT, the loss function essentially turns 626 00:28:45,400 --> 00:28:48,700 off. The network says, OK, close enough. Stop 627 00:28:48,700 --> 00:28:51,680 worrying about this. Oh. So it stops the encoder 628 00:28:51,680 --> 00:28:54,259 from hyper -focusing on minimizing the difference 629 00:28:54,259 --> 00:28:57,240 down to absolute zero. It gives it a little bit 630 00:28:57,240 --> 00:28:59,960 of breathing room, free bits of information space. 631 00:29:00,440 --> 00:29:02,920 So it can focus its learning capacity on actually 632 00:29:02,920 --> 00:29:05,359 predicting the rewards and reconstructing the 633 00:29:05,359 --> 00:29:07,359 image rather than just mathematically collapsing. 634 00:29:07,980 --> 00:29:10,819 Exactly. It prevents the network from over -optimizing 635 00:29:10,819 --> 00:29:13,380 a secondary metric at the expense of its actual 636 00:29:13,380 --> 00:29:16,009 task. Does using Freebits guarantee that the 637 00:29:16,009 --> 00:29:17,869 model remembers all the important information 638 00:29:17,869 --> 00:29:20,549 on the screen? No, it does not. Freebits is a 639 00:29:20,549 --> 00:29:22,630 regularization technique that works incredibly 640 00:29:22,630 --> 00:29:25,210 well empirically to balance the learning dynamics 641 00:29:25,210 --> 00:29:28,130 across different visual complexities, from simple 642 00:29:28,130 --> 00:29:31,130 2D shapes to complex 3D rendering. Right. It 643 00:29:31,130 --> 00:29:33,750 prevents collapse, but it does not impose a minimum 644 00:29:33,750 --> 00:29:36,009 amount of useful memory, nor does it guarantee 645 00:29:36,009 --> 00:29:38,390 that essential information is retained. If an 646 00:29:38,390 --> 00:29:40,890 object on screen is too small or seemingly irrelevant 647 00:29:40,890 --> 00:29:43,130 to the reward, the model might still forget it. 648 00:29:43,250 --> 00:29:46,150 It is staggering to step back and look at the 649 00:29:46,150 --> 00:29:48,309 sheer level of careful mathematical balancing 650 00:29:48,309 --> 00:29:52,349 required to make a single algorithm robust. We 651 00:29:52,349 --> 00:29:55,890 have SimLog squashing massive numbers, too -hot 652 00:29:55,890 --> 00:29:58,569 encoding turning continuous values into probabilities, 653 00:29:59,490 --> 00:30:01,789 percentile scaling normalizing rewards without 654 00:30:01,789 --> 00:30:04,569 inflating noise, and free bits preventing the 655 00:30:04,569 --> 00:30:06,710 whole representation from collapsing into a black 656 00:30:06,710 --> 00:30:09,750 hole. It's an engineering marvel. We have built 657 00:30:09,750 --> 00:30:12,619 the engine, We've explored the shocks, the stabilizers, 658 00:30:12,700 --> 00:30:15,759 the firewall. Now let's see this vehicle run 659 00:30:15,759 --> 00:30:19,059 on the track. This brings us to the crucible 660 00:30:19,059 --> 00:30:22,619 of the paper, the 150 task benchmark and the 661 00:30:22,619 --> 00:30:25,819 ultimate challenge, Minecraft. The empirical 662 00:30:25,819 --> 00:30:28,279 results in this paper are truly a milestone for 663 00:30:28,279 --> 00:30:31,400 the field. The paper evaluates Dreamer V3 across 664 00:30:31,400 --> 00:30:34,579 eight distinct domains, encompassing over 150 665 00:30:34,579 --> 00:30:37,420 tasks. We were talking about The classic suite 666 00:30:37,420 --> 00:30:40,539 of Atari 2600 games, continuous robot control 667 00:30:40,539 --> 00:30:43,539 tasks in the DeepMind control suite, 3D mazes 668 00:30:43,539 --> 00:30:46,700 and DM lab, procedural generation levels. It's 669 00:30:46,700 --> 00:30:49,619 a massive gauntlet of tests. It really is. But 670 00:30:49,619 --> 00:30:52,180 I want to pause here and be crystal clear for 671 00:30:52,180 --> 00:30:54,279 you, the listener, because it's very easy for 672 00:30:54,279 --> 00:30:57,319 AI news to get hyped out of proportion. We are 673 00:30:57,319 --> 00:31:00,980 not talking about one single godlike AI agent 674 00:31:00,980 --> 00:31:03,750 that was dropped into a supercomputer. boot it 675 00:31:03,750 --> 00:31:07,130 up all 150 games at the exact same time and learn 676 00:31:07,130 --> 00:31:10,049 them all simultaneously with one brain. I am 677 00:31:10,049 --> 00:31:12,410 very glad you emphasize that distinction because 678 00:31:12,410 --> 00:31:15,049 it is vital for understanding what was actually 679 00:31:15,049 --> 00:31:18,750 achieved. The 150 tasks use the same algorithm 680 00:31:18,750 --> 00:31:21,450 configuration. The same settings. Right. That 681 00:31:21,450 --> 00:31:24,710 means the researchers use the exact same hyperparameters, 682 00:31:24,970 --> 00:31:27,130 the same learning rates, the same simlog setup, 683 00:31:27,289 --> 00:31:29,869 the same one NAT free bits threshold for every 684 00:31:29,869 --> 00:31:33,009 game. However, The agents were trained separately 685 00:31:33,009 --> 00:31:36,549 on each task from scratch. You have an instance 686 00:31:36,549 --> 00:31:39,029 of Dreamer playing pong and a completely separate 687 00:31:39,029 --> 00:31:41,190 instance of Dreamer learning to make a simulated 688 00:31:41,190 --> 00:31:44,799 robot walk. And the absolute peak of this testing, 689 00:31:45,400 --> 00:31:47,700 the environment that broke so many previous algorithms, 690 00:31:48,299 --> 00:31:51,119 was Minecraft. Minecraft has become a notorious 691 00:31:51,119 --> 00:31:53,519 focal point for AI research over the last few 692 00:31:53,519 --> 00:31:55,859 years. I find this fascinating because human 693 00:31:55,859 --> 00:31:59,099 children play Minecraft effortlessly. A seven 694 00:31:59,099 --> 00:32:00,740 -year -old can figure it out in an afternoon. 695 00:32:00,930 --> 00:32:03,869 Why does a game made of simple, low -resolution 696 00:32:03,869 --> 00:32:07,170 blocks completely break state -of -the -art artificial 697 00:32:07,170 --> 00:32:10,250 intelligence? It breaks traditional RL algorithms 698 00:32:10,250 --> 00:32:12,750 because of the fundamental structure of the environment. 699 00:32:13,190 --> 00:32:15,730 First, it has incredibly sparse rewards. In a 700 00:32:15,730 --> 00:32:17,450 game like Pac -Man, you get a point for every 701 00:32:17,450 --> 00:32:19,829 dot you eat. The feedback is constant. Right. 702 00:32:20,029 --> 00:32:22,730 Eat dot, get point. In Minecraft, you get nothing 703 00:32:22,730 --> 00:32:26,069 for wandering around. Second, it is an open world 704 00:32:26,069 --> 00:32:28,269 with procedural generation, meaning the map is 705 00:32:28,269 --> 00:32:30,130 completely different every time you start a new 706 00:32:30,130 --> 00:32:33,049 episode. You can't memorize a path to a resource. 707 00:32:33,470 --> 00:32:36,289 Right. The map memorizer fails instantly. And 708 00:32:36,289 --> 00:32:38,849 third, it requires extremely long time horizons 709 00:32:38,849 --> 00:32:41,490 with deep, rigid dependencies. Think about the 710 00:32:41,490 --> 00:32:43,630 tech tree required to get a diamond in Minecraft. 711 00:32:44,250 --> 00:32:46,549 Oh man, it's a long process. You spawn in a random 712 00:32:46,549 --> 00:32:48,609 world. You have to find a tree. You have to punch 713 00:32:48,609 --> 00:32:50,650 the tree to get wood blocks. You have to open 714 00:32:50,650 --> 00:32:52,869 a crafting menu and turn the wood blocks into 715 00:32:52,869 --> 00:32:55,130 planks, then turn the planks into sticks. You 716 00:32:55,130 --> 00:32:57,049 have to combine sticks and planks to make a wooden 717 00:32:57,049 --> 00:32:59,910 pickaxe. Then you go underground. Then you have 718 00:32:59,910 --> 00:33:02,269 to find stone and mine it. Then upgrade to a 719 00:33:02,269 --> 00:33:04,690 stone pickaxe. Then find coal and iron, build 720 00:33:04,690 --> 00:33:07,490 a furnace, smelt the iron, make an iron pickaxe. 721 00:33:07,630 --> 00:33:09,410 Then you have to dig deep underground, navigating 722 00:33:09,410 --> 00:33:12,150 lava and monsters to finally mine a diamond block. 723 00:33:12,490 --> 00:33:15,509 It's a massive sequence of specific dependent 724 00:33:15,509 --> 00:33:19,079 actions. For an AI agent that starts by exploring 725 00:33:19,079 --> 00:33:22,700 randomly, just pushing random buttons, the mathematical 726 00:33:22,700 --> 00:33:25,259 probability of accidentally stumbling through 727 00:33:25,259 --> 00:33:28,480 that exact sequence of dozens of specific actions 728 00:33:28,480 --> 00:33:31,759 to find a diamond is astronomically low. It is 729 00:33:31,759 --> 00:33:34,259 virtually impossible to learn purely by random 730 00:33:34,259 --> 00:33:36,779 chance. Which is why previous AI systems that 731 00:33:36,779 --> 00:33:39,039 solved the Minecraft diamond challenge almost 732 00:33:39,039 --> 00:33:41,779 always needed massive human help, right? They 733 00:33:41,779 --> 00:33:44,329 used imitation learning. They had to watch thousands 734 00:33:44,329 --> 00:33:47,509 of hours of YouTube videos of human players playing 735 00:33:47,509 --> 00:33:50,450 Minecraft to learn the sequence of actions. Exactly. 736 00:33:51,130 --> 00:33:55,410 But Dreamer V3 did it from scratch. Applied completely 737 00:33:55,410 --> 00:33:57,990 out of the box with its fixed hyperparameters, 738 00:33:58,150 --> 00:34:00,890 Dreamer V3 is the first algorithm to successfully 739 00:34:00,890 --> 00:34:03,650 collect diamonds in Minecraft from scratch. It 740 00:34:03,650 --> 00:34:07,529 used no human demonstrations and it used no handcrafted 741 00:34:07,529 --> 00:34:09,429 adaptive curriculum to guide its learning step 742 00:34:09,429 --> 00:34:13,139 by step. That is mind blowing. But... As always, 743 00:34:13,260 --> 00:34:15,079 the devil is in the details of the experiment. 744 00:34:15,659 --> 00:34:17,400 We need to look at the specific environmental 745 00:34:17,400 --> 00:34:19,719 conditions used in the paper. Because they didn't 746 00:34:19,719 --> 00:34:22,280 just boot up standard retail Minecraft on a PlayStation 747 00:34:22,280 --> 00:34:25,380 and drop the AI in. No, they used a specifically 748 00:34:25,380 --> 00:34:27,639 adapted environment designed for AI research. 749 00:34:28,039 --> 00:34:30,159 This context is essential to understand the bounds 750 00:34:30,159 --> 00:34:32,420 of the achievement. The environment features 751 00:34:32,420 --> 00:34:35,340 a categorical action space rather than continuous 752 00:34:35,340 --> 00:34:37,599 analog mouse movements for looking around. OK, 753 00:34:37,980 --> 00:34:41,000 so simplified controls. It uses simplified, instantaneous 754 00:34:41,000 --> 00:34:43,219 crafting actions rather than manually placing 755 00:34:43,219 --> 00:34:46,280 items in a grid. It features faster block breaking 756 00:34:46,280 --> 00:34:49,400 than the standard game. And crucially, the researchers 757 00:34:49,400 --> 00:34:52,639 provided the agent with 12 rewarded milestones. 758 00:34:53,119 --> 00:34:55,739 12 rewarded milestones. So it doesn't just wander 759 00:34:55,739 --> 00:34:57,760 in a void for 10 hours and then suddenly get 760 00:34:57,760 --> 00:34:59,519 points for a diamond. It gets little breadcrumbs 761 00:34:59,519 --> 00:35:02,380 along the tech tree. How are those rewards structured? 762 00:35:02,639 --> 00:35:04,699 Does it get like one point for wood and then 763 00:35:04,699 --> 00:35:06,800 a massive jackpot of a million points when it 764 00:35:06,800 --> 00:35:09,300 finally hits the diamond? No. And this is a critical 765 00:35:09,300 --> 00:35:11,699 detail about the reward structure. Each of the 766 00:35:11,699 --> 00:35:14,960 12 milestones earns a reward of plus one on its 767 00:35:14,960 --> 00:35:17,219 first collection per episode. That's it. Wait. 768 00:35:17,699 --> 00:35:20,559 It gets plus one for punching a tree? And just 769 00:35:20,559 --> 00:35:23,159 plus one for finding the diamond? Yes. The diamond 770 00:35:23,159 --> 00:35:25,760 does not receive a uniquely massive reward relative 771 00:35:25,760 --> 00:35:28,460 to collecting wood, stone, or iron. It is just 772 00:35:28,460 --> 00:35:31,659 another plus one. The immense difficulty of the 773 00:35:31,659 --> 00:35:33,940 task isn't in following a massive gradient of 774 00:35:33,940 --> 00:35:36,639 points. The difficulty is purely in the long 775 00:35:36,639 --> 00:35:39,440 time horizon. and the complex dependencies required 776 00:35:39,440 --> 00:35:42,380 to reach that particular plus one deep in the 777 00:35:42,380 --> 00:35:44,760 tech tree. That honestly makes the achievement 778 00:35:44,760 --> 00:35:48,300 even more impressive. It wasn't gravitationally 779 00:35:48,300 --> 00:35:51,239 pulled toward the diamond by some massive game 780 00:35:51,239 --> 00:35:53,519 -breaking reward spike that overshadowed everything 781 00:35:53,519 --> 00:35:56,800 else. It just explored the tech tree naturally, 782 00:35:57,300 --> 00:35:59,900 using its world model to predict how items combine, 783 00:36:00,360 --> 00:36:02,619 and eventually found its way down to the diamond 784 00:36:02,619 --> 00:36:05,679 layer. And how much computing power did this 785 00:36:05,679 --> 00:36:08,480 take? Because usually, These breakthroughs require 786 00:36:08,480 --> 00:36:10,679 data centers that consume the power output of 787 00:36:10,679 --> 00:36:13,280 a small city. The published comparison reports 788 00:36:13,280 --> 00:36:16,199 training Grimmer V3 on the Minecraft task using 789 00:36:16,199 --> 00:36:20,239 a single NVIDIA A100 GPU for nine wall clock 790 00:36:20,239 --> 00:36:24,480 days. Nine days on a single graphics card. Granted, 791 00:36:24,880 --> 00:36:28,239 A100 is a very powerful, expensive enterprise 792 00:36:28,239 --> 00:36:31,480 grade graphics card, but it's a single unit that 793 00:36:31,480 --> 00:36:34,639 is accessible to hundreds of university research 794 00:36:34,639 --> 00:36:37,849 labs. not requiring a billion dollar super computer 795 00:36:37,849 --> 00:36:41,070 cluster from a massive tech conglomerate. It's 796 00:36:41,070 --> 00:36:43,409 surprisingly accessible. But we should clarify 797 00:36:43,409 --> 00:36:46,969 the time scale here. Nine days of wall clock 798 00:36:46,969 --> 00:36:49,750 time, meaning time passing for you and me in 799 00:36:49,750 --> 00:36:52,690 the real world, is not the same as the simulated 800 00:36:52,690 --> 00:36:55,889 play time the agent experienced, right? Correct. 801 00:36:56,170 --> 00:36:59,150 Simulations run much faster than real time. Within 802 00:36:59,150 --> 00:37:01,570 those nine wall clock days, the agent processed 803 00:37:01,570 --> 00:37:04,550 a hundred million environment steps. It experienced 804 00:37:04,550 --> 00:37:08,230 many, many lifetimes of simulated episodes. Wall 805 00:37:08,230 --> 00:37:10,909 clock training time, environment steps, and simulated 806 00:37:10,909 --> 00:37:13,969 play time are distinct metrics. But 100 million 807 00:37:13,969 --> 00:37:16,590 steps on a single GPU is highly efficient for 808 00:37:16,590 --> 00:37:19,789 this level of complex mastery. Drewer V3 is mastering 809 00:37:19,789 --> 00:37:23,570 simulations. It crushed Atari. It dominated DM 810 00:37:23,570 --> 00:37:26,449 Lab. It conquered Minecraft. But these are all 811 00:37:26,449 --> 00:37:29,659 fundamentally lines of code. What about the messy, 812 00:37:29,960 --> 00:37:33,460 chaotic, physical real world? The hardest environment 813 00:37:33,460 --> 00:37:36,420 of all. This transitions us perfectly to the 814 00:37:36,420 --> 00:37:39,880 second paper, V. J. Patu, and the notorious problem 815 00:37:39,880 --> 00:37:43,239 known in robotics is sim to real. The simulation 816 00:37:43,239 --> 00:37:45,480 gap, transferring a policy learned in a computer 817 00:37:45,480 --> 00:37:48,400 into a physical robot, is one of the most stubborn 818 00:37:48,400 --> 00:37:50,719 and expensive problems in the entire field of 819 00:37:50,719 --> 00:37:53,320 robotics. My intuition here is that video games, 820 00:37:53,480 --> 00:37:56,800 even complex ones, are perfectly clean. The math 821 00:37:56,800 --> 00:37:59,079 is absolute. If you press a button in Minecraft, 822 00:37:59,320 --> 00:38:01,539 the block breaks exactly the same way every single 823 00:38:01,539 --> 00:38:05,519 time. But the real world has wind. It has sticky 824 00:38:05,519 --> 00:38:08,500 gears. It has unpredictable friction on a tabletop. 825 00:38:08,900 --> 00:38:11,179 It has a camera lens that gets smudged. The real 826 00:38:11,179 --> 00:38:14,219 world is noisy. Well, that's true. I want to 827 00:38:14,219 --> 00:38:16,440 clarify a common misconception about the sim 828 00:38:16,440 --> 00:38:19,440 -to -real gap. It is a misconception that simulations 829 00:38:19,440 --> 00:38:22,000 are inherently perfectly clean or simplistic. 830 00:38:22,539 --> 00:38:24,400 Modern physics simulators can actually include 831 00:38:24,400 --> 00:38:27,639 incredibly complex fluid dynamics for wind, stochastic 832 00:38:27,639 --> 00:38:30,619 noise for sticky gears, dynamic friction profiles, 833 00:38:30,739 --> 00:38:32,960 and highly complex rendering effects for camera 834 00:38:32,960 --> 00:38:35,760 glare. We should not dismiss simulation as inherently 835 00:38:35,760 --> 00:38:38,239 lacking these features. Oh, that's interesting. 836 00:38:38,739 --> 00:38:41,260 So if the physics engines are actually really 837 00:38:41,260 --> 00:38:44,579 good, and can simulate wind and friction. Why 838 00:38:44,579 --> 00:38:46,800 do robots that train for millions of hours in 839 00:38:46,800 --> 00:38:49,340 simulation still fall over the moment you put 840 00:38:49,340 --> 00:38:51,400 them in a real physical room? What is the actual 841 00:38:51,400 --> 00:38:54,019 barrier? The core issues are model mismatch, 842 00:38:54,400 --> 00:38:56,579 distribution mismatch, and the massive costs 843 00:38:56,579 --> 00:38:59,579 associated with physical reality. No matter how 844 00:38:59,579 --> 00:39:02,159 advanced your physics engine is, it remains an 845 00:39:02,159 --> 00:39:04,420 approximation of the universe. It's still just 846 00:39:04,420 --> 00:39:07,400 a model. The distribution of data in the simulation 847 00:39:07,400 --> 00:39:09,679 will always have subtle compounding mismatches 848 00:39:09,679 --> 00:39:12,239 with the real world. A physical gear in a robot 849 00:39:12,239 --> 00:39:14,099 arm might wear down slightly over the course 850 00:39:14,099 --> 00:39:16,760 of three days, dynamically changing its friction 851 00:39:16,760 --> 00:39:18,940 profile in a way the static simulation didn't 852 00:39:18,940 --> 00:39:21,260 capture. The lighting in a room changes when 853 00:39:21,260 --> 00:39:23,820 a cloud passes over the sun. And a neural network 854 00:39:23,820 --> 00:39:25,679 that is overfit to the exact lighting of the 855 00:39:25,679 --> 00:39:28,519 simulation will suddenly get confused. Yes. And 856 00:39:28,519 --> 00:39:30,380 perhaps even more importantly in simulation, 857 00:39:30,599 --> 00:39:32,900 a virtual robot can attempt a task a million 858 00:39:32,900 --> 00:39:35,940 times, fall over, a million times, and you just 859 00:39:35,940 --> 00:39:38,119 hit a button to reset the environment in a fraction 860 00:39:38,119 --> 00:39:39,960 of a second. You can't do that in a physical 861 00:39:39,960 --> 00:39:42,860 lab, which is where the entirely different paradigm 862 00:39:42,860 --> 00:39:46,199 of VJEPA 2 comes in. They took a radically different 863 00:39:46,199 --> 00:39:48,920 approach to building a world model for real physical 864 00:39:48,920 --> 00:39:52,000 robots. Let's detail the VJEPA 2 architecture 865 00:39:52,000 --> 00:39:55,659 and its training stages. The VJEPA stands for 866 00:39:55,659 --> 00:39:58,260 Video Joint Embedding Predictive Architecture. 867 00:39:58,699 --> 00:40:01,079 It uses a two -stage approach that leans heavily 868 00:40:01,079 --> 00:40:04,530 on massive offline self -supervised learning 869 00:40:04,530 --> 00:40:07,750 rather than online interaction. Stage one? Stage 870 00:40:07,750 --> 00:40:10,389 one is video and image pre -training. They took 871 00:40:10,389 --> 00:40:13,050 the model and trained it on a data set containing 872 00:40:13,050 --> 00:40:16,329 over one million hours of uncurated internet 873 00:40:16,329 --> 00:40:20,090 video. One million hours. That is over a century 874 00:40:20,090 --> 00:40:22,909 of continuous non -stop video. It is a staggering 875 00:40:22,909 --> 00:40:25,369 amount of data. And crucially, this stage is 876 00:40:25,369 --> 00:40:27,650 completely unsupervised. There are no reward 877 00:40:27,650 --> 00:40:30,789 signals, no task labels, and no actions. The 878 00:40:30,789 --> 00:40:33,289 model is simply watching videos, people cooking, 879 00:40:33,550 --> 00:40:36,809 cars driving, nature scenes, sports, and learning 880 00:40:36,809 --> 00:40:39,449 to predict missing parts of the video in a latent 881 00:40:39,449 --> 00:40:42,110 representation space. So it's not trying to solve 882 00:40:42,110 --> 00:40:45,389 anything. No. It is passively building a massive, 883 00:40:45,789 --> 00:40:47,849 highly generalized internal understanding of 884 00:40:47,849 --> 00:40:50,610 visual physics. It learns how objects occlude 885 00:40:50,610 --> 00:40:53,269 each other, how gravity generally works, how 886 00:40:53,269 --> 00:40:56,489 light reflects. Okay, so it has this massive, 887 00:40:56,789 --> 00:40:59,090 century -deep visual foundation of how the world 888 00:40:59,090 --> 00:41:02,110 looks. But it doesn't know what a robot is. It 889 00:41:02,110 --> 00:41:04,269 doesn't know how to move a motor or complete 890 00:41:04,269 --> 00:41:08,130 a task. What is stage two? Stage two is the action 891 00:41:08,130 --> 00:41:11,250 condition post -training. They take this massive 892 00:41:11,250 --> 00:41:13,949 passively pre -trained visual model and fine 893 00:41:13,949 --> 00:41:16,829 -tune it using a remarkably small, highly specific 894 00:41:16,829 --> 00:41:20,409 data set. Less than 62 hours of droid robot videos. 895 00:41:20,809 --> 00:41:23,210 Droid being a specific open source data set of 896 00:41:23,210 --> 00:41:26,219 robot manipulation tasks, right? Yes. In this 897 00:41:26,219 --> 00:41:28,679 stage, they train the model, which they now call 898 00:41:28,679 --> 00:41:33,099 VJEPA2AC for action conditioned, to predict future 899 00:41:33,099 --> 00:41:35,420 video states based on the current visual state 900 00:41:35,420 --> 00:41:38,519 and a specific sequence of robot actions. Ah, 901 00:41:38,760 --> 00:41:41,239 now the actions come in. Because the underlying 902 00:41:41,239 --> 00:41:43,480 model already understands the complex visual 903 00:41:43,480 --> 00:41:45,539 physics of the world from that million hours 904 00:41:45,539 --> 00:41:48,440 of internet video, it is incredibly data efficient 905 00:41:48,440 --> 00:41:51,659 in the second stage. It only needs 62 hours of 906 00:41:51,659 --> 00:41:54,380 specific robot data to map its general physics 907 00:41:54,380 --> 00:41:57,159 knowledge to the specific kinematic actions of 908 00:41:57,159 --> 00:42:00,659 the robot arm. That is wildly efficient compared 909 00:42:00,659 --> 00:42:03,559 to RL algorithms that need hundreds of millions 910 00:42:03,559 --> 00:42:06,960 of steps. But this raises a major question for 911 00:42:06,960 --> 00:42:11,420 me. If VJEPA 2 isn't an RL algorithm, if there 912 00:42:11,420 --> 00:42:14,139 is no actor network learning a policy in a dream, 913 00:42:14,480 --> 00:42:17,400 no critic network judging the states, how does 914 00:42:17,400 --> 00:42:20,239 the robot actually do anything? A great question. 915 00:42:20,480 --> 00:42:22,400 How does it plan a movement to pick up a cup 916 00:42:22,400 --> 00:42:25,079 if it doesn't have an actor policy? It uses a 917 00:42:25,079 --> 00:42:27,340 technique called model predictive control, or 918 00:42:27,340 --> 00:42:30,599 MPC, utilizing a specific algorithm called the 919 00:42:30,599 --> 00:42:33,400 cross entropy method, or CEM. OK, let's break 920 00:42:33,400 --> 00:42:37,489 that down. Model predictive control using the 921 00:42:37,489 --> 00:42:40,230 cross entropy method. Let's make this tangible. 922 00:42:41,340 --> 00:42:43,860 I put the robot in a physical room. I give it 923 00:42:43,860 --> 00:42:46,539 an image of a goal, say a picture of a red block 924 00:42:46,539 --> 00:42:49,059 sitting inside a blue bowl. How does the robot 925 00:42:49,059 --> 00:42:52,099 plan the motor movements to make reality match 926 00:42:52,099 --> 00:42:54,980 that picture? The VJEPA2 model uses its action 927 00:42:54,980 --> 00:42:57,460 conditioned world model to imagine various sequences 928 00:42:57,460 --> 00:43:00,380 of actions. It generates thousands of random 929 00:43:00,380 --> 00:43:03,639 potential action sequences. Move left, move down, 930 00:43:03,820 --> 00:43:06,719 grip, move up. in its latent space. Exactly. 931 00:43:07,039 --> 00:43:09,360 It rolls those actions forward through its internal 932 00:43:09,360 --> 00:43:12,340 world model to predict what the future visual 933 00:43:12,340 --> 00:43:14,639 state will look like in its latent representation 934 00:43:14,639 --> 00:43:17,519 space for each sequence. It imagines thousands 935 00:43:17,519 --> 00:43:20,619 of different futures. Yes. Then it compares those 936 00:43:20,619 --> 00:43:22,739 imagined futures to the goal representation. 937 00:43:23,400 --> 00:43:26,219 It mathematically minimizes the L1 distance between 938 00:43:26,219 --> 00:43:28,679 its imagined future state and the target goal 939 00:43:28,679 --> 00:43:30,920 state. So it's literally trying to match the 940 00:43:30,920 --> 00:43:32,840 picture in its head to the picture you gave it. 941 00:43:33,079 --> 00:43:36,500 It sorts through the thousands of imagined futures 942 00:43:36,500 --> 00:43:38,880 and finds the one where the visual representation 943 00:43:38,880 --> 00:43:42,000 is mathematically closest to the goal. Exactly. 944 00:43:42,420 --> 00:43:44,440 But it doesn't just guess once. That's where 945 00:43:44,440 --> 00:43:47,059 the cross entropy method comes in. Okay, what 946 00:43:47,059 --> 00:43:50,820 does CEM do? CEM is an iterative optimization 947 00:43:50,820 --> 00:43:53,780 process. It samples a large batch of random action 948 00:43:53,780 --> 00:43:56,500 sequences. It evaluates all of them using the 949 00:43:56,500 --> 00:43:59,260 world model and takes the top percentage, the 950 00:43:59,260 --> 00:44:02,469 best performing, closest action sequences. the 951 00:44:02,469 --> 00:44:04,349 ones that almost put the block in the bowl. Right. 952 00:44:04,769 --> 00:44:07,250 It then fits a new probability distribution over 953 00:44:07,250 --> 00:44:09,889 those successful actions and samples a new batch 954 00:44:09,889 --> 00:44:12,269 of sequences from that refined distribution. 955 00:44:12,570 --> 00:44:15,489 It repeats this process over and over, rapidly 956 00:44:15,489 --> 00:44:17,889 narrowing down on a highly accurate sequence 957 00:44:17,889 --> 00:44:19,909 of actions that gets it to the goal. And then 958 00:44:19,909 --> 00:44:22,530 it executes it? It executes the first action 959 00:44:22,530 --> 00:44:25,309 of the best sequence in the real world. Then 960 00:44:25,309 --> 00:44:27,949 it observes the new physical state and replans 961 00:44:27,949 --> 00:44:30,110 the entire process from the new starting point. 962 00:44:30,360 --> 00:44:33,179 This allows it to correct for real -world noise 963 00:44:33,179 --> 00:44:35,900 or slight errors in its model on the fly. That 964 00:44:35,900 --> 00:44:39,420 is brilliant. The paper talks about this enabling 965 00:44:39,420 --> 00:44:42,300 zero -shot transfer to new environments. I really 966 00:44:42,300 --> 00:44:45,000 want to drill down on this because zero -shot 967 00:44:45,000 --> 00:44:47,099 is one of those AI buzzwords that gets thrown 968 00:44:47,099 --> 00:44:49,159 around in press releases to sound like magic. 969 00:44:49,760 --> 00:44:52,659 It absolutely does. Does this mean they unboxed 970 00:44:52,659 --> 00:44:55,840 a brand new robot arm? turned it on in a factory 971 00:44:55,840 --> 00:44:58,699 it had never seen, gave it zero prior experience 972 00:44:58,699 --> 00:45:01,219 of any kind, and it just flawlessly started sorting 973 00:45:01,219 --> 00:45:05,429 widgets? Definitely not. And we must define zero 974 00:45:05,429 --> 00:45:08,809 -shot precisely in this specific context. In 975 00:45:08,809 --> 00:45:12,389 the VJEPA2 paper, zero -shot refers only to the 976 00:45:12,389 --> 00:45:16,349 tested new labs and tasks. Meaning the environment 977 00:45:16,349 --> 00:45:20,010 was new, not the robot's brain. Right. The researchers 978 00:45:20,010 --> 00:45:22,849 placed the robot in visually novel physical environments, 979 00:45:23,230 --> 00:45:25,829 different rooms, different lighting, with novel 980 00:45:25,829 --> 00:45:28,809 objects it hadn't manipulated before and asked 981 00:45:28,809 --> 00:45:31,489 it to perform tasks it hadn't explicitly been 982 00:45:31,489 --> 00:45:34,030 fine -tuned to do in those specific settings. 983 00:45:34,610 --> 00:45:37,469 So it generalized to the new room, but it did 984 00:45:37,469 --> 00:45:41,030 have prior experience. Extensive prior experience. 985 00:45:41,570 --> 00:45:44,210 It does not mean the robot had zero prior knowledge. 986 00:45:44,710 --> 00:45:47,409 It was built on the foundation of 1 million hours 987 00:45:47,409 --> 00:45:50,750 of video, plus the 62 hours of action -conditioned 988 00:45:50,750 --> 00:45:53,769 droid robot data. It had plenty of experience 989 00:45:53,769 --> 00:45:56,590 regarding how its own mechanical arms work, how 990 00:45:56,590 --> 00:45:58,909 joints move, and how physical objects generally 991 00:45:58,909 --> 00:46:01,750 behave. So what does zero shot actually mean 992 00:46:01,750 --> 00:46:04,710 here? Zero shot here simply means it generalized 993 00:46:04,710 --> 00:46:07,510 its planning capabilities to a new desk, a new 994 00:46:07,510 --> 00:46:09,869 cup, or a new background without needing a human 995 00:46:09,869 --> 00:46:12,289 to collect data and retrain it specifically on 996 00:46:12,289 --> 00:46:14,820 that new desk. So it's a bounded success, but 997 00:46:14,820 --> 00:46:17,000 still an incredibly impressive one. Furthermore, 998 00:46:17,400 --> 00:46:20,139 we should note that this does not establish general 999 00:46:20,139 --> 00:46:23,059 universal robot competence or absolute safety. 1000 00:46:23,500 --> 00:46:26,440 The paper explicitly reports bounded experimental 1001 00:46:26,440 --> 00:46:29,820 findings. There are limits. Yes, there are limitations. 1002 00:46:30,019 --> 00:46:32,780 For example, if you move the camera too far outside 1003 00:46:32,780 --> 00:46:34,940 the distribution of angles it saw during training, 1004 00:46:35,340 --> 00:46:37,900 or if you change the coordinate frames too drastically, 1005 00:46:38,179 --> 00:46:41,420 the planning degrades. It is a massive step forward. 1006 00:46:41,750 --> 00:46:45,170 But it is not magic. It's a fascinating contrast 1007 00:46:45,170 --> 00:46:47,929 in paradigms when you synthesize what we've looked 1008 00:46:47,929 --> 00:46:51,110 at today. We have two completely different philosophies 1009 00:46:51,110 --> 00:46:53,789 for building a world model. Completely different. 1010 00:46:54,110 --> 00:46:57,010 On one hand, you have Dreamer V3. It learns its 1011 00:46:57,010 --> 00:46:59,650 world model entirely online. It gets dropped 1012 00:46:59,650 --> 00:47:03,210 into a game. It interacts. It fails. It dreams 1013 00:47:03,210 --> 00:47:06,130 about the game. It updates its actor critic policy 1014 00:47:06,130 --> 00:47:08,769 based on exploiting that dream. And it repeats 1015 00:47:08,769 --> 00:47:11,909 the cycle. It builds its understanding of reality 1016 00:47:11,909 --> 00:47:14,670 through active, ongoing trial and error in the 1017 00:47:14,670 --> 00:47:16,909 environment itself. Starting from scratch each 1018 00:47:16,909 --> 00:47:19,849 time, but adapting rapidly through active interaction. 1019 00:47:20,349 --> 00:47:23,369 And on the other hand, you have VJEPA 2 taking 1020 00:47:23,369 --> 00:47:26,789 the exact opposite route. It pre -trains a massive 1021 00:47:26,789 --> 00:47:30,349 generalized visual model offline, entirely passively, 1022 00:47:30,690 --> 00:47:32,949 by watching a century of internet video. Right, 1023 00:47:33,030 --> 00:47:35,630 the observer. Then it fine -tunes that model 1024 00:47:35,630 --> 00:47:38,869 with a tiny sliver of specific robot data and 1025 00:47:38,869 --> 00:47:41,170 uses the model directly for planning via model 1026 00:47:41,170 --> 00:47:44,329 predictive control. It doesn't use an actor or 1027 00:47:44,329 --> 00:47:47,590 a critic network at all. It just imagines futures 1028 00:47:47,590 --> 00:47:50,250 and chooses the one that matches the goal. It 1029 00:47:50,250 --> 00:47:53,090 is a remarkable divergence in methodology, but 1030 00:47:53,090 --> 00:47:55,230 it is essential to remind the listener that these 1031 00:47:55,230 --> 00:47:57,929 two papers use entirely different data budgets, 1032 00:47:58,449 --> 00:48:00,550 fundamentally different architectures, and exist 1033 00:48:00,550 --> 00:48:03,130 in different domains, simulation versus physical 1034 00:48:03,130 --> 00:48:05,650 robotics. So we aren't picking a winner. Exactly. 1035 00:48:06,010 --> 00:48:07,809 This is not a head -to -head competition where 1036 00:48:07,809 --> 00:48:10,250 one algorithm is objectively better than the 1037 00:48:10,250 --> 00:48:12,650 other. And that brings us to the end of our very 1038 00:48:12,650 --> 00:48:15,750 long journey today. Let's do a quick conceptual 1039 00:48:15,750 --> 00:48:19,000 recap of what we've unpacked. We started with 1040 00:48:19,000 --> 00:48:21,719 Kenneth Craig's cognitive theory, the idea that 1041 00:48:21,719 --> 00:48:24,139 true intelligence requires an internal representation 1042 00:48:24,139 --> 00:48:27,119 of reality to compress information and predict 1043 00:48:27,119 --> 00:48:30,059 the future. The foundation of it all. Then we 1044 00:48:30,059 --> 00:48:32,280 dove deep into the neural triumvirate of Dreamer 1045 00:48:32,280 --> 00:48:35,519 v3. We unpacked the hybrid latent state of the 1046 00:48:35,519 --> 00:48:39,199 RSSM, exploring how continuous memory and discrete 1047 00:48:39,199 --> 00:48:42,139 representations provide structure and how techniques 1048 00:48:42,139 --> 00:48:44,099 like the straight -through estimator allow the 1049 00:48:44,099 --> 00:48:47,179 math to bypass discrete cliffs. And we examine 1050 00:48:47,179 --> 00:48:49,599 the vital mathematical stabilizers that allow 1051 00:48:49,599 --> 00:48:52,820 the algorithm to generalize across 150 tasks 1052 00:48:52,820 --> 00:48:55,699 without human intervention. The SIM log transformations, 1053 00:48:55,920 --> 00:48:58,500 compressing massive reward spikes. The too -hot 1054 00:48:58,500 --> 00:49:01,199 encoding, decoupling gradient magnitudes by treating 1055 00:49:01,199 --> 00:49:03,920 continuous numbers like radio dials. Percentile 1056 00:49:03,920 --> 00:49:06,679 return scaling, carefully balancing exploration 1057 00:49:06,679 --> 00:49:10,730 without amplifying sparse noise. And KL balancing 1058 00:49:10,730 --> 00:49:13,309 with free bits to prevent the internal representation 1059 00:49:13,309 --> 00:49:15,960 from degenerating into a blank wall. We looked 1060 00:49:15,960 --> 00:49:18,539 at its bounded yet highly impressive success 1061 00:49:18,539 --> 00:49:22,019 as the first algorithm to conquer the rigid dependencies 1062 00:49:22,019 --> 00:49:25,760 of Minecraft from scratch on a single GPU relying 1063 00:49:25,760 --> 00:49:28,739 on fixed hyperparameters. That true milestone. 1064 00:49:29,280 --> 00:49:32,099 And then we crossed the simulation gap, noting 1065 00:49:32,099 --> 00:49:34,380 that the real problem isn't a lack of simulated 1066 00:49:34,380 --> 00:49:38,519 physics, but distribution mismatch and the prohibitive 1067 00:49:38,519 --> 00:49:41,599 cost of physical failure. This led us to Vjapa 1068 00:49:41,599 --> 00:49:45,179 2's alternative path. A path utilizing massive 1069 00:49:45,179 --> 00:49:48,099 offline self -supervised video pre -training, 1070 00:49:48,280 --> 00:49:50,659 followed by highly efficient action -conditioned 1071 00:49:50,659 --> 00:49:53,480 fine -tuning to achieve zero -shot generalization 1072 00:49:53,480 --> 00:49:56,599 in bounded physical robot control tasks via latent 1073 00:49:56,599 --> 00:49:58,820 space planning and the cross -entropy method. 1074 00:49:59,619 --> 00:50:02,400 We saw how the actor in Dreamer exploits flaws 1075 00:50:02,400 --> 00:50:05,679 in the model's imagination if an AI built a highly 1076 00:50:05,679 --> 00:50:08,739 efficient mathematically optimized but fundamentally 1077 00:50:08,739 --> 00:50:12,440 non -human imagination to solve complex tasks. 1078 00:50:12,769 --> 00:50:16,489 how will we ever truly audit its intuition? That 1079 00:50:16,489 --> 00:50:19,070 is the big question. Something to mull over while 1080 00:50:19,070 --> 00:50:21,250 you navigate your own internal world model on 1081 00:50:21,250 --> 00:50:23,809 your commute today. Keep diving deep, everyone. 1082 00:50:24,409 --> 00:50:25,050 Until next time.