1 00:00:00,000 --> 00:00:03,339 Imagine, for a second, trying to learn the fundamental 2 00:00:03,339 --> 00:00:06,740 laws of physics. But, you know, you have no teacher. 3 00:00:07,099 --> 00:00:09,220 Right. Just you and the world. Exactly. You have 4 00:00:09,220 --> 00:00:13,199 no textbook. And here is the real kicker. No 5 00:00:13,199 --> 00:00:15,779 one has even explicitly told you what an object 6 00:00:15,779 --> 00:00:18,800 actually is. Wow. Yeah. You don't even have a 7 00:00:18,800 --> 00:00:21,800 vocabulary for it. Right. You are just watching 8 00:00:21,800 --> 00:00:24,719 things move around in the world. And from that 9 00:00:24,719 --> 00:00:27,960 raw, unfiltered stream of visual data, you have 10 00:00:27,960 --> 00:00:30,679 to figure out how reality works. Well, enough 11 00:00:30,679 --> 00:00:33,020 to predict what happens next anyway. Exactly. 12 00:00:33,600 --> 00:00:35,659 And that massive challenge is at the heart of 13 00:00:35,659 --> 00:00:37,920 our deep dive today. It really is a staggering 14 00:00:37,920 --> 00:00:39,640 computational problem when you think about it. 15 00:00:39,659 --> 00:00:42,520 It is. We are dissecting a truly foundational 16 00:00:42,520 --> 00:00:45,000 piece of research that set out to tackle this 17 00:00:45,000 --> 00:00:47,880 exact problem for artificial agents. We have 18 00:00:47,880 --> 00:00:50,420 a stack of notes and the original text for a 19 00:00:50,420 --> 00:00:53,619 2016 paper titled, Unsupervised Learning for 20 00:00:53,619 --> 00:00:56,000 Physical Interaction Through Video Prediction. 21 00:00:56,299 --> 00:00:58,679 And we should mention who was behind this. Oh, 22 00:00:58,859 --> 00:01:01,600 absolutely. This comes from Chelsea Finn at UC 23 00:01:01,600 --> 00:01:04,540 Berkeley, Ian Goodfellow, representing OpenAI, 24 00:01:05,159 --> 00:01:07,560 and Sergey Levine from Google Brain and Berkeley. 25 00:01:08,299 --> 00:01:10,519 Yeah. The pedigree of those authors alone signals 26 00:01:10,519 --> 00:01:12,640 the density of this work. I mean, you have minds 27 00:01:12,640 --> 00:01:15,739 that fundamentally shaped deep learning architecture 28 00:01:15,739 --> 00:01:17,900 coming together here. To address a bottleneck 29 00:01:17,900 --> 00:01:20,579 that was basically suffocating the development 30 00:01:20,579 --> 00:01:23,969 of autonomous robotics. Exactly. Because if you 31 00:01:23,969 --> 00:01:26,829 want a robotic arm to operate in an unstructured 32 00:01:26,829 --> 00:01:29,629 environment, like a messy kitchen or a warehouse, 33 00:01:30,189 --> 00:01:32,510 it needs a working model of physics. It has to 34 00:01:32,510 --> 00:01:34,969 know how things move. So the mission of this 35 00:01:34,969 --> 00:01:38,090 deep dive is to unpack exactly how they achieve 36 00:01:38,090 --> 00:01:40,269 that. We are going to look at the neural network 37 00:01:40,269 --> 00:01:42,950 architecture they designed. A system meant to 38 00:01:42,950 --> 00:01:45,890 teach a machine to develop a visual imagination. 39 00:01:46,129 --> 00:01:49,680 Yes, visual imagination. The ability to predict 40 00:01:49,680 --> 00:01:52,280 the future outcomes of its own physical actions 41 00:01:52,280 --> 00:01:56,120 using entirely unlabeled raw video data. No humans 42 00:01:56,120 --> 00:01:58,620 pointing things out. Just a machine watching 43 00:01:58,620 --> 00:02:01,219 video and calculating the most probable future. 44 00:02:01,640 --> 00:02:04,480 I love that phrase, visual imagination. What 45 00:02:04,480 --> 00:02:06,260 does it actually mean in this context, though? 46 00:02:06,519 --> 00:02:09,259 Well, the term is highly specific here. It means 47 00:02:09,259 --> 00:02:11,860 the model isn't just a passive observer extrapolating 48 00:02:11,860 --> 00:02:14,039 a trajectory, you know. Right. It's not just 49 00:02:14,039 --> 00:02:16,080 watching a ball rolling, guessing where it stops. 50 00:02:16,180 --> 00:02:19,659 Exactly. It is an active, predictive model designed 51 00:02:19,659 --> 00:02:22,520 for planning. It is constructing potential futures 52 00:02:22,520 --> 00:02:24,780 conditioned on what it might choose to do. That 53 00:02:24,780 --> 00:02:27,759 is so cool. But to understand how they engineered 54 00:02:27,759 --> 00:02:30,400 that, we have to look at the exact wall the field 55 00:02:30,400 --> 00:02:33,379 was hitting prior to this paper. Yeah, the supervised 56 00:02:33,379 --> 00:02:35,259 learning bottleneck. That was the big issue. 57 00:02:35,439 --> 00:02:37,139 Right. Because if you go back to the standard 58 00:02:37,139 --> 00:02:39,240 methodology for teaching agents about physical 59 00:02:39,240 --> 00:02:42,500 interaction, it relied entirely on labeled data 60 00:02:42,500 --> 00:02:44,939 sets. Very tedious work. Right. If you wanted 61 00:02:44,939 --> 00:02:47,099 a robot to understand how a block moves when 62 00:02:47,099 --> 00:02:49,759 pushed, you had to feed it data where a human 63 00:02:49,759 --> 00:02:52,599 had meticulously provided the answer key. And 64 00:02:52,599 --> 00:02:55,159 that answer key is excruciating to generate. 65 00:02:55,219 --> 00:02:57,860 I mean, we are talking about ground truth object 66 00:02:57,860 --> 00:03:00,879 poses. Like the exact 3D coordinates of everything 67 00:03:00,879 --> 00:03:03,719 in the scene. Yes. Yeah. And we are also talking 68 00:03:03,719 --> 00:03:07,060 about segmentation masks. That is where a researcher 69 00:03:07,060 --> 00:03:10,840 has literally gone frame by frame drawing boundaries 70 00:03:10,840 --> 00:03:13,439 around objects. Oh, wow. Frame by frame. Yeah. 71 00:03:13,500 --> 00:03:17,259 explicitly defining for the computer, hey, these 72 00:03:17,259 --> 00:03:20,159 pixels belong to the block, and these pixels 73 00:03:20,159 --> 00:03:22,939 belong to the table. That sounds incredibly brittle. 74 00:03:23,479 --> 00:03:25,400 Like trying to teach a kid physics by forcing 75 00:03:25,400 --> 00:03:27,520 them to memorize flashcards of falling apples 76 00:03:27,520 --> 00:03:29,400 instead of just letting them play with blocks. 77 00:03:29,849 --> 00:03:32,590 That is a great analogy because if you train 78 00:03:32,590 --> 00:03:35,270 a model on a data set of wooden blocks with perfectly 79 00:03:35,270 --> 00:03:37,889 labeled poses, and then you put a plastic toy 80 00:03:37,889 --> 00:03:40,110 car in front of the robot. The model completely 81 00:03:40,110 --> 00:03:43,189 fails. It fails instantly. It has no pose data 82 00:03:43,189 --> 00:03:45,830 for the car. It doesn't even know the car is 83 00:03:45,830 --> 00:03:48,210 a discrete object because it hasn't been provided 84 00:03:48,210 --> 00:03:50,669 a segmentation mask for it. So the supervised 85 00:03:50,669 --> 00:03:52,849 approach simply does not scale to the entropy 86 00:03:52,849 --> 00:03:55,389 of the real world. You can't manually label every 87 00:03:55,389 --> 00:03:57,870 object in existence. Not in every possible lighting 88 00:03:57,870 --> 00:04:00,409 condition, no. So that limitation is exactly 89 00:04:00,409 --> 00:04:02,530 what drove this shift toward unsupervised learning 90 00:04:02,530 --> 00:04:04,750 from raw video. If we connect this to the bigger 91 00:04:04,750 --> 00:04:07,830 picture, if an agent can learn purely from the 92 00:04:07,830 --> 00:04:11,449 RGB pixel data streaming into its camera, it 93 00:04:11,449 --> 00:04:14,469 bypasses the manual labeling bottleneck entirely. 94 00:04:15,069 --> 00:04:17,110 It can just autonomously collect an infinite 95 00:04:17,110 --> 00:04:19,850 amount of experience. Just by moving its manipulators 96 00:04:19,850 --> 00:04:22,110 and recording the visual consequences. But the 97 00:04:22,110 --> 00:04:24,149 moment you transition to raw video, the math 98 00:04:24,149 --> 00:04:26,449 becomes exceptionally hostile, doesn't it? Oh, 99 00:04:26,470 --> 00:04:29,230 the dimensionality just explodes. Right. Because 100 00:04:29,230 --> 00:04:32,189 even if you downsample a video feed to 64 by 101 00:04:32,189 --> 00:04:35,430 64 pixels, which they do in this paper, you are 102 00:04:35,430 --> 00:04:37,370 dealing with three color channels per pixel. 103 00:04:37,829 --> 00:04:41,149 That is 12 ,288 dimensions of continuous data. 104 00:04:41,470 --> 00:04:44,410 For a single frame. And you are asking the network 105 00:04:44,410 --> 00:04:47,889 to predict the exact state of those 12 ,000 dimensions 106 00:04:47,889 --> 00:04:50,170 multiple frames into the future. And you are 107 00:04:50,170 --> 00:04:52,370 asking it to do this in an environment governed 108 00:04:52,370 --> 00:04:56,230 by partial observability and stochasticity. Wait, 109 00:04:56,310 --> 00:04:58,699 break that down for me. Partial observability. 110 00:04:59,420 --> 00:05:02,100 Basically, objects occlude one another. So if 111 00:05:02,100 --> 00:05:03,959 a robotic gripper moves in front of a block, 112 00:05:04,439 --> 00:05:06,480 the pixels representing the block disappear from 113 00:05:06,480 --> 00:05:09,579 the data array. Oh, right, because the gripper 114 00:05:09,579 --> 00:05:11,879 is blocking the camera's view of it. Exactly. 115 00:05:12,319 --> 00:05:15,079 But the network has to maintain an internal representation 116 00:05:15,079 --> 00:05:17,259 of that block's continued existence behind the 117 00:05:17,259 --> 00:05:19,300 gripper. It has to have object permanence. Yes. 118 00:05:19,529 --> 00:05:22,370 And then add to that the stochastic nature of 119 00:05:22,370 --> 00:05:25,050 physics. If you push an irregular object, the 120 00:05:25,050 --> 00:05:27,490 exact angle of its rotation might vary slightly 121 00:05:27,490 --> 00:05:30,050 every time. Just due to microscopic friction 122 00:05:30,050 --> 00:05:31,930 differences and things like that. Precisely. 123 00:05:32,290 --> 00:05:35,269 The real world is messy. Which explains why earlier 124 00:05:35,269 --> 00:05:38,470 attempts at this failed so dramatically. The 125 00:05:38,470 --> 00:05:40,990 paper specifically critiques prior video prediction 126 00:05:40,990 --> 00:05:43,470 models, notably the work by Matthew and colleagues 127 00:05:43,470 --> 00:05:46,529 and Owen colleagues. Yes, both of those teams 128 00:05:46,529 --> 00:05:49,170 approached the problem through the lens of reconstruction. 129 00:05:49,569 --> 00:05:51,970 Reconstruction. So they fed the current frame 130 00:05:51,970 --> 00:05:54,250 into the network, compressed it into a hidden 131 00:05:54,250 --> 00:05:57,689 state, and then mathematically attempted to redraw 132 00:05:57,689 --> 00:05:59,970 the next frame from scratch. They forced the 133 00:05:59,970 --> 00:06:02,670 network to act as a generative artist. That sounds 134 00:06:02,670 --> 00:06:04,949 exhausting for a computer. It is an incredibly 135 00:06:04,949 --> 00:06:07,870 inefficient use of network capacity. If you take 136 00:06:07,870 --> 00:06:10,170 the reconstruction approach, the neural network 137 00:06:10,170 --> 00:06:12,550 has to use its internal parameter space to store 138 00:06:12,550 --> 00:06:15,029 the exact appearance, texture, color, and lighting 139 00:06:15,029 --> 00:06:17,170 of every object. And the background, too, right? 140 00:06:17,310 --> 00:06:20,250 And the background. Then it has to compute the 141 00:06:20,250 --> 00:06:23,129 new positions and redraw all those textures in 142 00:06:23,129 --> 00:06:26,089 the new coordinates. So if the model spends like 143 00:06:26,089 --> 00:06:28,389 90 % of its parameters just trying to memorize 144 00:06:28,389 --> 00:06:31,029 what the background table looks like so it can 145 00:06:31,029 --> 00:06:33,740 accurately redraw the wood grain, It has very 146 00:06:33,740 --> 00:06:36,040 little computational power left to actually model 147 00:06:36,040 --> 00:06:38,500 the complex dynamics of the objects colliding 148 00:06:38,500 --> 00:06:40,740 on top of the table. That makes total sense. 149 00:06:41,220 --> 00:06:43,620 Furthermore, if you show a reconstruction model 150 00:06:43,620 --> 00:06:46,240 a novel object it wasn't trained on, it will 151 00:06:46,240 --> 00:06:48,680 fail to predict the future, right? Because it 152 00:06:48,680 --> 00:06:50,579 has no mathematical representation of how to 153 00:06:50,579 --> 00:06:53,000 draw that new texture. Okay, let's unpack this 154 00:06:53,000 --> 00:06:55,519 because that fundamental flaw is what makes the 155 00:06:55,519 --> 00:06:57,959 core innovation of Finn, Goodfellow, and Levine's 156 00:06:57,959 --> 00:07:00,819 paper so elegant. They completely abandon the 157 00:07:00,819 --> 00:07:03,300 idea of reconstruction. They introduce a paradigm 158 00:07:03,300 --> 00:07:07,839 based on advection. Yes. They recognize that 159 00:07:07,839 --> 00:07:09,860 the appearance of the objects is already present 160 00:07:09,860 --> 00:07:12,920 in the previous frame. You don't need to redraw 161 00:07:12,920 --> 00:07:15,300 the block. You just need to calculate the vectors 162 00:07:15,300 --> 00:07:17,779 of motion and move the pixels that are already 163 00:07:17,779 --> 00:07:21,199 there. Advection. It is a term lifted straight 164 00:07:21,199 --> 00:07:24,529 out of fluid dynamics. Instead of generating 165 00:07:24,529 --> 00:07:28,029 pixels, the network outputs a set of transformation 166 00:07:28,029 --> 00:07:30,550 rules that dictate how pixels in the current 167 00:07:30,550 --> 00:07:34,129 image should flow into the next image. It completely 168 00:07:34,129 --> 00:07:36,350 decouples appearance from motion. And why is 169 00:07:36,350 --> 00:07:38,629 that decoupling so important? That decoupling 170 00:07:38,629 --> 00:07:41,509 is the absolute secret to generalization. When 171 00:07:41,509 --> 00:07:44,310 the network explicitly models motion rather than 172 00:07:44,310 --> 00:07:46,350 appearance, it becomes invariant to the visual 173 00:07:46,350 --> 00:07:49,129 identity of the objects. Meaning it doesn't matter 174 00:07:49,129 --> 00:07:52,129 if it... is looking at a red block or some bizarrely 175 00:07:52,129 --> 00:07:55,430 shaped novelty mug. Exactly. It just perceives 176 00:07:55,430 --> 00:07:58,269 a cluster of pixels that, based on its physical 177 00:07:58,269 --> 00:08:00,610 intuition of the collision it's witnessing, needs 178 00:08:00,610 --> 00:08:03,050 to be displaced by a certain vector. It just 179 00:08:03,050 --> 00:08:05,509 perceives a cluster of pixels that, based on 180 00:08:05,509 --> 00:08:07,730 its physical intuition of the collision it's 181 00:08:07,730 --> 00:08:10,050 witnessing, needs to be displaced by a certain 182 00:08:10,050 --> 00:08:13,389 vector. That conceptual leap from redrawing to 183 00:08:13,389 --> 00:08:16,779 moving is massive. But executing it mathematically 184 00:08:16,779 --> 00:08:18,939 is where the architecture gets really dense. 185 00:08:19,259 --> 00:08:21,379 Very dense. Because the authors didn't just propose 186 00:08:21,379 --> 00:08:24,720 one way to move pixels, they designed three distinct 187 00:08:24,720 --> 00:08:26,720 motion prediction models to handle advection. 188 00:08:27,399 --> 00:08:29,540 We really need to analyze these deeply to understand 189 00:08:29,540 --> 00:08:32,840 how they capture physical motion. Right. So the 190 00:08:32,840 --> 00:08:36,419 first model they propose is dynamic neural advection, 191 00:08:36,539 --> 00:08:39,700 or DNA. Okay. DNA. How does this one work? In 192 00:08:39,700 --> 00:08:42,139 the DNA architecture, the network computes a 193 00:08:42,139 --> 00:08:44,620 highly granular prediction. For every single 194 00:08:44,620 --> 00:08:47,240 pixel location in the target future frame, the 195 00:08:47,240 --> 00:08:50,100 model predicts a unique distribution over locations 196 00:08:50,100 --> 00:08:53,379 in the previous frame. So if the network is calculating 197 00:08:53,379 --> 00:08:56,779 the pixel value for coordinate 10, 10 in the 198 00:08:56,779 --> 00:08:59,639 new frame? It outputs a probability distribution 199 00:08:59,639 --> 00:09:02,139 pointing to the previous frame. Like saying, 200 00:09:02,460 --> 00:09:04,940 I calculate a 70 % probability that the color 201 00:09:04,940 --> 00:09:07,759 for this spot should come from coordinate 9, 202 00:09:07,759 --> 00:09:10,679 10, and a 30 % probability it should come from 203 00:09:10,679 --> 00:09:14,840 8, 10. Exactly. It is literally taking an expectation 204 00:09:14,840 --> 00:09:17,179 over a local region to determine the new pixel 205 00:09:17,179 --> 00:09:20,000 value. It's like a micromanager assigning a specific 206 00:09:20,000 --> 00:09:22,320 new desk to every single employee in an office 207 00:09:22,320 --> 00:09:24,340 independently. That's a good way to look at it. 208 00:09:24,340 --> 00:09:26,259 And that local region, which they denoted as 209 00:09:26,259 --> 00:09:28,779 Kappa, is a crucial regularizing constraint. 210 00:09:29,279 --> 00:09:31,600 So the network isn't searching the entire previous 211 00:09:31,600 --> 00:09:33,860 frame? No, it is mathematically constrained to 212 00:09:33,860 --> 00:09:36,419 only pull pixels from a small surrounding neighborhood. 213 00:09:36,679 --> 00:09:39,059 Which reflects the physical reality of high -framing 214 00:09:39,059 --> 00:09:41,940 video, right? Objects don't teleport. Precisely. 215 00:09:42,139 --> 00:09:44,159 A pixel is only going to move a few coordinates 216 00:09:44,159 --> 00:09:46,860 between fractions of a second. Constraining the 217 00:09:46,860 --> 00:09:48,840 search space keeps the parameter count manageable. 218 00:09:49,480 --> 00:09:51,639 But the DNA approach has a glaring structural 219 00:09:51,639 --> 00:09:54,120 weakness when you think about macroscopic physics. 220 00:09:54,759 --> 00:09:57,059 By predicting a unique origin distribution for 221 00:09:57,059 --> 00:09:59,740 every single pixel independently, the model completely 222 00:09:59,740 --> 00:10:02,340 ignores the concept of cohesive matter. It does. 223 00:10:02,639 --> 00:10:05,840 Like, if a robotic arm pushes a solid box, all 224 00:10:05,840 --> 00:10:08,200 the pixels making up that box should move along 225 00:10:08,200 --> 00:10:11,100 parallel vectors. They are bound together. Right. 226 00:10:11,600 --> 00:10:14,120 DNA wastes immense computational energy calculating 227 00:10:14,120 --> 00:10:16,700 independent trajectories for thousands of pixels 228 00:10:16,700 --> 00:10:19,559 that are ultimately locked in a rigid body. So 229 00:10:19,559 --> 00:10:22,059 how did they fix that? Well, that exact inefficiency 230 00:10:22,059 --> 00:10:24,120 is what drives the design of their second model, 231 00:10:24,559 --> 00:10:27,220 convolutional dynamic neural advection, or cDNA. 232 00:10:27,340 --> 00:10:29,700 cDNA. The authors recognized that they needed 233 00:10:29,700 --> 00:10:32,179 an architecture with an inductive bias toward 234 00:10:32,179 --> 00:10:36,039 object -centric rigid motion. So instead of mapping 235 00:10:36,039 --> 00:10:39,840 every pixel independently, cDNA relies on tied 236 00:10:39,840 --> 00:10:42,200 weights applied across the entire image? What's 237 00:10:42,200 --> 00:10:45,059 fascinating here is how cDNA leverages convolution 238 00:10:45,059 --> 00:10:48,759 to enforce uniform motion. Instead of pixel -specific 239 00:10:48,759 --> 00:10:51,000 distributions, the network predicts multiple 240 00:10:51,000 --> 00:10:54,659 discrete motion distributions. And each distribution 241 00:10:54,659 --> 00:10:57,360 is applied universally across the spatial dimensions 242 00:10:57,360 --> 00:10:59,940 of the image. Exactly. You could think of it 243 00:10:59,940 --> 00:11:02,500 like applying a uniform vector field over the 244 00:11:02,500 --> 00:11:05,710 entire frame. the network might output 10 different 245 00:11:05,710 --> 00:11:08,789 convolutional kernels. Kernel 1 might represent 246 00:11:08,789 --> 00:11:11,789 a pure lateral shift to the right. And kernel 247 00:11:11,789 --> 00:11:15,139 2 might represent a shift downward. Right. When 248 00:11:15,139 --> 00:11:17,159 you apply these kernels to the previous frame, 249 00:11:17,679 --> 00:11:20,379 you generate multiple candidate futures. Each 250 00:11:20,379 --> 00:11:23,120 assumes that a cohesive mass of pixels moved 251 00:11:23,120 --> 00:11:26,940 uniformly in a specific direction. It explicitly 252 00:11:26,940 --> 00:11:29,460 assumes that groups of pixels belong to rigid 253 00:11:29,460 --> 00:11:32,470 objects moving together. But relying purely on 254 00:11:32,470 --> 00:11:34,370 convolutional shifts leads a massive blind spot, 255 00:11:34,429 --> 00:11:37,289 doesn't it? It does. For rotational dynamics 256 00:11:37,289 --> 00:11:39,909 and scaling. Because a convolution kernel is 257 00:11:39,909 --> 00:11:42,370 highly effective at pure translation, just sliding 258 00:11:42,370 --> 00:11:45,149 pixels along an X or Y axis. But what if the 259 00:11:45,149 --> 00:11:47,230 robotic arm strikes an object off -center, causing 260 00:11:47,230 --> 00:11:49,450 it to spin? Or what if an object is moving along 261 00:11:49,450 --> 00:11:51,990 the Z axis toward the camera, meaning its pixel 262 00:11:51,990 --> 00:11:55,250 footprint needs to expand? Right. A basic 2D 263 00:11:55,250 --> 00:11:57,909 convolutional shift cannot natively express a 264 00:11:57,909 --> 00:12:00,700 zoom or a rotation. If you try to force a sliding 265 00:12:00,700 --> 00:12:03,179 kernel to rotate an object, the pixels will shear 266 00:12:03,179 --> 00:12:06,200 and the image will degrade instantly. Which brings 267 00:12:06,200 --> 00:12:09,059 us to the third architecture, spatial transformer 268 00:12:09,059 --> 00:12:13,059 predictors or STP. Okay, so how does STP handle 269 00:12:13,059 --> 00:12:16,039 rotation? This model abandons convolutional shifts 270 00:12:16,039 --> 00:12:19,120 entirely. It generates parameters for 2D affine 271 00:12:19,120 --> 00:12:21,700 image transformations. And an affine transformation 272 00:12:21,700 --> 00:12:24,120 matrix can mathematically capture translation, 273 00:12:24,779 --> 00:12:27,220 but also rotation, scaling, and shearing. Exactly. 274 00:12:27,820 --> 00:12:30,639 So SCP generates a set of these matrices and 275 00:12:30,639 --> 00:12:33,440 then uses them to construct a warping grid. A 276 00:12:33,440 --> 00:12:36,259 warping grid? Yeah. The warping grid maps the 277 00:12:36,259 --> 00:12:38,700 spatial coordinates of the input image to the 278 00:12:38,700 --> 00:12:40,940 target coordinates of the output image. Okay, 279 00:12:40,940 --> 00:12:43,840 but how does it actually move the discrete pixels 280 00:12:43,840 --> 00:12:46,610 according to this continuous grid? They utilize 281 00:12:46,610 --> 00:12:49,830 a bilinear sampling kernel. This is vital because 282 00:12:49,830 --> 00:12:51,809 the affine transformation might calculate that 283 00:12:51,809 --> 00:12:55,090 a pixel should move to coordinate 10 .5, 10 .5. 284 00:12:55,289 --> 00:12:58,049 And you can't have half a pixel. Right. The bilinear 285 00:12:58,049 --> 00:13:00,129 sampler takes a weighted average of the four 286 00:13:00,129 --> 00:13:02,629 nearest integer pixel coordinates to smoothly 287 00:13:02,629 --> 00:13:05,629 interpolate the new value. So it makes the entire 288 00:13:05,629 --> 00:13:08,230 warping process differentiable. Meaning the neural 289 00:13:08,230 --> 00:13:10,370 network can back propagate errors through the 290 00:13:10,370 --> 00:13:12,830 transformation and actually learn how to optimize 291 00:13:12,830 --> 00:13:15,539 the affine parameters. Okay, so to synthesize 292 00:13:15,539 --> 00:13:20,230 these advection engines, DNA moves pixels individually 293 00:13:20,230 --> 00:13:24,399 but ignores rigid bodies. Yes. cDNA slides rigid 294 00:13:24,399 --> 00:13:26,620 groups of pixels via convolution but struggles 295 00:13:26,620 --> 00:13:30,519 with rotation. Correct. And STP warps the image 296 00:13:30,519 --> 00:13:33,720 using affine math to capture complex rotation 297 00:13:33,720 --> 00:13:36,879 and scaling. You've got it. But here is the critical 298 00:13:36,879 --> 00:13:41,309 architectural puzzle I am stuck on. CDNA and 299 00:13:41,309 --> 00:13:44,370 STP don't just output one transformed image. 300 00:13:44,809 --> 00:13:47,110 They output multiple candidate images based on 301 00:13:47,110 --> 00:13:49,990 the different kernels or affi matrices they generate. 302 00:13:50,210 --> 00:13:52,409 That is right. How does the network collapse 303 00:13:52,409 --> 00:13:55,830 10 different contradictory futures into a single 304 00:13:55,830 --> 00:13:58,610 coherent frame? This is handled by a compositing 305 00:13:58,610 --> 00:14:01,370 mask. It's arguably the most elegant structural 306 00:14:01,370 --> 00:14:04,289 decision in the entire paper. A compositing mask? 307 00:14:04,529 --> 00:14:07,700 Yes. So simultaneously with predicting the motion 308 00:14:07,700 --> 00:14:10,299 transformations, the network outputs a set of 309 00:14:10,299 --> 00:14:13,360 masking channels. And then for every spatial 310 00:14:13,360 --> 00:14:16,299 location, the network generates a weighting across 311 00:14:16,299 --> 00:14:19,539 all the generated candidate images. And it modulates 312 00:14:19,539 --> 00:14:22,220 this using a channel -wise softmax function. 313 00:14:22,559 --> 00:14:25,879 Exactly. The softmask forces the weights at any 314 00:14:25,879 --> 00:14:28,759 given pixel coordinate to sum to exactly one. 315 00:14:29,360 --> 00:14:32,019 So the network is dynamically blending the layers. 316 00:14:32,259 --> 00:14:35,289 It could say... At this specific pixel coordinate, 317 00:14:35,529 --> 00:14:37,990 take 95 % of the pixel value from the affine 318 00:14:37,990 --> 00:14:41,570 rotation layer and 5 % from the lateral translation 319 00:14:41,570 --> 00:14:45,549 layer. Yes, but the real genius lies in a specific 320 00:14:45,549 --> 00:14:48,490 layer they permanently hard -code into the compositing 321 00:14:48,490 --> 00:14:51,889 stack, the background mask. The background mask? 322 00:14:52,669 --> 00:14:54,950 What does that do? They explicitly provide the 323 00:14:54,950 --> 00:14:57,250 network with an unmodified copy of the previous 324 00:14:57,250 --> 00:15:00,049 frame and give the compositing mask the option 325 00:15:00,049 --> 00:15:02,899 to select it. Which completely alters the learning 326 00:15:02,899 --> 00:15:05,240 dynamics. It really does. By giving the network 327 00:15:05,240 --> 00:15:08,519 a zero -cost bypass lane to just copy static 328 00:15:08,519 --> 00:15:11,440 pixels, you mathematically incentivize it to 329 00:15:11,440 --> 00:15:14,639 focus its transformative power only on the regions 330 00:15:14,639 --> 00:15:17,299 of the image that are actively changing. Because 331 00:15:17,299 --> 00:15:19,240 most of the environment, you know, the table, 332 00:15:19,419 --> 00:15:21,919 the bins, the walls, it's static. Right. It's 333 00:15:21,919 --> 00:15:24,360 an emergent property of loss minimization. The 334 00:15:24,360 --> 00:15:26,440 most computationally efficient way to minimize 335 00:15:26,440 --> 00:15:28,440 prediction error is to route all those static 336 00:15:28,440 --> 00:15:30,750 pixels through the background mask. So the network 337 00:15:30,750 --> 00:15:33,009 naturally learns to isolate the moving pixels 338 00:15:33,009 --> 00:15:35,690 and route them through the advection transformations. 339 00:15:36,450 --> 00:15:38,909 And in doing so, it implicitly learns to segment 340 00:15:38,909 --> 00:15:41,570 the foreground object from the background without 341 00:15:41,570 --> 00:15:44,789 ever being provided a segmentation label. That 342 00:15:44,789 --> 00:15:47,129 is wild. It discovers the boundary of the object 343 00:15:47,129 --> 00:15:49,129 simply because the physics of the scene demand 344 00:15:49,129 --> 00:15:52,250 it. Exactly. It extracts the structural reality 345 00:15:52,250 --> 00:15:54,570 of the world purely through mathematical efficiency. 346 00:15:54,990 --> 00:15:57,230 But having these brilliant transformation modules 347 00:15:57,230 --> 00:16:00,220 isn't enough on its own. They need a memory engine 348 00:16:00,220 --> 00:16:02,340 to actually track these spatial movements over 349 00:16:02,340 --> 00:16:06,059 time, right? They do. And if they used a standard 350 00:16:06,059 --> 00:16:09,379 long short -term memory network, a standard LSTM, 351 00:16:09,480 --> 00:16:11,840 it would flatten the 2D image into a massive 352 00:16:11,840 --> 00:16:14,559 1D vector. It would completely destroy the spatial 353 00:16:14,559 --> 00:16:17,200 relationships. Yes. The model would forget that 354 00:16:17,200 --> 00:16:19,799 a pixel on the far left is physically adjacent 355 00:16:19,799 --> 00:16:21,940 to another pixel on the far left. And if you 356 00:16:21,940 --> 00:16:24,860 destroy the spatial topology, you destroy the 357 00:16:24,860 --> 00:16:27,399 ability to leverage the spatial invariance of 358 00:16:27,399 --> 00:16:29,919 physical loss. Right, because the physics governing 359 00:16:29,919 --> 00:16:31,860 a block falling on the left side of the camera 360 00:16:31,860 --> 00:16:34,519 view are identical to the physics governing a 361 00:16:34,519 --> 00:16:37,220 block on the right side. Grody doesn't care where 362 00:16:37,220 --> 00:16:39,940 in the frame you are. Exactly. If you flatten 363 00:16:39,940 --> 00:16:42,080 the image into a 1D vector and feed it through 364 00:16:42,080 --> 00:16:44,559 fully connected layers, the network has to independently 365 00:16:44,559 --> 00:16:47,120 learn the laws of physics for every single spatial 366 00:16:47,120 --> 00:16:50,139 portnet. The wave matrices would become astronomically 367 00:16:50,139 --> 00:16:53,179 large. Which is why they utilize stacked convolvenal 368 00:16:53,179 --> 00:16:57,039 LSTMs. Okay, conv LSTMs. How does that change 369 00:16:57,039 --> 00:17:00,230 the game? Inside a convolve LSTM, the hidden 370 00:17:00,230 --> 00:17:02,370 state is not a flat vector. It is maintained 371 00:17:02,370 --> 00:17:05,309 as a 3D tensor, height, width, and feature channels. 372 00:17:05,990 --> 00:17:07,869 And the gates that control the flow of memory, 373 00:17:08,410 --> 00:17:10,230 the input gate, the forget gate, the output gate, 374 00:17:10,809 --> 00:17:13,430 they are calculated using convolutions rather 375 00:17:13,430 --> 00:17:16,089 than matrix multiplications. Yes. This means 376 00:17:16,089 --> 00:17:18,690 the network slides the same set of learned physical 377 00:17:18,690 --> 00:17:21,269 rules across the entire spatial extent of the 378 00:17:21,269 --> 00:17:24,069 hidden state. It learns the dynamics of a collision 379 00:17:24,069 --> 00:17:28,119 once. encodes it in a convolutional kernel and 380 00:17:28,119 --> 00:17:30,440 can apply it anywhere in the visual field. The 381 00:17:30,440 --> 00:17:32,380 efficiency gains from that structural choice 382 00:17:32,380 --> 00:17:35,500 are staggering. The authors note that their convolve 383 00:17:35,500 --> 00:17:38,839 LSTM advection models require roughly 12 .5 million 384 00:17:38,839 --> 00:17:41,839 parameters. 12 .5 million. That sounds like a 385 00:17:41,839 --> 00:17:45,039 lot, but wait until we compare it. Right. They 386 00:17:45,039 --> 00:17:47,180 compare that directly to the prior reconstruction 387 00:17:47,180 --> 00:17:50,140 model by Owen colleagues, which utilized a standard 388 00:17:50,140 --> 00:17:53,059 fully connected LSTM. And how big was that one? 389 00:17:53,240 --> 00:17:56,339 That model required 78 million parameters just 390 00:17:56,339 --> 00:18:00,720 to function. 78 million versus 12 .5? And as 391 00:18:00,720 --> 00:18:03,180 we'll see in the data later, it performed significantly 392 00:18:03,180 --> 00:18:05,960 worse despite being six times larger. Because 393 00:18:05,960 --> 00:18:08,619 the 78 million parameter model was fighting the 394 00:18:08,619 --> 00:18:11,420 geometry of the data, whereas the 12 .5 million 395 00:18:11,420 --> 00:18:13,380 parameter model was structurally aligned with 396 00:18:13,380 --> 00:18:16,160 it. Okay, now there is one crucial missing link 397 00:18:16,160 --> 00:18:18,599 in this architecture. We established earlier 398 00:18:18,599 --> 00:18:21,640 that this is an action -conditioned model. Right. 399 00:18:21,720 --> 00:18:24,660 The robot's actions matter. The robot has an 400 00:18:24,660 --> 00:18:27,400 internal state, the XYZ coordinates of its gripper, 401 00:18:27,559 --> 00:18:30,140 the angle of its joints, and it receives a control 402 00:18:30,140 --> 00:18:32,779 action like a velocity command. Which is a highly 403 00:18:32,779 --> 00:18:34,980 compressed 1D vector of floating point numbers. 404 00:18:35,519 --> 00:18:38,740 So how do you mathematically inject a 1D vector 405 00:18:38,740 --> 00:18:41,200 of state and action data into a neural network 406 00:18:41,200 --> 00:18:43,900 whose entire architecture is built around processing 407 00:18:43,900 --> 00:18:46,960 2D spatial feature maps? You can't just... add 408 00:18:46,960 --> 00:18:49,220 a joint angle to a pixel value. No, you can't. 409 00:18:49,299 --> 00:18:51,819 You have to map the continuous low -dimensional 410 00:18:51,819 --> 00:18:54,299 action space into the high -dimensional spatial 411 00:18:54,299 --> 00:18:56,839 topology of the network. They achieve this through 412 00:18:56,839 --> 00:18:59,099 spatial tiling. Let's break down the mechanics 413 00:18:59,099 --> 00:19:01,400 of that tiling, because it dictates how the network 414 00:19:01,400 --> 00:19:04,839 actually understands the command. Sure. If the 415 00:19:04,839 --> 00:19:07,579 combined state and action vector has, say, 10 416 00:19:07,579 --> 00:19:10,960 dimensions, they reshape that vector into a 1 417 00:19:10,960 --> 00:19:15,140 by 1 by 10 tensor. OK. Then they broadcast, or 418 00:19:15,140 --> 00:19:17,339 tile, that vector across the spatial dimensions 419 00:19:17,339 --> 00:19:20,000 of the network's internal feature maps. So if 420 00:19:20,000 --> 00:19:23,660 the hidden state at a specific layer is a 64x64 421 00:19:23,660 --> 00:19:27,440 grid, they duplicate the action vector 4096 times 422 00:19:27,440 --> 00:19:31,940 to create a 64x64x10 tensor. Exactly. Okay. And 423 00:19:31,940 --> 00:19:34,000 then they concatenate it. They attach it along 424 00:19:34,000 --> 00:19:36,180 the channel dimension of the visual feature maps. 425 00:19:36,740 --> 00:19:39,329 This is the crucial step. By tiling the action 426 00:19:39,329 --> 00:19:42,509 data spatially and concatenating it, the subsequent 427 00:19:42,509 --> 00:19:44,309 convolutional kernels in the network now have 428 00:19:44,309 --> 00:19:47,829 direct access to the action values at every single 429 00:19:47,829 --> 00:19:51,009 spatial location. Yes. When the convolutional 430 00:19:51,009 --> 00:19:53,009 filter slides over the feature map, it isn't 431 00:19:53,009 --> 00:19:55,009 just looking at visual patterns. It is looking 432 00:19:55,009 --> 00:19:58,390 at visual patterns and the embedded action command 433 00:19:58,390 --> 00:20:00,950 simultaneously. This allows the network to learn 434 00:20:00,950 --> 00:20:04,150 conditional dependencies. Like what? Like it 435 00:20:04,150 --> 00:20:07,019 can learn weightings such as If the embedded 436 00:20:07,019 --> 00:20:09,980 action vector indicates a high velocity along 437 00:20:09,980 --> 00:20:13,019 the positive x -axis, and I detect the visual 438 00:20:13,019 --> 00:20:15,240 features of my robotic gripper interacting with 439 00:20:15,240 --> 00:20:19,059 an object, I should heavily weight the cDNA transformation 440 00:20:19,059 --> 00:20:22,019 kernel that translates pixels to the right. Beautifully 441 00:20:22,019 --> 00:20:25,500 said. The concatenation directly biases the advection 442 00:20:25,500 --> 00:20:27,819 engines. It is a wonderfully seamless way to 443 00:20:27,819 --> 00:20:29,880 merge proprioception with visual processing. 444 00:20:30,980 --> 00:20:34,500 The architecture is fully assembled. We have 445 00:20:34,500 --> 00:20:38,819 conv LSTMs preserving the spatial geometry, action 446 00:20:38,819 --> 00:20:41,819 vectors tiled and concatenated to guide the physics, 447 00:20:42,700 --> 00:20:45,500 and advection modules generating composited pixel 448 00:20:45,500 --> 00:20:47,759 shifts. Now they just had to prove it worked. 449 00:20:48,339 --> 00:20:50,440 Right. To validate this, the researchers had 450 00:20:50,440 --> 00:20:53,480 to generate a data set capable of testing physical 451 00:20:53,480 --> 00:20:57,279 intuition. Chapters 4 and 5 .1 of the paper detail 452 00:20:57,279 --> 00:21:00,799 the robotic pushing data set. The sheer scale 453 00:21:00,799 --> 00:21:02,700 of data collection here is a testament to the 454 00:21:02,700 --> 00:21:05,039 unsupervised methodology. Because they didn't 455 00:21:05,039 --> 00:21:06,819 have to label anything, they could just let the 456 00:21:06,819 --> 00:21:10,000 hardware run. Exactly. They utilized ten separate, 457 00:21:10,059 --> 00:21:12,200 seven degree of freedom robotic arms. Ten of 458 00:21:12,200 --> 00:21:14,440 them. Yeah, they placed them in front of bins 459 00:21:14,440 --> 00:21:16,920 containing hundreds of disparate objects, toys, 460 00:21:17,119 --> 00:21:19,819 blocks, tools, and they simply programmed the 461 00:21:19,819 --> 00:21:22,700 arms to execute random pushes and sweeps. Recording 462 00:21:22,700 --> 00:21:25,059 the raw RGB video and the corresponding joint 463 00:21:25,059 --> 00:21:27,089 states and actions the whole time. The result 464 00:21:27,089 --> 00:21:30,450 is 57 ,000 continuous interaction sequences. 465 00:21:30,589 --> 00:21:33,009 Totaling roughly 1 .5 million frames of video. 466 00:21:33,390 --> 00:21:36,329 Yes. And to evaluate the models, they set up 467 00:21:36,329 --> 00:21:38,789 an action -conditioned prediction task. How did 468 00:21:38,789 --> 00:21:41,289 that work during testing? The network is fed 469 00:21:41,289 --> 00:21:43,890 two initial frames to establish baseline velocity. 470 00:21:44,430 --> 00:21:46,789 It is also fed the initial pose of the gripper 471 00:21:46,789 --> 00:21:49,650 and a sequence of planned actions. The model 472 00:21:49,650 --> 00:21:51,930 must then recursively predict the future visual 473 00:21:51,930 --> 00:21:54,450 states. Right. They trained it to predict eight 474 00:21:54,450 --> 00:21:56,670 steps into the future, and during testing, they 475 00:21:56,670 --> 00:21:59,250 pushed it to predict up to 18 steps. That is 476 00:21:59,250 --> 00:22:02,769 a long horizon for video prediction. The quantitative 477 00:22:02,769 --> 00:22:05,269 evaluation relies on standard visual metrics, 478 00:22:05,730 --> 00:22:08,630 right? Peak signal to noise ratio, or PSNR, and 479 00:22:08,630 --> 00:22:12,170 structural similarity index, SSIM. PSNR essentially 480 00:22:12,170 --> 00:22:14,970 measures the absolute pixel level error, while 481 00:22:14,970 --> 00:22:17,450 SSIM evaluates the preservation of structural 482 00:22:17,450 --> 00:22:20,589 patterns and contrast. And across both metrics, 483 00:22:21,170 --> 00:22:25,670 the advection models, DNA, cDNA, and STP, drastically 484 00:22:25,670 --> 00:22:27,650 outperformed the prior reconstruction -based 485 00:22:27,650 --> 00:22:30,650 models. It wasn't even close. Let's analyze exactly 486 00:22:30,650 --> 00:22:33,569 how the prior models failed during this test. 487 00:22:34,289 --> 00:22:36,710 The multi -scale reconstruction model by Matu, 488 00:22:36,970 --> 00:22:40,150 it actually performed adequately for exactly 489 00:22:40,150 --> 00:22:44,029 one time step. But the moment it had to predict 490 00:22:44,029 --> 00:22:47,500 step two, it collapsed. Why? That is the compounding 491 00:22:47,500 --> 00:22:49,720 error problem inherent to reconstruction. Oh, 492 00:22:49,720 --> 00:22:51,839 because it uses its own flawed prediction as 493 00:22:51,839 --> 00:22:55,039 the next input. Exactly. If the model generates 494 00:22:55,039 --> 00:22:57,880 the first future frame from scratch and gets 495 00:22:57,880 --> 00:23:00,019 the edge of an object slightly wrong, maybe off 496 00:23:00,019 --> 00:23:03,019 by a few pixels, or the color gradient is slightly 497 00:23:03,019 --> 00:23:05,700 blurred, it must use that flawed degraded image 498 00:23:05,700 --> 00:23:07,900 as the input for the next prediction. Right. 499 00:23:08,039 --> 00:23:09,960 It tries to reconstruct the object again based 500 00:23:09,960 --> 00:23:12,880 on the flawed memory, and the error multiplies 501 00:23:12,880 --> 00:23:15,289 exponentially. Within three steps, the object 502 00:23:15,289 --> 00:23:18,609 diffuses into visual noise. And the fully connected 503 00:23:18,609 --> 00:23:21,930 LSTM model by O failed on a much more fundamental 504 00:23:21,930 --> 00:23:24,509 level. The paper indicates that it basically 505 00:23:24,509 --> 00:23:27,950 ignored the objects entirely and just reconstructed 506 00:23:27,950 --> 00:23:31,089 the static bin. Because of the 78 million parameters 507 00:23:31,089 --> 00:23:34,230 it required just to map the flattened 1D vector 508 00:23:34,230 --> 00:23:37,410 back to 2D space. It exhausted its capacity just 509 00:23:37,410 --> 00:23:39,690 attempting to memorize the background textures. 510 00:23:39,950 --> 00:23:42,589 It simply lacked the representational bandwidth 511 00:23:42,589 --> 00:23:44,829 to track the high -frequency spatial changes 512 00:23:44,829 --> 00:23:47,069 of the small objects interacting in the bin. 513 00:23:47,710 --> 00:23:51,990 Conversely, because CDNA and SDP explicitly advect 514 00:23:51,990 --> 00:23:54,630 the existing pixels and utilize the background 515 00:23:54,630 --> 00:23:57,450 mask, they trivially maintain the background 516 00:23:57,450 --> 00:24:00,509 and focus their capacity entirely on the collision 517 00:24:00,509 --> 00:24:03,809 dynamics. But the true validation of the paper's 518 00:24:03,809 --> 00:24:06,109 core hypothesis comes from the generalization 519 00:24:06,109 --> 00:24:08,519 test. Oh, this is the best part. They evaluated 520 00:24:08,519 --> 00:24:10,599 the advection models on a test set featuring 521 00:24:10,599 --> 00:24:13,500 completely novel objects. Items the robotic arms 522 00:24:13,500 --> 00:24:16,759 had never interacted with during the 1 .5 million 523 00:24:16,759 --> 00:24:19,279 training frames. And the advection models maintained 524 00:24:19,279 --> 00:24:21,940 their predictive accuracy. This is the definitive 525 00:24:21,940 --> 00:24:24,920 proof of the paradigm shift. If the network had 526 00:24:24,920 --> 00:24:26,779 been memorizing visual features, it would have 527 00:24:26,779 --> 00:24:29,140 failed instantly upon seeing a novel geometry. 528 00:24:29,259 --> 00:24:30,759 Because it would know how to draw it. Right. 529 00:24:31,680 --> 00:24:34,059 But because it learned the mechanics of pixel 530 00:24:34,059 --> 00:24:36,799 advection based on physical interaction, it generalized 531 00:24:36,799 --> 00:24:39,430 the physics. It deduced that when the pixels 532 00:24:39,430 --> 00:24:42,210 representing the robotic arm intersect with any 533 00:24:42,210 --> 00:24:45,289 cluster of pixels, that cluster must be displaced, 534 00:24:45,769 --> 00:24:48,190 regardless of its visual identity. It learned 535 00:24:48,190 --> 00:24:51,740 what a push is, essentially. Now, I have a bone 536 00:24:51,740 --> 00:24:54,680 to pick here. Because anyone analyzing the actual 537 00:24:54,680 --> 00:24:57,359 visual outputs of these models, especially at 538 00:24:57,359 --> 00:25:01,039 steps 10, 15, or 18, will immediately notice 539 00:25:01,039 --> 00:25:04,140 a specific degradation. The blur. The predicted 540 00:25:04,140 --> 00:25:06,700 objects become increasingly blurry over time. 541 00:25:06,769 --> 00:25:10,730 They smear out. So what does this all mean? If 542 00:25:10,730 --> 00:25:13,750 you were building a robotic planner, is a blurry 543 00:25:13,750 --> 00:25:16,809 future actually useful? Does the blur indicate 544 00:25:16,809 --> 00:25:18,950 that the model's physical intuition is breaking 545 00:25:18,950 --> 00:25:21,230 down? Oh, I'm glad you brought this up. The authors 546 00:25:21,230 --> 00:25:23,970 address this directly, and the mathematical explanation 547 00:25:23,970 --> 00:25:28,089 is crucial. The blur is not a failure of the 548 00:25:28,089 --> 00:25:31,150 network's physical modeling. It's not? No. It 549 00:25:31,150 --> 00:25:33,569 is a direct artifact of the loss function used 550 00:25:33,569 --> 00:25:36,349 to optimize the network during training. Specifically, 551 00:25:36,710 --> 00:25:39,390 the L2 reconstruction loss, which calculates 552 00:25:39,390 --> 00:25:41,150 the mean squared error between the predicted 553 00:25:41,150 --> 00:25:43,690 pixels and the ground truth video. OK, let's 554 00:25:43,690 --> 00:25:47,009 unpack the statistical consequence of mean squared 555 00:25:47,009 --> 00:25:50,730 error in a stochastic environment. The real world 556 00:25:50,730 --> 00:25:53,390 is not strictly deterministic from the perspective 557 00:25:53,390 --> 00:25:57,660 of a 64 by 64 video feed. Right. If the robotic 558 00:25:57,660 --> 00:26:01,480 arm pushes in a regular object, there's a distribution 559 00:26:01,480 --> 00:26:04,559 of possible physical outcomes. The object might 560 00:26:04,559 --> 00:26:07,240 tumble slightly left or slightly right. But the 561 00:26:07,240 --> 00:26:09,380 network is predicting a single deterministic 562 00:26:09,380 --> 00:26:11,960 frame, and it is being penalized based on its 563 00:26:11,960 --> 00:26:15,240 distance to a stochastic reality. So mathematically, 564 00:26:15,859 --> 00:26:18,619 the prediction that minimizes the L2 loss, the 565 00:26:18,619 --> 00:26:21,420 mean squared error, over a distribution of possible 566 00:26:21,420 --> 00:26:25,039 futures is the conditional expectation, or the 567 00:26:25,039 --> 00:26:28,880 mean. of those futures. Exactly. Ah, I get it 568 00:26:28,880 --> 00:26:31,880 now. If there is a 50 % probability the object 569 00:26:31,880 --> 00:26:34,799 ends up on the left and a 50 % probability it 570 00:26:34,799 --> 00:26:37,599 ends up on the right, predicting it will be sharply 571 00:26:37,599 --> 00:26:41,119 on the left incurs a massive penalty if it actually 572 00:26:41,119 --> 00:26:44,500 goes right. Yes. The safest mathematical bet, 573 00:26:44,759 --> 00:26:46,960 the one that minimizes the squared error, is 574 00:26:46,960 --> 00:26:50,019 to predict a faint semi -transparent object covering 575 00:26:50,019 --> 00:26:53,599 both the left and right positions. Wow. The network 576 00:26:53,599 --> 00:26:56,160 is literally encoding its epistemic uncertainty 577 00:26:56,160 --> 00:26:58,299 about the stochastic physics into the visual 578 00:26:58,299 --> 00:27:02,380 domain as blur. It is outputting a probability 579 00:27:02,380 --> 00:27:04,980 cloud of where the object might be. So while 580 00:27:04,980 --> 00:27:07,420 a sharply defined but highly inaccurate future 581 00:27:07,420 --> 00:27:10,400 would look better to a human observer, the blurry 582 00:27:10,400 --> 00:27:13,980 future is statistically more accurate. And much 583 00:27:13,980 --> 00:27:17,160 safer for a robotic planner because it maps the 584 00:27:17,160 --> 00:27:19,599 variance of the physical interaction. It's an 585 00:27:19,599 --> 00:27:21,849 honest representation of uncertainty. And there 586 00:27:21,849 --> 00:27:24,190 is a phenomenal qualitative experiment detailed 587 00:27:24,190 --> 00:27:27,789 in Figure 6 that showcases the depth of this 588 00:27:27,789 --> 00:27:30,049 predictive capability. Oh, this is a great one. 589 00:27:30,589 --> 00:27:33,390 They took a single initial state and altered 590 00:27:33,390 --> 00:27:35,529 the action vector to see how the model would 591 00:27:35,529 --> 00:27:38,069 extrapolate the physics. In the baseline test, 592 00:27:38,130 --> 00:27:40,670 they fed the network a zero action command. The 593 00:27:40,670 --> 00:27:43,049 robot was instructed to hold still. And the network 594 00:27:43,049 --> 00:27:45,890 correctly predicted a static future. The compositing 595 00:27:45,890 --> 00:27:47,670 mask routed everything through the background 596 00:27:47,670 --> 00:27:50,210 layer. It demonstrated a learned understanding 597 00:27:50,210 --> 00:27:52,869 of inertia. But the real revelation occurred 598 00:27:52,869 --> 00:27:54,869 when they fed the network an action command that 599 00:27:54,869 --> 00:27:57,910 was significantly out of distribution. They instructed 600 00:27:57,910 --> 00:28:01,670 the robot to execute a swing that was 150 % faster 601 00:28:01,670 --> 00:28:04,369 and harder than anything it had executed in the 602 00:28:04,369 --> 00:28:06,769 training data. So it had never seen a hit that 603 00:28:06,769 --> 00:28:09,130 hard? Never. Even though the network had never 604 00:28:09,130 --> 00:28:11,609 observed a collision at that velocity from that 605 00:28:11,609 --> 00:28:14,150 specific initial state, the advection kernels 606 00:28:14,150 --> 00:28:17,049 extrapolated the dynamics perfectly. The network 607 00:28:17,049 --> 00:28:19,289 predicted a future where the robotic arm traversed 608 00:28:19,289 --> 00:28:21,690 the frame much faster and the objects in the 609 00:28:21,690 --> 00:28:23,789 bin were displaced with significantly more force. 610 00:28:24,210 --> 00:28:26,470 Moving much further across the screen. It isn't 611 00:28:26,470 --> 00:28:29,970 just regurgitating memorized sequences. It is 612 00:28:30,130 --> 00:28:32,910 mathematically interpolating the vector math 613 00:28:32,910 --> 00:28:35,250 of a harder collision based on the continuous 614 00:28:35,250 --> 00:28:37,910 parameters of the action space. It is executing 615 00:28:37,910 --> 00:28:40,589 a physical simulation entirely within its hidden 616 00:28:40,589 --> 00:28:42,890 states. That is mind -blowing. But it naturally 617 00:28:42,890 --> 00:28:44,910 leads to the question of scope. Right. How far 618 00:28:44,910 --> 00:28:47,250 can this go? Exactly. If this architecture is 619 00:28:47,250 --> 00:28:49,849 fundamentally just learning to map motion transformations 620 00:28:49,849 --> 00:28:53,190 to minimize pixel error, is it strictly bound 621 00:28:53,190 --> 00:28:55,849 to robotic manipulation? Or can the advection 622 00:28:55,849 --> 00:28:58,029 engines parse physical dynamics in environments 623 00:28:58,029 --> 00:29:01,009 with completely different topologies and no explicit 624 00:29:01,009 --> 00:29:04,119 action data? To test that exact boundary, The 625 00:29:04,119 --> 00:29:06,220 researchers apply the architecture to the human 626 00:29:06,220 --> 00:29:09,720 3 .6M dataset. This is where it gets really interesting. 627 00:29:09,940 --> 00:29:12,700 This dataset consists of complex, multi -jointed 628 00:29:12,700 --> 00:29:15,680 human actors moving around a room, walking, and 629 00:29:15,680 --> 00:29:18,059 gesturing. The critical difference in this experiment 630 00:29:18,059 --> 00:29:21,140 is the absence of the Atkin vector. Right. The 631 00:29:21,140 --> 00:29:23,259 network is no longer being conditioned on a known 632 00:29:23,259 --> 00:29:26,400 control input. It is operating in a purely observational 633 00:29:26,400 --> 00:29:29,220 mode. It has fed 10 consecutive frames of a human 634 00:29:29,220 --> 00:29:31,819 moving, and must predict the next 10 frames. 635 00:29:31,859 --> 00:29:33,920 Which is roughly one second of future motion. 636 00:29:34,119 --> 00:29:36,519 Based entirely on the momentum and trajectories 637 00:29:36,519 --> 00:29:39,220 established in the initial visual context. And 638 00:29:39,220 --> 00:29:41,839 we should note the generalization protocol here 639 00:29:41,839 --> 00:29:44,740 was extremely strict. Right, the networks were 640 00:29:44,740 --> 00:29:47,880 trained on several human subjects but were evaluated 641 00:29:47,880 --> 00:29:50,359 on a held out subject. A completely different 642 00:29:50,359 --> 00:29:52,960 human with different bodily proportions wearing 643 00:29:52,960 --> 00:29:56,160 clothing the network had never encountered. And 644 00:29:56,160 --> 00:29:58,440 quantitatively, the advection models once again 645 00:29:58,440 --> 00:30:00,900 outperformed the baseline reconstruction architectures. 646 00:30:01,680 --> 00:30:03,640 This raises an important question, though, about 647 00:30:03,640 --> 00:30:06,200 what the network actually learned. Because the 648 00:30:06,200 --> 00:30:08,900 most profound insight from the human motion experiments 649 00:30:08,900 --> 00:30:11,279 doesn't come from the error metrics. Where does 650 00:30:11,279 --> 00:30:13,880 it come from? It comes from visualizing the internal 651 00:30:13,880 --> 00:30:16,519 compositing masks generated by the network. Oh, 652 00:30:16,519 --> 00:30:18,859 right. When you look at the background mass the 653 00:30:18,859 --> 00:30:20,839 network generates to process the human movement, 654 00:30:21,440 --> 00:30:24,309 it perfectly isolates the human figure. It carves 655 00:30:24,309 --> 00:30:26,390 out a highly precise silhouette of the moving 656 00:30:26,390 --> 00:30:29,150 person, separating them entirely from the static 657 00:30:29,150 --> 00:30:31,950 room. And this is a system with zero prior knowledge 658 00:30:31,950 --> 00:30:34,869 of human anatomy? None. It was given no pose 659 00:30:34,869 --> 00:30:37,710 data, no skeletal models, and no segmentation 660 00:30:37,710 --> 00:30:40,349 labels. It simply evaluated the optical flow 661 00:30:40,349 --> 00:30:43,309 of the pixels, recognized that a complex cluster 662 00:30:43,309 --> 00:30:46,329 of pixels was moving with interconnected rigidity, 663 00:30:46,890 --> 00:30:48,769 while the surrounding pixels remained static. 664 00:30:48,970 --> 00:30:51,609 And determined that the mathematically optimal 665 00:30:51,609 --> 00:30:53,650 way to predict the next frame was to segment 666 00:30:53,650 --> 00:30:56,150 that cluster. It structurally discovered the 667 00:30:56,150 --> 00:30:59,190 concept of a human body through the pure optimization 668 00:30:59,190 --> 00:31:01,829 of its advection math. The network finds the 669 00:31:01,829 --> 00:31:03,910 signal in the noise purely through the mechanics 670 00:31:03,910 --> 00:31:06,349 of the universe. That is just incredible. So 671 00:31:06,349 --> 00:31:09,670 to synthesize the massive leap forward this paper 672 00:31:09,670 --> 00:31:12,509 represents, prior to this research, the field 673 00:31:12,509 --> 00:31:15,630 was mired in the supervised bottleneck. forcing 674 00:31:15,630 --> 00:31:18,529 networks to laboriously reconstruct high dimensional 675 00:31:18,529 --> 00:31:21,470 appearances from scratch. And Finn, Goodfellow, 676 00:31:21,589 --> 00:31:24,049 and Levine completely redefined the problem. 677 00:31:24,430 --> 00:31:27,210 By decoupling appearance from motion and explicitly 678 00:31:27,210 --> 00:31:30,509 engineering architectures like cDNA and STP that 679 00:31:30,509 --> 00:31:33,230 are biased toward rigid advection and spatial 680 00:31:33,230 --> 00:31:36,269 invariance, they proved that deep neural networks 681 00:31:36,269 --> 00:31:38,690 can derive a generalized intuition for physical 682 00:31:38,690 --> 00:31:41,509 mechanics entirely from unlabeled unstructured 683 00:31:41,509 --> 00:31:44,329 raw video. And as the authors note in their conclusion, 684 00:31:44,829 --> 00:31:47,309 predicting video is not the end goal, right? 685 00:31:47,349 --> 00:31:50,210 It is the substrate for intelligent planning. 686 00:31:51,029 --> 00:31:53,670 Exactly. But they also acknowledge a structural 687 00:31:53,670 --> 00:31:56,049 limitation that points toward the next frontier 688 00:31:56,049 --> 00:31:59,549 of this research. Which is? Well, while the compositing 689 00:31:59,549 --> 00:32:02,589 masks naturally group pixels together, the architecture 690 00:32:02,589 --> 00:32:05,309 doesn't explicitly output a low dimensional state 691 00:32:05,309 --> 00:32:08,349 representation. So it understands that the pixels 692 00:32:08,349 --> 00:32:11,359 move together. But it doesn't compress that understanding 693 00:32:11,359 --> 00:32:14,220 into a definitive data point that says object 694 00:32:14,220 --> 00:32:17,819 A is located at coordinate x, y. Right. And for 695 00:32:17,819 --> 00:32:20,079 downstream applications like reinforcement learning 696 00:32:20,079 --> 00:32:22,660 or an agent attempts to optimize a policy over 697 00:32:22,660 --> 00:32:26,119 millions of trials, processing full 64 by 64 698 00:32:26,119 --> 00:32:28,920 video predictions at every step is computationally 699 00:32:28,920 --> 00:32:32,640 crushing. But if you could augment this unsupervised 700 00:32:32,640 --> 00:32:35,359 architecture. to not only predict the advection, 701 00:32:35,819 --> 00:32:38,920 but to explicitly output the centroid coordinates 702 00:32:38,920 --> 00:32:41,720 of the objects it has segmented. You would provide 703 00:32:41,720 --> 00:32:44,160 reinforcement learning algorithms with an incredibly 704 00:32:44,160 --> 00:32:47,000 clean, perfectly distilled representation of 705 00:32:47,000 --> 00:32:49,180 the physical state. Without ever needing a human 706 00:32:49,180 --> 00:32:51,559 to label a single frame. It lays the groundwork 707 00:32:51,559 --> 00:32:54,079 for agents that can drop into a novel environment, 708 00:32:54,380 --> 00:32:56,920 look around for a few minutes, internally map 709 00:32:56,920 --> 00:32:59,200 the physical rules governing the objects in that 710 00:32:59,200 --> 00:33:02,210 specific space. and immediately begin optimizing 711 00:33:02,210 --> 00:33:05,589 highly complex manipulative tasks. It is the 712 00:33:05,589 --> 00:33:09,789 foundation of true autonomous adaptability. So, 713 00:33:10,150 --> 00:33:12,670 as we conclude this analysis, I want you to consider 714 00:33:12,670 --> 00:33:15,390 the broader implications of what this architecture 715 00:33:15,390 --> 00:33:18,809 fundamentally achieves. We have just dissected 716 00:33:18,809 --> 00:33:22,109 how a neural network utilizing roughly 12 million 717 00:33:22,109 --> 00:33:25,130 parameters mathematically derived the rules of 718 00:33:25,130 --> 00:33:29,119 inertia, collision, and object permanence. simply 719 00:33:29,119 --> 00:33:32,140 by observing how patches of pixels advect over 720 00:33:32,140 --> 00:33:35,839 time in raw video data. It learned the physics 721 00:33:35,839 --> 00:33:39,380 of reality merely by watching. Exactly. If an 722 00:33:39,380 --> 00:33:42,000 unsupervised architecture can extract the fundamental 723 00:33:42,000 --> 00:33:45,180 invisible laws of physical mechanics from the 724 00:33:45,180 --> 00:33:47,940 chaotic visual noise of the real world, what 725 00:33:47,940 --> 00:33:51,339 other invisible fundamental laws might a similar 726 00:33:51,339 --> 00:33:54,339 architecture eventually discover? That is the 727 00:33:54,339 --> 00:33:57,529 real question. If you feed an unsupervised advection 728 00:33:57,529 --> 00:34:00,910 model massive streams of satellite imagery, could 729 00:34:00,910 --> 00:34:03,710 it independently deduce the complex interconnected 730 00:34:03,710 --> 00:34:06,369 rules of macroeconomics and supply chain logistics 731 00:34:06,369 --> 00:34:09,289 simply by analyzing the flow of ships and trucks? 732 00:34:10,369 --> 00:34:13,869 Or biology? Right. If you feed it raw video of 733 00:34:13,869 --> 00:34:17,150 biological cellular processes, could it map the 734 00:34:17,150 --> 00:34:19,690 mechanics of disease propagation without prior 735 00:34:19,690 --> 00:34:23,159 medical data? When you strip away the human engineered 736 00:34:23,159 --> 00:34:26,099 labels and allow a purely observational system 737 00:34:26,099 --> 00:34:29,380 to analyze the continuous flow of reality, what 738 00:34:29,380 --> 00:34:32,159 other invisible structures will it learn to imagine?