1 00:00:00,000 --> 00:00:03,100 You know, it is kind of funny how we think about 2 00:00:03,100 --> 00:00:07,080 physical intelligence. Oh, totally. Like, imagine 3 00:00:07,080 --> 00:00:09,519 you're just standing in your kitchen and you 4 00:00:09,519 --> 00:00:11,439 accidentally knock a glass off the counter. Right, 5 00:00:11,500 --> 00:00:14,320 classic. What happens next? Your hand just darts 6 00:00:14,320 --> 00:00:16,980 out. You sort of fumble for a split second and 7 00:00:16,980 --> 00:00:19,160 you catch it right before it hits the floor. 8 00:00:19,260 --> 00:00:20,820 It's pure instinct. Yeah, you don't even think 9 00:00:20,820 --> 00:00:23,679 about it. Exactly. But if we were to, like, slow 10 00:00:23,679 --> 00:00:27,719 that moment down and actually examine the computational 11 00:00:27,719 --> 00:00:30,480 load of your brain, You certainly aren't rendering 12 00:00:30,480 --> 00:00:33,359 this hyper -realistic, high -definition 3D model 13 00:00:33,359 --> 00:00:35,420 of the glass shattering on the tile just to figure 14 00:00:35,420 --> 00:00:37,859 out your arm trajectory. No, absolutely not. 15 00:00:38,060 --> 00:00:39,700 You don't render the future, right? You just 16 00:00:39,700 --> 00:00:42,700 react to the latent geometry of the present. 17 00:00:43,000 --> 00:00:46,240 Which is incredibly efficient, by the way. Your 18 00:00:46,240 --> 00:00:48,640 brain just discards everything irrelevant. Right, 19 00:00:48,679 --> 00:00:51,219 like the lighting. Exactly. The reflection of 20 00:00:51,219 --> 00:00:53,859 the kitchen light on the glass, the exact color 21 00:00:53,859 --> 00:00:56,979 of the liquid inside. It ignores all of that 22 00:00:56,979 --> 00:00:59,219 and isolates the only variables that actually 23 00:00:59,219 --> 00:01:02,259 matter for physical control. So things like velocity, 24 00:01:02,700 --> 00:01:06,120 mass, spatial coordinates. And yet the biggest 25 00:01:06,120 --> 00:01:08,780 tech companies in the world right now are trying 26 00:01:08,780 --> 00:01:11,060 to force robots to do exactly what your brain 27 00:01:11,060 --> 00:01:14,000 refuses to do. It is a massive contradiction. 28 00:01:14,299 --> 00:01:16,560 Yeah. They're building these models with billions 29 00:01:16,560 --> 00:01:19,099 of parameters, essentially demanding that a robot 30 00:01:19,099 --> 00:01:22,239 imagine the future in this photorealistic detail 31 00:01:22,239 --> 00:01:25,030 just to figure out how to pick up an apple. Which 32 00:01:25,030 --> 00:01:28,510 is wild. It is. So today's mission is to unpack 33 00:01:28,510 --> 00:01:31,709 this major paradigm shift called Lajepa. We're 34 00:01:31,709 --> 00:01:33,750 doing a deep dive into a groundbreaking text 35 00:01:33,750 --> 00:01:37,549 titled Lajepa, Compact Robot Policies with Leighton 36 00:01:37,549 --> 00:01:40,290 World Modeling. Yeah. Put together by Mateo Clemente 37 00:01:40,290 --> 00:01:42,750 and a team of researchers from Huawei Technologies, 38 00:01:43,170 --> 00:01:45,030 University of Toronto, McGill University, and 39 00:01:45,030 --> 00:01:48,349 Miele. Right. And for anyone tracking the explosion 40 00:01:48,349 --> 00:01:50,920 of AI or, you know, If you're just curious about 41 00:01:50,920 --> 00:01:53,599 when we're actually going to get useful physical 42 00:01:53,599 --> 00:01:56,640 robots in our homes or our factories, this deep 43 00:01:56,640 --> 00:01:58,840 dive is going to give you a massive aha moment. 44 00:01:59,099 --> 00:02:01,459 It really is a critical piece of research. We're 45 00:02:01,459 --> 00:02:03,019 going to explore how machines actually learn 46 00:02:03,019 --> 00:02:05,480 to interact with the physical world and why teaching 47 00:02:05,480 --> 00:02:08,500 a robot to move doesn't mean teaching it to see 48 00:02:08,500 --> 00:02:11,099 every single pixel. Because it directly challenges 49 00:02:11,099 --> 00:02:14,120 the whole prevailing dogma of the current AI 50 00:02:14,120 --> 00:02:17,020 era, which is basically scale is all you need. 51 00:02:17,340 --> 00:02:20,370 Just throw more parameters at it. Exactly. We've 52 00:02:20,370 --> 00:02:22,250 fallen into this trap of believing that if a 53 00:02:22,250 --> 00:02:24,389 model isn't working perfectly, we just need to 54 00:02:24,389 --> 00:02:26,729 feed it more data and inflate it to 10 or 20 55 00:02:26,729 --> 00:02:28,750 billion parameters. Which is just brute force. 56 00:02:28,949 --> 00:02:32,270 It is. But Clement's team asked a much more rigorous 57 00:02:32,270 --> 00:02:34,629 engineering question. They asked, what if our 58 00:02:34,629 --> 00:02:37,169 foundational architecture for robot control is 59 00:02:37,169 --> 00:02:39,069 just completely misaligned with the physics of 60 00:02:39,069 --> 00:02:42,430 the real world? So are we overcomplicating how 61 00:02:42,430 --> 00:02:46,590 robots think? Short answer, yes. What if a lean, 62 00:02:46,969 --> 00:02:49,189 elegant model can drastically outperform these 63 00:02:49,189 --> 00:02:51,689 massive, energy -hungry giants simply by thinking 64 00:02:51,689 --> 00:02:55,129 in a more appropriate feature space? I love a 65 00:02:55,129 --> 00:02:57,430 good underdog story where elegant design beats 66 00:02:57,430 --> 00:02:59,830 brute force, but before we look at their solution, 67 00:02:59,830 --> 00:03:02,069 I feel like we really need to thoroughly unpack 68 00:03:02,069 --> 00:03:04,270 the problem with the current state of these world 69 00:03:04,270 --> 00:03:06,710 action models. Yeah, to really appreciate what 70 00:03:06,710 --> 00:03:08,830 Lejeppe does, we have to look closely at the 71 00:03:08,830 --> 00:03:11,789 status quo of robot policies. Right now, the 72 00:03:11,789 --> 00:03:14,349 landscape is heavily dominated by two monolithic 73 00:03:14,349 --> 00:03:16,789 approaches. The first one is the Vision Language 74 00:03:16,789 --> 00:03:19,969 Action Model, or VLA. Let me stop you right there. 75 00:03:20,169 --> 00:03:23,669 Because when I look at VLAs, it just seems like 76 00:03:23,669 --> 00:03:25,930 we are brute forcing an architecture designed 77 00:03:25,930 --> 00:03:29,129 for text onto a physical machine. You're not 78 00:03:29,129 --> 00:03:32,550 wrong. I mean, we're taking the transformer backbone 79 00:03:32,550 --> 00:03:35,770 of a massive LLM, aligning it with visual tokens, 80 00:03:35,849 --> 00:03:37,949 and then instead of asking it to predict the 81 00:03:37,949 --> 00:03:41,090 next word in a sentence, we're asking it to predict 82 00:03:41,090 --> 00:03:44,539 a continuous action. That is... precisely the 83 00:03:44,539 --> 00:03:47,639 friction. You're inheriting this massive architecture, 84 00:03:48,020 --> 00:03:50,520 often tens of billions of parameters, that was 85 00:03:50,520 --> 00:03:53,439 optimized for discrete semantic concepts. Because 86 00:03:53,439 --> 00:03:56,080 language is discrete. Exactly. A word is a word. 87 00:03:56,740 --> 00:03:59,759 But physical space is continuous. the torque 88 00:03:59,759 --> 00:04:02,919 applied to a robotic joint is continuous. Right. 89 00:04:03,139 --> 00:04:05,620 So when you force an LLM architecture to output 90 00:04:05,620 --> 00:04:08,180 a continuous physical action, you are dragging 91 00:04:08,180 --> 00:04:10,780 along this immense amount of computational baggage 92 00:04:10,780 --> 00:04:13,520 that is completely irrelevant to the actual physics 93 00:04:13,520 --> 00:04:16,319 of the task. And that leads to the second dominant 94 00:04:16,319 --> 00:04:19,600 approach, right? The world action models or WAMs? 95 00:04:19,720 --> 00:04:22,199 Yes, WAMs. The logic here seems to be that instead 96 00:04:22,199 --> 00:04:25,680 of text, we should adapt video generators. like 97 00:04:25,680 --> 00:04:27,639 taking the underlying architectures of tools 98 00:04:27,639 --> 00:04:30,540 that make highly realistic video and adapting 99 00:04:30,540 --> 00:04:34,180 them to output robot control actions. The philosophical 100 00:04:34,180 --> 00:04:36,680 argument there is that if a model can accurately 101 00:04:36,680 --> 00:04:38,959 generate a video of a robot pushing a block, 102 00:04:39,360 --> 00:04:41,959 it must possess a deep internal model of physics. 103 00:04:42,300 --> 00:04:45,040 I mean, it sounds logical at first. It's an attractive 104 00:04:45,040 --> 00:04:47,920 philosophy for sure. But it completely conflates 105 00:04:47,920 --> 00:04:50,259 two very different computational goals, which 106 00:04:50,259 --> 00:04:54,240 are generation versus control. OK, unpack that 107 00:04:54,240 --> 00:04:56,720 for me. Let's look at the mechanics of these 108 00:04:56,720 --> 00:04:59,939 video -based world action models. The vast majority 109 00:04:59,939 --> 00:05:02,620 of them rely on variational autoencoders, or 110 00:05:02,620 --> 00:05:06,519 VAEs, as their visual backbone. Right. Now, a 111 00:05:06,519 --> 00:05:09,540 VAE is fundamentally optimized for reconstruction. 112 00:05:10,100 --> 00:05:12,579 Its loss function, which is the mathematical 113 00:05:12,579 --> 00:05:14,360 penalty it tries to minimize during training, 114 00:05:15,000 --> 00:05:18,399 is tied directly to pixel -level fidelity. So 115 00:05:18,399 --> 00:05:20,819 it's constantly evaluating itself by asking, 116 00:05:20,819 --> 00:05:23,290 like, did I get the pixels exactly right compared 117 00:05:23,290 --> 00:05:26,290 to the original image? Exactly. It allocates 118 00:05:26,290 --> 00:05:29,069 immense computational capacity to high frequency 119 00:05:29,069 --> 00:05:31,970 visual details. Like textures and stuff? Yeah. 120 00:05:32,129 --> 00:05:35,430 It's working incredibly hard to accurately synthesize 121 00:05:35,430 --> 00:05:38,430 the texture of a wooden table or the exact gradient 122 00:05:38,430 --> 00:05:41,170 of a shadow cast by an object or the reflections 123 00:05:41,170 --> 00:05:44,250 on a piece of metal. It does this because its 124 00:05:44,250 --> 00:05:46,230 original design purpose was to create a video 125 00:05:46,230 --> 00:05:50,069 that looks visually flawless to human eyes. Okay, 126 00:05:50,069 --> 00:05:52,470 let's unpack this with an analogy, because it 127 00:05:52,470 --> 00:05:55,329 sounds like using an ultra -high definition IMAX 128 00:05:55,329 --> 00:05:58,009 3D camera just to navigate a dark hallway to 129 00:05:58,009 --> 00:05:59,750 the bathroom in the middle of the night. That 130 00:05:59,750 --> 00:06:02,569 is a highly accurate way to visualize the bottleneck. 131 00:06:02,689 --> 00:06:05,870 I mean, do robots really need to imagine photorealistic 132 00:06:05,870 --> 00:06:08,610 futures just to move their arms? If I'm a robot 133 00:06:08,610 --> 00:06:10,769 whose only job is to push a wooden block, I do 134 00:06:10,769 --> 00:06:13,769 not care about the exact pixel color of a shadow. 135 00:06:13,790 --> 00:06:16,310 You really don't. I just need to know the spatial 136 00:06:16,310 --> 00:06:18,829 boundary of the block, right? So we have this 137 00:06:18,829 --> 00:06:21,389 massive mismatch. We're forcing the robot to 138 00:06:21,389 --> 00:06:24,250 process and reconstruct pixels when it only needs 139 00:06:24,250 --> 00:06:27,410 spatial and temporal understanding. And the cost 140 00:06:27,410 --> 00:06:30,629 of that mismatch is paid in inference time. The 141 00:06:30,629 --> 00:06:33,689 hidden costs. Right. When you have a video generator 142 00:06:33,689 --> 00:06:37,329 model with 10 billion parameters, every single 143 00:06:37,329 --> 00:06:39,430 time the robot needs to adjust its trajectory, 144 00:06:39,910 --> 00:06:42,610 it has to run a signal through that massive generative 145 00:06:42,610 --> 00:06:45,910 backbone. It's computationally exhausting. So 146 00:06:45,910 --> 00:06:49,170 if we remove the need to generate pixels, how 147 00:06:49,170 --> 00:06:52,050 much faster could a robot react? So much faster. 148 00:06:52,569 --> 00:06:55,389 In robotics, latency is fatal. If a robot is 149 00:06:55,389 --> 00:06:57,870 performing a dynamic task, like catching something 150 00:06:57,870 --> 00:07:01,829 that's slipping, a delay of even 50 or 100 milliseconds 151 00:07:01,829 --> 00:07:04,529 means the object just hits the floor. Because 152 00:07:04,529 --> 00:07:07,449 the brain is too bloated. Exactly. It's parameter 153 00:07:07,449 --> 00:07:10,290 bloat. Because it's busy processing high -frequency 154 00:07:10,290 --> 00:07:13,110 pixel data. Which brings us to the core innovation 155 00:07:13,110 --> 00:07:16,149 of the paper. If robots shouldn't dream in high 156 00:07:16,149 --> 00:07:18,829 definition pixels, what should they dream in? 157 00:07:19,269 --> 00:07:21,689 And this is where we enter the latent space with 158 00:07:21,689 --> 00:07:25,910 Lejeppa. OK. So that stands for Latent Action 159 00:07:25,910 --> 00:07:29,350 JEPA, right? Yes. The foundation of this approach 160 00:07:29,350 --> 00:07:32,970 is completely abandoning the VAE and that pixel 161 00:07:32,970 --> 00:07:34,949 reconstruction loss function we just talked about. 162 00:07:35,069 --> 00:07:39,110 So no more IMAX cameras. Exactly. Instead... 163 00:07:39,100 --> 00:07:42,019 La Jeppa learns future feature prediction and 164 00:07:42,019 --> 00:07:43,860 action generation directly in what they call 165 00:07:43,860 --> 00:07:48,139 a dense spatial and temporal feature space. Let's 166 00:07:48,139 --> 00:07:50,360 break down that visual backbone because it uses 167 00:07:50,360 --> 00:07:54,620 a pre -trained VJeppa 2 .1 encoder, specifically 168 00:07:54,620 --> 00:07:58,100 a VITL encoder. Crucially, there is absolutely 169 00:07:58,100 --> 00:08:01,759 no separate RGB pixel pathway here. None at all. 170 00:08:02,120 --> 00:08:04,879 And from what we know about VJEPA, it is fundamentally 171 00:08:04,879 --> 00:08:07,540 different from a VAE because it doesn't care 172 00:08:07,540 --> 00:08:09,699 about reconstructing pixels, not even slightly. 173 00:08:10,379 --> 00:08:13,060 So what does it do instead? VJEPA is designed 174 00:08:13,060 --> 00:08:16,439 for downstream tasks, things like object segmentation, 175 00:08:16,779 --> 00:08:19,319 depth estimation, complex video understanding. 176 00:08:19,839 --> 00:08:21,980 It's trained using a self -supervised method 177 00:08:21,980 --> 00:08:25,519 where parts of a video are masked out. Like hiding 178 00:08:25,519 --> 00:08:28,220 pieces of a puzzle. Sort of, yeah. And then the 179 00:08:28,220 --> 00:08:30,139 model has to predict the abstract embeddings 180 00:08:30,139 --> 00:08:33,009 of those missing parts. OK, so it's predicting 181 00:08:33,009 --> 00:08:35,169 the mathematical representation of the missing 182 00:08:35,169 --> 00:08:38,269 data rather than trying to like literally draw 183 00:08:38,269 --> 00:08:41,090 the missing pixels. Right. And because it operates 184 00:08:41,090 --> 00:08:43,529 purely in this embedding space, it naturally 185 00:08:43,529 --> 00:08:46,049 learns to discard all that high frequency visual 186 00:08:46,049 --> 00:08:48,809 noise. The lighting, the shadows. Yes, all of 187 00:08:48,809 --> 00:08:51,840 it. It focuses entirely on the semantic and geometric 188 00:08:51,840 --> 00:08:54,840 structure of the scene. So in the La Jepa architecture, 189 00:08:55,039 --> 00:08:57,700 the raw camera feeds enter the VJEPA encoder. 190 00:08:57,820 --> 00:09:00,259 They get translated into this abstract latent 191 00:09:00,259 --> 00:09:02,759 space. And the actual decision -making policy 192 00:09:02,759 --> 00:09:05,600 never touches a raw pixel again. I really want 193 00:09:05,600 --> 00:09:07,679 to drill down into that decision -making policy 194 00:09:07,679 --> 00:09:10,840 because they introduce this world action architecture. 195 00:09:11,299 --> 00:09:14,259 And it consists of two diffusion transformers, 196 00:09:14,360 --> 00:09:18,919 or DITs, a world DIT and an action DIT. Right. 197 00:09:19,000 --> 00:09:21,639 Now, usually when people build these, they take 198 00:09:21,639 --> 00:09:24,379 a massive pre -trained language model and try 199 00:09:24,379 --> 00:09:27,399 to finagle it into outputting actions. But they 200 00:09:27,399 --> 00:09:29,600 didn't do that here. No, they built both the 201 00:09:29,600 --> 00:09:32,320 world DIT and the action DIT completely from 202 00:09:32,320 --> 00:09:34,559 scratch. Wow, from scratch. Yeah. And that's 203 00:09:34,559 --> 00:09:36,580 a crucial design choice because they aren't chained 204 00:09:36,580 --> 00:09:38,899 to the parameter bloat of a pre -trained LLM. 205 00:09:38,919 --> 00:09:41,500 They can tailor the capacity specifically for 206 00:09:41,500 --> 00:09:45,500 control. So they use VJEPPA as this frozen visual 207 00:09:45,500 --> 00:09:47,860 understanding. But the parts that actually model 208 00:09:47,860 --> 00:09:50,289 the physics are lightweight. Exactly, purpose 209 00:09:50,289 --> 00:09:53,049 -built. So how did these two DITs actually work 210 00:09:53,049 --> 00:09:55,110 together during training? The paper mentions 211 00:09:55,110 --> 00:09:57,710 this term joint flow matching to predict both 212 00:09:57,710 --> 00:10:00,629 actions and future features. Right, so flow matching 213 00:10:00,629 --> 00:10:03,330 is distinct from standard diffusion. Standard 214 00:10:03,330 --> 00:10:05,789 diffusion involves slowly adding noise to data 215 00:10:05,789 --> 00:10:08,649 and then training a network to iteratively remove 216 00:10:08,649 --> 00:10:11,500 it. Like unblurring an image. Yeah. But flow 217 00:10:11,500 --> 00:10:14,620 matching is a more generalized, continuous framework. 218 00:10:14,820 --> 00:10:18,000 Think of it as learning the continuous trajectory 219 00:10:18,000 --> 00:10:20,419 of how the current state evolves into the future 220 00:10:20,419 --> 00:10:23,419 state. So during training, it watches a demonstration 221 00:10:23,419 --> 00:10:26,620 of, say, a robot pushing a block. It learns the 222 00:10:26,620 --> 00:10:28,940 action required. And at the same time, it learns 223 00:10:28,940 --> 00:10:30,899 how the abstract representation of the world 224 00:10:30,899 --> 00:10:33,059 is going to change because of that action. Yes. 225 00:10:33,120 --> 00:10:36,049 And they're trained jointly. So the world DIT 226 00:10:36,049 --> 00:10:39,009 and the action DIT are interacting, sharing representations. 227 00:10:39,669 --> 00:10:41,429 But this is where they introduce a fascinating 228 00:10:41,429 --> 00:10:44,029 architectural constraint. They use a very specific 229 00:10:44,029 --> 00:10:47,490 attention mask. OK. An attention mask basically 230 00:10:47,490 --> 00:10:49,470 dictates what the transformer is allowed to look 231 00:10:49,470 --> 00:10:53,129 at, right? Correct. In Legepa, the action tokens, 232 00:10:53,350 --> 00:10:55,090 the parts figuring out how to move the joints, 233 00:10:55,549 --> 00:10:58,090 they can look at the observed context, the present 234 00:10:58,090 --> 00:11:01,600 state. OK, makes sense. However, they are completely, 235 00:11:01,600 --> 00:11:04,679 strictly blocked from attending to any of the 236 00:11:04,679 --> 00:11:07,240 future tokens. OK, wait. Here's where it gets 237 00:11:07,240 --> 00:11:09,559 really interesting to me, because this seems 238 00:11:09,559 --> 00:11:12,379 like a massive contradiction. How so? Well, the 239 00:11:12,379 --> 00:11:14,559 whole premise of this joint training is to force 240 00:11:14,559 --> 00:11:17,039 the network to predict the future state. But 241 00:11:17,039 --> 00:11:19,700 you are explicitly blinding the action module 242 00:11:19,700 --> 00:11:22,940 to that exact future. It does sound like a paradox. 243 00:11:23,200 --> 00:11:25,659 And furthermore, they mention that during inference, 244 00:11:25,740 --> 00:11:27,580 when the robot is actually running in the real 245 00:11:27,580 --> 00:11:29,879 world, they completely drop the future prediction 246 00:11:29,879 --> 00:11:32,740 module entirely. Yep, they throw it out. So if 247 00:11:32,740 --> 00:11:34,820 you're going to blind the action module to the 248 00:11:34,820 --> 00:11:37,419 future and then throw the future prediction away 249 00:11:37,419 --> 00:11:40,059 when it matters, why bother predicting the future 250 00:11:40,059 --> 00:11:42,480 during training at all? That is the big question. 251 00:11:42,879 --> 00:11:45,159 And the answer is that predictive supervision 252 00:11:45,159 --> 00:11:48,279 actually shapes the present observation representations. 253 00:11:48,799 --> 00:11:51,029 Shapes them how? By forcing the shared world 254 00:11:51,029 --> 00:11:53,529 DIT parameters to accurately predict the future 255 00:11:53,529 --> 00:11:55,850 during training, you are forcing the network 256 00:11:55,850 --> 00:11:58,210 to profoundly deepen its understanding of the 257 00:11:58,210 --> 00:12:01,149 present state. Oh, I see. So the struggle to 258 00:12:01,149 --> 00:12:03,529 predict the future acts as a forcing function 259 00:12:03,529 --> 00:12:05,990 to understand the present more accurately. Exactly. 260 00:12:06,409 --> 00:12:08,909 The network learns that a block on a table isn't 261 00:12:08,909 --> 00:12:11,690 just static pixels. To predict how it will move, 262 00:12:11,850 --> 00:12:14,629 it has to implicitly encode its mass, the friction, 263 00:12:14,850 --> 00:12:17,470 the kinematics, all into the present -day feature 264 00:12:17,470 --> 00:12:19,710 representation. It's kind of analogous to training 265 00:12:19,710 --> 00:12:22,129 a chess grandmaster. If you force a player to 266 00:12:22,129 --> 00:12:24,169 constantly visualize the board 10 moves ahead 267 00:12:24,169 --> 00:12:26,509 during practice, the way they perceive the current 268 00:12:26,509 --> 00:12:28,990 layout becomes fundamentally different. That's 269 00:12:28,990 --> 00:12:31,129 a perfect analogy. They don't just see a knight, 270 00:12:31,269 --> 00:12:34,190 they see, like, a web of kinetic potential. Right. 271 00:12:34,299 --> 00:12:37,019 the present representation becomes imbued with 272 00:12:37,019 --> 00:12:39,899 dynamics. So when the action module looks at 273 00:12:39,899 --> 00:12:42,860 the present context, it's reading an incredibly 274 00:12:42,860 --> 00:12:46,059 rich, dynamically aware space. Even though it 275 00:12:46,059 --> 00:12:49,000 can't see the future tokens. Exactly. And the 276 00:12:49,000 --> 00:12:51,539 brilliance of this pays off at inference. Because 277 00:12:51,539 --> 00:12:53,799 it already extracts these rich features from 278 00:12:53,799 --> 00:12:56,340 the present, it doesn't need to waste time actively 279 00:12:56,340 --> 00:12:59,019 predicting the future during execution. It just 280 00:12:59,019 --> 00:13:01,200 drops the future module entirely. And that saves 281 00:13:01,200 --> 00:13:03,730 massive compute time. That's how it achieves 282 00:13:03,730 --> 00:13:06,309 such staggering speed without losing physical 283 00:13:06,309 --> 00:13:10,009 intuition. That's just brilliant theory. But 284 00:13:10,009 --> 00:13:11,970 you know, to prove this architecture actually 285 00:13:11,970 --> 00:13:14,529 works, the researchers didn't just build a massive 286 00:13:14,529 --> 00:13:16,289 model and cross their fingers. No, they were 287 00:13:16,289 --> 00:13:18,590 very methodical. Yeah, they ran a step -by -step 288 00:13:18,590 --> 00:13:21,129 design study, a real master class, using this 289 00:13:21,129 --> 00:13:23,309 comeback -first approach. Which is so rare right 290 00:13:23,309 --> 00:13:25,960 now. Everyone just wants to scale immediately. 291 00:13:26,340 --> 00:13:28,220 But Clemente's team wanted to make the architecture 292 00:13:28,220 --> 00:13:31,960 viable for a tiny policy first. Just 0 .1 billion 293 00:13:31,960 --> 00:13:34,940 trainable parameters, right? Size S. Yes. Size 294 00:13:34,940 --> 00:13:38,480 S. Testing a 100 million parameter model when 295 00:13:38,480 --> 00:13:41,019 everyone else is bragging about 10 billion parameter 296 00:13:41,019 --> 00:13:43,120 models. It's crazy. It's like trying to prove 297 00:13:43,120 --> 00:13:45,820 a highly tuned, efficient engine can outmaneuver 298 00:13:45,820 --> 00:13:48,620 a brute force rocket. They wanted to ensure the 299 00:13:48,620 --> 00:13:50,600 foundational recipe was perfect before scaling. 300 00:13:50,879 --> 00:13:53,379 So let's walk through the specific design choices 301 00:13:53,379 --> 00:13:56,580 they tested on these major benchmarks. They used 302 00:13:56,580 --> 00:14:00,639 RoboTwin 2 .0 and Libero, right? Right. RoboTwin 303 00:14:00,639 --> 00:14:04,600 tests bimanual manipulation to robot arms across 304 00:14:04,600 --> 00:14:06,720 clean environments and randomized environments, 305 00:14:06,799 --> 00:14:08,860 which are full of visual clutter. And what was 306 00:14:08,860 --> 00:14:11,139 the first thing they tested? The feature space. 307 00:14:11,440 --> 00:14:16,039 They pitted VGEPA 2 .1 features against WAN 2 308 00:14:16,039 --> 00:14:19,480 .2 VAE Latents. So ultimate test of abstract 309 00:14:19,480 --> 00:14:22,240 geometry versus pixel reconstruction. Exactly. 310 00:14:22,879 --> 00:14:25,840 And VJEPA completely crushed the VAE, especially 311 00:14:25,840 --> 00:14:27,820 out of distribution. Do we have the stats on 312 00:14:27,820 --> 00:14:31,080 that? Yeah. VJEPA improved clean robo -twin success 313 00:14:31,080 --> 00:14:34,779 by 15 .9 percentage points. Wow. And in randomized 314 00:14:34,779 --> 00:14:37,879 scenes, it improved by 16 .0 points over the 315 00:14:37,879 --> 00:14:40,779 VAE. Because the VAE is mathematically desperate 316 00:14:40,779 --> 00:14:42,740 to reconstruct all the random clutter, right? 317 00:14:42,740 --> 00:14:44,799 It just gets overwhelmed. Exactly. It wastes 318 00:14:44,799 --> 00:14:46,899 its attention on textures instead of semantic 319 00:14:46,899 --> 00:14:48,919 boundaries. OK. So they locked in the feature 320 00:14:48,919 --> 00:14:51,850 space. The next major choice is temporal encoding. 321 00:14:52,230 --> 00:14:55,129 They compared static, tube, and tube motion encoding. 322 00:14:55,909 --> 00:14:57,889 Right. Static treats every frame as an isolated 323 00:14:57,889 --> 00:15:01,360 photo. No momentum. Useless for physics. Basically. 324 00:15:01,860 --> 00:15:04,139 Tube encoding is better. It uses a two -frame 325 00:15:04,139 --> 00:15:07,179 clip to give an immediate velocity cue. But tube 326 00:15:07,179 --> 00:15:09,639 motion is the real winner. How does tube motion 327 00:15:09,639 --> 00:15:12,860 work? It jointly encodes 16 future frames, spaced 328 00:15:12,860 --> 00:15:15,580 two frames apart. So it forces the model to grapple 329 00:15:15,580 --> 00:15:18,299 with a continuous, long -horizon kinetic flow. 330 00:15:18,500 --> 00:15:20,840 Which gives the policy a real motion cue. So 331 00:15:20,840 --> 00:15:24,000 we have VGEPA latency and tube motion. But what 332 00:15:24,000 --> 00:15:26,340 about robustness? A real robot's going to have 333 00:15:26,340 --> 00:15:28,600 sensor failures. That's where they introduced 334 00:15:28,600 --> 00:15:30,940 camera dropout and proprioception dropout during 335 00:15:30,940 --> 00:15:33,179 training. Right. So dropping eligible cameras 336 00:15:33,179 --> 00:15:35,759 like just temporarily turning off the wrist camera 337 00:15:35,759 --> 00:15:38,100 feed during training? Exactly. It forces the 338 00:15:38,100 --> 00:15:40,899 model not to over rely on a single view. So it's 339 00:15:40,899 --> 00:15:43,100 essentially like making a Jedi train with a blast 340 00:15:43,100 --> 00:15:45,860 helmet on or, you know, a basketball player practicing 341 00:15:45,860 --> 00:15:48,919 with one eye closed. That is exactly what it's 342 00:15:48,919 --> 00:15:51,679 like. By temporarily taking away a sense during 343 00:15:51,679 --> 00:15:53,899 training, the robot becomes practically bulletproof 344 00:15:53,899 --> 00:15:55,759 when it has all its sensors back in the real 345 00:15:55,759 --> 00:15:59,330 world. It's engineered resilience. And they also 346 00:15:59,330 --> 00:16:01,750 dropped proprioception, the robot's internal 347 00:16:01,750 --> 00:16:04,710 joint states, 10 % of the time, forcing it to 348 00:16:04,710 --> 00:16:06,990 deduce its own arm position purely from vision. 349 00:16:07,309 --> 00:16:10,149 That is so clever. They also highlight the impact 350 00:16:10,149 --> 00:16:13,269 of image augmentation paired with something called 351 00:16:13,269 --> 00:16:17,320 encoder Laura. Low rank adaptation, yes. The 352 00:16:17,320 --> 00:16:20,500 massive V -JEPA encoder is mostly frozen, but 353 00:16:20,500 --> 00:16:23,639 LoRa injects tiny trainable matrices so it can 354 00:16:23,639 --> 00:16:26,820 adapt to the specific geometry of, say, a robot 355 00:16:26,820 --> 00:16:28,720 cripple without forgetting its pre -training. 356 00:16:29,059 --> 00:16:31,240 And when paired with image augmentation, like 357 00:16:31,240 --> 00:16:34,379 changing brightness or blur, that raised randomized 358 00:16:34,379 --> 00:16:37,559 robo -twin success by an incredible 16 .0 points. 359 00:16:37,879 --> 00:16:41,039 The synergy was just remarkable. So with the 360 00:16:41,039 --> 00:16:43,340 perfect hyper -efficient recipe locked in, it 361 00:16:43,340 --> 00:16:45,919 was finally time to test La Jepa against the 362 00:16:45,919 --> 00:16:48,250 heavyweights. But when they started testing, 363 00:16:48,470 --> 00:16:50,789 they stumbled onto a massive flaw in the benchmark 364 00:16:50,789 --> 00:16:53,370 itself. This is honestly one of the wildest parts 365 00:16:53,370 --> 00:16:55,450 of the paper. Let's talk about the benchmarks 366 00:16:55,450 --> 00:16:58,830 first. They scaled Logepi to three sizes, size 367 00:16:58,830 --> 00:17:03,029 S at 0 .10 billion parameters, size B at 0 .34 368 00:17:03,029 --> 00:17:06,849 billion, and size L at 1 .14 billion. Still tiny 369 00:17:06,849 --> 00:17:09,329 compared to the competition. Right. And they 370 00:17:09,329 --> 00:17:12,109 tested on RoboTwin and LibroPlus, which is supposed 371 00:17:12,109 --> 00:17:14,869 to test robustness to lighting, camera viewpoints, 372 00:17:14,910 --> 00:17:17,819 things like that. And Logepa outperformed every 373 00:17:17,819 --> 00:17:20,700 baseline without embodied pre -training. But 374 00:17:20,700 --> 00:17:23,359 during this evaluation, they found a structural 375 00:17:23,359 --> 00:17:27,920 flaw, the Libero Plus scandal. Yes, the instruction 376 00:17:27,920 --> 00:17:31,000 leakage. Tell me about this. OK, so the benchmark 377 00:17:31,000 --> 00:17:33,700 is supposed to give a natural language instruction, 378 00:17:34,339 --> 00:17:36,619 like pick up the black bowl while changing the 379 00:17:36,619 --> 00:17:38,940 lighting or camera. Right. But they discovered 380 00:17:38,940 --> 00:17:41,480 the officially released Libero Plus evaluator 381 00:17:41,480 --> 00:17:43,980 was literally corrupting the text instructions 382 00:17:43,980 --> 00:17:48,380 for 8 ,493 out of 10 ,030 variants. Oh my god. 383 00:17:48,779 --> 00:17:52,619 8 ,400 out of 10 ,000. Yes. The software was 384 00:17:52,619 --> 00:17:54,920 accidentally taking the internal parameters used 385 00:17:54,920 --> 00:17:57,619 for the visual changes and appending them directly 386 00:17:57,619 --> 00:18:01,079 to the text instruction. So instead of pick up 387 00:18:01,079 --> 00:18:04,000 the black bowl, the evaluator was feeding the 388 00:18:04,000 --> 00:18:06,920 robots gibberish. Exactly. It was feeding them 389 00:18:06,920 --> 00:18:09,359 stuff like place it on the plate, light one. 390 00:18:10,059 --> 00:18:11,859 Or if it was a camera test, I would have penned 391 00:18:11,859 --> 00:18:16,460 something like view 0010024. Wait, wait, wait. 392 00:18:16,680 --> 00:18:18,720 So what does this all mean for the other models? 393 00:18:18,839 --> 00:18:20,400 I mean, were they failing because they couldn't 394 00:18:20,400 --> 00:18:23,200 see the bowl or because they were just horribly 395 00:18:23,200 --> 00:18:25,920 confused by the words light one? They were completely 396 00:18:25,920 --> 00:18:29,440 confused by the text. Remember, VLA models and 397 00:18:29,440 --> 00:18:31,920 video generator whams are fundamentally built 398 00:18:31,920 --> 00:18:35,440 on LLMs, or text -conditioned diffusion. They 399 00:18:35,440 --> 00:18:39,029 are hypersensitive to text. Right. If you feed 400 00:18:39,029 --> 00:18:41,529 a prompt ending in light one, it's going to try 401 00:18:41,529 --> 00:18:43,869 to hallucinate a new light source or just collapse. 402 00:18:44,190 --> 00:18:46,789 Exactly. The benchmark was completely obscuring 403 00:18:46,789 --> 00:18:49,609 the true visual robustness of these models. Because 404 00:18:49,609 --> 00:18:52,109 it was a string concatenation error, not a failure 405 00:18:52,109 --> 00:18:54,680 of physics. Right. Now, Clemente's team could 406 00:18:54,680 --> 00:18:57,299 have just taken their win because La Jeppa isn't 407 00:18:57,299 --> 00:18:59,559 as reliant on text conditioning, but they took 408 00:18:59,559 --> 00:19:01,700 the time to correct the text and re -evaluate 409 00:19:01,700 --> 00:19:04,200 all the open baselines. That is some serious 410 00:19:04,200 --> 00:19:06,559 scientific rigor. It is. And the recalibration 411 00:19:06,559 --> 00:19:10,400 was huge. FastSAM, for example, a massive 12 412 00:19:10,400 --> 00:19:14,940 .4 billion parameter model, it gained 19 .3 points 413 00:19:14,940 --> 00:19:16,880 once the instructions were actually corrected. 414 00:19:17,500 --> 00:19:20,859 19 .3 points. Just from fixing a bug in the text. 415 00:19:20,900 --> 00:19:23,869 That is insane. But... Here is the kicker. They 416 00:19:23,869 --> 00:19:25,670 leveled the playing field, gave everyone their 417 00:19:25,670 --> 00:19:28,650 corrected scores, and La Jeppe still won. The 418 00:19:28,650 --> 00:19:31,930 large size L beat the massive NAMWAM model by 419 00:19:31,930 --> 00:19:35,210 1 .8 points, right? Yes, and NAMWAM is over 7 420 00:19:35,210 --> 00:19:37,970 billion parameters. La Jeppe did it with four 421 00:19:37,970 --> 00:19:41,049 times fewer deployed parameters. So winning in 422 00:19:41,049 --> 00:19:42,970 simulation is great, obviously, but the ultimate 423 00:19:42,970 --> 00:19:44,789 test for a robot brain is the physical world, 424 00:19:44,890 --> 00:19:47,589 where speed and contact really matter. Let's 425 00:19:47,589 --> 00:19:50,309 talk real -world speed. This is where La Jeppe's 426 00:19:50,309 --> 00:19:52,980 efficiency really shines. Because it drops the 427 00:19:52,980 --> 00:19:55,240 future prediction and doesn't reconstruct pixels, 428 00:19:55,440 --> 00:19:58,319 it updates far fewer parameters. The paper says 429 00:19:58,319 --> 00:20:01,299 it runs up to five times faster than Fast Wham, 430 00:20:01,599 --> 00:20:04,019 hitting like 17 to 25 milliseconds of latency. 431 00:20:04,440 --> 00:20:08,039 17 milliseconds is an incredibly fast control 432 00:20:08,039 --> 00:20:10,339 loop. If you want a robot to chop vegetables 433 00:20:10,339 --> 00:20:12,880 in your kitchen or work on an assembly line, 434 00:20:13,519 --> 00:20:15,640 a delay of even half a second because the brain 435 00:20:15,640 --> 00:20:18,240 is too big means dropped plates or injuries. 436 00:20:18,640 --> 00:20:21,839 Speed is safety. Absolutely. And they proved 437 00:20:21,839 --> 00:20:24,839 this with real robot experiments on a Franca 438 00:20:24,839 --> 00:20:28,240 tabletop arm. They designed four specific tasks. 439 00:20:28,740 --> 00:20:31,500 Let's run through those. First was cube placement. 440 00:20:32,180 --> 00:20:34,980 Right. A basic control task. Picking up a cube, 441 00:20:35,279 --> 00:20:37,839 placing it in a bowl. Baseline hand -eye coordination. 442 00:20:38,720 --> 00:20:40,900 Then there was fruit picking, which is harder. 443 00:20:41,099 --> 00:20:43,539 The robot has to lift a steamer lid, set it down 444 00:20:43,539 --> 00:20:45,759 without tipping it, and pick the correct plum. 445 00:20:46,180 --> 00:20:48,779 So multi -stage planning and object permanence. 446 00:20:49,099 --> 00:20:52,240 Exactly. The third task was push T, and this 447 00:20:52,240 --> 00:20:55,140 one is contact heavy, pushing a T -shaped block 448 00:20:55,140 --> 00:20:57,980 across a table to an outline. Pushing is so hard 449 00:20:57,980 --> 00:21:00,779 for robots because of slipstick friction, right? 450 00:21:00,940 --> 00:21:02,720 The block starts rotating, you have to constantly 451 00:21:02,720 --> 00:21:05,380 adjust. Yes, and this is exactly where that 17 452 00:21:05,380 --> 00:21:08,779 millisecond latency is crucial. Fast wham at 453 00:21:08,779 --> 00:21:11,339 88 milliseconds is too slow to catch the micro 454 00:21:11,339 --> 00:21:14,319 rotations of the block. La Jeppe's fast inference 455 00:21:14,319 --> 00:21:16,599 allows for better continuous adjustment during 456 00:21:16,599 --> 00:21:19,180 that heavy contact. Wow. And the fourth task. 457 00:21:19,339 --> 00:21:23,460 Dish manipulation. This was a long horizon, uninterrupted 458 00:21:23,460 --> 00:21:26,099 rollout. It had to place plates, then pick up 459 00:21:26,099 --> 00:21:28,920 a spoon and a fork correctly from a holder. And 460 00:21:28,920 --> 00:21:31,339 how did it compare in the real world? La Jepa 461 00:21:31,339 --> 00:21:33,839 achieved the highest average success among all 462 00:21:33,839 --> 00:21:36,319 policies without embodied pre -training. In fact, 463 00:21:36,380 --> 00:21:38,920 it was comparable to the massive Pi .5 model, 464 00:21:38,980 --> 00:21:41,900 which does have massive embodied pre -training. 465 00:21:42,440 --> 00:21:45,140 So La Jepa proves we can do way more with way 466 00:21:45,140 --> 00:21:48,150 less. But... To be scientifically rigorous here, 467 00:21:48,289 --> 00:21:51,650 we have to ask, what can't it do yet, and where 468 00:21:51,650 --> 00:21:53,609 does this all go from here? Right, the paper 469 00:21:53,609 --> 00:21:57,069 does outline limitations. For one, they didn't 470 00:21:57,069 --> 00:21:59,650 do exhaustive hyperparameter tuning, but the 471 00:21:59,650 --> 00:22:02,849 most interesting finding was this non -monotonic 472 00:22:02,849 --> 00:22:06,269 scaling on RoboTwin. Non -monotonic, meaning 473 00:22:06,269 --> 00:22:08,430 that making the model bigger didn't always make 474 00:22:08,430 --> 00:22:12,210 it better. Interestingly, the medium B size sometimes 475 00:22:12,210 --> 00:22:14,950 beat the large L size. Why does that happen? 476 00:22:15,289 --> 00:22:18,710 It comes down to overfitting. The Action DIT 477 00:22:18,710 --> 00:22:20,950 is predicting a relatively simple output, just 478 00:22:20,950 --> 00:22:23,990 the joint twerks. It's a low dimensional problem 479 00:22:23,990 --> 00:22:26,250 compared to the massive visual feature space. 480 00:22:26,650 --> 00:22:29,089 So if you give the Action DIT too many parameters, 481 00:22:29,289 --> 00:22:31,349 it just memorizes the training demonstrations 482 00:22:31,349 --> 00:22:34,049 down to the millimeter instead of learning the 483 00:22:34,049 --> 00:22:37,809 physics concepts? Exactly. An overly large action 484 00:22:37,809 --> 00:22:40,549 DIT easily overfits the action distribution when 485 00:22:40,549 --> 00:22:43,109 the target data is limited. When it sees a slight 486 00:22:43,109 --> 00:22:45,509 variation in testing, it fails because it's too 487 00:22:45,509 --> 00:22:47,750 rigid. Which brings us to the ultimate missing 488 00:22:47,750 --> 00:22:51,630 piece, embodied pre -training. La Jepa only trained 489 00:22:51,630 --> 00:22:54,670 on target demonstrations, right? Right. The paper 490 00:22:54,670 --> 00:22:57,309 explicitly calls for testing how La Jepa scales 491 00:22:57,309 --> 00:23:00,869 when fed massive, large -scale robot pre -training 492 00:23:00,869 --> 00:23:03,289 datasets across multiple embodiments. Things 493 00:23:03,289 --> 00:23:06,130 like the Open X Embodiment dataset. It's like 494 00:23:06,130 --> 00:23:08,349 they've built this incredibly efficient, highly 495 00:23:08,349 --> 00:23:11,049 tuned Formula One engine, but they've only tested 496 00:23:11,049 --> 00:23:13,589 it on the local go -kart track with a few laps 497 00:23:13,589 --> 00:23:16,069 of data. The next step is letting it drive the 498 00:23:16,069 --> 00:23:19,250 whole world. And because it uses abstract latents 499 00:23:19,250 --> 00:23:21,670 instead of pixels, it should theoretically scale 500 00:23:21,670 --> 00:23:23,650 across different robot arms and environments 501 00:23:23,650 --> 00:23:26,349 much more gracefully. Well, we've covered a massive 502 00:23:26,349 --> 00:23:28,130 amount of ground today. I mean, we started with 503 00:23:28,130 --> 00:23:30,210 robots bogged down by dreaming in high -definition 504 00:23:30,210 --> 00:23:32,589 video. Trying to render IMAX movies in their 505 00:23:32,589 --> 00:23:34,750 heads. Right. And we discovered how shifting 506 00:23:34,750 --> 00:23:37,490 to a latent control -focused architecture like 507 00:23:37,490 --> 00:23:40,950 La Jeppe makes them smaller, up to 5x faster, 508 00:23:41,549 --> 00:23:44,569 and vastly more capable. It really is a master 509 00:23:44,569 --> 00:23:47,650 class in alignment. You now know that in the 510 00:23:47,650 --> 00:23:51,650 future of AI, bigger isn't always better. Architectural 511 00:23:51,650 --> 00:23:54,750 alignment to the actual task is what wins. Absolutely. 512 00:23:55,549 --> 00:23:57,710 But I want to leave you with a deep philosophical 513 00:23:57,710 --> 00:24:01,690 question directly building on this text. If Lejeppe 514 00:24:01,690 --> 00:24:03,569 achieves state -of -the -art physical control 515 00:24:03,569 --> 00:24:06,730 by deliberately hiding the future from the action 516 00:24:06,730 --> 00:24:09,650 module and completely ignoring pixel -perfect 517 00:24:09,650 --> 00:24:12,829 reality in favor of abstract, latent features, 518 00:24:13,730 --> 00:24:16,029 what does that say about human muscle memory? 519 00:24:16,950 --> 00:24:19,829 Oh, that's interesting. When you catch that falling 520 00:24:19,829 --> 00:24:22,559 glass in the kitchen, do you render the high 521 00:24:22,559 --> 00:24:24,440 -def pixels of the glass hitting the floor in 522 00:24:24,440 --> 00:24:27,440 your mind's eye, or do you just instinctively 523 00:24:27,440 --> 00:24:30,039 feel the latent flow of the environment? Just 524 00:24:30,039 --> 00:24:33,390 sensing the momentum. Exactly. Maybe the way 525 00:24:33,390 --> 00:24:35,789 to build the ultimate robot is to realize that 526 00:24:35,789 --> 00:24:38,150 true physical intuition is completely blind. 527 00:24:39,140 --> 00:24:41,460 Something to think about. Thank you for joining 528 00:24:41,460 --> 00:24:43,660 us on this deep dive, and we'll catch you next 529 00:24:43,660 --> 00:24:43,799 time.