EPISODE TRANSCRIPT
World Models and V-JEPA: Learning from Video (Yann LeCun & Adrien Bardes et al.)
Automatically transcribed from the episode audio. Some words and names may be inaccurate.
Listen to the episodeDownload textDownload captions
You know, if you watch a magic trick, say a magician is holding a coin and they close their hand and then open it to show the coin has just vanished into thin air. Right, a classic disappearing act. Yeah, exactly. There is this very specific sensation you experience right then. Your brain just kind of short circuits for a split second. Yeah, it does. Because you know it's a trick, right? You aren't actually confused about whether reality has fundamentally broken. And the reason you know it's a trick is because you possess this unshakable model of how physical reality works. Absolutely. Objects do not just vanish.
Right. They have permanence, they obey gravity, they occupy three -dimensional space, you know? It is a direct violation of your internal physics engine. Yes, a physics engine. Like when that coin disappears, you don't have to consciously sit there and calculate the mass of the coin or the density of the air or the trajectory of a drop. Your brain just flags the visual input immediately with this massive error message. It's like error 404, reality not found. Exactly. Says that is not how the world operates. Yeah.
You have a baseline expectation of reality and the magician just violated it on purpose. Right. And that instinct, that foundational understanding of reality is something every single one of us has. We completely take it for granted. I mean, Toddler has it Oh for sure a cat looking for a toy under a couch has it but then you step into the world of artificial intelligence as it exists right now and Suddenly that foundational physics engine is just well, it's completely missing. It's just not there at all We are dealing with these incredibly articulate systems that can write a passable sonnet or you know code a Python script in seconds. But if you ask them about the physical mechanics of stacking a heavy wooden box on top of a fragile raw egg, they don't actually know what happens in the real world. No, they don't. They don't wince at the idea of the egg breaking. They don't wince because they don't have a model of an egg, or a box, or gravity for that matter. It is the absolute definition of a shallow understanding. Shallow rights. When a large language model tells you the egg will break, it isn't running an internal simulation of fragile shells and heavy masses.
It's essentially this brilliant scholar who has been locked in a dark room their entire life. Oh, I like that analogy. Right. just reading every book ever written, but having never actually bumped their shin on a coffee table. Yeah, they have no physical context. Exactly. They know the word pain, they know the word gravity, but they have zero experiential grounding for what those words physically entail in the real world. Well, welcome to this deep dive. Today, we are looking at how to actually build that missing physics engine. We are basically opening up the blueprints for a completely new kind of digital brain. That's a huge shift. It really is. And our sources for this are coming straight from the absolute cutting edge. We're looking at a visionary 2022 position paper by Yann LeCun, who is the chief AI scientist at Metta. It's titled A Path Towards Autonomous Machine Intelligence.
A very ambitious title. Yeah, no kidding. And to see if his wild theories actually hold water, we are pairing it with a 2024 empirical paper, also from Metta, exploring something called VJPA. Which is super exciting. It is because that second paper is the real -world application of these exact theories to video representation. Right. Our mission today is to meticulously map out this radical new architecture for you. We are going to decode the philosophy behind how machines can actually learn the way animals do. Which is vastly different from how current AI learns, right? Totally different. We'll unpack the specific, module -by -module cognitive architecture required to build a system with actual common sense. And then we're going to look at the hard proof that this new path to intelligence is already working in the lab. So this isn't just about making a better chat bot to write your emails. Not at all. It's about a massive pivot in the entire field. moving away from systems that just memorize statistical patterns and moving toward systems that truly understand the underlying causal mechanisms of the physical world. Okay, let's start with the philosophy of intelligence, which Lacoon tackles right at the beginning of his 2022 paper.
He points out something that is incredibly obvious but, uh, I guess often ignored in computer science. Animals, and obviously humans, learn incredibly efficiently. So efficiently. And we do it mostly just by passively observing our environment. Like, in the first few months of life, a human infant is just soaking in data, staring at things, dropping things from their high chair. Over and over again. Right. They are acquiring this massive background knowledge about physical constraints. And efficiency really is the key word there.
Think about how biological learning compares to standard machine learning today. An infant doesn't need to drop a million different objects off a high chair to learn about the concept of gravity. Right, they don't need a massive data set of falling cups. Exactly. They observe a toy falling a few times and they rapidly build an internal model that says, okay, unsupported things move downward. That accumulated background knowledge, that intuitive physics is what we call common sense. Common sense, right. It tells an agent what is likely to happen, what is plausible, and what is utterly impossible. And current AI just does not have this efficiency at all. Let's take reinforcement learning, for example. That's how we train AI to play chess or video games.
Yeah, that's the standard method. The system tries something, gets a score, tries again, gets a better score. And over millions and millions of trials, it learns the optimal moves. But Lacoon points out a massive flaw with this when you try to bring it into the real world. A fatal flaw, really. Real -world interactions are expensive. They are slow, and they are incredibly dangerous. You can't put a robot in a physical kitchen and let it learn how to chop vegetables by trial and error. You absolutely cannot. In a video game, if the AI makes a mistake, the character dies, the simulation resets, and it tries again a millisecond later. No harm, no foul. Right.
But in a physical kitchen, if a robot learning by trial and error moves its arm too fast, it destroys the kitchen counter. breaks the knife, and probably shatters its own motor. Yeah, the penalty for a wrong action is just way too high. You cannot afford to learn that jumping off a cliff is bad by jumping off a cliff. You'd only learn it once. Exactly. You have to be able to predict the fatal mistake before you make it. So the system has to have a way to simulate reality internally before taking action externally. Yes. And that brings us to the core concept of a world model. Now, to be clear, this isn't a brand new idea in the grand scheme of science. All right, it goes way back. The psychologist Kenneth Craig actually proposed the idea of internal models and human cognition way back in 1943. Wow, 1943.
Yeah. He argued that the nervous system is essentially a calculating machine capable of modeling external events. But taking that psychological concept and actually applying it to modern, differentiable machine learning that is the frontier Lacan is exploring here. Okay, so an intelligent agent must have an internal simulator of the world to predict the consequences of actions it has in that take -in. That is the whole ballgame. Okay, I have to stop and play devil's advocate for a second, because I use large language models, you know, chat bots, every single day, and if you talk to them, they really do seem to have common sense. They do a very good impression of it, yes. Right. If I type into a prompt, what happens if I place a hot iron flat on a silk shirt and leave it there for an hour? The AI will immediately tell me the shirt will burn.
It will catch fire. It's a terrible idea. Of course it will. It can answer incredibly complex physical, logical and even emotional questions. So why isn't that enough? Why do we say it lacks common sense if it consistently gives me the exact right answer about physical reality? This is perhaps the biggest point of confusion in the public discourse about AI right now. So it is vital we unpack it. Lacoon argues that the knowledge of language model possesses is fundamentally ungrounded. Ungrounded. OK. Yes, it can tell you the silk shirt will burn. But it is not telling you that because it has an internal simulation of heat transfer, material flammability, and the passage of time. It's not running a physics calculation in the background. Not at all. It tells you that because the words hot iron, silk, and burn appear in close statistical proximity across the billions of web pages it ingested during training. Ah, I see. So it's like trying to learn how to ride a bicycle by reading the dictionary definition of the word pedal and the word balance. That is a brilliant way to put it. You might be able to write a beautiful essay about how a bicycle works, but the second you actually sit on one, you're going to fall right over. You don't have the grounded physical intuition.
Exactly. Human common sense isn't derived from reading text. It's built from interacting with a physical reality. True intelligence requires a model of that reality, not just statistical word associations. That makes a lot of sense. Think about what happens if you give a language model a completely novel physical puzzle. Something bizarre that has never been explicitly described in any text on the internet. Like some weird Rube Goldberg machine scenario. Yeah, exactly. It often fails completely or it hallucinates a nonsensical answer because it has no internal physics engine to fall back on. It cannot run a mental simulation of a novel scenario. It can only search for text patterns that look like they might be the answer. It's the difference between memorizing the answer key to a physics test and actually understanding the laws of physics.
Precisely. If the problem isn't on the answer key, the memorizer is totally lost. But the person who understands physics? can solve a problem they've never seen before. And that ability to solve the unseen problem is why a world model is the missing link. Intelligent agents must be able to fill in the blanks of what they can't currently observe and predict the outcomes of actions in completely novel situations. So we need an AI that doesn't just read about the world but actually models it. But how do you build a digital physics engine? Like, where does this world model actually live inside the computer code? This is where Lacoon's paper gets really exciting. Because he doesn't just philosophize, right? He lays out a highly specific architectural blueprint. He does. He proposes a cognitive architecture made of six distinct modules. And the key phrase he uses to describe all of them is that they are differentiable. We should pause on that word because differentiable is the magic property that makes modern deep learning possible. It really is the secret sauce of the whole field.
Let's unpack that for a second. For anyone who isn't a machine learning engineer, what does it actually mean for a module to be differentiable, and why is it mandatory for this architecture? Okay, imagine you are blindfolded, standing on the side of a mountain, and your goal is to get to the very bottom of the valley. Okay, blindfolded on a mountain. Got it. Because you can't see, you just feel the ground with your foot. If the ground slopes down to the left, you take a step left. You are calculating the gradient, or the slope, of the terrain and moving in the direction that takes you lower. You're just following the downward slope. Right. In machine learning, the valley is the lowest possible error in the system's performance. If a system is differentiable, it means we can mathematically calculate the slope of its errors. So it can feel its way down the mountain. Exactly. We can feed data through the whole architecture, see how wrong its prediction was at the end, and then send a signal backwards to all the modules saying, adjust your internal settings just a tiny bit in this specific direction to lower the error next time. OK, so if the modules weren't differentiable, if they were just rigid old school computer code with rigid if -then rules they couldn't learn. Right, there'd be no slope to feel. But because they are differentiable, the entire six -module system can organically adjust and teach itself just by experiencing data and minimizing its errors. Yes. The whole system can be trained end -to -end. So let's actually walk through what these six differentiable pieces do and how they interact. Let's do it.
The first one is the perception module. Its job is to take in the raw signals from the sensors. For a robot, this would be carapixels, microphone, audio, maybe tactile sensors. Or for a digital agent, it might just be raw data streams. Exactly. But its critical function is not just to ingest data. It is to estimate the current state of the world by extracting only what is relevant. to the task at hand. So it's a filter. A very powerful filter. Like if my task is driving a car, my perception module needs to extract the positions of other cars, the state of the stop lights, the pedestrians on the corner. Right, the stuff that matter. And it explicitly needs to ignore the exact shape of the clouds in the sky or the specific brand of shoes a pedestrian is wearing because that information is totally useless for the task of driving. If I tried to process all of that, my brain would just overload.
That filtering is essential. If the agent tries to model every single molecule in its environment, it will fail. So the perception module creates a filtered, compressed representation of the current state of the world. Then it passes that filtered representation to the absolute heart of the system. The world model. The simulator, yes. The world model has a two -fold job. First, it estimates missing information about the current state. It basically fills in the blanks. What do you mean by filling in the blanks? If you are looking at a table and a laptop is obscuring part of a coffee mug, your world model tells you the rest of the mug exists behind the laptop.
It doesn't just assume the mug is sliced in half. Oh, right. Object permanence again. Exactly. And second, and most importantly, it predicts plausible future states of the world based on proposed actions. Okay, so the world model is spinning up these simulations. If I turn the steering wheel left, the car moves left into that lane. But how does the system know if moving into that lane is a good idea or a terrible idea? Like, what if there's a truck there? That evaluation is handled by the third module, which is the cost module. The cost module computes a single mathematical value, which LeCun calls energy.
But you can think of it intuitively as discomfort or pain, or basically just divergence from a goal. OK, so it's a pain meter. The agent's entire existence is built around minimizing this cost. Now, the cost module is actually split into two distinct subparts. First, you have the intrinsic cost. What's that? These are the immutable, hardwired biological drives. For a human, intrinsic costs are physical pain, hunger, extreme cold. For a robot, intrinsic costs might be battery depletion, extreme heat in its processors, or excessive physical force on its mechanical joints. It's the built -in alarm system, the factory settings that say, under no circumstances should you let the battery hit zero and please do not crush your own arm. Yes, exactly. It measures the pain of the current state. But there's a second part to the cost module, which is the trainable critic.
The trainable critic. That sounds ominous. It's crucial, actually. While the intrinsic cost tells you if you are currently in pain, the trainable critic learns to anticipate future costs. Ah, so it looks ahead. Right. It looks at the current state and says, wait, we aren't in pain right now, but I recognize this situation. Yeah. The last time we were in this state, it led to a massive spike in intrinsic costs a few seconds later. It learns to associate certain states with impending disaster or success. I love this. So the perception module sees the world. The world model predicts what will happen if we take an action. The cost module looks at that prediction and screams, no, that future hurts. That is literally the workflow. But what is actually deciding on the action? What is the steering wheel here?
That is the fourth module, the actor. The actor. The actor is the module that proposes sequences of actions. It says, what if I do A, then B, then C? It feeds that proposed sequence to the world model. The world model simulates the outcomes. The cost module evaluates how painful those outcomes are. And then what? The actor then searches through various different proposals until it finds the sequence of actions that results in the lowest possible cost. Once it finds that optimal sequence, it sends the very first action of that sequence out to the real world, to the robot's motors, for example. To keep all of this straight, there has to be some kind of scratch pad, right? Like a place to hold these thoughts. Yes, that's the fifth module, the short -term memory. It simply keeps track of the current state. the predicted future states, and the associated costs. So the actor and world model have a reference point as they run these simulations. It's the agent's working memory. Exactly. And finally, the boss of the whole operation, the sixth module. The configurator. Think of this as the executive control. It takes the overall overarching goal, say, navigate to the kitchen, and it primes all the other modules to achieve that specific goal.
How does it prime them? It tells the perception model what to pay attention to. It tunes the world model to pull up the physics of indoor environments. And it adjusts the parameters of the cost module so the system knows that getting closer to the kitchen reduces the cost. OK, this is a beautiful, intricate system. Perception, world model, cost, actor, memory, configurator. But honestly, when I'm going through my daily life, I am definitely not running intense multi -step simulations of the future for every single thing I do. You'd be exhausted if you did. Right.
If someone throws a baseball at my head, I don't pause, calculate the trajectory of the ball, run a simulation of it hitting my nose, consult my cost module to determine that a broken nose is high cost, and then instruct my actor module to move my neck muscles. I just duck. And Lacun addresses this perfectly. He defines two completely different operating modes for this architecture, mode one and mode two. Which immediately made me think of Daniel Kahneman's famous book thinking fast and slow with system one and system two.
The paper explicitly makes that connection. What you just described with the baseball is mode one. Mode one is reactive behavior. So it skips the simulation entirely. In this mode, the perception module extracts the state of the world and it sends that state straight to the actor. The actor immediately outputs an action. The world model and the cost module aren't even engaged. It bypasses the simulator entirely. This is your reflexes or, you know, it's driving a route you've driven a thousand times where you were basically on autopilot, letting muscle memory take over. Correct. So mode one is fast. It's cheap on computing power, but it only works for situations you already know how to handle perfectly. Right. But mode two is reasoning and planning. This is where the true intelligence of the world model shines.
In mode two, the actor doesn't just act. It proposes an action sequence. The world model predicts the futures. The cost module evaluates. The actor optimizes. All of this happens internally before a single physical action is taken in the real world. So mode two is condom in system two. That's me trying to parallel park a massive rental SUV in a tiny spot in a crowded city. The worst feeling. It is. I am not relying on muscle memory there. I am sitting there, gripping the wheel, mentally simulating, OK, if I cut the wheel hard left right now, my front right bumper is going to click the car in front of me. High cost. OK, let's rewind the simulation. I need to reverse straight back for two more feet, then cut the wheel left. Yes. You are consciously running your world model, checking the cost of each hypothetical move. And taking a step back, this represents a massive philosophical shift for how we design AI. In this architecture, complex behavior is not hard -coded by a human programmer writing if -then rules. Not at all. Nor is the agent just randomly trying things to maximize a meaningless score in a video game. The agent's intelligence, its reasoning, is driven entirely by energy minimization finding the path through reality that satisfies its intrinsic biological or programmed drives.
Reasing itself is literally defined as this internal simulation and optimization process. It plans because it has to avoid pain. This sounds flawless in theory. You have a world model that simulates the future and an actor that picks the best future. But we kind of glossed over a really massive problem. We did the reality check. Right. Predicting the future is incredibly messy. The real world is not a clean, sterile chessboard. It is chaotic. So how exactly do we train a machine's world model to predict a chaotic world without it getting totally overwhelmed by unpredictable, irrelevant details? That is the million -dollar technical roadblock. For a long time, we just didn't know how to do it efficiently. And this brings us to the core technical innovation proposed by Lacombe, the transition away from predicting pixels and toward predicting features. Let's dive deep into this. Because to train a world model, you have to use a technique called self -supervised learning, or SSL. Right. The basic idea is simple. You take a piece of data, you hide part of it from the machine, and you ask the machine to predict the hidden part. It's basically pattern completion. If it guesses wrong, it adjusts its internal gradients to guess better next time. That's the basic loop. But LeCun points out a fatal flaw in how we historically applied this to images and video. Historically, we used generative models. If we wanted to teach a model how physics works in a video, we'd show it frame one and ask it to generate frame two. And think about what that computationally demands. If you ask a system to generate frame two, you are forcing the machine to predict every single pixel in the next frame, every color value, every shadow, every edge. Lacun uses a fantastic analogy in the paper that really made this click for me.
Imagine you are driving a car down a suburban street. For your internal world model to be useful and keep you safe, you need it to predict the position, the speed, and the trajectory of the cars around you. Right, the big heavy objects. You need it to predict that the pedestrian on the corner might step into the crosswalk. But you do not want your brain wasting massive amounts of energy trying to predict the exact rustling of the leaves on the trees blowing in the wind by the side of the road. or the specific ripples on a puddle you're driving past. Exactly. And that is why generative pixel -predicting models fail when trying to build complex world models.
A pixel prediction model cannot distinguish between the important semantic features like the heavy car moving toward you and the irrelevant, stopastic details like the leaves rustling. It's all just pixels to the model. If it is forced to predict the next frame at the pixel level, it is forced to predict the leaves. And because the movement of leaves in the wind is chaotic, random, and fundamentally unpredictable, the model gets overwhelmed. It completely crashes its capacity. It dedicates all of its neural power trying to model the noise, and it fails to learn the underlying physics of the actual scene. So it's trying so hard to draw the exact texture of the tree bark in the next second that it doesn't notice the Mack truck barreling toward it. That is exactly what happens.
So Lacune's proposed solution is an entirely new mathematical structure called JPA. JPA, which stands for the Joint Embedding Predictive Architecture. Yes. And the absolute defining characteristic of JPA is that it is non -generative. Instead of predicting the actual output, the pixels of frame two, from the input of frame one, it predicts the abstract representation of the output. We need to unpack this carefully because this is the real paradigm shift. How does JPA actually work mechanically? Let's break down the architecture. JPA consists of three main parts. an X encoder, a Y encoder, and a predictor. Let's say X is what you observe right now, and Y is what you want to predict in the future. Okay, so X is frame one of our driving video, and Y is frame two of the driving video. Right. The X encoder process is frame one. But instead of just passing pixels along, it compresses frame one into a mathematical abstract representation. It extracts the features. Okay. At the same time, during training, the Y encoder processes frame 2, and it also creates an abstract representation of frame 2. So they're both working in abstract space now. Crucially, yes. The predictor module is then asked to look at the representation of X and guess the representation of Y. The predictor is never, ever asked to recreate the pixels of Y. So going back to the driving analogy, the X encoder looks at the street scene. And because it's generating an abstract representation, it has the freedom to just ignore the rustling leaves entirely. It literally filters them out. It looks at the scene and creates a mathematical vector that basically says there is a large metal object moving forward at 40 miles per hour. The predictor then takes that abstract idea and says, OK, based on the laws of physics, the future representation of this scene will be large metal object moving forward at 40 miles per hour, but positioned 10 feet further down the road. It never had to draw the car, and it never had to draw the leaves.
It just predicted the state of the features. That is the elegance of it. By doing the prediction in this abstract representation space, the model can safely eliminate unpredictable, irrelevant details. It filters out the noise so it can focus on the underlying causal dynamics. It's so smart. It is. But as with everything in machine learning, Solving one problem creates another. Oh, of course it does. There is a massive, gaping, technical trap hidden in this approach. If you design a system where the only goal is to make the predictor's output match the encoder's output, the system will eventually find a way to cheat. It suffers from what we call informational collapse. Right.
I was reading this and I had this exact thought. If the model is just a lazy student trying to minimize its prediction error and it gets to define its own abstract representations, Why wouldn't it just output a constant zero for everything? That is exactly what a lazy neural net does. Okay. The representation of frame one is zero. The representation of frame two is zero. My prediction is zero. Oh, look. My prediction perfectly matches the reality. Zero error. I'm a genius. It is a very real, very frustrating problem in self -supervised learning. If the encoders just output a constant zero, or if they just ignore the input entirely and output a static signal, the error drops to zero. But the system has learned absolutely nothing about the video. It has collapsed the information. So how do you fix it? To prevent this, JEPA relies on specific regularization techniques to force the encoders to actually do their jobs. The paper heavily features one called VYCRAG, which stands for Variance and Variance Covariance Regularization. OK, that is a terrifying amount of jargon. Let's break VYCRAG down into plain English. How does it stop the lazy student from just outputting zeros?
It sounds intimidating. but the intuition is actually quite beautiful. It's essentially a set of strict rules applied to the mathematical vectors the encoders produce. Let's take them one by one. First, variance. Okay, what does variance do? This rule forces the representations of different images to be spread out or varied across the mathematical space. If you show the encoder a dog and then you show it a car, the variance rule mathematically penalizes the system if the representation for the dog is too close to the representation for the car. Ah, so it can't just group everything together. Right.
It says you cannot output zero for everything. You must maintain distinct, diverse signals for different inputs. Okay, so variance prevents the collapse to a single point. What about invariance? Invariance is the prediction rule. It simply says that the output of the predictor, which is trying to guess the future, must match the output of the encoder, the actual future. It forces the prediction to be accurate. Simple enough, make a good guess. And the last one, covariance. Covariance is about efficiency. Imagine an encoder produces a representation with a hundred different variables. If variable one tells you the object is a dog and variable two also tells you the object is a dog, variable two is totally redundant. It's wasting space. Right, you don't need to say dog twice. The covariance rule forces the different variables within a single representation to be independent of each other. It de -correlates them. It says, don't use two variables to represent the same concept. Make every single variable capture a unique, independent piece of information about the scene. So PhALIG is this brilliant balancing act. It forces the system to create representations that are highly informative and diverse using variance and covariance, but also highly predictable using invariance. It forces the system to actually learn the meaningful features of the world. Exactly. And there's one more crucial piece to the JPEG predictor module we need to discuss, the latent variable, which is usually denoted by the letter Z. What does Z do? The latent variable Z handled the fundamental uncertainty of the real world. Think about it. The world is not strictly deterministic. No, it's definitely not. If you are watching a car approach a fork in the road, there is no single guaranteed future.
The car might go left or the car might go right. Both are entirely plausible, valid futures based on the current state. If the predictor is only allowed to make one single guess, what does it do? If it guesses left and the car goes right, it gets penalized. If it tries to average the two and guess that the car drives straight into the median, that's completely wrong physically. It's a lose -lose situation for the model. So how does Z solve this? The latent variable Z is essentially a dial that the predictor can turn to explore different valid futures. It allows the predictor to say, OK, if the environment's hidden variable z is equal to 1, then my prediction is the car goes left. If z is equal to 2, my prediction is the car goes right. Oh, that's clever. It allows the world model to represent a multiplicity of plausible futures simultaneously.
And this is absolutely essential for the motor 2 planning we discussed earlier. The actor module can plug different Z's values into the world model to game out different scenarios before taking action. This whole architecture is staggering. Predicting abstract concepts rather than exact pixels, using vis reg to avoid informational collapse, using latent variables to handle the chaos of a fork in the road, it makes total logical sense on paper. It is a very tight theory. But Lacombe published this manifesto in 2022. For a couple years, this was just a really elegant blueprint. Which brings us to the second paper.
Does this conceptual philosophical blueprint actually work when you write the code and run it on a supercomputer? That is the big question. Let's turn to the 2024 meta paper on VJP to see this theory put to the ultimate test. Yes, moving from theory to reality. VJP, which stands for VideoJP, is the empirical validation of Lacoon's 2022 vision. The researchers at Meta set out to test feature prediction as a standalone objective for unsupervised learning. And they really went all in, right? They did. They completely abandoned pixel reconstruction. No generative modeling whatsoever. They wanted to see if modern tools could make JPEG work at scale on complex video data. And when we say they tested it at scale, we mean massive scale. They trained this model on a data set they constructed called VideoMix 2M. This is two million video clips pulled from various public data sets. It's a huge amount of dynamic, real world visual data. It's a lot of data to crunch. And they used a specific architecture for their encoders called a vision transformer or a VIT. Let's briefly demystify that term because vision transformer backbone sounds like sci -fi jargon. It does sound intense. But a transformer is actually the exact same underlying neural network architecture that powers large language models like chat GPT. OK, so it's a proven technology.
Very proven. It's incredibly good at looking at a sequence of data. like a sequence of words and figuring out which parts of the sequence are most important to pay attention to. A vision transformer, or VIT, takes that exact same mechanism and applies it to images and video. How does it do that with a picture? It chops a video frame up into a grid of little patches, treats each patch like a word, and learns how all the patches relate to each other across space and time. When we say it's the backbone, we just mean it's the core engine doing the X encoding and we encoding. So they have this power cell vision transformer engine and they have 2 million videos. But how do they actually train it? The training method they use is called multi -block masking. And I really want to spend some time on this because when I read the section of the paper I thought it was completely wild. It's a very aggressive training method. To train the model they draw up a massive 90 % of a video clip. They literally black out 90 % of the video and they ask the predictor to guess the abstract representations of the missing 90 % based on seeing only the remaining 10%. It is an incredibly punishing task. Imagine trying to watch a movie which we're looking through a tiny fragmented keyhole that randomly jumps around the screen revealing only 10 % of the picture at any given moment. From that tiny amount of information, you have to understand the entire plot and physics of the scene. That sounds impossible for a human. But the way they mask it is the real secret sauce here. The paper does a fantastic job of ablating, which just means testing and removing different masking strategies to see what works best. Right, because if you mask it the wrong way, the model just cheats. Exactly. For example, if you just randomly drop 90 % of the individual pixels across the screen, like static, it's actually too easy for the model. Why is that easy? Because it can just look at the remaining pixels right next to the missing ones and interpolate the answer.
If a pixel is blue, the missing pixel right next to it is probably also blue. That's called spatial redundancy. It teaches the model about local textures, but it doesn't teach it about the global scene. Oh, I see. Or if you mask out an entire frame of the video, the model can just look at the frame right before it and the frame right after it and average them together. Like the dog was at point A, then it's at point C, so in the masked frame, it must be at point B. It's temporal redundancy. It's too easy. So how do they fix that? The multi -block masking strategy solves this by masking out large continuous blocks across both space and time. They drop huge contiguous chunks of the video for several frames at a time.
So there's no neighboring pixels to cheat from. Exactly. The predictor can't just rely on neighbors and it can't rely on adjacent frames. It forces the network to truly understand the global semantics of the scene, the actual objects, and how they move in order to make accurate predictions in the feature space. Okay, let's talk about the results of this brutal training regime. They took this VJT model, specifically their largest one, which is called a VIT -H16, meaning a huge vision transformer. They trained it only on video using this feature prediction method, and then they froze the backbone. Freezing the backbone is a crucial experimental step. It means they stopped training the core feature extractor.
They didn't allow it to adjust its weights anymore. They basically said, OK, pencil's down, training is over. Let's see how smart your representations actually are. And then what did they do? They attached a tiny, simple classifier on top of the frozen backbone and tested it on standard benchmark data sets. And the numbers it put up were just incredible. It scored 81 .9 % accuracy on a dataset called Kinetics 400 and 72 .2 % on a dataset called Something Something B2. To put those numbers in context for anyone not reading AI leaderboards all day, those are state -of -the -art results for this type of architecture.
But the sheer numbers aren't what's interesting, are they? No, they aren't. The contrast between those two specific datasets is what is so illuminating about what VGP actually learned. Let's compare them. Kinetics 400 is what we call an appearance -based task. It's videos of people doing things. But you can often guess the action just by recognizing the static objects in the frame. If you see a swimming pool and a person in a swimsuit, the action is probably swimming. You don't necessarily need to understand the temporal dynamics of the swimming stroke. A model that only understands static images could do decently well on kinetics.
Right. But something V2 is completely different. Something V2 is specifically designed to be a motion -based task. The categories aren't swimming or playing tennis. What are they? The categories are things like pushing something from left to right or pushing something from right to left or pretending to open a book. The objects themselves don't give away the answer at all. You must understand fine -grained temporal dynamics and the direction of motion to get the right answer. And this is where VJP absolutely crushed the competition. The paper shows it outperformed models like video MAE and OmnimeE. Now, those MAE models, our massive systems, train the old way they are trained to reconstruct the missing pixels. Right, pixel generators. On the motion -heavy something -something V2 data set, VJPR outperformed those pixel reconstruction baselines by up to 6%. It's also way more label -efficient. And this is a huge point for the engineers listening. It trains twice as fast as the massive pixel prediction models. Which makes perfect logical sense when you circle back to Lacoon's original theory from the 2022 paper.
If you do not force the network to waste massive amounts of compute cycles reconstructing the rustling leaves on the trees, the ripples in the puddles, and the exact textures of the carpet, the training is vastly more efficient. It learns the core semantic features, the things that actually matter for understanding physics and motion faster and better. Exactly. Okay, but I have to play the skeptic again here because as I was reading this a nagging question kept popping into my head. If the system is never ever outputting pixels, If it's only ever predicting abstract math vectors, how do the researchers even know it's learning what a dog or a car actually looks like? That's a very fair question. We say it understands the semantics of the scene, but couldn't it just be predicting complex math that happens to correlate with the benchmark test but means absolutely nothing to humanize? How do we know it's not just cheating in some mathematical dimension we can't interpret? Does it actually know what a dog is? That is exactly the question the meta researchers anticipated, and it leads to what I think is the most fascinating and conceptually beautiful experiment in the entire VGPA paper.
Oh, I agree. They designed an offline visualization experiment to prove exact that the model was seeing in its mind's eye. You can see this vividly in Figure 6 of the paper. They took the frozen VJPOM model, which, remember, has never generated a single pixel in its life. It has only ever trafficked in abstract feature vectors. And they trained a separate secondary mechanism called a diffusion decoder. Let's explain what a diffusion decoder is for the audience, because it's a key part of the translator. A diffusion model is the technology behind popular AI image generators, like Mid Journey or Deli. It works by taking a field of pure static noise and slowly refining it step by step until it forms an image guided by a prompt. Okay, so it draws pictures. Right.
In this experiment, the researchers used a diffusion decoder not to generate art from a text prompt, but to act as a translator. The sole job of this decoder was to take VJPA's abstract mathematical predictions and translate them back into interpretable pixels so humans could see them. Okay, so they essentially built a visualization tool to read the VJPA's mind. They show VJPA a video. VJPA creates an abstract thought about what happens next, and the decoder draws a picture of that thought. Exactly. And the setup they used to test it was brilliant. They took a video of a dog running across a grassy field. They put a massive opaque white block right in the middle of the screen, hiding the dog completely as it ran behind the block. Completely hidden. They fed this heavily masked video to VJPA and asked its predictor module to guess the features of what was happening behind the block. Then, they fed those abstract feature predictions into the diffusion decoder to see what VJPA was thinking was behind the mask. And what did the decoder draw? It correctly rendered the dog behind the mask. And it wasn't just a blurry static image of a dog shape. It rendered the consistent existence, the correct position, and the continuous motion of the dog running through the hidden area. That is just... I mean, it's mind -blowing. It proved that VGPA genuinely understands object permanence and spatial relations. It knew there was a dog, it knew the trajectory the dog was traveling, and it knew what the dog looked like even when the dog was completely scrubbed from the visual input by the mask. That's incredible. And the most incredible part is that it learned all of this foundational physics without ever being told to reconstruct a pixel during its training.
It learned it purely by trying to make its internal abstract representations predictable over time using V -CREG. It is an absolute triumph of the predictive feature principle. It shows that by forcing a system to predict the future state of the world in a compressed abstract space, the system naturally, organically learns the physical constraints of reality. It learns that dogs do not cease to exist when they run behind a tree. They continue to move through space and time. VGPA basically built a rudimentary physics engine entirely on its own. It did. So VGPA proves that predicting features works brilliantly. It understands motion. It understands object permanence.
But this brings us to a scale problem. VGPA is dealing with short video clips. It's predicting a few seconds into the future. True human -level intelligence, the kind of autonomous agent Lacun envisions, isn't just knowing what happens in the next two seconds. It's about planning your entire day. It's about taking actions now that pay off hours from now. Exactly. How does this architecture scale up to human -level reasoning over long time horizons? To answer that, we have to return to Lacun's blueprint to look at something called HJPA. Yes, HJPA stands for hierarchical JPA, because you're absolutely right to point out the scale issue. Predicting two seconds into the future is great for mode one reactive behavior or very short -term mode two planning. But if you try to use a single -world model to plan an entire day, the uncertainty compounds way too quickly. So what's the solution? To predict over long time horizons, the architecture must stack multiple JPA models hierarchically. LeCun uses this fantastic analogy in the paper about planning a commute to work. Let's explore this because it perfectly illustrates why hierarchy is mandatory. Let's say my overall goal is to travel from my house to my office in Paris. A nice commute. Not bad, right. If I only have a single low -level world model, a system that predicts the world millisecond by millisecond, how do I plan my commute? I would have to predict every single millisecond muscle twitch of my leg as I walk to the train. I would have to predict the exact angle of the steering wheel for my entire drive. I would have to predict the movement of every single leaf blowing across the road for the next hour to ensure I don't hit anything.
And that is computationally impossible. You would never make it out the front door. The simulation would take longer to run than the commute itself. Because there's just too much data. Right. The uncertainty compounds so fast at the millisecond level that you literally cannot predict what the traffic lights will be doing an hour in advance. The low -level details become noise that totally prevents long -term planning. So instead of predicting everything, we use hierarchical abstraction. We stack the models. The top level of the hierarchy, the highest level JPEG, doesn't care about my leg muscles or the leaves on the road. It just looks at the highest level of abstraction. The state is I am at home. The goal is I am at work.
It predicts that if I take the train in one hour, I will be at my desk. Then it breaks that high level prediction down and passes it to the next level down. Exactly. The top level sets a sub goal. Get to the train station. The mid -level JPA takes that sub -goal and plans the route. Walk down this street, turn left at the bakery. It ignores the millisecond muscle twitches, but it focuses on minute -by -minute navigation. And then it passes it down again. Finally, it passes the immediate instructions down to the lowest level. Only the very lowest level JPA deals with the millisecond muscle controls to move your leg, and that level only operates in the immediate present, dealing with the exact unevenness of the sidewalk right in front of your foot. So the higher you go in the hierarchy, the more abstract the representation becomes, and the more details are intentionally ignored.
And because you are ignoring the details, you can predict much further into the future. This is exactly how HJPA is designed. Reasoning in this system is literally defined as a cascade of energy minimization across these hierarchies. The configurator module sets the high -level goal, which defines the high -level cost. The top -level actor searches through its abstract latent variables to find a sequence of high -level subgoals that lowers that cost. And it passes those subgoals down to the mid -level. Which does its own search to minimize the cost of the subgoal. all the way down to the low -level actions that actually move the robot. And this cascading search for the lowest cost brings us to what is easily the most provocative and speculative section of Lacoon's entire paper. I want to spend some time here because it really blew my mind.
He introduces the idea of machine emotions. Yes. It is a profound philosophical implication of the architecture he's built. When I read this, I was floored. Because historically, in sci -fi and even in AI philosophy, we usually think of emotions as this deeply human, biological, almost spiritual thing that separates us from cold, calculating machines. But Lacoon argues the exact opposite. He argues that in an autonomous agent driven by energy minimization, equivalents of emotions are not just possible, they are an inevitable, unavoidable architectural requirement. Let's trace the logic, because it's airtight. Think about what the cost module does in this architecture.
It measures energy, which is a mathematical representation of discomfort or divergence from a goal. We established that the intrinsic cost module measures immediate discomfort. For a biological organism, a high intrinsic cost is physical pain, hunger, or exhaustion. A low intrinsic cost is pleasure, satiety, or comfort. For a robot, a high intrinsic cost is a depleted battery, a damaged motor, or a system overheating. OK, so the machine experiences a mathematical equivalent of physical pain when its battery is low. But it goes deeper when you add in the trainable critic. The trainable critic, as we discussed, anticipates future costs based on the world model simulations. Right. I was looking at it. So imagine the robot is standing at the top of a steep flight of stairs. The actor proposes moving forward. The world model simulates the robot tumbling down the stairs. The trainable critic looks at that simulation and says, If we do that, we are going to experience massive intrinsic costs like physical damage five seconds from now. So the trainable critic outputs a massive spike of high cost right now in anticipation of the future event. And this is the crux of LeCun's argument. What do we call the sudden acute anticipation of future pain or disaster?
In humans, we call it fear or anxiety. Wow. What do we call the anticipation of a highly successful outcome that minimizes future cost? We call it hope. or elation. Because the agent's behavior is entirely driven by avoiding these anticipated costs, the machine will necessarily exhibit behaviors that look exactly like fear, joy, anxiety, or satisfaction. It's practically biological. If the trainable critic spikes, The robot might freeze, back away, or reroute behaving exactly as an animal does when it experiences fear. This is so wild. The idea is that machine emotions aren't just going to be something we program in as a user interface trick to make robots seem friendly to humans. The argument is that emotions are functionally required for the machine to be autonomous and intelligent. It needs to feel the anticipated cost of its actions to make good plans. If it doesn't feel fear about falling down the stairs, It will just fall down the stairs.
It fundamentally reframes how we view both artificial intelligence and biological intelligence. It strips away the mysticism of human emotion and suggests that our own feelings, our dread, our joy, our panic are just highly evolved cost modules. That's a little existentially terrifying, actually. They are evolutionary mechanisms designed to keep us alive in a chaotic environment by forcing us to anticipate and react to future states. We are just biological HJPAs trying to minimize our own energy costs. OK, let's zoom out. We have covered an immense amount of ground today, unpacking the blueprints for a completely new paradigm of intelligence. We started with the realization that current chatbots, as incredibly impressive as they are at generating text, fundamentally lack the grounded common sense of a toddler.
They don't have a physics engine. And to solve that, we unpacked LeCun's blueprint for an autonomous agent. We saw a shift away from reactive encyclopedic system system one towards systems with a configurable hierarchical world model that can actively reason and plan system two. We dove into the six module architecture. Perception filtering the world, the world model predicting the future, the cost module feeling the pain of those predictions, the actor choosing the safest path, all guided by memory and the configurator. And crucially, we explored how to build that world model. We saw that the key is not generating pixels, which leads to the system getting overwhelmed by noise and informational collapse. The rustling leaves problem. Exactly. The key is JPA, predicting abstract features using techniques like FIREG to ensure those features are meaningful. And then we looked at the hard proof. Mehta's 2024 VJPA paper proved that this works in the real world. A model trained purely to predict the abstract features of heavily masked videos developed a profound, grounded understanding of object permanence, spatial relations, and temporal dynamics, vastly outperforming models that try to reconstruct pixels. It learned that the dog still exists behind the mask. And finally, we scale that all the way up to HJPA. showing how stacking these predictive models creates the capacity for long -term planning, sub -goal decomposition, and the inevitable architectural emergence of machine emotions. So the next time you see an AI do something incredibly smart or, let's be honest, incredibly stupid, I want you to pause. Don't just ask, wow, how did it know what word to say next? Ask yourself a much deeper question. Does this machine just know the statistical likelihood of the next word in a sentence, or does it actually possess a world model of the physical reality it is talking about? Does it understand gravity, or is it just parroting the dictionary? Because until it has that world model, it is just a magic trick without the physics engine. That is the perfect takeaway.
The shift from statistical mimicry to causal predictive world modeling is the defining challenge of the next decade of artificial intelligence. Without a doubt. But I want to leave you with one final deeply provocative thought from the very end of Lacoon's paper. We talked about the configurator module, the executive control that oversees and tunes the one single world model for whatever specific task is at hand. Lacoon suggests a highly speculative idea about the nature of human consciousness itself based on this architecture. It's a huge thought experiment.
He hypothesizes that the human illusion of consciousness, that feeling of a single unified self directing our attention, might simply be the biological side effect of possessing a single configurator -like module in our prefrontal cortex. Because the human brain only has one main world model engine, the configurator has to constantly tune it and retune it for the single task at hand, creating a bottleneck of attention. And we experience that bottleneck as conscious awareness. It's a staggering thought. He suggests that if our biological brains were simply large enough to hold thousands of independent non -configurable world models running in parallel for every possible situation simultaneously, we wouldn't need a configurator to shift our attention. And as a result, we might not experience consciousness at all. We would just perfectly react to everything at once. It's wild to think about. Which leaves us with a profound thought to mull over. As we meticulously map out this cognitive architecture in the lab, as we build these configurators, these cost modules, and these GPays to give machines common sense, are we inadvertently building the very mechanism that makes us self -aware? Think about that the next time you watch a magic trick.
We'll see you on the next Deep Dive.