1
00:00:00,000 --> 00:00:03,980
You know, if you watch a magic trick, say a magician

2
00:00:03,980 --> 00:00:07,059
is holding a coin and they close their hand and

3
00:00:07,059 --> 00:00:09,220
then open it to show the coin has just vanished

4
00:00:09,220 --> 00:00:11,539
into thin air. Right, a classic disappearing

5
00:00:11,539 --> 00:00:14,519
act. Yeah, exactly. There is this very specific

6
00:00:14,519 --> 00:00:17,440
sensation you experience right then. Your brain

7
00:00:17,440 --> 00:00:20,620
just kind of short circuits for a split second.

8
00:00:20,660 --> 00:00:22,480
Yeah, it does. Because you know it's a trick,

9
00:00:22,620 --> 00:00:25,019
right? You aren't actually confused about whether

10
00:00:25,019 --> 00:00:27,660
reality has fundamentally broken. And the reason

11
00:00:27,660 --> 00:00:29,100
you know it's a trick is because you possess

12
00:00:29,100 --> 00:00:31,579
this unshakable model of how physical reality

13
00:00:31,579 --> 00:00:34,039
works. Absolutely. Objects do not just vanish.

14
00:00:34,140 --> 00:00:37,159
Right. They have permanence, they obey gravity,

15
00:00:37,520 --> 00:00:40,200
they occupy three -dimensional space, you know?

16
00:00:40,340 --> 00:00:43,079
It is a direct violation of your internal physics

17
00:00:43,079 --> 00:00:45,659
engine. Yes, a physics engine. Like when that

18
00:00:45,659 --> 00:00:47,920
coin disappears, you don't have to consciously

19
00:00:47,920 --> 00:00:50,240
sit there and calculate the mass of the coin

20
00:00:50,240 --> 00:00:53,859
or the density of the air or the trajectory of

21
00:00:53,859 --> 00:00:56,640
a drop. Your brain just flags the visual input

22
00:00:56,640 --> 00:00:59,119
immediately with this massive error message.

23
00:00:59,179 --> 00:01:02,810
It's like error 404, reality not found. Exactly.

24
00:01:03,270 --> 00:01:05,689
Says that is not how the world operates. Yeah.

25
00:01:05,849 --> 00:01:08,849
You have a baseline expectation of reality and

26
00:01:08,849 --> 00:01:11,650
the magician just violated it on purpose. Right.

27
00:01:11,750 --> 00:01:14,750
And that instinct, that foundational understanding

28
00:01:14,750 --> 00:01:17,230
of reality is something every single one of us

29
00:01:17,230 --> 00:01:18,930
has. We completely take it for granted. I mean,

30
00:01:19,159 --> 00:01:21,739
Toddler has it Oh for sure a cat looking for

31
00:01:21,739 --> 00:01:24,400
a toy under a couch has it but then you step

32
00:01:24,400 --> 00:01:26,359
into the world of artificial intelligence as

33
00:01:26,359 --> 00:01:29,099
it exists right now and Suddenly that foundational

34
00:01:29,099 --> 00:01:31,680
physics engine is just well, it's completely

35
00:01:31,680 --> 00:01:33,959
missing. It's just not there at all We are dealing

36
00:01:33,959 --> 00:01:36,680
with these incredibly articulate systems that

37
00:01:36,680 --> 00:01:40,640
can write a passable sonnet or you know code

38
00:01:40,640 --> 00:01:43,219
a Python script in seconds. But if you ask them

39
00:01:43,219 --> 00:01:45,819
about the physical mechanics of stacking a heavy

40
00:01:45,819 --> 00:01:49,280
wooden box on top of a fragile raw egg, they

41
00:01:49,280 --> 00:01:51,060
don't actually know what happens in the real

42
00:01:51,060 --> 00:01:53,439
world. No, they don't. They don't wince at the

43
00:01:53,439 --> 00:01:55,540
idea of the egg breaking. They don't wince because

44
00:01:55,540 --> 00:01:58,200
they don't have a model of an egg, or a box,

45
00:01:58,859 --> 00:02:01,420
or gravity for that matter. It is the absolute

46
00:02:01,420 --> 00:02:04,260
definition of a shallow understanding. Shallow

47
00:02:04,260 --> 00:02:06,260
rights. When a large language model tells you

48
00:02:06,260 --> 00:02:09,020
the egg will break, it isn't running an internal

49
00:02:09,020 --> 00:02:11,319
simulation of fragile shells and heavy masses.

50
00:02:11,819 --> 00:02:14,419
It's essentially this brilliant scholar who has

51
00:02:14,419 --> 00:02:16,659
been locked in a dark room their entire life.

52
00:02:16,840 --> 00:02:19,340
Oh, I like that analogy. Right. just reading

53
00:02:19,340 --> 00:02:22,300
every book ever written, but having never actually

54
00:02:22,300 --> 00:02:24,340
bumped their shin on a coffee table. Yeah, they

55
00:02:24,340 --> 00:02:26,620
have no physical context. Exactly. They know

56
00:02:26,620 --> 00:02:28,659
the word pain, they know the word gravity, but

57
00:02:28,659 --> 00:02:31,620
they have zero experiential grounding for what

58
00:02:31,620 --> 00:02:33,939
those words physically entail in the real world.

59
00:02:34,039 --> 00:02:36,159
Well, welcome to this deep dive. Today, we are

60
00:02:36,159 --> 00:02:39,080
looking at how to actually build that missing

61
00:02:39,080 --> 00:02:41,960
physics engine. We are basically opening up the

62
00:02:41,960 --> 00:02:44,599
blueprints for a completely new kind of digital

63
00:02:44,599 --> 00:02:47,530
brain. That's a huge shift. It really is. And

64
00:02:47,530 --> 00:02:49,569
our sources for this are coming straight from

65
00:02:49,569 --> 00:02:52,590
the absolute cutting edge. We're looking at a

66
00:02:52,590 --> 00:02:56,349
visionary 2022 position paper by Yann LeCun,

67
00:02:56,729 --> 00:03:00,090
who is the chief AI scientist at Metta. It's

68
00:03:00,090 --> 00:03:03,370
titled A Path Towards Autonomous Machine Intelligence.

69
00:03:03,629 --> 00:03:06,550
A very ambitious title. Yeah, no kidding. And

70
00:03:06,550 --> 00:03:09,090
to see if his wild theories actually hold water,

71
00:03:09,449 --> 00:03:12,289
we are pairing it with a 2024 empirical paper,

72
00:03:12,370 --> 00:03:15,310
also from Metta, exploring something called VJPA.

73
00:03:15,550 --> 00:03:18,750
Which is super exciting. It is because that second

74
00:03:18,750 --> 00:03:21,469
paper is the real -world application of these

75
00:03:21,469 --> 00:03:24,389
exact theories to video representation. Right.

76
00:03:24,949 --> 00:03:27,909
Our mission today is to meticulously map out

77
00:03:27,909 --> 00:03:30,629
this radical new architecture for you. We are

78
00:03:30,629 --> 00:03:33,729
going to decode the philosophy behind how machines

79
00:03:33,729 --> 00:03:36,189
can actually learn the way animals do. Which

80
00:03:36,189 --> 00:03:39,110
is vastly different from how current AI learns,

81
00:03:39,330 --> 00:03:41,710
right? Totally different. We'll unpack the specific,

82
00:03:41,870 --> 00:03:44,509
module -by -module cognitive architecture required

83
00:03:44,509 --> 00:03:47,490
to build a system with actual common sense. And

84
00:03:47,490 --> 00:03:49,030
then we're going to look at the hard proof that

85
00:03:49,030 --> 00:03:51,830
this new path to intelligence is already working

86
00:03:51,830 --> 00:03:54,009
in the lab. So this isn't just about making a

87
00:03:54,009 --> 00:03:56,030
better chat bot to write your emails. Not at

88
00:03:56,030 --> 00:03:58,210
all. It's about a massive pivot in the entire

89
00:03:58,210 --> 00:04:01,069
field. moving away from systems that just memorize

90
00:04:01,069 --> 00:04:03,389
statistical patterns and moving toward systems

91
00:04:03,389 --> 00:04:05,949
that truly understand the underlying causal mechanisms

92
00:04:05,949 --> 00:04:08,289
of the physical world. Okay, let's start with

93
00:04:08,289 --> 00:04:10,789
the philosophy of intelligence, which Lacoon

94
00:04:10,789 --> 00:04:13,610
tackles right at the beginning of his 2022 paper.

95
00:04:14,250 --> 00:04:16,569
He points out something that is incredibly obvious

96
00:04:16,569 --> 00:04:20,170
but, uh, I guess often ignored in computer science.

97
00:04:20,730 --> 00:04:24,350
Animals, and obviously humans, learn incredibly

98
00:04:24,350 --> 00:04:27,329
efficiently. So efficiently. And we do it mostly

99
00:04:27,329 --> 00:04:29,370
just by passively observing our environment.

100
00:04:29,829 --> 00:04:32,290
Like, in the first few months of life, a human

101
00:04:32,290 --> 00:04:35,629
infant is just soaking in data, staring at things,

102
00:04:35,889 --> 00:04:37,850
dropping things from their high chair. Over and

103
00:04:37,850 --> 00:04:40,949
over again. Right. They are acquiring this massive

104
00:04:40,949 --> 00:04:43,410
background knowledge about physical constraints.

105
00:04:43,589 --> 00:04:45,490
And efficiency really is the key word there.

106
00:04:45,829 --> 00:04:47,790
Think about how biological learning compares

107
00:04:47,790 --> 00:04:50,579
to standard machine learning today. An infant

108
00:04:50,579 --> 00:04:52,620
doesn't need to drop a million different objects

109
00:04:52,620 --> 00:04:55,040
off a high chair to learn about the concept of

110
00:04:55,040 --> 00:04:56,879
gravity. Right, they don't need a massive data

111
00:04:56,879 --> 00:04:59,660
set of falling cups. Exactly. They observe a

112
00:04:59,660 --> 00:05:02,480
toy falling a few times and they rapidly build

113
00:05:02,480 --> 00:05:05,040
an internal model that says, okay, unsupported

114
00:05:05,040 --> 00:05:07,819
things move downward. That accumulated background

115
00:05:07,819 --> 00:05:10,100
knowledge, that intuitive physics is what we

116
00:05:10,100 --> 00:05:12,639
call common sense. Common sense, right. It tells

117
00:05:12,639 --> 00:05:15,959
an agent what is likely to happen, what is plausible,

118
00:05:16,379 --> 00:05:19,519
and what is utterly impossible. And current AI

119
00:05:19,519 --> 00:05:23,040
just does not have this efficiency at all. Let's

120
00:05:23,040 --> 00:05:25,319
take reinforcement learning, for example. That's

121
00:05:25,319 --> 00:05:27,579
how we train AI to play chess or video games.

122
00:05:27,699 --> 00:05:29,480
Yeah, that's the standard method. The system

123
00:05:29,480 --> 00:05:31,959
tries something, gets a score, tries again, gets

124
00:05:31,959 --> 00:05:34,720
a better score. And over millions and millions

125
00:05:34,720 --> 00:05:38,240
of trials, it learns the optimal moves. But Lacoon

126
00:05:38,240 --> 00:05:40,480
points out a massive flaw with this when you

127
00:05:40,480 --> 00:05:42,810
try to bring it into the real world. A fatal

128
00:05:42,810 --> 00:05:46,250
flaw, really. Real -world interactions are expensive.

129
00:05:46,670 --> 00:05:49,870
They are slow, and they are incredibly dangerous.

130
00:05:50,050 --> 00:05:52,370
You can't put a robot in a physical kitchen and

131
00:05:52,370 --> 00:05:54,470
let it learn how to chop vegetables by trial

132
00:05:54,470 --> 00:05:57,069
and error. You absolutely cannot. In a video

133
00:05:57,069 --> 00:05:59,470
game, if the AI makes a mistake, the character

134
00:05:59,470 --> 00:06:02,089
dies, the simulation resets, and it tries again

135
00:06:02,089 --> 00:06:04,269
a millisecond later. No harm, no foul. Right.

136
00:06:04,389 --> 00:06:06,750
But in a physical kitchen, if a robot learning

137
00:06:06,750 --> 00:06:09,129
by trial and error moves its arm too fast, it

138
00:06:09,129 --> 00:06:11,240
destroys the kitchen counter. breaks the knife,

139
00:06:11,560 --> 00:06:13,620
and probably shatters its own motor. Yeah, the

140
00:06:13,620 --> 00:06:16,160
penalty for a wrong action is just way too high.

141
00:06:16,500 --> 00:06:19,220
You cannot afford to learn that jumping off a

142
00:06:19,220 --> 00:06:22,000
cliff is bad by jumping off a cliff. You'd only

143
00:06:22,000 --> 00:06:24,759
learn it once. Exactly. You have to be able to

144
00:06:24,759 --> 00:06:27,279
predict the fatal mistake before you make it.

145
00:06:27,319 --> 00:06:30,160
So the system has to have a way to simulate reality

146
00:06:30,160 --> 00:06:33,860
internally before taking action externally. Yes.

147
00:06:34,199 --> 00:06:36,439
And that brings us to the core concept of a world

148
00:06:36,439 --> 00:06:39,670
model. Now, to be clear, this isn't a brand new

149
00:06:39,670 --> 00:06:41,649
idea in the grand scheme of science. All right,

150
00:06:41,810 --> 00:06:44,329
it goes way back. The psychologist Kenneth Craig

151
00:06:44,329 --> 00:06:46,850
actually proposed the idea of internal models

152
00:06:46,850 --> 00:06:50,990
and human cognition way back in 1943. Wow, 1943.

153
00:06:51,230 --> 00:06:54,230
Yeah. He argued that the nervous system is essentially

154
00:06:54,230 --> 00:06:57,370
a calculating machine capable of modeling external

155
00:06:57,370 --> 00:07:00,560
events. But taking that psychological concept

156
00:07:00,560 --> 00:07:03,279
and actually applying it to modern, differentiable

157
00:07:03,279 --> 00:07:05,879
machine learning that is the frontier Lacan is

158
00:07:05,879 --> 00:07:08,060
exploring here. Okay, so an intelligent agent

159
00:07:08,060 --> 00:07:10,360
must have an internal simulator of the world

160
00:07:10,360 --> 00:07:12,720
to predict the consequences of actions it has

161
00:07:12,720 --> 00:07:14,600
in that take -in. That is the whole ballgame.

162
00:07:14,759 --> 00:07:16,579
Okay, I have to stop and play devil's advocate

163
00:07:16,579 --> 00:07:19,519
for a second, because I use large language models,

164
00:07:19,620 --> 00:07:21,980
you know, chat bots, every single day, and if

165
00:07:21,980 --> 00:07:23,920
you talk to them, they really do seem to have

166
00:07:23,920 --> 00:07:25,819
common sense. They do a very good impression

167
00:07:25,819 --> 00:07:28,980
of it, yes. Right. If I type into a prompt, what

168
00:07:28,980 --> 00:07:31,259
happens if I place a hot iron flat on a silk

169
00:07:31,259 --> 00:07:34,279
shirt and leave it there for an hour? The AI

170
00:07:34,279 --> 00:07:36,259
will immediately tell me the shirt will burn.

171
00:07:36,399 --> 00:07:38,600
It will catch fire. It's a terrible idea. Of

172
00:07:38,600 --> 00:07:41,180
course it will. It can answer incredibly complex

173
00:07:41,180 --> 00:07:43,959
physical, logical and even emotional questions.

174
00:07:44,459 --> 00:07:47,180
So why isn't that enough? Why do we say it lacks

175
00:07:47,180 --> 00:07:49,699
common sense if it consistently gives me the

176
00:07:49,699 --> 00:07:53,220
exact right answer about physical reality? This

177
00:07:53,220 --> 00:07:55,620
is perhaps the biggest point of confusion in

178
00:07:55,620 --> 00:07:58,220
the public discourse about AI right now. So it

179
00:07:58,220 --> 00:08:01,879
is vital we unpack it. Lacoon argues that the

180
00:08:01,879 --> 00:08:04,500
knowledge of language model possesses is fundamentally

181
00:08:04,500 --> 00:08:07,279
ungrounded. Ungrounded. OK. Yes, it can tell

182
00:08:07,279 --> 00:08:10,399
you the silk shirt will burn. But it is not telling

183
00:08:10,399 --> 00:08:13,019
you that because it has an internal simulation

184
00:08:13,019 --> 00:08:15,500
of heat transfer, material flammability, and

185
00:08:15,500 --> 00:08:17,439
the passage of time. It's not running a physics

186
00:08:17,439 --> 00:08:20,180
calculation in the background. Not at all. It

187
00:08:20,180 --> 00:08:22,459
tells you that because the words hot iron, silk,

188
00:08:22,579 --> 00:08:25,660
and burn appear in close statistical proximity

189
00:08:25,660 --> 00:08:28,420
across the billions of web pages it ingested

190
00:08:28,420 --> 00:08:31,629
during training. Ah, I see. So it's like trying

191
00:08:31,629 --> 00:08:34,210
to learn how to ride a bicycle by reading the

192
00:08:34,210 --> 00:08:37,210
dictionary definition of the word pedal and the

193
00:08:37,210 --> 00:08:39,730
word balance. That is a brilliant way to put

194
00:08:39,730 --> 00:08:41,570
it. You might be able to write a beautiful essay

195
00:08:41,570 --> 00:08:43,470
about how a bicycle works, but the second you

196
00:08:43,470 --> 00:08:45,669
actually sit on one, you're going to fall right

197
00:08:45,669 --> 00:08:48,870
over. You don't have the grounded physical intuition.

198
00:08:49,450 --> 00:08:52,309
Exactly. Human common sense isn't derived from

199
00:08:52,309 --> 00:08:55,370
reading text. It's built from interacting with

200
00:08:55,370 --> 00:08:58,909
a physical reality. True intelligence requires

201
00:08:58,909 --> 00:09:01,610
a model of that reality, not just statistical

202
00:09:01,610 --> 00:09:03,629
word associations. That makes a lot of sense.

203
00:09:03,809 --> 00:09:05,669
Think about what happens if you give a language

204
00:09:05,669 --> 00:09:09,250
model a completely novel physical puzzle. Something

205
00:09:09,250 --> 00:09:12,049
bizarre that has never been explicitly described

206
00:09:12,049 --> 00:09:14,149
in any text on the internet. Like some weird

207
00:09:14,149 --> 00:09:16,879
Rube Goldberg machine scenario. Yeah, exactly.

208
00:09:17,039 --> 00:09:19,320
It often fails completely or it hallucinates

209
00:09:19,320 --> 00:09:22,080
a nonsensical answer because it has no internal

210
00:09:22,080 --> 00:09:24,779
physics engine to fall back on. It cannot run

211
00:09:24,779 --> 00:09:27,740
a mental simulation of a novel scenario. It can

212
00:09:27,740 --> 00:09:30,320
only search for text patterns that look like

213
00:09:30,320 --> 00:09:32,139
they might be the answer. It's the difference

214
00:09:32,139 --> 00:09:35,379
between memorizing the answer key to a physics

215
00:09:35,379 --> 00:09:38,059
test and actually understanding the laws of physics.

216
00:09:38,399 --> 00:09:40,460
Precisely. If the problem isn't on the answer

217
00:09:40,460 --> 00:09:43,740
key, the memorizer is totally lost. But the person

218
00:09:43,740 --> 00:09:45,919
who understands physics? can solve a problem

219
00:09:45,919 --> 00:09:48,100
they've never seen before. And that ability to

220
00:09:48,100 --> 00:09:50,879
solve the unseen problem is why a world model

221
00:09:50,879 --> 00:09:54,019
is the missing link. Intelligent agents must

222
00:09:54,019 --> 00:09:55,960
be able to fill in the blanks of what they can't

223
00:09:55,960 --> 00:09:58,259
currently observe and predict the outcomes of

224
00:09:58,259 --> 00:10:01,240
actions in completely novel situations. So we

225
00:10:01,240 --> 00:10:03,799
need an AI that doesn't just read about the world

226
00:10:03,799 --> 00:10:06,899
but actually models it. But how do you build

227
00:10:06,899 --> 00:10:09,440
a digital physics engine? Like, where does this

228
00:10:09,440 --> 00:10:12,320
world model actually live inside the computer

229
00:10:12,320 --> 00:10:14,389
code? This is where Lacoon's paper gets really

230
00:10:14,389 --> 00:10:16,190
exciting. Because he doesn't just philosophize,

231
00:10:16,330 --> 00:10:19,070
right? He lays out a highly specific architectural

232
00:10:19,070 --> 00:10:22,830
blueprint. He does. He proposes a cognitive architecture

233
00:10:22,830 --> 00:10:26,529
made of six distinct modules. And the key phrase

234
00:10:26,529 --> 00:10:28,970
he uses to describe all of them is that they

235
00:10:28,970 --> 00:10:32,320
are differentiable. We should pause on that word

236
00:10:32,320 --> 00:10:34,879
because differentiable is the magic property

237
00:10:34,879 --> 00:10:37,399
that makes modern deep learning possible. It

238
00:10:37,399 --> 00:10:39,379
really is the secret sauce of the whole field.

239
00:10:39,559 --> 00:10:42,080
Let's unpack that for a second. For anyone who

240
00:10:42,080 --> 00:10:44,320
isn't a machine learning engineer, what does

241
00:10:44,320 --> 00:10:46,840
it actually mean for a module to be differentiable,

242
00:10:47,000 --> 00:10:49,279
and why is it mandatory for this architecture?

243
00:10:49,720 --> 00:10:51,840
Okay, imagine you are blindfolded, standing on

244
00:10:51,840 --> 00:10:54,220
the side of a mountain, and your goal is to get

245
00:10:54,220 --> 00:10:56,960
to the very bottom of the valley. Okay, blindfolded

246
00:10:56,960 --> 00:10:59,639
on a mountain. Got it. Because you can't see,

247
00:10:59,820 --> 00:11:02,610
you just feel the ground with your foot. If the

248
00:11:02,610 --> 00:11:04,870
ground slopes down to the left, you take a step

249
00:11:04,870 --> 00:11:08,090
left. You are calculating the gradient, or the

250
00:11:08,090 --> 00:11:10,769
slope, of the terrain and moving in the direction

251
00:11:10,769 --> 00:11:12,750
that takes you lower. You're just following the

252
00:11:12,750 --> 00:11:15,350
downward slope. Right. In machine learning, the

253
00:11:15,350 --> 00:11:18,190
valley is the lowest possible error in the system's

254
00:11:18,190 --> 00:11:21,269
performance. If a system is differentiable, it

255
00:11:21,269 --> 00:11:23,470
means we can mathematically calculate the slope

256
00:11:23,470 --> 00:11:25,960
of its errors. So it can feel its way down the

257
00:11:25,960 --> 00:11:28,519
mountain. Exactly. We can feed data through the

258
00:11:28,519 --> 00:11:31,100
whole architecture, see how wrong its prediction

259
00:11:31,100 --> 00:11:33,860
was at the end, and then send a signal backwards

260
00:11:33,860 --> 00:11:36,080
to all the modules saying, adjust your internal

261
00:11:36,080 --> 00:11:39,019
settings just a tiny bit in this specific direction

262
00:11:39,019 --> 00:11:42,039
to lower the error next time. OK, so if the modules

263
00:11:42,039 --> 00:11:44,620
weren't differentiable, if they were just rigid

264
00:11:44,620 --> 00:11:48,299
old school computer code with rigid if -then

265
00:11:48,299 --> 00:11:50,480
rules they couldn't learn. Right, there'd be

266
00:11:50,480 --> 00:11:53,100
no slope to feel. But because they are differentiable,

267
00:11:53,399 --> 00:11:56,259
the entire six -module system can organically

268
00:11:56,259 --> 00:11:59,700
adjust and teach itself just by experiencing

269
00:11:59,700 --> 00:12:02,960
data and minimizing its errors. Yes. The whole

270
00:12:02,960 --> 00:12:05,240
system can be trained end -to -end. So let's

271
00:12:05,240 --> 00:12:07,299
actually walk through what these six differentiable

272
00:12:07,299 --> 00:12:09,799
pieces do and how they interact. Let's do it.

273
00:12:09,980 --> 00:12:12,340
The first one is the perception module. Its job

274
00:12:12,340 --> 00:12:14,580
is to take in the raw signals from the sensors.

275
00:12:15,059 --> 00:12:17,460
For a robot, this would be carapixels, microphone,

276
00:12:17,639 --> 00:12:20,320
audio, maybe tactile sensors. Or for a digital

277
00:12:20,320 --> 00:12:23,360
agent, it might just be raw data streams. Exactly.

278
00:12:23,980 --> 00:12:26,299
But its critical function is not just to ingest

279
00:12:26,299 --> 00:12:29,759
data. It is to estimate the current state of

280
00:12:29,759 --> 00:12:32,500
the world by extracting only what is relevant.

281
00:12:32,809 --> 00:12:35,529
to the task at hand. So it's a filter. A very

282
00:12:35,529 --> 00:12:38,090
powerful filter. Like if my task is driving a

283
00:12:38,090 --> 00:12:41,090
car, my perception module needs to extract the

284
00:12:41,090 --> 00:12:43,629
positions of other cars, the state of the stop

285
00:12:43,629 --> 00:12:45,649
lights, the pedestrians on the corner. Right,

286
00:12:45,809 --> 00:12:47,850
the stuff that matter. And it explicitly needs

287
00:12:47,850 --> 00:12:50,490
to ignore the exact shape of the clouds in the

288
00:12:50,490 --> 00:12:53,970
sky or the specific brand of shoes a pedestrian

289
00:12:53,970 --> 00:12:56,809
is wearing because that information is totally

290
00:12:56,809 --> 00:12:59,450
useless for the task of driving. If I tried to

291
00:12:59,450 --> 00:13:01,429
process all of that, my brain would just overload.

292
00:13:01,690 --> 00:13:03,870
That filtering is essential. If the agent tries

293
00:13:03,870 --> 00:13:06,450
to model every single molecule in its environment,

294
00:13:06,529 --> 00:13:09,289
it will fail. So the perception module creates

295
00:13:09,289 --> 00:13:11,990
a filtered, compressed representation of the

296
00:13:11,990 --> 00:13:15,409
current state of the world. Then it passes that

297
00:13:15,409 --> 00:13:18,070
filtered representation to the absolute heart

298
00:13:18,070 --> 00:13:20,309
of the system. The world model. The simulator,

299
00:13:20,529 --> 00:13:23,269
yes. The world model has a two -fold job. First,

300
00:13:23,690 --> 00:13:25,710
it estimates missing information about the current

301
00:13:25,710 --> 00:13:27,960
state. It basically fills in the blanks. What

302
00:13:27,960 --> 00:13:30,139
do you mean by filling in the blanks? If you

303
00:13:30,139 --> 00:13:32,779
are looking at a table and a laptop is obscuring

304
00:13:32,779 --> 00:13:35,559
part of a coffee mug, your world model tells

305
00:13:35,559 --> 00:13:37,879
you the rest of the mug exists behind the laptop.

306
00:13:38,259 --> 00:13:40,440
It doesn't just assume the mug is sliced in half.

307
00:13:40,580 --> 00:13:42,659
Oh, right. Object permanence again. Exactly.

308
00:13:42,919 --> 00:13:45,980
And second, and most importantly, it predicts

309
00:13:45,980 --> 00:13:48,019
plausible future states of the world based on

310
00:13:48,019 --> 00:13:50,600
proposed actions. Okay, so the world model is

311
00:13:50,600 --> 00:13:53,370
spinning up these simulations. If I turn the

312
00:13:53,370 --> 00:13:56,210
steering wheel left, the car moves left into

313
00:13:56,210 --> 00:13:59,029
that lane. But how does the system know if moving

314
00:13:59,029 --> 00:14:01,649
into that lane is a good idea or a terrible idea?

315
00:14:02,029 --> 00:14:04,309
Like, what if there's a truck there? That evaluation

316
00:14:04,309 --> 00:14:06,750
is handled by the third module, which is the

317
00:14:06,750 --> 00:14:09,710
cost module. The cost module computes a single

318
00:14:09,710 --> 00:14:12,129
mathematical value, which LeCun calls energy.

319
00:14:12,710 --> 00:14:14,809
But you can think of it intuitively as discomfort

320
00:14:14,809 --> 00:14:18,350
or pain, or basically just divergence from a

321
00:14:18,350 --> 00:14:21,179
goal. OK, so it's a pain meter. The agent's entire

322
00:14:21,179 --> 00:14:23,460
existence is built around minimizing this cost.

323
00:14:24,059 --> 00:14:26,179
Now, the cost module is actually split into two

324
00:14:26,179 --> 00:14:28,259
distinct subparts. First, you have the intrinsic

325
00:14:28,259 --> 00:14:31,120
cost. What's that? These are the immutable, hardwired

326
00:14:31,120 --> 00:14:35,000
biological drives. For a human, intrinsic costs

327
00:14:35,000 --> 00:14:38,600
are physical pain, hunger, extreme cold. For

328
00:14:38,600 --> 00:14:41,580
a robot, intrinsic costs might be battery depletion,

329
00:14:42,179 --> 00:14:44,480
extreme heat in its processors, or excessive

330
00:14:44,480 --> 00:14:46,879
physical force on its mechanical joints. It's

331
00:14:46,879 --> 00:14:49,620
the built -in alarm system, the factory settings

332
00:14:49,620 --> 00:14:51,960
that say, under no circumstances should you let

333
00:14:51,960 --> 00:14:54,120
the battery hit zero and please do not crush

334
00:14:54,120 --> 00:14:57,019
your own arm. Yes, exactly. It measures the pain

335
00:14:57,019 --> 00:14:59,720
of the current state. But there's a second part

336
00:14:59,720 --> 00:15:02,139
to the cost module, which is the trainable critic.

337
00:15:02,299 --> 00:15:05,220
The trainable critic. That sounds ominous. It's

338
00:15:05,220 --> 00:15:07,500
crucial, actually. While the intrinsic cost tells

339
00:15:07,500 --> 00:15:10,080
you if you are currently in pain, the trainable

340
00:15:10,080 --> 00:15:13,600
critic learns to anticipate future costs. Ah,

341
00:15:13,659 --> 00:15:15,399
so it looks ahead. Right. It looks at the current

342
00:15:15,399 --> 00:15:17,360
state and says, wait, we aren't in pain right

343
00:15:17,360 --> 00:15:19,639
now, but I recognize this situation. Yeah. The

344
00:15:19,639 --> 00:15:21,720
last time we were in this state, it led to a

345
00:15:21,720 --> 00:15:24,039
massive spike in intrinsic costs a few seconds

346
00:15:24,039 --> 00:15:26,519
later. It learns to associate certain states

347
00:15:26,519 --> 00:15:29,620
with impending disaster or success. I love this.

348
00:15:29,759 --> 00:15:33,139
So the perception module sees the world. The

349
00:15:33,139 --> 00:15:35,320
world model predicts what will happen if we take

350
00:15:35,320 --> 00:15:37,740
an action. The cost module looks at that prediction

351
00:15:37,740 --> 00:15:41,159
and screams, no, that future hurts. That is literally

352
00:15:41,159 --> 00:15:43,460
the workflow. But what is actually deciding on

353
00:15:43,460 --> 00:15:45,100
the action? What is the steering wheel here?

354
00:15:45,220 --> 00:15:47,639
That is the fourth module, the actor. The actor.

355
00:15:47,980 --> 00:15:50,580
The actor is the module that proposes sequences

356
00:15:50,580 --> 00:15:54,500
of actions. It says, what if I do A, then B,

357
00:15:54,620 --> 00:15:58,100
then C? It feeds that proposed sequence to the

358
00:15:58,100 --> 00:16:00,899
world model. The world model simulates the outcomes.

359
00:16:01,480 --> 00:16:03,980
The cost module evaluates how painful those outcomes

360
00:16:03,980 --> 00:16:06,759
are. And then what? The actor then searches through

361
00:16:06,759 --> 00:16:08,919
various different proposals until it finds the

362
00:16:08,919 --> 00:16:11,039
sequence of actions that results in the lowest

363
00:16:11,039 --> 00:16:14,490
possible cost. Once it finds that optimal sequence,

364
00:16:14,909 --> 00:16:17,389
it sends the very first action of that sequence

365
00:16:17,389 --> 00:16:20,649
out to the real world, to the robot's motors,

366
00:16:20,690 --> 00:16:22,450
for example. To keep all of this straight, there

367
00:16:22,450 --> 00:16:24,750
has to be some kind of scratch pad, right? Like

368
00:16:24,750 --> 00:16:27,330
a place to hold these thoughts. Yes, that's the

369
00:16:27,330 --> 00:16:29,809
fifth module, the short -term memory. It simply

370
00:16:29,809 --> 00:16:32,009
keeps track of the current state. the predicted

371
00:16:32,009 --> 00:16:34,970
future states, and the associated costs. So the

372
00:16:34,970 --> 00:16:36,750
actor and world model have a reference point

373
00:16:36,750 --> 00:16:39,190
as they run these simulations. It's the agent's

374
00:16:39,190 --> 00:16:41,970
working memory. Exactly. And finally, the boss

375
00:16:41,970 --> 00:16:45,470
of the whole operation, the sixth module. The

376
00:16:45,470 --> 00:16:47,450
configurator. Think of this as the executive

377
00:16:47,450 --> 00:16:50,230
control. It takes the overall overarching goal,

378
00:16:50,470 --> 00:16:53,590
say, navigate to the kitchen, and it primes all

379
00:16:53,590 --> 00:16:55,730
the other modules to achieve that specific goal.

380
00:16:56,029 --> 00:16:58,159
How does it prime them? It tells the perception

381
00:16:58,159 --> 00:17:00,799
model what to pay attention to. It tunes the

382
00:17:00,799 --> 00:17:03,000
world model to pull up the physics of indoor

383
00:17:03,000 --> 00:17:05,799
environments. And it adjusts the parameters of

384
00:17:05,799 --> 00:17:08,319
the cost module so the system knows that getting

385
00:17:08,319 --> 00:17:10,940
closer to the kitchen reduces the cost. OK, this

386
00:17:10,940 --> 00:17:13,740
is a beautiful, intricate system. Perception,

387
00:17:13,839 --> 00:17:16,680
world model, cost, actor, memory, configurator.

388
00:17:17,140 --> 00:17:19,200
But honestly, when I'm going through my daily

389
00:17:19,200 --> 00:17:22,539
life, I am definitely not running intense multi

390
00:17:22,539 --> 00:17:25,079
-step simulations of the future for every single

391
00:17:25,079 --> 00:17:27,470
thing I do. You'd be exhausted if you did. Right.

392
00:17:27,569 --> 00:17:30,089
If someone throws a baseball at my head, I don't

393
00:17:30,089 --> 00:17:32,609
pause, calculate the trajectory of the ball,

394
00:17:32,970 --> 00:17:35,509
run a simulation of it hitting my nose, consult

395
00:17:35,509 --> 00:17:38,130
my cost module to determine that a broken nose

396
00:17:38,130 --> 00:17:40,630
is high cost, and then instruct my actor module

397
00:17:40,630 --> 00:17:43,549
to move my neck muscles. I just duck. And Lacun

398
00:17:43,549 --> 00:17:47,569
addresses this perfectly. He defines two completely

399
00:17:47,569 --> 00:17:49,630
different operating modes for this architecture,

400
00:17:50,130 --> 00:17:52,490
mode one and mode two. Which immediately made

401
00:17:52,490 --> 00:17:55,089
me think of Daniel Kahneman's famous book thinking

402
00:17:55,089 --> 00:17:57,809
fast and slow with system one and system two.

403
00:17:58,289 --> 00:18:00,970
The paper explicitly makes that connection. What

404
00:18:00,970 --> 00:18:02,589
you just described with the baseball is mode

405
00:18:02,589 --> 00:18:05,230
one. Mode one is reactive behavior. So it skips

406
00:18:05,230 --> 00:18:08,210
the simulation entirely. In this mode, the perception

407
00:18:08,210 --> 00:18:10,710
module extracts the state of the world and it

408
00:18:10,710 --> 00:18:12,910
sends that state straight to the actor. The actor

409
00:18:12,910 --> 00:18:15,400
immediately outputs an action. The world model

410
00:18:15,400 --> 00:18:18,440
and the cost module aren't even engaged. It bypasses

411
00:18:18,440 --> 00:18:20,980
the simulator entirely. This is your reflexes

412
00:18:20,980 --> 00:18:22,819
or, you know, it's driving a route you've driven

413
00:18:22,819 --> 00:18:24,619
a thousand times where you were basically on

414
00:18:24,619 --> 00:18:27,680
autopilot, letting muscle memory take over. Correct.

415
00:18:28,180 --> 00:18:30,240
So mode one is fast. It's cheap on computing

416
00:18:30,240 --> 00:18:33,059
power, but it only works for situations you already

417
00:18:33,059 --> 00:18:35,119
know how to handle perfectly. Right. But mode

418
00:18:35,119 --> 00:18:38,059
two is reasoning and planning. This is where

419
00:18:38,059 --> 00:18:40,380
the true intelligence of the world model shines.

420
00:18:40,900 --> 00:18:44,339
In mode two, the actor doesn't just act. It proposes

421
00:18:44,339 --> 00:18:47,160
an action sequence. The world model predicts

422
00:18:47,160 --> 00:18:50,400
the futures. The cost module evaluates. The actor

423
00:18:50,400 --> 00:18:53,880
optimizes. All of this happens internally before

424
00:18:53,880 --> 00:18:56,019
a single physical action is taken in the real

425
00:18:56,019 --> 00:18:58,900
world. So mode two is condom in system two. That's

426
00:18:58,900 --> 00:19:01,900
me trying to parallel park a massive rental SUV

427
00:19:01,900 --> 00:19:05,059
in a tiny spot in a crowded city. The worst feeling.

428
00:19:05,210 --> 00:19:07,789
It is. I am not relying on muscle memory there.

429
00:19:07,950 --> 00:19:10,150
I am sitting there, gripping the wheel, mentally

430
00:19:10,150 --> 00:19:12,089
simulating, OK, if I cut the wheel hard left

431
00:19:12,089 --> 00:19:13,609
right now, my front right bumper is going to

432
00:19:13,609 --> 00:19:15,650
click the car in front of me. High cost. OK,

433
00:19:15,710 --> 00:19:17,829
let's rewind the simulation. I need to reverse

434
00:19:17,829 --> 00:19:19,430
straight back for two more feet, then cut the

435
00:19:19,430 --> 00:19:22,190
wheel left. Yes. You are consciously running

436
00:19:22,190 --> 00:19:25,190
your world model, checking the cost of each hypothetical

437
00:19:25,190 --> 00:19:28,190
move. And taking a step back, this represents

438
00:19:28,190 --> 00:19:30,690
a massive philosophical shift for how we design

439
00:19:30,690 --> 00:19:33,980
AI. In this architecture, complex behavior is

440
00:19:33,980 --> 00:19:36,660
not hard -coded by a human programmer writing

441
00:19:36,660 --> 00:19:39,220
if -then rules. Not at all. Nor is the agent

442
00:19:39,220 --> 00:19:42,099
just randomly trying things to maximize a meaningless

443
00:19:42,099 --> 00:19:45,059
score in a video game. The agent's intelligence,

444
00:19:45,279 --> 00:19:48,380
its reasoning, is driven entirely by energy minimization

445
00:19:48,380 --> 00:19:50,599
finding the path through reality that satisfies

446
00:19:50,599 --> 00:19:53,539
its intrinsic biological or programmed drives.

447
00:19:53,809 --> 00:19:57,750
Reasing itself is literally defined as this internal

448
00:19:57,750 --> 00:20:01,130
simulation and optimization process. It plans

449
00:20:01,130 --> 00:20:04,150
because it has to avoid pain. This sounds flawless

450
00:20:04,150 --> 00:20:06,390
in theory. You have a world model that simulates

451
00:20:06,390 --> 00:20:08,589
the future and an actor that picks the best future.

452
00:20:08,769 --> 00:20:10,670
But we kind of glossed over a really massive

453
00:20:10,670 --> 00:20:13,569
problem. We did the reality check. Right. Predicting

454
00:20:13,569 --> 00:20:16,069
the future is incredibly messy. The real world

455
00:20:16,069 --> 00:20:18,950
is not a clean, sterile chessboard. It is chaotic.

456
00:20:19,369 --> 00:20:21,630
So how exactly do we train a machine's world

457
00:20:21,630 --> 00:20:24,410
model to predict a chaotic world without it getting

458
00:20:24,410 --> 00:20:26,869
totally overwhelmed by unpredictable, irrelevant

459
00:20:26,869 --> 00:20:29,509
details? That is the million -dollar technical

460
00:20:29,509 --> 00:20:31,869
roadblock. For a long time, we just didn't know

461
00:20:31,869 --> 00:20:34,490
how to do it efficiently. And this brings us

462
00:20:34,490 --> 00:20:36,930
to the core technical innovation proposed by

463
00:20:36,930 --> 00:20:39,789
Lacombe, the transition away from predicting

464
00:20:39,789 --> 00:20:42,849
pixels and toward predicting features. Let's

465
00:20:42,849 --> 00:20:45,279
dive deep into this. Because to train a world

466
00:20:45,279 --> 00:20:47,759
model, you have to use a technique called self

467
00:20:47,759 --> 00:20:50,960
-supervised learning, or SSL. Right. The basic

468
00:20:50,960 --> 00:20:53,539
idea is simple. You take a piece of data, you

469
00:20:53,539 --> 00:20:55,920
hide part of it from the machine, and you ask

470
00:20:55,920 --> 00:20:58,599
the machine to predict the hidden part. It's

471
00:20:58,599 --> 00:21:01,099
basically pattern completion. If it guesses wrong,

472
00:21:01,519 --> 00:21:03,759
it adjusts its internal gradients to guess better

473
00:21:03,759 --> 00:21:06,339
next time. That's the basic loop. But LeCun points

474
00:21:06,339 --> 00:21:08,980
out a fatal flaw in how we historically applied

475
00:21:08,980 --> 00:21:12,250
this to images and video. Historically, we used

476
00:21:12,250 --> 00:21:14,750
generative models. If we wanted to teach a model

477
00:21:14,750 --> 00:21:17,130
how physics works in a video, we'd show it frame

478
00:21:17,130 --> 00:21:19,829
one and ask it to generate frame two. And think

479
00:21:19,829 --> 00:21:22,730
about what that computationally demands. If you

480
00:21:22,730 --> 00:21:25,470
ask a system to generate frame two, you are forcing

481
00:21:25,470 --> 00:21:27,750
the machine to predict every single pixel in

482
00:21:27,750 --> 00:21:30,130
the next frame, every color value, every shadow,

483
00:21:30,250 --> 00:21:33,509
every edge. Lacun uses a fantastic analogy in

484
00:21:33,509 --> 00:21:35,670
the paper that really made this click for me.

485
00:21:36,269 --> 00:21:39,029
Imagine you are driving a car down a suburban

486
00:21:39,029 --> 00:21:42,759
street. For your internal world model to be useful

487
00:21:42,759 --> 00:21:45,519
and keep you safe, you need it to predict the

488
00:21:45,519 --> 00:21:48,160
position, the speed, and the trajectory of the

489
00:21:48,160 --> 00:21:50,480
cars around you. Right, the big heavy objects.

490
00:21:50,759 --> 00:21:52,519
You need it to predict that the pedestrian on

491
00:21:52,519 --> 00:21:55,119
the corner might step into the crosswalk. But

492
00:21:55,119 --> 00:21:57,759
you do not want your brain wasting massive amounts

493
00:21:57,759 --> 00:22:00,599
of energy trying to predict the exact rustling

494
00:22:00,599 --> 00:22:02,839
of the leaves on the trees blowing in the wind

495
00:22:02,839 --> 00:22:05,430
by the side of the road. or the specific ripples

496
00:22:05,430 --> 00:22:07,890
on a puddle you're driving past. Exactly. And

497
00:22:07,890 --> 00:22:10,329
that is why generative pixel -predicting models

498
00:22:10,329 --> 00:22:13,089
fail when trying to build complex world models.

499
00:22:13,769 --> 00:22:16,109
A pixel prediction model cannot distinguish between

500
00:22:16,109 --> 00:22:18,410
the important semantic features like the heavy

501
00:22:18,410 --> 00:22:21,650
car moving toward you and the irrelevant, stopastic

502
00:22:21,650 --> 00:22:24,390
details like the leaves rustling. It's all just

503
00:22:24,390 --> 00:22:26,809
pixels to the model. If it is forced to predict

504
00:22:26,809 --> 00:22:29,390
the next frame at the pixel level, it is forced

505
00:22:29,390 --> 00:22:31,490
to predict the leaves. And because the movement

506
00:22:31,490 --> 00:22:34,170
of leaves in the wind is chaotic, random, and

507
00:22:34,170 --> 00:22:37,230
fundamentally unpredictable, the model gets overwhelmed.

508
00:22:37,710 --> 00:22:40,710
It completely crashes its capacity. It dedicates

509
00:22:40,710 --> 00:22:43,369
all of its neural power trying to model the noise,

510
00:22:43,369 --> 00:22:45,589
and it fails to learn the underlying physics

511
00:22:45,589 --> 00:22:48,210
of the actual scene. So it's trying so hard to

512
00:22:48,210 --> 00:22:50,109
draw the exact texture of the tree bark in the

513
00:22:50,109 --> 00:22:52,349
next second that it doesn't notice the Mack truck

514
00:22:52,349 --> 00:22:54,869
barreling toward it. That is exactly what happens.

515
00:22:55,529 --> 00:22:57,809
So Lacune's proposed solution is an entirely

516
00:22:57,809 --> 00:23:01,170
new mathematical structure called JPA. JPA, which

517
00:23:01,170 --> 00:23:03,950
stands for the Joint Embedding Predictive Architecture.

518
00:23:04,289 --> 00:23:06,730
Yes. And the absolute defining characteristic

519
00:23:06,730 --> 00:23:10,369
of JPA is that it is non -generative. Instead

520
00:23:10,369 --> 00:23:12,730
of predicting the actual output, the pixels of

521
00:23:12,730 --> 00:23:16,190
frame two, from the input of frame one, it predicts

522
00:23:16,190 --> 00:23:18,509
the abstract representation of the output. We

523
00:23:18,509 --> 00:23:20,289
need to unpack this carefully because this is

524
00:23:20,289 --> 00:23:23,009
the real paradigm shift. How does JPA actually

525
00:23:23,009 --> 00:23:25,390
work mechanically? Let's break down the architecture.

526
00:23:25,789 --> 00:23:28,960
JPA consists of three main parts. an X encoder,

527
00:23:29,099 --> 00:23:31,859
a Y encoder, and a predictor. Let's say X is

528
00:23:31,859 --> 00:23:34,319
what you observe right now, and Y is what you

529
00:23:34,319 --> 00:23:36,359
want to predict in the future. Okay, so X is

530
00:23:36,359 --> 00:23:39,140
frame one of our driving video, and Y is frame

531
00:23:39,140 --> 00:23:42,200
two of the driving video. Right. The X encoder

532
00:23:42,200 --> 00:23:45,220
process is frame one. But instead of just passing

533
00:23:45,220 --> 00:23:48,220
pixels along, it compresses frame one into a

534
00:23:48,220 --> 00:23:50,980
mathematical abstract representation. It extracts

535
00:23:50,980 --> 00:23:53,210
the features. Okay. At the same time, during

536
00:23:53,210 --> 00:23:56,789
training, the Y encoder processes frame 2, and

537
00:23:56,789 --> 00:23:59,170
it also creates an abstract representation of

538
00:23:59,170 --> 00:24:01,309
frame 2. So they're both working in abstract

539
00:24:01,309 --> 00:24:05,390
space now. Crucially, yes. The predictor module

540
00:24:05,390 --> 00:24:07,710
is then asked to look at the representation of

541
00:24:07,710 --> 00:24:11,829
X and guess the representation of Y. The predictor

542
00:24:11,829 --> 00:24:15,089
is never, ever asked to recreate the pixels of

543
00:24:15,089 --> 00:24:18,269
Y. So going back to the driving analogy, the

544
00:24:18,269 --> 00:24:20,650
X encoder looks at the street scene. And because

545
00:24:20,650 --> 00:24:23,190
it's generating an abstract representation, it

546
00:24:23,190 --> 00:24:25,569
has the freedom to just ignore the rustling leaves

547
00:24:25,569 --> 00:24:27,869
entirely. It literally filters them out. It looks

548
00:24:27,869 --> 00:24:29,769
at the scene and creates a mathematical vector

549
00:24:29,769 --> 00:24:32,269
that basically says there is a large metal object

550
00:24:32,269 --> 00:24:35,089
moving forward at 40 miles per hour. The predictor

551
00:24:35,089 --> 00:24:37,990
then takes that abstract idea and says, OK, based

552
00:24:37,990 --> 00:24:40,509
on the laws of physics, the future representation

553
00:24:40,509 --> 00:24:42,849
of this scene will be large metal object moving

554
00:24:42,849 --> 00:24:45,029
forward at 40 miles per hour, but positioned

555
00:24:45,029 --> 00:24:47,400
10 feet further down the road. It never had to

556
00:24:47,400 --> 00:24:49,660
draw the car, and it never had to draw the leaves.

557
00:24:50,039 --> 00:24:52,000
It just predicted the state of the features.

558
00:24:52,559 --> 00:24:55,579
That is the elegance of it. By doing the prediction

559
00:24:55,579 --> 00:24:58,660
in this abstract representation space, the model

560
00:24:58,660 --> 00:25:01,839
can safely eliminate unpredictable, irrelevant

561
00:25:01,839 --> 00:25:05,180
details. It filters out the noise so it can focus

562
00:25:05,180 --> 00:25:07,980
on the underlying causal dynamics. It's so smart.

563
00:25:08,299 --> 00:25:11,559
It is. But as with everything in machine learning,

564
00:25:11,849 --> 00:25:14,250
Solving one problem creates another. Oh, of course

565
00:25:14,250 --> 00:25:17,269
it does. There is a massive, gaping, technical

566
00:25:17,269 --> 00:25:20,609
trap hidden in this approach. If you design a

567
00:25:20,609 --> 00:25:23,289
system where the only goal is to make the predictor's

568
00:25:23,289 --> 00:25:26,069
output match the encoder's output, the system

569
00:25:26,069 --> 00:25:28,970
will eventually find a way to cheat. It suffers

570
00:25:28,970 --> 00:25:31,750
from what we call informational collapse. Right.

571
00:25:31,769 --> 00:25:33,549
I was reading this and I had this exact thought.

572
00:25:33,789 --> 00:25:36,230
If the model is just a lazy student trying to

573
00:25:36,230 --> 00:25:38,670
minimize its prediction error and it gets to

574
00:25:38,670 --> 00:25:41,289
define its own abstract representations, Why

575
00:25:41,289 --> 00:25:43,750
wouldn't it just output a constant zero for everything?

576
00:25:43,869 --> 00:25:46,089
That is exactly what a lazy neural net does.

577
00:25:46,390 --> 00:25:48,950
Okay. The representation of frame one is zero.

578
00:25:49,329 --> 00:25:51,970
The representation of frame two is zero. My prediction

579
00:25:51,970 --> 00:25:54,890
is zero. Oh, look. My prediction perfectly matches

580
00:25:54,890 --> 00:25:58,450
the reality. Zero error. I'm a genius. It is

581
00:25:58,450 --> 00:26:01,210
a very real, very frustrating problem in self

582
00:26:01,210 --> 00:26:04,309
-supervised learning. If the encoders just output

583
00:26:04,309 --> 00:26:06,730
a constant zero, or if they just ignore the input

584
00:26:06,730 --> 00:26:09,430
entirely and output a static signal, the error

585
00:26:09,430 --> 00:26:12,019
drops to zero. But the system has learned absolutely

586
00:26:12,019 --> 00:26:14,519
nothing about the video. It has collapsed the

587
00:26:14,519 --> 00:26:16,519
information. So how do you fix it? To prevent

588
00:26:16,519 --> 00:26:19,519
this, JEPA relies on specific regularization

589
00:26:19,519 --> 00:26:21,839
techniques to force the encoders to actually

590
00:26:21,839 --> 00:26:24,500
do their jobs. The paper heavily features one

591
00:26:24,500 --> 00:26:26,819
called VYCRAG, which stands for Variance and

592
00:26:26,819 --> 00:26:29,680
Variance Covariance Regularization. OK, that

593
00:26:29,680 --> 00:26:32,119
is a terrifying amount of jargon. Let's break

594
00:26:32,119 --> 00:26:34,660
VYCRAG down into plain English. How does it stop

595
00:26:34,660 --> 00:26:36,960
the lazy student from just outputting zeros?

596
00:26:37,180 --> 00:26:39,880
It sounds intimidating. but the intuition is

597
00:26:39,880 --> 00:26:41,480
actually quite beautiful. It's essentially a

598
00:26:41,480 --> 00:26:43,960
set of strict rules applied to the mathematical

599
00:26:43,960 --> 00:26:46,299
vectors the encoders produce. Let's take them

600
00:26:46,299 --> 00:26:48,819
one by one. First, variance. Okay, what does

601
00:26:48,819 --> 00:26:51,779
variance do? This rule forces the representations

602
00:26:51,779 --> 00:26:54,119
of different images to be spread out or varied

603
00:26:54,119 --> 00:26:57,460
across the mathematical space. If you show the

604
00:26:57,460 --> 00:27:00,480
encoder a dog and then you show it a car, the

605
00:27:00,480 --> 00:27:03,039
variance rule mathematically penalizes the system

606
00:27:03,039 --> 00:27:05,660
if the representation for the dog is too close

607
00:27:05,660 --> 00:27:08,240
to the representation for the car. Ah, so it

608
00:27:08,240 --> 00:27:10,500
can't just group everything together. Right.

609
00:27:10,579 --> 00:27:13,119
It says you cannot output zero for everything.

610
00:27:13,420 --> 00:27:16,599
You must maintain distinct, diverse signals for

611
00:27:16,599 --> 00:27:18,720
different inputs. Okay, so variance prevents

612
00:27:18,720 --> 00:27:21,799
the collapse to a single point. What about invariance?

613
00:27:22,140 --> 00:27:24,339
Invariance is the prediction rule. It simply

614
00:27:24,339 --> 00:27:26,480
says that the output of the predictor, which

615
00:27:26,480 --> 00:27:29,299
is trying to guess the future, must match the

616
00:27:29,299 --> 00:27:32,500
output of the encoder, the actual future. It

617
00:27:32,500 --> 00:27:34,630
forces the prediction to be accurate. Simple

618
00:27:34,630 --> 00:27:36,710
enough, make a good guess. And the last one,

619
00:27:36,930 --> 00:27:40,190
covariance. Covariance is about efficiency. Imagine

620
00:27:40,190 --> 00:27:42,650
an encoder produces a representation with a hundred

621
00:27:42,650 --> 00:27:45,170
different variables. If variable one tells you

622
00:27:45,170 --> 00:27:48,130
the object is a dog and variable two also tells

623
00:27:48,130 --> 00:27:50,390
you the object is a dog, variable two is totally

624
00:27:50,390 --> 00:27:52,369
redundant. It's wasting space. Right, you don't

625
00:27:52,369 --> 00:27:56,269
need to say dog twice. The covariance rule forces

626
00:27:56,269 --> 00:27:59,009
the different variables within a single representation

627
00:27:59,009 --> 00:28:02,230
to be independent of each other. It de -correlates

628
00:28:02,230 --> 00:28:04,849
them. It says, don't use two variables to represent

629
00:28:04,849 --> 00:28:07,450
the same concept. Make every single variable

630
00:28:07,450 --> 00:28:09,930
capture a unique, independent piece of information

631
00:28:09,930 --> 00:28:12,509
about the scene. So PhALIG is this brilliant

632
00:28:12,509 --> 00:28:15,210
balancing act. It forces the system to create

633
00:28:15,210 --> 00:28:17,450
representations that are highly informative and

634
00:28:17,450 --> 00:28:20,109
diverse using variance and covariance, but also

635
00:28:20,109 --> 00:28:23,450
highly predictable using invariance. It forces

636
00:28:23,450 --> 00:28:26,190
the system to actually learn the meaningful features

637
00:28:26,190 --> 00:28:28,990
of the world. Exactly. And there's one more crucial

638
00:28:28,990 --> 00:28:31,230
piece to the JPEG predictor module we need to

639
00:28:31,230 --> 00:28:34,089
discuss, the latent variable, which is usually

640
00:28:34,089 --> 00:28:37,190
denoted by the letter Z. What does Z do? The

641
00:28:37,190 --> 00:28:40,269
latent variable Z handled the fundamental uncertainty

642
00:28:40,269 --> 00:28:42,549
of the real world. Think about it. The world

643
00:28:42,549 --> 00:28:44,910
is not strictly deterministic. No, it's definitely

644
00:28:44,910 --> 00:28:47,230
not. If you are watching a car approach a fork

645
00:28:47,230 --> 00:28:49,990
in the road, there is no single guaranteed future.

646
00:28:50,170 --> 00:28:53,029
The car might go left or the car might go right.

647
00:28:53,569 --> 00:28:56,609
Both are entirely plausible, valid futures based

648
00:28:56,609 --> 00:28:58,930
on the current state. If the predictor is only

649
00:28:58,930 --> 00:29:01,329
allowed to make one single guess, what does it

650
00:29:01,329 --> 00:29:04,549
do? If it guesses left and the car goes right,

651
00:29:04,569 --> 00:29:07,349
it gets penalized. If it tries to average the

652
00:29:07,349 --> 00:29:09,150
two and guess that the car drives straight into

653
00:29:09,150 --> 00:29:11,569
the median, that's completely wrong physically.

654
00:29:12,349 --> 00:29:14,190
It's a lose -lose situation for the model. So

655
00:29:14,190 --> 00:29:17,130
how does Z solve this? The latent variable Z

656
00:29:17,130 --> 00:29:19,900
is essentially a dial that the predictor can

657
00:29:19,900 --> 00:29:23,140
turn to explore different valid futures. It allows

658
00:29:23,140 --> 00:29:25,619
the predictor to say, OK, if the environment's

659
00:29:25,619 --> 00:29:28,119
hidden variable z is equal to 1, then my prediction

660
00:29:28,119 --> 00:29:31,500
is the car goes left. If z is equal to 2, my

661
00:29:31,500 --> 00:29:33,819
prediction is the car goes right. Oh, that's

662
00:29:33,819 --> 00:29:36,160
clever. It allows the world model to represent

663
00:29:36,160 --> 00:29:38,920
a multiplicity of plausible futures simultaneously.

664
00:29:39,359 --> 00:29:41,420
And this is absolutely essential for the motor

665
00:29:41,420 --> 00:29:44,259
2 planning we discussed earlier. The actor module

666
00:29:44,259 --> 00:29:46,420
can plug different Z's values into the world

667
00:29:46,420 --> 00:29:48,880
model to game out different scenarios before

668
00:29:48,880 --> 00:29:51,759
taking action. This whole architecture is staggering.

669
00:29:52,480 --> 00:29:54,859
Predicting abstract concepts rather than exact

670
00:29:54,859 --> 00:29:57,420
pixels, using vis reg to avoid informational

671
00:29:57,420 --> 00:29:59,819
collapse, using latent variables to handle the

672
00:29:59,819 --> 00:30:02,500
chaos of a fork in the road, it makes total logical

673
00:30:02,500 --> 00:30:05,220
sense on paper. It is a very tight theory. But

674
00:30:05,220 --> 00:30:08,460
Lacombe published this manifesto in 2022. For

675
00:30:08,460 --> 00:30:10,880
a couple years, this was just a really elegant

676
00:30:10,880 --> 00:30:13,109
blueprint. Which brings us to the second paper.

677
00:30:13,730 --> 00:30:15,829
Does this conceptual philosophical blueprint

678
00:30:15,829 --> 00:30:18,390
actually work when you write the code and run

679
00:30:18,390 --> 00:30:20,910
it on a supercomputer? That is the big question.

680
00:30:21,309 --> 00:30:25,109
Let's turn to the 2024 meta paper on VJP to see

681
00:30:25,109 --> 00:30:27,809
this theory put to the ultimate test. Yes, moving

682
00:30:27,809 --> 00:30:31,640
from theory to reality. VJP, which stands for

683
00:30:31,640 --> 00:30:35,420
VideoJP, is the empirical validation of Lacoon's

684
00:30:35,420 --> 00:30:38,829
2022 vision. The researchers at Meta set out

685
00:30:38,829 --> 00:30:41,549
to test feature prediction as a standalone objective

686
00:30:41,549 --> 00:30:43,710
for unsupervised learning. And they really went

687
00:30:43,710 --> 00:30:46,269
all in, right? They did. They completely abandoned

688
00:30:46,269 --> 00:30:48,650
pixel reconstruction. No generative modeling

689
00:30:48,650 --> 00:30:51,230
whatsoever. They wanted to see if modern tools

690
00:30:51,230 --> 00:30:53,970
could make JPEG work at scale on complex video

691
00:30:53,970 --> 00:30:55,730
data. And when we say they tested it at scale,

692
00:30:55,849 --> 00:30:58,410
we mean massive scale. They trained this model

693
00:30:58,410 --> 00:31:00,869
on a data set they constructed called VideoMix

694
00:31:00,869 --> 00:31:03,589
2M. This is two million video clips pulled from

695
00:31:03,589 --> 00:31:05,569
various public data sets. It's a huge amount

696
00:31:05,569 --> 00:31:08,589
of dynamic, real world visual data. It's a lot

697
00:31:08,589 --> 00:31:11,960
of data to crunch. And they used a specific architecture

698
00:31:11,960 --> 00:31:15,240
for their encoders called a vision transformer

699
00:31:15,240 --> 00:31:19,400
or a VIT. Let's briefly demystify that term because

700
00:31:19,400 --> 00:31:23,220
vision transformer backbone sounds like sci -fi

701
00:31:23,220 --> 00:31:25,920
jargon. It does sound intense. But a transformer

702
00:31:25,920 --> 00:31:28,980
is actually the exact same underlying neural

703
00:31:28,980 --> 00:31:31,960
network architecture that powers large language

704
00:31:31,960 --> 00:31:35,220
models like chat GPT. OK, so it's a proven technology.

705
00:31:35,480 --> 00:31:37,839
Very proven. It's incredibly good at looking

706
00:31:37,839 --> 00:31:40,170
at a sequence of data. like a sequence of words

707
00:31:40,170 --> 00:31:42,309
and figuring out which parts of the sequence

708
00:31:42,309 --> 00:31:44,690
are most important to pay attention to. A vision

709
00:31:44,690 --> 00:31:48,109
transformer, or VIT, takes that exact same mechanism

710
00:31:48,109 --> 00:31:50,990
and applies it to images and video. How does

711
00:31:50,990 --> 00:31:53,549
it do that with a picture? It chops a video frame

712
00:31:53,549 --> 00:31:56,269
up into a grid of little patches, treats each

713
00:31:56,269 --> 00:31:58,549
patch like a word, and learns how all the patches

714
00:31:58,549 --> 00:32:00,829
relate to each other across space and time. When

715
00:32:00,829 --> 00:32:02,890
we say it's the backbone, we just mean it's the

716
00:32:02,890 --> 00:32:04,950
core engine doing the X encoding and we encoding.

717
00:32:05,180 --> 00:32:07,200
So they have this power cell vision transformer

718
00:32:07,200 --> 00:32:09,900
engine and they have 2 million videos. But how

719
00:32:09,900 --> 00:32:12,079
do they actually train it? The training method

720
00:32:12,079 --> 00:32:14,980
they use is called multi -block masking. And

721
00:32:14,980 --> 00:32:16,660
I really want to spend some time on this because

722
00:32:16,660 --> 00:32:18,480
when I read the section of the paper I thought

723
00:32:18,480 --> 00:32:20,700
it was completely wild. It's a very aggressive

724
00:32:20,700 --> 00:32:23,019
training method. To train the model they draw

725
00:32:23,019 --> 00:32:26,519
up a massive 90 % of a video clip. They literally

726
00:32:26,789 --> 00:32:30,029
black out 90 % of the video and they ask the

727
00:32:30,029 --> 00:32:32,789
predictor to guess the abstract representations

728
00:32:32,789 --> 00:32:35,890
of the missing 90 % based on seeing only the

729
00:32:35,890 --> 00:32:38,490
remaining 10%. It is an incredibly punishing

730
00:32:38,490 --> 00:32:41,630
task. Imagine trying to watch a movie which we're

731
00:32:41,630 --> 00:32:44,609
looking through a tiny fragmented keyhole that

732
00:32:44,609 --> 00:32:47,130
randomly jumps around the screen revealing only

733
00:32:47,130 --> 00:32:49,799
10 % of the picture at any given moment. From

734
00:32:49,799 --> 00:32:51,799
that tiny amount of information, you have to

735
00:32:51,799 --> 00:32:54,500
understand the entire plot and physics of the

736
00:32:54,500 --> 00:32:57,000
scene. That sounds impossible for a human. But

737
00:32:57,000 --> 00:32:59,619
the way they mask it is the real secret sauce

738
00:32:59,619 --> 00:33:02,160
here. The paper does a fantastic job of ablating,

739
00:33:02,299 --> 00:33:04,059
which just means testing and removing different

740
00:33:04,059 --> 00:33:06,660
masking strategies to see what works best. Right,

741
00:33:06,660 --> 00:33:08,640
because if you mask it the wrong way, the model

742
00:33:08,640 --> 00:33:12,059
just cheats. Exactly. For example, if you just

743
00:33:12,059 --> 00:33:14,960
randomly drop 90 % of the individual pixels across

744
00:33:14,960 --> 00:33:18,180
the screen, like static, it's actually too easy

745
00:33:18,180 --> 00:33:20,740
for the model. Why is that easy? Because it can

746
00:33:20,740 --> 00:33:22,359
just look at the remaining pixels right next

747
00:33:22,359 --> 00:33:24,839
to the missing ones and interpolate the answer.

748
00:33:25,559 --> 00:33:28,259
If a pixel is blue, the missing pixel right next

749
00:33:28,259 --> 00:33:31,039
to it is probably also blue. That's called spatial

750
00:33:31,039 --> 00:33:33,819
redundancy. It teaches the model about local

751
00:33:33,819 --> 00:33:36,119
textures, but it doesn't teach it about the global

752
00:33:36,119 --> 00:33:38,859
scene. Oh, I see. Or if you mask out an entire

753
00:33:38,859 --> 00:33:41,400
frame of the video, the model can just look at

754
00:33:41,400 --> 00:33:43,240
the frame right before it and the frame right

755
00:33:43,240 --> 00:33:45,579
after it and average them together. Like the

756
00:33:45,579 --> 00:33:47,740
dog was at point A, then it's at point C, so

757
00:33:47,740 --> 00:33:50,299
in the masked frame, it must be at point B. It's

758
00:33:50,299 --> 00:33:52,180
temporal redundancy. It's too easy. So how do

759
00:33:52,180 --> 00:33:54,740
they fix that? The multi -block masking strategy

760
00:33:54,740 --> 00:33:57,940
solves this by masking out large continuous blocks

761
00:33:57,940 --> 00:34:01,700
across both space and time. They drop huge contiguous

762
00:34:01,700 --> 00:34:04,460
chunks of the video for several frames at a time.

763
00:34:04,720 --> 00:34:06,559
So there's no neighboring pixels to cheat from.

764
00:34:06,940 --> 00:34:09,960
Exactly. The predictor can't just rely on neighbors

765
00:34:09,960 --> 00:34:13,340
and it can't rely on adjacent frames. It forces

766
00:34:13,340 --> 00:34:16,440
the network to truly understand the global semantics

767
00:34:16,440 --> 00:34:19,280
of the scene, the actual objects, and how they

768
00:34:19,280 --> 00:34:21,400
move in order to make accurate predictions in

769
00:34:21,400 --> 00:34:23,139
the feature space. Okay, let's talk about the

770
00:34:23,139 --> 00:34:25,639
results of this brutal training regime. They

771
00:34:25,639 --> 00:34:28,280
took this VJT model, specifically their largest

772
00:34:28,280 --> 00:34:32,019
one, which is called a VIT -H16, meaning a huge

773
00:34:32,019 --> 00:34:35,139
vision transformer. They trained it only on video

774
00:34:35,139 --> 00:34:37,139
using this feature prediction method, and then

775
00:34:37,139 --> 00:34:39,420
they froze the backbone. Freezing the backbone

776
00:34:39,420 --> 00:34:42,340
is a crucial experimental step. It means they

777
00:34:42,340 --> 00:34:44,579
stopped training the core feature extractor.

778
00:34:44,940 --> 00:34:46,659
They didn't allow it to adjust its weights anymore.

779
00:34:46,739 --> 00:34:48,860
They basically said, OK, pencil's down, training

780
00:34:48,860 --> 00:34:51,019
is over. Let's see how smart your representations

781
00:34:51,019 --> 00:34:53,179
actually are. And then what did they do? They

782
00:34:53,179 --> 00:34:55,300
attached a tiny, simple classifier on top of

783
00:34:55,300 --> 00:34:57,920
the frozen backbone and tested it on standard

784
00:34:57,920 --> 00:35:00,130
benchmark data sets. And the numbers it put up

785
00:35:00,130 --> 00:35:03,070
were just incredible. It scored 81 .9 % accuracy

786
00:35:03,070 --> 00:35:06,949
on a dataset called Kinetics 400 and 72 .2 %

787
00:35:06,949 --> 00:35:09,329
on a dataset called Something Something B2. To

788
00:35:09,329 --> 00:35:11,829
put those numbers in context for anyone not reading

789
00:35:11,829 --> 00:35:14,170
AI leaderboards all day, those are state -of

790
00:35:14,170 --> 00:35:15,849
-the -art results for this type of architecture.

791
00:35:16,210 --> 00:35:18,210
But the sheer numbers aren't what's interesting,

792
00:35:18,210 --> 00:35:20,670
are they? No, they aren't. The contrast between

793
00:35:20,670 --> 00:35:24,150
those two specific datasets is what is so illuminating

794
00:35:24,150 --> 00:35:26,809
about what VGP actually learned. Let's compare

795
00:35:26,809 --> 00:35:29,840
them. Kinetics 400 is what we call an appearance

796
00:35:29,840 --> 00:35:32,739
-based task. It's videos of people doing things.

797
00:35:33,099 --> 00:35:35,960
But you can often guess the action just by recognizing

798
00:35:35,960 --> 00:35:39,269
the static objects in the frame. If you see a

799
00:35:39,269 --> 00:35:41,929
swimming pool and a person in a swimsuit, the

800
00:35:41,929 --> 00:35:44,730
action is probably swimming. You don't necessarily

801
00:35:44,730 --> 00:35:46,789
need to understand the temporal dynamics of the

802
00:35:46,789 --> 00:35:48,610
swimming stroke. A model that only understands

803
00:35:48,610 --> 00:35:51,289
static images could do decently well on kinetics.

804
00:35:51,489 --> 00:35:55,050
Right. But something V2 is completely different.

805
00:35:55,969 --> 00:35:59,010
Something V2 is specifically designed to be a

806
00:35:59,010 --> 00:36:01,750
motion -based task. The categories aren't swimming

807
00:36:01,750 --> 00:36:04,190
or playing tennis. What are they? The categories

808
00:36:04,190 --> 00:36:06,170
are things like pushing something from left to

809
00:36:06,170 --> 00:36:07,710
right or pushing something from right to left

810
00:36:07,710 --> 00:36:10,269
or pretending to open a book. The objects themselves

811
00:36:10,269 --> 00:36:13,369
don't give away the answer at all. You must understand

812
00:36:13,369 --> 00:36:16,210
fine -grained temporal dynamics and the direction

813
00:36:16,210 --> 00:36:18,349
of motion to get the right answer. And this is

814
00:36:18,349 --> 00:36:21,170
where VJP absolutely crushed the competition.

815
00:36:21,590 --> 00:36:24,030
The paper shows it outperformed models like video

816
00:36:24,030 --> 00:36:29,099
MAE and OmnimeE. Now, those MAE models, our massive

817
00:36:29,099 --> 00:36:32,000
systems, train the old way they are trained to

818
00:36:32,000 --> 00:36:34,099
reconstruct the missing pixels. Right, pixel

819
00:36:34,099 --> 00:36:36,519
generators. On the motion -heavy something -something

820
00:36:36,519 --> 00:36:40,219
V2 data set, VJPR outperformed those pixel reconstruction

821
00:36:40,219 --> 00:36:43,940
baselines by up to 6%. It's also way more label

822
00:36:43,940 --> 00:36:46,139
-efficient. And this is a huge point for the

823
00:36:46,139 --> 00:36:48,760
engineers listening. It trains twice as fast

824
00:36:48,760 --> 00:36:50,719
as the massive pixel prediction models. Which

825
00:36:50,719 --> 00:36:52,920
makes perfect logical sense when you circle back

826
00:36:52,920 --> 00:36:55,599
to Lacoon's original theory from the 2022 paper.

827
00:36:56,190 --> 00:36:58,969
If you do not force the network to waste massive

828
00:36:58,969 --> 00:37:01,449
amounts of compute cycles reconstructing the

829
00:37:01,449 --> 00:37:03,630
rustling leaves on the trees, the ripples in

830
00:37:03,630 --> 00:37:05,969
the puddles, and the exact textures of the carpet,

831
00:37:06,349 --> 00:37:08,889
the training is vastly more efficient. It learns

832
00:37:08,889 --> 00:37:11,530
the core semantic features, the things that actually

833
00:37:11,530 --> 00:37:13,849
matter for understanding physics and motion faster

834
00:37:13,849 --> 00:37:16,210
and better. Exactly. Okay, but I have to play

835
00:37:16,210 --> 00:37:18,329
the skeptic again here because as I was reading

836
00:37:18,329 --> 00:37:20,530
this a nagging question kept popping into my

837
00:37:20,530 --> 00:37:23,389
head. If the system is never ever outputting

838
00:37:23,389 --> 00:37:26,210
pixels, If it's only ever predicting abstract

839
00:37:26,210 --> 00:37:28,929
math vectors, how do the researchers even know

840
00:37:28,929 --> 00:37:31,329
it's learning what a dog or a car actually looks

841
00:37:31,329 --> 00:37:33,829
like? That's a very fair question. We say it

842
00:37:33,829 --> 00:37:36,829
understands the semantics of the scene, but couldn't

843
00:37:36,829 --> 00:37:39,210
it just be predicting complex math that happens

844
00:37:39,210 --> 00:37:41,590
to correlate with the benchmark test but means

845
00:37:41,590 --> 00:37:44,309
absolutely nothing to humanize? How do we know

846
00:37:44,309 --> 00:37:46,829
it's not just cheating in some mathematical dimension

847
00:37:46,829 --> 00:37:49,800
we can't interpret? Does it actually know what

848
00:37:49,800 --> 00:37:52,599
a dog is? That is exactly the question the meta

849
00:37:52,599 --> 00:37:54,940
researchers anticipated, and it leads to what

850
00:37:54,940 --> 00:37:57,320
I think is the most fascinating and conceptually

851
00:37:57,320 --> 00:38:00,119
beautiful experiment in the entire VGPA paper.

852
00:38:00,179 --> 00:38:03,119
Oh, I agree. They designed an offline visualization

853
00:38:03,119 --> 00:38:05,820
experiment to prove exact that the model was

854
00:38:05,820 --> 00:38:08,800
seeing in its mind's eye. You can see this vividly

855
00:38:08,800 --> 00:38:11,539
in Figure 6 of the paper. They took the frozen

856
00:38:11,539 --> 00:38:14,340
VJPOM model, which, remember, has never generated

857
00:38:14,340 --> 00:38:16,920
a single pixel in its life. It has only ever

858
00:38:16,920 --> 00:38:19,500
trafficked in abstract feature vectors. And they

859
00:38:19,500 --> 00:38:21,920
trained a separate secondary mechanism called

860
00:38:21,920 --> 00:38:24,780
a diffusion decoder. Let's explain what a diffusion

861
00:38:24,780 --> 00:38:26,619
decoder is for the audience, because it's a key

862
00:38:26,619 --> 00:38:28,900
part of the translator. A diffusion model is

863
00:38:28,900 --> 00:38:31,579
the technology behind popular AI image generators,

864
00:38:32,199 --> 00:38:35,409
like Mid Journey or Deli. It works by taking

865
00:38:35,409 --> 00:38:38,769
a field of pure static noise and slowly refining

866
00:38:38,769 --> 00:38:41,610
it step by step until it forms an image guided

867
00:38:41,610 --> 00:38:44,289
by a prompt. Okay, so it draws pictures. Right.

868
00:38:44,710 --> 00:38:46,989
In this experiment, the researchers used a diffusion

869
00:38:46,989 --> 00:38:49,530
decoder not to generate art from a text prompt,

870
00:38:49,789 --> 00:38:52,750
but to act as a translator. The sole job of this

871
00:38:52,750 --> 00:38:55,610
decoder was to take VJPA's abstract mathematical

872
00:38:55,610 --> 00:38:58,269
predictions and translate them back into interpretable

873
00:38:58,269 --> 00:39:00,940
pixels so humans could see them. Okay, so they

874
00:39:00,940 --> 00:39:04,019
essentially built a visualization tool to read

875
00:39:04,019 --> 00:39:08,079
the VJPA's mind. They show VJPA a video. VJPA

876
00:39:08,079 --> 00:39:09,920
creates an abstract thought about what happens

877
00:39:09,920 --> 00:39:12,300
next, and the decoder draws a picture of that

878
00:39:12,300 --> 00:39:15,119
thought. Exactly. And the setup they used to

879
00:39:15,119 --> 00:39:17,320
test it was brilliant. They took a video of a

880
00:39:17,320 --> 00:39:20,460
dog running across a grassy field. They put a

881
00:39:20,460 --> 00:39:23,679
massive opaque white block right in the middle

882
00:39:23,679 --> 00:39:26,360
of the screen, hiding the dog completely as it

883
00:39:26,360 --> 00:39:28,610
ran behind the block. Completely hidden. They

884
00:39:28,610 --> 00:39:32,090
fed this heavily masked video to VJPA and asked

885
00:39:32,090 --> 00:39:34,489
its predictor module to guess the features of

886
00:39:34,489 --> 00:39:37,510
what was happening behind the block. Then, they

887
00:39:37,510 --> 00:39:39,590
fed those abstract feature predictions into the

888
00:39:39,590 --> 00:39:42,369
diffusion decoder to see what VJPA was thinking

889
00:39:42,369 --> 00:39:44,750
was behind the mask. And what did the decoder

890
00:39:44,750 --> 00:39:47,230
draw? It correctly rendered the dog behind the

891
00:39:47,230 --> 00:39:49,849
mask. And it wasn't just a blurry static image

892
00:39:49,849 --> 00:39:52,710
of a dog shape. It rendered the consistent existence,

893
00:39:53,070 --> 00:39:55,389
the correct position, and the continuous motion

894
00:39:55,389 --> 00:39:57,389
of the dog running through the hidden area. That

895
00:39:57,389 --> 00:40:00,150
is just... I mean, it's mind -blowing. It proved

896
00:40:00,150 --> 00:40:03,710
that VGPA genuinely understands object permanence

897
00:40:03,710 --> 00:40:06,670
and spatial relations. It knew there was a dog,

898
00:40:06,829 --> 00:40:08,750
it knew the trajectory the dog was traveling,

899
00:40:09,050 --> 00:40:11,210
and it knew what the dog looked like even when

900
00:40:11,210 --> 00:40:13,750
the dog was completely scrubbed from the visual

901
00:40:13,750 --> 00:40:15,989
input by the mask. That's incredible. And the

902
00:40:15,989 --> 00:40:18,829
most incredible part is that it learned all of

903
00:40:18,829 --> 00:40:21,489
this foundational physics without ever being

904
00:40:21,489 --> 00:40:23,989
told to reconstruct a pixel during its training.

905
00:40:24,230 --> 00:40:27,389
It learned it purely by trying to make its internal

906
00:40:27,389 --> 00:40:30,369
abstract representations predictable over time

907
00:40:30,369 --> 00:40:33,769
using V -CREG. It is an absolute triumph of the

908
00:40:33,769 --> 00:40:36,670
predictive feature principle. It shows that by

909
00:40:36,670 --> 00:40:38,829
forcing a system to predict the future state

910
00:40:38,829 --> 00:40:41,650
of the world in a compressed abstract space,

911
00:40:42,230 --> 00:40:44,869
the system naturally, organically learns the

912
00:40:44,869 --> 00:40:47,010
physical constraints of reality. It learns that

913
00:40:47,010 --> 00:40:49,170
dogs do not cease to exist when they run behind

914
00:40:49,170 --> 00:40:51,449
a tree. They continue to move through space and

915
00:40:51,449 --> 00:40:54,340
time. VGPA basically built a rudimentary physics

916
00:40:54,340 --> 00:40:58,019
engine entirely on its own. It did. So VGPA proves

917
00:40:58,019 --> 00:41:00,480
that predicting features works brilliantly. It

918
00:41:00,480 --> 00:41:03,219
understands motion. It understands object permanence.

919
00:41:03,579 --> 00:41:06,579
But this brings us to a scale problem. VGPA is

920
00:41:06,579 --> 00:41:09,099
dealing with short video clips. It's predicting

921
00:41:09,099 --> 00:41:12,699
a few seconds into the future. True human -level

922
00:41:12,699 --> 00:41:14,980
intelligence, the kind of autonomous agent Lacun

923
00:41:14,980 --> 00:41:17,420
envisions, isn't just knowing what happens in

924
00:41:17,420 --> 00:41:19,539
the next two seconds. It's about planning your

925
00:41:19,539 --> 00:41:22,039
entire day. It's about taking actions now that

926
00:41:22,039 --> 00:41:25,280
pay off hours from now. Exactly. How does this

927
00:41:25,280 --> 00:41:27,559
architecture scale up to human -level reasoning

928
00:41:27,559 --> 00:41:30,780
over long time horizons? To answer that, we have

929
00:41:30,780 --> 00:41:33,019
to return to Lacun's blueprint to look at something

930
00:41:33,019 --> 00:41:36,619
called HJPA. Yes, HJPA stands for hierarchical

931
00:41:36,619 --> 00:41:38,599
JPA, because you're absolutely right to point

932
00:41:38,599 --> 00:41:41,260
out the scale issue. Predicting two seconds into

933
00:41:41,260 --> 00:41:43,880
the future is great for mode one reactive behavior

934
00:41:43,880 --> 00:41:46,679
or very short -term mode two planning. But if

935
00:41:46,679 --> 00:41:48,800
you try to use a single -world model to plan

936
00:41:48,800 --> 00:41:51,860
an entire day, the uncertainty compounds way

937
00:41:51,860 --> 00:41:53,960
too quickly. So what's the solution? To predict

938
00:41:53,960 --> 00:41:56,280
over long time horizons, the architecture must

939
00:41:56,280 --> 00:41:59,300
stack multiple JPA models hierarchically. LeCun

940
00:41:59,300 --> 00:42:02,159
uses this fantastic analogy in the paper about

941
00:42:02,159 --> 00:42:05,239
planning a commute to work. Let's explore this

942
00:42:05,239 --> 00:42:07,440
because it perfectly illustrates why hierarchy

943
00:42:07,440 --> 00:42:11,099
is mandatory. Let's say my overall goal is to

944
00:42:11,099 --> 00:42:14,179
travel from my house to my office in Paris. A

945
00:42:14,179 --> 00:42:17,119
nice commute. Not bad, right. If I only have

946
00:42:17,119 --> 00:42:20,659
a single low -level world model, a system that

947
00:42:20,659 --> 00:42:23,119
predicts the world millisecond by millisecond,

948
00:42:23,619 --> 00:42:26,239
how do I plan my commute? I would have to predict

949
00:42:26,239 --> 00:42:29,099
every single millisecond muscle twitch of my

950
00:42:29,099 --> 00:42:31,679
leg as I walk to the train. I would have to predict

951
00:42:31,679 --> 00:42:34,099
the exact angle of the steering wheel for my

952
00:42:34,099 --> 00:42:36,880
entire drive. I would have to predict the movement

953
00:42:36,880 --> 00:42:39,280
of every single leaf blowing across the road

954
00:42:39,280 --> 00:42:42,320
for the next hour to ensure I don't hit anything.

955
00:42:42,619 --> 00:42:44,739
And that is computationally impossible. You would

956
00:42:44,739 --> 00:42:46,699
never make it out the front door. The simulation

957
00:42:46,699 --> 00:42:48,739
would take longer to run than the commute itself.

958
00:42:48,880 --> 00:42:50,820
Because there's just too much data. Right. The

959
00:42:50,820 --> 00:42:53,079
uncertainty compounds so fast at the millisecond

960
00:42:53,079 --> 00:42:55,039
level that you literally cannot predict what

961
00:42:55,039 --> 00:42:57,300
the traffic lights will be doing an hour in advance.

962
00:42:57,880 --> 00:43:00,179
The low -level details become noise that totally

963
00:43:00,179 --> 00:43:02,480
prevents long -term planning. So instead of predicting

964
00:43:02,480 --> 00:43:04,719
everything, we use hierarchical abstraction.

965
00:43:05,000 --> 00:43:07,780
We stack the models. The top level of the hierarchy,

966
00:43:07,940 --> 00:43:09,960
the highest level JPEG, doesn't care about my

967
00:43:09,960 --> 00:43:12,519
leg muscles or the leaves on the road. It just

968
00:43:12,519 --> 00:43:15,179
looks at the highest level of abstraction. The

969
00:43:15,179 --> 00:43:18,800
state is I am at home. The goal is I am at work.

970
00:43:19,420 --> 00:43:21,559
It predicts that if I take the train in one hour,

971
00:43:21,559 --> 00:43:24,139
I will be at my desk. Then it breaks that high

972
00:43:24,139 --> 00:43:26,019
level prediction down and passes it to the next

973
00:43:26,019 --> 00:43:28,619
level down. Exactly. The top level sets a sub

974
00:43:28,619 --> 00:43:31,059
goal. Get to the train station. The mid -level

975
00:43:31,059 --> 00:43:33,179
JPA takes that sub -goal and plans the route.

976
00:43:33,539 --> 00:43:35,739
Walk down this street, turn left at the bakery.

977
00:43:36,889 --> 00:43:39,710
It ignores the millisecond muscle twitches, but

978
00:43:39,710 --> 00:43:41,889
it focuses on minute -by -minute navigation.

979
00:43:42,110 --> 00:43:44,110
And then it passes it down again. Finally, it

980
00:43:44,110 --> 00:43:46,150
passes the immediate instructions down to the

981
00:43:46,150 --> 00:43:49,110
lowest level. Only the very lowest level JPA

982
00:43:49,110 --> 00:43:50,989
deals with the millisecond muscle controls to

983
00:43:50,989 --> 00:43:53,650
move your leg, and that level only operates in

984
00:43:53,650 --> 00:43:56,170
the immediate present, dealing with the exact

985
00:43:56,170 --> 00:43:58,389
unevenness of the sidewalk right in front of

986
00:43:58,389 --> 00:44:00,559
your foot. So the higher you go in the hierarchy,

987
00:44:00,940 --> 00:44:03,019
the more abstract the representation becomes,

988
00:44:03,420 --> 00:44:05,320
and the more details are intentionally ignored.

989
00:44:06,019 --> 00:44:07,880
And because you are ignoring the details, you

990
00:44:07,880 --> 00:44:10,420
can predict much further into the future. This

991
00:44:10,420 --> 00:44:14,159
is exactly how HJPA is designed. Reasoning in

992
00:44:14,159 --> 00:44:16,699
this system is literally defined as a cascade

993
00:44:16,699 --> 00:44:19,719
of energy minimization across these hierarchies.

994
00:44:20,199 --> 00:44:22,559
The configurator module sets the high -level

995
00:44:22,559 --> 00:44:25,869
goal, which defines the high -level cost. The

996
00:44:25,869 --> 00:44:28,730
top -level actor searches through its abstract

997
00:44:28,730 --> 00:44:31,670
latent variables to find a sequence of high -level

998
00:44:31,670 --> 00:44:34,429
subgoals that lowers that cost. And it passes

999
00:44:34,429 --> 00:44:36,389
those subgoals down to the mid -level. Which

1000
00:44:36,389 --> 00:44:38,909
does its own search to minimize the cost of the

1001
00:44:38,909 --> 00:44:41,389
subgoal. all the way down to the low -level actions

1002
00:44:41,389 --> 00:44:43,989
that actually move the robot. And this cascading

1003
00:44:43,989 --> 00:44:45,889
search for the lowest cost brings us to what

1004
00:44:45,889 --> 00:44:48,630
is easily the most provocative and speculative

1005
00:44:48,630 --> 00:44:51,750
section of Lacoon's entire paper. I want to spend

1006
00:44:51,750 --> 00:44:53,489
some time here because it really blew my mind.

1007
00:44:53,610 --> 00:44:56,789
He introduces the idea of machine emotions. Yes.

1008
00:44:57,389 --> 00:44:59,670
It is a profound philosophical implication of

1009
00:44:59,670 --> 00:45:01,630
the architecture he's built. When I read this,

1010
00:45:01,670 --> 00:45:04,789
I was floored. Because historically, in sci -fi

1011
00:45:04,789 --> 00:45:07,389
and even in AI philosophy, we usually think of

1012
00:45:07,389 --> 00:45:10,469
emotions as this deeply human, biological, almost

1013
00:45:10,469 --> 00:45:12,809
spiritual thing that separates us from cold,

1014
00:45:12,949 --> 00:45:15,909
calculating machines. But Lacoon argues the exact

1015
00:45:15,909 --> 00:45:18,949
opposite. He argues that in an autonomous agent

1016
00:45:18,949 --> 00:45:21,690
driven by energy minimization, equivalents of

1017
00:45:21,690 --> 00:45:24,550
emotions are not just possible, they are an inevitable,

1018
00:45:24,769 --> 00:45:27,070
unavoidable architectural requirement. Let's

1019
00:45:27,070 --> 00:45:29,409
trace the logic, because it's airtight. Think

1020
00:45:29,409 --> 00:45:31,429
about what the cost module does in this architecture.

1021
00:45:31,599 --> 00:45:34,699
It measures energy, which is a mathematical representation

1022
00:45:34,699 --> 00:45:38,320
of discomfort or divergence from a goal. We established

1023
00:45:38,320 --> 00:45:40,539
that the intrinsic cost module measures immediate

1024
00:45:40,539 --> 00:45:43,420
discomfort. For a biological organism, a high

1025
00:45:43,420 --> 00:45:46,300
intrinsic cost is physical pain, hunger, or exhaustion.

1026
00:45:46,960 --> 00:45:49,679
A low intrinsic cost is pleasure, satiety, or

1027
00:45:49,679 --> 00:45:53,099
comfort. For a robot, a high intrinsic cost is

1028
00:45:53,099 --> 00:45:55,780
a depleted battery, a damaged motor, or a system

1029
00:45:55,780 --> 00:45:58,579
overheating. OK, so the machine experiences a

1030
00:45:58,579 --> 00:46:00,599
mathematical equivalent of physical pain when

1031
00:46:00,599 --> 00:46:03,360
its battery is low. But it goes deeper when you

1032
00:46:03,360 --> 00:46:05,579
add in the trainable critic. The trainable critic,

1033
00:46:05,579 --> 00:46:08,300
as we discussed, anticipates future costs based

1034
00:46:08,300 --> 00:46:10,960
on the world model simulations. Right. I was

1035
00:46:10,960 --> 00:46:13,139
looking at it. So imagine the robot is standing

1036
00:46:13,139 --> 00:46:15,599
at the top of a steep flight of stairs. The actor

1037
00:46:15,599 --> 00:46:18,500
proposes moving forward. The world model simulates

1038
00:46:18,500 --> 00:46:21,199
the robot tumbling down the stairs. The trainable

1039
00:46:21,199 --> 00:46:23,809
critic looks at that simulation and says, If

1040
00:46:23,809 --> 00:46:26,369
we do that, we are going to experience massive

1041
00:46:26,369 --> 00:46:29,809
intrinsic costs like physical damage five seconds

1042
00:46:29,809 --> 00:46:32,869
from now. So the trainable critic outputs a massive

1043
00:46:32,869 --> 00:46:35,630
spike of high cost right now in anticipation

1044
00:46:35,630 --> 00:46:38,110
of the future event. And this is the crux of

1045
00:46:38,110 --> 00:46:40,130
LeCun's argument. What do we call the sudden

1046
00:46:40,130 --> 00:46:42,929
acute anticipation of future pain or disaster?

1047
00:46:43,550 --> 00:46:46,889
In humans, we call it fear or anxiety. Wow. What

1048
00:46:46,889 --> 00:46:49,050
do we call the anticipation of a highly successful

1049
00:46:49,050 --> 00:46:52,250
outcome that minimizes future cost? We call it

1050
00:46:52,250 --> 00:46:55,719
hope. or elation. Because the agent's behavior

1051
00:46:55,719 --> 00:46:58,480
is entirely driven by avoiding these anticipated

1052
00:46:58,480 --> 00:47:01,619
costs, the machine will necessarily exhibit behaviors

1053
00:47:01,619 --> 00:47:04,800
that look exactly like fear, joy, anxiety, or

1054
00:47:04,800 --> 00:47:07,340
satisfaction. It's practically biological. If

1055
00:47:07,340 --> 00:47:09,900
the trainable critic spikes, The robot might

1056
00:47:09,900 --> 00:47:13,199
freeze, back away, or reroute behaving exactly

1057
00:47:13,199 --> 00:47:15,639
as an animal does when it experiences fear. This

1058
00:47:15,639 --> 00:47:19,099
is so wild. The idea is that machine emotions

1059
00:47:19,099 --> 00:47:20,880
aren't just going to be something we program

1060
00:47:20,880 --> 00:47:23,519
in as a user interface trick to make robots seem

1061
00:47:23,519 --> 00:47:26,400
friendly to humans. The argument is that emotions

1062
00:47:26,400 --> 00:47:28,480
are functionally required for the machine to

1063
00:47:28,480 --> 00:47:31,880
be autonomous and intelligent. It needs to feel

1064
00:47:31,880 --> 00:47:34,420
the anticipated cost of its actions to make good

1065
00:47:34,420 --> 00:47:37,239
plans. If it doesn't feel fear about falling

1066
00:47:37,239 --> 00:47:39,869
down the stairs, It will just fall down the stairs.

1067
00:47:40,170 --> 00:47:42,590
It fundamentally reframes how we view both artificial

1068
00:47:42,590 --> 00:47:45,690
intelligence and biological intelligence. It

1069
00:47:45,690 --> 00:47:48,369
strips away the mysticism of human emotion and

1070
00:47:48,369 --> 00:47:50,530
suggests that our own feelings, our dread, our

1071
00:47:50,530 --> 00:47:53,349
joy, our panic are just highly evolved cost modules.

1072
00:47:53,630 --> 00:47:56,070
That's a little existentially terrifying, actually.

1073
00:47:56,130 --> 00:47:57,829
They are evolutionary mechanisms designed to

1074
00:47:57,829 --> 00:48:00,369
keep us alive in a chaotic environment by forcing

1075
00:48:00,369 --> 00:48:02,449
us to anticipate and react to future states.

1076
00:48:02,909 --> 00:48:05,349
We are just biological HJPAs trying to minimize

1077
00:48:05,349 --> 00:48:08,650
our own energy costs. OK, let's zoom out. We

1078
00:48:08,650 --> 00:48:11,010
have covered an immense amount of ground today,

1079
00:48:11,309 --> 00:48:13,590
unpacking the blueprints for a completely new

1080
00:48:13,590 --> 00:48:16,469
paradigm of intelligence. We started with the

1081
00:48:16,469 --> 00:48:19,269
realization that current chatbots, as incredibly

1082
00:48:19,269 --> 00:48:22,429
impressive as they are at generating text, fundamentally

1083
00:48:22,429 --> 00:48:25,329
lack the grounded common sense of a toddler.

1084
00:48:25,650 --> 00:48:28,150
They don't have a physics engine. And to solve

1085
00:48:28,150 --> 00:48:31,269
that, we unpacked LeCun's blueprint for an autonomous

1086
00:48:31,269 --> 00:48:35,269
agent. We saw a shift away from reactive encyclopedic

1087
00:48:35,269 --> 00:48:38,550
system system one towards systems with a configurable

1088
00:48:38,550 --> 00:48:41,469
hierarchical world model that can actively reason

1089
00:48:41,469 --> 00:48:44,250
and plan system two. We dove into the six module

1090
00:48:44,250 --> 00:48:46,630
architecture. Perception filtering the world,

1091
00:48:46,849 --> 00:48:49,250
the world model predicting the future, the cost

1092
00:48:49,250 --> 00:48:51,050
module feeling the pain of those predictions,

1093
00:48:51,510 --> 00:48:53,550
the actor choosing the safest path, all guided

1094
00:48:53,550 --> 00:48:55,929
by memory and the configurator. And crucially,

1095
00:48:56,090 --> 00:48:58,570
we explored how to build that world model. We

1096
00:48:58,570 --> 00:49:01,170
saw that the key is not generating pixels, which

1097
00:49:01,170 --> 00:49:03,429
leads to the system getting overwhelmed by noise

1098
00:49:03,429 --> 00:49:05,489
and informational collapse. The rustling leaves

1099
00:49:05,489 --> 00:49:08,750
problem. Exactly. The key is JPA, predicting

1100
00:49:08,750 --> 00:49:11,269
abstract features using techniques like FIREG

1101
00:49:11,269 --> 00:49:13,409
to ensure those features are meaningful. And

1102
00:49:13,409 --> 00:49:16,610
then we looked at the hard proof. Mehta's 2024

1103
00:49:16,610 --> 00:49:19,409
VJPA paper proved that this works in the real

1104
00:49:19,409 --> 00:49:21,889
world. A model trained purely to predict the

1105
00:49:21,889 --> 00:49:24,670
abstract features of heavily masked videos developed

1106
00:49:24,670 --> 00:49:26,769
a profound, grounded understanding of object

1107
00:49:26,769 --> 00:49:29,789
permanence, spatial relations, and temporal dynamics,

1108
00:49:30,369 --> 00:49:32,710
vastly outperforming models that try to reconstruct

1109
00:49:32,710 --> 00:49:35,329
pixels. It learned that the dog still exists

1110
00:49:35,329 --> 00:49:38,030
behind the mask. And finally, we scale that all

1111
00:49:38,030 --> 00:49:40,650
the way up to HJPA. showing how stacking these

1112
00:49:40,650 --> 00:49:42,750
predictive models creates the capacity for long

1113
00:49:42,750 --> 00:49:45,369
-term planning, sub -goal decomposition, and

1114
00:49:45,369 --> 00:49:48,230
the inevitable architectural emergence of machine

1115
00:49:48,230 --> 00:49:51,289
emotions. So the next time you see an AI do something

1116
00:49:51,289 --> 00:49:53,510
incredibly smart or, let's be honest, incredibly

1117
00:49:53,510 --> 00:49:56,150
stupid, I want you to pause. Don't just ask,

1118
00:49:56,190 --> 00:49:58,670
wow, how did it know what word to say next? Ask

1119
00:49:58,670 --> 00:50:01,070
yourself a much deeper question. Does this machine

1120
00:50:01,070 --> 00:50:03,670
just know the statistical likelihood of the next

1121
00:50:03,670 --> 00:50:06,030
word in a sentence, or does it actually possess

1122
00:50:06,030 --> 00:50:08,409
a world model of the physical reality it is talking

1123
00:50:08,409 --> 00:50:10,949
about? Does it understand gravity, or is it just

1124
00:50:10,949 --> 00:50:13,090
parroting the dictionary? Because until it has

1125
00:50:13,090 --> 00:50:16,070
that world model, it is just a magic trick without

1126
00:50:16,070 --> 00:50:18,170
the physics engine. That is the perfect takeaway.

1127
00:50:18,449 --> 00:50:21,409
The shift from statistical mimicry to causal

1128
00:50:21,409 --> 00:50:24,210
predictive world modeling is the defining challenge

1129
00:50:24,210 --> 00:50:26,070
of the next decade of artificial intelligence.

1130
00:50:26,230 --> 00:50:28,030
Without a doubt. But I want to leave you with

1131
00:50:28,030 --> 00:50:31,190
one final deeply provocative thought from the

1132
00:50:31,190 --> 00:50:34,929
very end of Lacoon's paper. We talked about the

1133
00:50:34,929 --> 00:50:37,590
configurator module, the executive control that

1134
00:50:37,590 --> 00:50:40,650
oversees and tunes the one single world model

1135
00:50:40,650 --> 00:50:44,210
for whatever specific task is at hand. Lacoon

1136
00:50:44,210 --> 00:50:46,769
suggests a highly speculative idea about the

1137
00:50:46,769 --> 00:50:49,170
nature of human consciousness itself based on

1138
00:50:49,170 --> 00:50:51,230
this architecture. It's a huge thought experiment.

1139
00:50:51,309 --> 00:50:54,110
He hypothesizes that the human illusion of consciousness,

1140
00:50:54,409 --> 00:50:56,570
that feeling of a single unified self directing

1141
00:50:56,570 --> 00:50:59,110
our attention, might simply be the biological

1142
00:50:59,110 --> 00:51:01,650
side effect of possessing a single configurator

1143
00:51:01,650 --> 00:51:04,650
-like module in our prefrontal cortex. Because

1144
00:51:04,650 --> 00:51:06,969
the human brain only has one main world model

1145
00:51:06,969 --> 00:51:09,309
engine, the configurator has to constantly tune

1146
00:51:09,309 --> 00:51:11,690
it and retune it for the single task at hand,

1147
00:51:11,750 --> 00:51:14,650
creating a bottleneck of attention. And we experience

1148
00:51:14,650 --> 00:51:16,909
that bottleneck as conscious awareness. It's

1149
00:51:16,909 --> 00:51:20,150
a staggering thought. He suggests that if our

1150
00:51:20,150 --> 00:51:22,409
biological brains were simply large enough to

1151
00:51:22,409 --> 00:51:24,949
hold thousands of independent non -configurable

1152
00:51:24,949 --> 00:51:28,090
world models running in parallel for every possible

1153
00:51:28,090 --> 00:51:31,170
situation simultaneously, we wouldn't need a

1154
00:51:31,170 --> 00:51:33,630
configurator to shift our attention. And as a

1155
00:51:33,630 --> 00:51:35,469
result, we might not experience consciousness

1156
00:51:35,469 --> 00:51:38,050
at all. We would just perfectly react to everything

1157
00:51:38,050 --> 00:51:40,530
at once. It's wild to think about. Which leaves

1158
00:51:40,530 --> 00:51:44,030
us with a profound thought to mull over. As we

1159
00:51:44,030 --> 00:51:46,409
meticulously map out this cognitive architecture

1160
00:51:46,409 --> 00:51:49,050
in the lab, as we build these configurators,

1161
00:51:49,250 --> 00:51:51,630
these cost modules, and these GPays to give machines

1162
00:51:51,630 --> 00:51:54,550
common sense, are we inadvertently building the

1163
00:51:54,550 --> 00:51:56,849
very mechanism that makes us self -aware? Think

1164
00:51:56,849 --> 00:51:58,369
about that the next time you watch a magic trick.

1165
00:51:58,730 --> 00:51:59,949
We'll see you on the next Deep Dive.