1
00:00:00,000 --> 00:00:02,660
You know, usually when we talk about getting

2
00:00:02,660 --> 00:00:04,820
a medical diagnosis, there's this underlying

3
00:00:04,820 --> 00:00:07,879
expectation of absolute precision. Right, like

4
00:00:07,879 --> 00:00:09,919
it's a math problem. Exactly. It's almost like

5
00:00:09,919 --> 00:00:12,240
engineering. You break your arm, you go to the

6
00:00:12,240 --> 00:00:14,960
hospital, they take the x -ray, and it shows

7
00:00:14,960 --> 00:00:17,859
that really obvious jagged white line on the

8
00:00:17,859 --> 00:00:19,559
bone. Yeah, and the doctor just points right

9
00:00:19,559 --> 00:00:21,719
at it. Right. They point at it and say, you know,

10
00:00:21,780 --> 00:00:24,280
there it is. That is your problem. It's a binary

11
00:00:24,280 --> 00:00:27,199
state. Broken or not broken? Very definitive.

12
00:00:27,820 --> 00:00:30,339
And honestly, I think that's comforting. We,

13
00:00:30,460 --> 00:00:32,780
as humans, we really, really like things to be

14
00:00:32,780 --> 00:00:35,399
visible like that. Oh, absolutely. We want to

15
00:00:35,399 --> 00:00:37,600
perfectly categorize our problems. Yeah. We want

16
00:00:37,600 --> 00:00:39,679
to point to the exact cause of a phenomenon and

17
00:00:39,679 --> 00:00:42,960
say with, I mean, absolute certainty, that specific

18
00:00:42,960 --> 00:00:44,899
thing right there is the reason this is happening.

19
00:00:45,320 --> 00:00:47,600
But the reality is most things aren't an x -ray.

20
00:00:47,939 --> 00:00:50,520
No. They aren't. When you step into the world

21
00:00:50,520 --> 00:00:53,600
of complex systems, you look at neurodevelopment

22
00:00:53,600 --> 00:00:56,439
or systemic trauma or even like macroeconomics,

23
00:00:56,939 --> 00:00:59,439
suddenly that beautiful clean x -ray machine

24
00:00:59,439 --> 00:01:02,159
is just totally broken. It's useless. Right.

25
00:01:02,700 --> 00:01:05,079
We're looking at a diagnostic landscape that

26
00:01:05,079 --> 00:01:07,180
is, well, it's completely murky. You can't just

27
00:01:07,180 --> 00:01:09,319
point to a single pixel and find the answer.

28
00:01:09,459 --> 00:01:12,140
Because the cause is distributed. Exactly. It's

29
00:01:12,140 --> 00:01:14,239
distributed across maybe a hundred different

30
00:01:14,239 --> 00:01:16,920
factors. It's hidden in the connections between

31
00:01:16,920 --> 00:01:19,439
things. Which is the absolute definition of diagnostic

32
00:01:19,439 --> 00:01:21,480
muddy waters. And you know what's so fascinating

33
00:01:21,480 --> 00:01:24,500
here is that this exact tension. Yeah. This tension

34
00:01:24,500 --> 00:01:27,579
between wanting a clean observable cause and

35
00:01:27,579 --> 00:01:29,920
dealing with a really complex hidden reality,

36
00:01:30,340 --> 00:01:32,459
that is the precise problem that almost killed

37
00:01:32,459 --> 00:01:34,939
the field of artificial intelligence in its infancy.

38
00:01:35,159 --> 00:01:38,019
Wow. Which brings us perfectly to the core of

39
00:01:38,019 --> 00:01:40,280
what we're doing today. So welcome to the deep

40
00:01:40,280 --> 00:01:42,450
dive. Glad to be here. Today, we are looking

41
00:01:42,450 --> 00:01:45,049
at how a small group of researchers back in 1986

42
00:01:45,049 --> 00:01:47,670
figured out how to solve that exact problem of

43
00:01:47,670 --> 00:01:51,409
hidden murky causes. And in the process, they

44
00:01:51,409 --> 00:01:53,590
accidentally laid the mathematical bedrock for

45
00:01:53,590 --> 00:01:55,909
basically every single piece of modern AI that

46
00:01:55,909 --> 00:01:58,890
we use today. It's wild to think about. We're

47
00:01:58,890 --> 00:02:01,569
unpacking this really legendary paper published

48
00:02:01,569 --> 00:02:03,810
in the journal Nature. It was written by David

49
00:02:03,810 --> 00:02:06,390
Rumelhart, Jeffrey Hinton, and Ronald Williams.

50
00:02:06,870 --> 00:02:09,840
A truly foundational text. Learning representations

51
00:02:09,840 --> 00:02:13,400
by backpropagating errors. Right. And for you

52
00:02:13,400 --> 00:02:15,419
listening, we are not just going to skim the

53
00:02:15,419 --> 00:02:18,020
surface on this one. We are going to meticulously

54
00:02:18,020 --> 00:02:20,919
unpack how these three researchers managed to

55
00:02:20,919 --> 00:02:22,979
build an algorithm that allowed a machine to

56
00:02:22,979 --> 00:02:25,659
create its own internal representations of the

57
00:02:25,659 --> 00:02:27,960
world. Because if you interact with any modern

58
00:02:27,960 --> 00:02:31,560
artificial intelligence today, a voice assistant,

59
00:02:31,780 --> 00:02:34,400
an image generator, a large language model. Basically

60
00:02:34,400 --> 00:02:36,840
anything smart on your phone. Exactly. You are

61
00:02:36,840 --> 00:02:39,300
interacting directly with the mathematical descendants

62
00:02:39,300 --> 00:02:42,680
of the ideas laid out in this exact 1986 document.

63
00:02:42,960 --> 00:02:45,479
But to really grasp the magnitude of what Hinton,

64
00:02:45,780 --> 00:02:47,479
Rumelhart, and Williams achieved, I feel like

65
00:02:47,479 --> 00:02:51,039
we have to look at the dark ages they were working

66
00:02:51,039 --> 00:02:54,159
in. The AI winter. Right, the AI winter. We have

67
00:02:54,159 --> 00:02:56,840
to look at what existed before 1986. The paper

68
00:02:56,840 --> 00:02:58,840
spends some time talking about earlier methods,

69
00:02:59,280 --> 00:03:01,879
specifically something called a perceptron. Yes,

70
00:03:01,919 --> 00:03:04,889
the perceptron convergence procedure. That was

71
00:03:04,889 --> 00:03:06,669
the prevailing model for a really long time.

72
00:03:06,969 --> 00:03:09,689
And it had some issues, right? It was deeply,

73
00:03:09,689 --> 00:03:13,250
deeply flawed. So I was trying to visualize perceptrons

74
00:03:13,250 --> 00:03:15,590
while going through the research for this, and

75
00:03:15,590 --> 00:03:18,229
I kept picturing them like those really rigid,

76
00:03:18,669 --> 00:03:21,889
strict flow charts. Oh, interesting. Like what

77
00:03:21,889 --> 00:03:24,210
kind? Like the ones you see in, you know, old

78
00:03:24,210 --> 00:03:26,189
1950s corporate training manuals, where it's

79
00:03:26,189 --> 00:03:29,150
like, if X happens, do Y. If Y happens, do Z.

80
00:03:29,229 --> 00:03:31,469
There is absolutely no room for interpretation.

81
00:03:32,090 --> 00:03:35,430
That's, yeah. Am I far off there? Were perceptrons

82
00:03:35,430 --> 00:03:37,629
basically just rigid flow charts that couldn't

83
00:03:37,629 --> 00:03:40,090
read between the lines? That is actually a brilliant

84
00:03:40,090 --> 00:03:42,750
way to visualize it. A perceptron was essentially

85
00:03:42,750 --> 00:03:45,849
a flat system. You had input data on one side

86
00:03:45,849 --> 00:03:47,889
and you wanted an output on the other. Okay.

87
00:03:48,169 --> 00:03:50,629
They did have intermediate steps with the researchers

88
00:03:50,629 --> 00:03:52,770
called feature analyzers sitting between the

89
00:03:52,770 --> 00:03:54,889
input and the output. So there was a middle man.

90
00:03:55,090 --> 00:03:57,650
There was, but here is the critical limitation.

91
00:03:58,509 --> 00:04:00,849
These intermediate steps were entirely designed

92
00:04:00,849 --> 00:04:04,639
by humans. Wait, entirely? Entirely. So a programmer

93
00:04:04,639 --> 00:04:07,979
had to sit down and mathematically dictate exactly

94
00:04:07,979 --> 00:04:10,460
what the machine should look for. Exactly. The

95
00:04:10,460 --> 00:04:12,659
human hard -coded the connections. The state

96
00:04:12,659 --> 00:04:15,219
of those intermediate units was completely dictated

97
00:04:15,219 --> 00:04:18,939
by the input based on human -made rules. So they

98
00:04:18,939 --> 00:04:21,720
didn't really learn anything themselves. No.

99
00:04:22,139 --> 00:04:23,720
They weren't what Rumelhart and his team would

100
00:04:23,720 --> 00:04:26,560
call true hidden units. Because they couldn't

101
00:04:26,560 --> 00:04:28,740
actually hide anything. They were completely

102
00:04:28,740 --> 00:04:32,230
transparent. just blindly following orders. It's

103
00:04:32,230 --> 00:04:34,730
like going back to the medical thing, giving

104
00:04:34,730 --> 00:04:37,129
someone an x -ray and telling them to look for

105
00:04:37,129 --> 00:04:39,990
one highly specific 45 degree white line. And

106
00:04:39,990 --> 00:04:41,930
if they don't see exactly that line, they just

107
00:04:41,930 --> 00:04:44,290
fail the patient. Even if the bone is clearly

108
00:04:44,290 --> 00:04:46,850
shattered in a different way. Exactly. The machine

109
00:04:46,850 --> 00:04:49,209
couldn't learn that, hey, maybe a weird shadow

110
00:04:49,209 --> 00:04:51,629
over here combined with a tiny hairline fracture

111
00:04:51,629 --> 00:04:54,709
over there also means a broken bone. Precisely.

112
00:04:55,290 --> 00:04:58,550
They couldn't learn representations. If the human

113
00:04:58,550 --> 00:05:01,709
didn't pre -program the exact feature, the perceptron

114
00:05:01,709 --> 00:05:04,069
was completely blind to it. Which seems like

115
00:05:04,069 --> 00:05:07,089
a huge bottleneck. It caused a massive stagnation

116
00:05:07,089 --> 00:05:10,410
in AI research. In fact, prominent figures like

117
00:05:10,410 --> 00:05:13,709
Marvin Minsky and Seymour Papert actually wrote

118
00:05:13,709 --> 00:05:16,410
a whole book mathematically proving that perceptrons

119
00:05:16,410 --> 00:05:19,129
were fundamentally incapable of solving certain

120
00:05:19,129 --> 00:05:21,290
basic logic puzzles. Just because they couldn't

121
00:05:21,290 --> 00:05:23,910
learn those hidden features. Exactly. It was

122
00:05:23,910 --> 00:05:25,990
a dead end. Which brings us to the breakthrough.

123
00:05:26,279 --> 00:05:29,319
The mission of this 1986 paper was to introduce

124
00:05:29,319 --> 00:05:31,959
a new procedure called backpropagation. Yes.

125
00:05:32,079 --> 00:05:34,060
And the goal here was to let the network itself

126
00:05:34,060 --> 00:05:36,879
decide under what circumstances an intermediate

127
00:05:36,879 --> 00:05:39,740
unit should fire. Autonomy. Right. It's the difference

128
00:05:39,740 --> 00:05:42,259
between giving a traveler a map with a single

129
00:05:42,259 --> 00:05:44,720
red line drawn on it and giving them a compass,

130
00:05:44,879 --> 00:05:46,459
a machete, and telling them to just find the

131
00:05:46,459 --> 00:05:48,220
most efficient path through the jungle themselves.

132
00:05:48,339 --> 00:05:50,300
And that is exactly what the author set out to

133
00:05:50,300 --> 00:05:53,339
demonstrate. They wanted a general purpose. relatively

134
00:05:53,339 --> 00:05:55,699
simple procedure that was powerful enough to

135
00:05:55,699 --> 00:05:58,240
construct appropriate internal representations

136
00:05:58,240 --> 00:06:01,740
without a human holding its hand. But to understand

137
00:06:01,740 --> 00:06:04,000
how a network actually pulls this off, like how

138
00:06:04,000 --> 00:06:06,399
it basically invents abstract concepts out of

139
00:06:06,399 --> 00:06:08,879
thin air, we can't just talk in metaphors all

140
00:06:08,879 --> 00:06:10,819
day. No, we have to get into the mechanics. Right.

141
00:06:10,879 --> 00:06:13,459
We need to build the machine with you, the listener,

142
00:06:13,500 --> 00:06:15,860
right now from the ground up, which means we

143
00:06:15,860 --> 00:06:17,600
need to start with how this thing is actually

144
00:06:17,600 --> 00:06:20,439
physically or I guess mathematically structured.

145
00:06:20,560 --> 00:06:23,480
Let's build the architecture. So the simplest

146
00:06:23,480 --> 00:06:25,740
form of this learning procedure applies to what

147
00:06:25,740 --> 00:06:29,060
they call layered networks. I want you to visualize

148
00:06:29,060 --> 00:06:31,160
a multi -story building. Okay, I'm picturing

149
00:06:31,160 --> 00:06:33,779
an office building. Perfect. On the ground floor

150
00:06:33,779 --> 00:06:35,959
you have your input layer. And this is where

151
00:06:35,959 --> 00:06:39,420
the raw data comes in, right? Right. Then above

152
00:06:39,420 --> 00:06:41,779
that you have any number of intermediate floors.

153
00:06:42,240 --> 00:06:45,060
These are the hidden layers. They're the black

154
00:06:45,060 --> 00:06:48,519
box of the system. Okay. And finally on the penthouse

155
00:06:48,519 --> 00:06:50,779
floor you have the output layer. which gives

156
00:06:50,779 --> 00:06:54,600
you your final answer. And there are incredibly

157
00:06:54,600 --> 00:06:56,620
strict rules about how these floors communicate

158
00:06:56,620 --> 00:06:58,100
with each other, right? Like the researchers

159
00:06:58,100 --> 00:07:00,980
were very explicit that connections within a

160
00:07:00,980 --> 00:07:03,259
single layer are absolutely forbidden. That's

161
00:07:03,259 --> 00:07:06,019
correct. So two nodes sitting right next to each

162
00:07:06,019 --> 00:07:08,360
other on the same hidden floor cannot talk to

163
00:07:08,360 --> 00:07:10,500
each other at all. No gossiping on the same floor.

164
00:07:10,779 --> 00:07:13,720
Right. And furthermore, in this specific model,

165
00:07:14,339 --> 00:07:16,639
connections cannot go backward. A higher floor

166
00:07:16,639 --> 00:07:19,269
cannot send a signal down to a lower floor. The

167
00:07:19,269 --> 00:07:21,889
signal can only flow upward. It's a strictly

168
00:07:21,889 --> 00:07:23,930
one -way street, which is why it's called a feed

169
00:07:23,930 --> 00:07:26,350
-forward network. Feed -forward, okay. Though

170
00:07:26,350 --> 00:07:28,410
I did notice one fascinating little loophole

171
00:07:28,410 --> 00:07:30,850
in the rules when I was reading. It says connections

172
00:07:30,850 --> 00:07:34,589
can skip intermediate layers. Oh, yes. Good catch.

173
00:07:34,769 --> 00:07:36,790
So the ground floor could theoretically have

174
00:07:36,790 --> 00:07:39,930
a direct express elevator to the penthouse, completely

175
00:07:39,930 --> 00:07:42,209
bypassing the hidden floors, while other signals

176
00:07:42,209 --> 00:07:44,389
have to take the stairs layer by layer. Yes,

177
00:07:44,410 --> 00:07:47,129
that's a crucial architectural flexibility. It

178
00:07:47,129 --> 00:07:49,709
allows the network to weigh raw data directly

179
00:07:49,709 --> 00:07:53,959
against processed data, if it needs to. But let's

180
00:07:53,959 --> 00:07:55,920
look at how the actual information moves through

181
00:07:55,920 --> 00:07:58,259
this structure. Let's do it. It always starts

182
00:07:58,259 --> 00:08:02,000
with an input vector. And let's clarify that

183
00:08:02,000 --> 00:08:04,220
term for a second, because vector sounds intimidating.

184
00:08:05,079 --> 00:08:07,259
If you're listening in this, an input vector

185
00:08:07,259 --> 00:08:09,860
is just a list of numbers. That's it. A simple

186
00:08:09,860 --> 00:08:12,769
list, yes. Like, if we are trying to teach the

187
00:08:12,769 --> 00:08:14,990
network to recognize an image, that vector is

188
00:08:14,990 --> 00:08:17,149
literally just the color value of every single

189
00:08:17,149 --> 00:08:19,810
pixel in the image. Right. Or if it's a medical

190
00:08:19,810 --> 00:08:21,870
diagnostic tool, the vector is a list of your

191
00:08:21,870 --> 00:08:24,129
vital signs. Heart rate, blood pressure, temperature.

192
00:08:24,470 --> 00:08:26,509
You take that list of numbers and you lock them

193
00:08:26,509 --> 00:08:28,990
into the units on the bottom layer. And once

194
00:08:28,990 --> 00:08:32,149
that bottom layer is locked in, the network basically

195
00:08:32,149 --> 00:08:34,870
wakes up. The states of the units in the next

196
00:08:34,870 --> 00:08:37,470
layer up are determined. And the timing here

197
00:08:37,470 --> 00:08:41,309
is actually critical. How so? All the units within

198
00:08:41,309 --> 00:08:44,009
a single layer calculate their states in parallel,

199
00:08:44,309 --> 00:08:47,970
simultaneously. OK, right. But the layers themselves

200
00:08:47,970 --> 00:08:50,549
fire in sequence. Yes. It's like an assembly

201
00:08:50,549 --> 00:08:53,250
line in a factory. Layer one sorts all the raw

202
00:08:53,250 --> 00:08:55,909
materials at the exact same time. But layer two,

203
00:08:55,990 --> 00:08:57,889
you know, the welders, they can't start doing

204
00:08:57,889 --> 00:09:00,210
anything until a layer one completely finishes

205
00:09:00,210 --> 00:09:02,809
passing the materials up to them. Exactly. And

206
00:09:02,809 --> 00:09:05,129
then layer three, the painters have to wait for

207
00:09:05,129 --> 00:09:07,889
layer two. The signal just kind of washes over

208
00:09:07,889 --> 00:09:10,750
the network from bottom to top. The assembly

209
00:09:10,750 --> 00:09:13,190
line is the perfect analogy. So let's zoom in

210
00:09:13,190 --> 00:09:15,590
on a single worker on that assembly line. Let's

211
00:09:15,590 --> 00:09:18,090
look at one individual artificial neuron, or

212
00:09:18,090 --> 00:09:20,789
what they call a unit, sitting in a hidden layer.

213
00:09:20,990 --> 00:09:24,049
OK, focusing on one guy. Right. How does it actually

214
00:09:24,049 --> 00:09:26,309
know what to do with the materials being handed

215
00:09:26,309 --> 00:09:28,629
up to it? Well, this is where we get into the

216
00:09:28,629 --> 00:09:30,370
math, but we are going to keep this grounded

217
00:09:30,370 --> 00:09:32,750
for you. The first thing this unit has to do

218
00:09:32,750 --> 00:09:34,970
is figure out its total input. It's just sitting

219
00:09:34,970 --> 00:09:37,090
there, and it has all these wires coming up to

220
00:09:37,090 --> 00:09:39,789
it from the floor below. And the signal traveling

221
00:09:39,789 --> 00:09:42,389
up each wire is multiplied by a specific weight.

222
00:09:43,149 --> 00:09:45,409
I always like to think of the weight as a volume

223
00:09:45,409 --> 00:09:48,090
knob on the wire. A volume knob? I like that.

224
00:09:48,529 --> 00:09:50,870
Yeah, so a signal comes up from the floor below.

225
00:09:51,370 --> 00:09:54,070
If the volume knob, the weight, is turned way

226
00:09:54,070 --> 00:09:56,730
up to a high positive number, that signal comes

227
00:09:56,730 --> 00:09:59,129
through loud and clear. Right. If the weight

228
00:09:59,129 --> 00:10:01,929
is near zero, the unit basically ignores the

229
00:10:01,929 --> 00:10:04,269
signal. But if the weight is a negative number,

230
00:10:04,610 --> 00:10:07,149
it actually inverts the signal. It acts as an

231
00:10:07,149 --> 00:10:10,409
inhibitor, telling the unit to quiet down. That's

232
00:10:10,409 --> 00:10:12,570
exactly how it functions. So the unit simply

233
00:10:12,570 --> 00:10:15,409
listens to all these incoming volume -adjusted

234
00:10:15,409 --> 00:10:17,549
signals from the layer below, and it adds them

235
00:10:17,549 --> 00:10:20,049
all together. Just a straight linear sum. Just

236
00:10:20,049 --> 00:10:23,090
addition. Right. That sum is its total input.

237
00:10:23,919 --> 00:10:26,879
But there is a catch here. There's this brilliant

238
00:10:26,879 --> 00:10:28,980
little mathematical trick the researchers use

239
00:10:28,980 --> 00:10:32,159
involving something called a bias. Yes, the bias

240
00:10:32,159 --> 00:10:35,360
is fundamental. In earlier models, he had to

241
00:10:35,360 --> 00:10:38,600
worry about a neuron's threshold. Like a biological

242
00:10:38,600 --> 00:10:41,440
neuron. Exactly. Just like a biological brain,

243
00:10:41,879 --> 00:10:44,120
a neuron builds up electrical charge, and it

244
00:10:44,120 --> 00:10:46,940
won't actually fire and send a signal until that

245
00:10:46,940 --> 00:10:50,779
charge crosses a specific threshold. So historically,

246
00:10:51,220 --> 00:10:53,100
programmers had to write two completely different

247
00:10:53,100 --> 00:10:55,950
sets of rules. one mathematical rule for how

248
00:10:55,950 --> 00:10:57,929
the network should adjust the volume knobs on

249
00:10:57,929 --> 00:11:00,750
the wires, and a completely separate rule for

250
00:11:00,750 --> 00:11:03,090
how the network should adjust the internal threshold

251
00:11:03,090 --> 00:11:05,710
of the neuron itself. Which was incredibly clunky.

252
00:11:05,870 --> 00:11:07,909
Yeah. Sounds like a nightmare to program. It

253
00:11:07,909 --> 00:11:10,990
was. But Rumelhart and Hinton bypassed this entirely.

254
00:11:11,850 --> 00:11:15,409
They introduced an extra phantom input to every

255
00:11:15,409 --> 00:11:17,730
single unit in the network. And this phantom

256
00:11:17,730 --> 00:11:20,950
input is bizarre because it never, ever changes.

257
00:11:21,070 --> 00:11:23,980
It is permanently glued to the number one. Exactly.

258
00:11:24,259 --> 00:11:27,340
It's a mathematical constant. And the weight

259
00:11:27,340 --> 00:11:30,840
on the wire connecting this constant, 1, to the

260
00:11:30,840 --> 00:11:33,919
unit is what they call the bias. Wait, I want

261
00:11:33,919 --> 00:11:35,740
to make sure I'm fully grasping why this is so

262
00:11:35,740 --> 00:11:39,000
clever. If you treat the threshold as just another

263
00:11:39,000 --> 00:11:42,200
weight on a wire that's always turned on, you

264
00:11:42,200 --> 00:11:45,200
don't need a second set of learning rules. Precisely.

265
00:11:45,500 --> 00:11:48,159
You solve both problems with one equation. Because

266
00:11:48,159 --> 00:11:50,019
the network already knows how to adjust weights

267
00:11:50,019 --> 00:11:52,870
to get better answers. So if a neuron is firing

268
00:11:52,870 --> 00:11:55,509
too easily and ruining the output, the network

269
00:11:55,509 --> 00:11:58,509
doesn't have to go into a special threshold adjustment

270
00:11:58,509 --> 00:12:01,190
mode. Right. It just treats it like any other

271
00:12:01,190 --> 00:12:04,370
wire. It just turns down the volume knob on that

272
00:12:04,370 --> 00:12:06,870
one specific phantom wire. It makes the bias

273
00:12:06,870 --> 00:12:09,570
weight highly negative, which constantly pulls

274
00:12:09,570 --> 00:12:12,129
the neurons total sum down, effectively making

275
00:12:12,129 --> 00:12:14,889
it harder for the other normal wires to trigger

276
00:12:14,889 --> 00:12:17,450
it. It is mathematically identical to raising

277
00:12:17,450 --> 00:12:20,830
a threshold, but... From the algorithm's perspective,

278
00:12:20,990 --> 00:12:24,129
it's just tuning another volume knob. It completely

279
00:12:24,129 --> 00:12:26,429
streamlines the learning procedure. It's so elegant.

280
00:12:27,029 --> 00:12:29,250
OK. So our unit has listened to all the incoming

281
00:12:29,250 --> 00:12:32,049
signals. It adjusted them by their weights, added

282
00:12:32,049 --> 00:12:34,230
them all up, and factored in its bias. Right.

283
00:12:34,450 --> 00:12:36,649
So it has a single number representing its total

284
00:12:36,649 --> 00:12:39,830
input. Let's say that number is 15. Does it just

285
00:12:39,830 --> 00:12:43,529
shout 15 up to the next floor? No. It cannot

286
00:12:43,529 --> 00:12:47,190
pass that linear sum directly up. It has to process

287
00:12:47,190 --> 00:12:49,830
it through a nonlinear function. This is often

288
00:12:49,830 --> 00:12:52,769
called a squashing function. And here is where

289
00:12:52,769 --> 00:12:54,809
I found myself scratching my head while reading

290
00:12:54,809 --> 00:12:57,690
this part of the paper. Why's that? Well, the

291
00:12:57,690 --> 00:13:00,049
paper says you can basically use any function

292
00:13:00,049 --> 00:13:02,429
as long as it has a smooth, bounded derivative,

293
00:13:02,909 --> 00:13:05,750
but they choose a very specific, somewhat complicated

294
00:13:05,750 --> 00:13:07,889
-looking S -curve called a logistic function.

295
00:13:08,190 --> 00:13:10,990
Right. The sigmoid. Yeah, why not just use a

296
00:13:10,990 --> 00:13:13,029
hard cutoff? Like, if the total input is greater

297
00:13:13,029 --> 00:13:15,409
than zero, fire a one. If it's less than zero,

298
00:13:15,429 --> 00:13:18,570
fire a zero. Why do we need this curvy, smooth

299
00:13:18,570 --> 00:13:22,200
mathematical slope? That is the pivotal question

300
00:13:22,200 --> 00:13:25,379
of early AI. The hard cutoff you just described,

301
00:13:25,679 --> 00:13:28,460
a step function, is exactly what the old failing

302
00:13:28,460 --> 00:13:33,200
perceptrons used. Oh. Yes. Really? The problem

303
00:13:33,200 --> 00:13:35,799
with a hard cutoff is that it gives you absolutely

304
00:13:35,799 --> 00:13:39,299
no nuance. Imagine you're trying to tune a radio.

305
00:13:39,399 --> 00:13:43,519
Okay. If your only feedback is no music or perfect

306
00:13:43,519 --> 00:13:45,679
music, you have no idea which way to turn the

307
00:13:45,679 --> 00:13:48,200
dial when you only hear static. You need to know

308
00:13:48,200 --> 00:13:50,600
if turning the dial slightly to the left makes

309
00:13:50,600 --> 00:13:53,720
the static slightly better. Ah, I see. You need

310
00:13:53,720 --> 00:13:56,200
a gradient. You need to know the rate of change.

311
00:13:56,580 --> 00:14:00,039
Exactly. A hard step function has no slope. It's

312
00:14:00,039 --> 00:14:02,820
just a flat line of zero. A sudden cliff straight

313
00:14:02,820 --> 00:14:06,080
up, and then a flat line of one. You can't calculate

314
00:14:06,080 --> 00:14:09,070
a useful derivative from a vertical cliff. That

315
00:14:09,070 --> 00:14:11,190
makes perfect sense. The logistic function they

316
00:14:11,190 --> 00:14:13,889
chose acts like a dimmer switch. It squashes

317
00:14:13,889 --> 00:14:16,490
any possible input number into a smooth range

318
00:14:16,490 --> 00:14:20,110
strictly between 0 and 1. So if the total input

319
00:14:20,110 --> 00:14:23,009
is a massive positive number like 1 ,000, the

320
00:14:23,009 --> 00:14:25,450
output gets incredibly close to 1 but never quite

321
00:14:25,450 --> 00:14:28,129
breaches it. Right. If the input is negative

322
00:14:28,129 --> 00:14:31,289
1 ,000, it gets infinitesimally close to 0. And

323
00:14:31,289 --> 00:14:34,309
if the input is exactly 0, the output is perfectly

324
00:14:34,309 --> 00:14:36,990
0 .5. Right. And because it's a smooth curve,

325
00:14:37,289 --> 00:14:40,429
if you were at an output of, say, 0 .8 and you

326
00:14:40,429 --> 00:14:43,149
change the input by a tiny fraction, the output

327
00:14:43,149 --> 00:14:46,210
changes by a predictable tiny fraction. Which

328
00:14:46,210 --> 00:14:49,230
you need for learning. This smooth continuous

329
00:14:49,230 --> 00:14:52,690
rate of change is the absolute bedrock requirement

330
00:14:52,690 --> 00:14:54,409
for the entire learning process we were about

331
00:14:54,409 --> 00:14:57,850
to discuss. Plus, as a bonus, the calculus for

332
00:14:57,850 --> 00:15:00,750
finding the derivative of this specific s -curve

333
00:15:00,750 --> 00:15:03,409
is incredibly simple to compute, which was vital

334
00:15:03,409 --> 00:15:06,230
for computers in 1986. But they didn't have the

335
00:15:06,230 --> 00:15:08,289
processing power we have now? Not even close.

336
00:15:08,970 --> 00:15:12,429
OK, let me pause and synthesize this so you listening

337
00:15:12,429 --> 00:15:14,789
have the complete picture in your head. We've

338
00:15:14,789 --> 00:15:17,370
built the machine. We feed raw numbers into the

339
00:15:17,370 --> 00:15:19,889
ground floor. Those numbers travel up wires.

340
00:15:20,570 --> 00:15:24,669
Every wire has a volume knob, a weight. At every

341
00:15:24,669 --> 00:15:26,990
single unit on the hidden floors, those volume

342
00:15:26,990 --> 00:15:29,490
-adjusted numbers are added together, offset

343
00:15:29,490 --> 00:15:32,460
by a bias. and then squashed through a smooth

344
00:15:32,460 --> 00:15:35,139
S -curve to create a new signal between 0 and

345
00:15:35,139 --> 00:15:37,860
1. Exactly. That new signal is passed up to the

346
00:15:37,860 --> 00:15:40,519
next floor, where the process repeats, all the

347
00:15:40,519 --> 00:15:42,419
way to the penthouse, where the final units light

348
00:15:42,419 --> 00:15:44,600
up with an answer. And that entire sequence,

349
00:15:45,019 --> 00:15:48,139
from raw data to final answer, is the complete

350
00:15:48,139 --> 00:15:51,899
forward pass. Which is great. It's a very impressive

351
00:15:51,899 --> 00:15:55,019
calculator. But here is the trillion -dollar

352
00:15:55,019 --> 00:15:57,899
problem that stalled AI for decades. What happens

353
00:15:57,899 --> 00:16:00,179
when the answer in the penthouse is completely

354
00:16:00,179 --> 00:16:03,460
spectacularly wrong? And it often is at first.

355
00:16:03,779 --> 00:16:06,039
Right. You feed the ground floor a beautiful

356
00:16:06,039 --> 00:16:08,480
picture of a golden retriever, and the penthouse

357
00:16:08,480 --> 00:16:11,039
lights up and shouts, this is a toaster. The

358
00:16:11,039 --> 00:16:13,299
forward pass is completely useless if the machine

359
00:16:13,299 --> 00:16:15,470
cannot learn from its mistakes. Right. How do

360
00:16:15,470 --> 00:16:17,830
we tell the network it messed up? And infinitely

361
00:16:17,830 --> 00:16:20,309
more difficult, how do we look at a network with

362
00:16:20,309 --> 00:16:22,950
potentially millions of interconnected volume

363
00:16:22,950 --> 00:16:25,389
knobs and tell it exactly which knobs to turn

364
00:16:25,389 --> 00:16:28,090
and in which direction to make sure it doesn't

365
00:16:28,090 --> 00:16:31,210
call a dog a toaster next time? That is the heart

366
00:16:31,210 --> 00:16:33,690
of the breakthrough. That is backpropagation.

367
00:16:34,029 --> 00:16:36,169
And it requires us to move into the second major

368
00:16:36,169 --> 00:16:39,269
phase of the paper, the backward pass. Let's

369
00:16:39,269 --> 00:16:42,419
do it. Step one seems obvious, right? We have

370
00:16:42,419 --> 00:16:45,000
to mathematically define exactly how wrong the

371
00:16:45,000 --> 00:16:48,460
network is. Yes. The researchers call this the

372
00:16:48,460 --> 00:16:51,279
total error. The ultimate goal is to find a set

373
00:16:51,279 --> 00:16:53,320
of weights that ensures the network's output

374
00:16:53,320 --> 00:16:56,700
exactly matches the desired output for every

375
00:16:56,700 --> 00:17:00,419
possible case. So they measure the gap. If the

376
00:17:00,419 --> 00:17:02,879
network spat out a 0 .2 for the toaster node,

377
00:17:03,299 --> 00:17:06,160
but we wanted a zero because it's a dog, not

378
00:17:06,160 --> 00:17:09,819
a toaster, the gap is 0 .2. Exactly. But they

379
00:17:09,819 --> 00:17:12,599
don't just add up those raw gaps. They take the

380
00:17:12,599 --> 00:17:14,619
difference between the actual output and the

381
00:17:14,619 --> 00:17:17,579
desired output and they square it. Now, I was

382
00:17:17,579 --> 00:17:19,559
looking at this error calculation and I had a

383
00:17:19,559 --> 00:17:22,380
moment of, wait, why square it? Why not just

384
00:17:22,380 --> 00:17:24,839
take the absolute value of the mistake? It's

385
00:17:24,839 --> 00:17:27,640
a common question. But then I realized, if you

386
00:17:27,640 --> 00:17:30,920
guess too high by two on one image and too low

387
00:17:30,920 --> 00:17:33,240
by two on the next image, and you just add the

388
00:17:33,240 --> 00:17:35,920
raw errors together, plus two and minus two equals

389
00:17:35,920 --> 00:17:38,640
zero. Precisely. The network would look at a

390
00:17:38,640 --> 00:17:41,099
total error of zero and think, I am a genius,

391
00:17:41,259 --> 00:17:44,440
I solved it, when it actually failed twice. Squaring

392
00:17:44,440 --> 00:17:46,440
the number forces every single mistake, whether

393
00:17:46,440 --> 00:17:48,720
it's too high or too low, to become a positive

394
00:17:48,720 --> 00:17:52,160
number. It forces errors to accumulate. But squaring

395
00:17:52,160 --> 00:17:55,099
it serves a second, equally vital purpose. What's

396
00:17:55,099 --> 00:17:58,059
that? Imagine you have a tiny error of 0 .1.

397
00:17:58,359 --> 00:18:01,160
If you square it, it becomes 0 .01. It shrinks.

398
00:18:01,440 --> 00:18:04,009
Right. But if you have a massive glaring error

399
00:18:04,009 --> 00:18:07,710
of 0 .8 and you square it, it becomes 0 .64.

400
00:18:08,750 --> 00:18:11,490
Squaring the gap severely penalizes large errors

401
00:18:11,490 --> 00:18:15,170
much more than small ones. Oh, wow. It acts as

402
00:18:15,170 --> 00:18:17,430
a mathematical triage, forcing the network to

403
00:18:17,430 --> 00:18:19,849
focus its energy on fixing its most catastrophic

404
00:18:19,849 --> 00:18:22,690
mistakes first, rather than obsessing over tiny

405
00:18:22,690 --> 00:18:26,170
micro adjustments. That is so smart. OK, so we

406
00:18:26,170 --> 00:18:28,789
sum up all those squared errors across every

407
00:18:28,789 --> 00:18:30,849
output node and every single training image.

408
00:18:31,250 --> 00:18:33,789
We now have a massive number representing our

409
00:18:33,789 --> 00:18:36,710
total error. And this gives us our singular goal.

410
00:18:37,009 --> 00:18:39,609
We want to minimize that total error. We want

411
00:18:39,609 --> 00:18:41,950
to drive it as close to zero as possible. This

412
00:18:41,950 --> 00:18:45,289
process is called gradient descent. I always

413
00:18:45,289 --> 00:18:48,230
picture this as a topographical map. Imagine

414
00:18:48,230 --> 00:18:51,509
the total error is the altitude of a massive

415
00:18:51,509 --> 00:18:54,900
jagged mountain range. OK, I can see that. The

416
00:18:54,900 --> 00:18:57,019
different volume knobs, the weights in the network,

417
00:18:57,339 --> 00:19:00,640
are the latitude and longitude coordinates. Our

418
00:19:00,640 --> 00:19:02,799
goal is to find the exact coordinates that put

419
00:19:02,799 --> 00:19:05,180
us at the bottom of the deepest valley in that

420
00:19:05,180 --> 00:19:08,220
entire mountain range. We want to descend the

421
00:19:08,220 --> 00:19:11,200
gradient. But remember the network is blindfolded.

422
00:19:11,299 --> 00:19:13,319
It can't just look at the map from above. To

423
00:19:13,319 --> 00:19:15,980
know which way is down, it needs to feel the

424
00:19:15,980 --> 00:19:17,920
slope of the ground directly beneath its feet.

425
00:19:18,779 --> 00:19:20,279
Mathematically, what does that mean? It means

426
00:19:20,279 --> 00:19:22,220
it needs to calculate the partial derivative

427
00:19:22,220 --> 00:19:25,180
of the total error with respect to every single

428
00:19:25,180 --> 00:19:27,980
weight in the network. Meaning, for every single

429
00:19:27,980 --> 00:19:30,799
wire, the network needs to ask, if I turn this

430
00:19:30,799 --> 00:19:33,640
specific volume knob up just a tiny little bit,

431
00:19:34,119 --> 00:19:36,839
does the total altitude go up or does it go down?

432
00:19:37,059 --> 00:19:39,539
Exactly. Now, calculating that slope for the

433
00:19:39,539 --> 00:19:41,200
weights directly connected to the penthouse,

434
00:19:41,259 --> 00:19:44,119
the output layer, is relatively straightforward.

435
00:19:44,980 --> 00:19:47,000
Because they're right there at the end. Right.

436
00:19:47,099 --> 00:19:48,660
You just look at the final mistake and the final

437
00:19:48,660 --> 00:19:51,380
signal. But how do you calculate the exact blame

438
00:19:51,380 --> 00:19:54,299
for a single volume knob buried three layers

439
00:19:54,299 --> 00:19:56,750
deep in the hidden floors? Yeah, that's where

440
00:19:56,750 --> 00:19:59,130
it gets messy. Anamu's signal was mixed with

441
00:19:59,130 --> 00:20:01,750
a hundred other signals squashed through an S

442
00:20:01,750 --> 00:20:04,569
-curve and mixed again. That feels impossible.

443
00:20:05,029 --> 00:20:06,990
It did feel impossible to many people at the

444
00:20:06,990 --> 00:20:09,890
time, but Rumelhart, Hinton, and Williams solved

445
00:20:09,890 --> 00:20:12,930
it by meticulously applying the chain rule of

446
00:20:12,930 --> 00:20:15,750
calculus in reverse. Okay, I want everyone listening

447
00:20:15,750 --> 00:20:17,470
to stick with us here because we are going to

448
00:20:17,470 --> 00:20:20,490
translate this calculus into intuition. We are

449
00:20:20,490 --> 00:20:23,309
going to trace the error backward without reading

450
00:20:23,309 --> 00:20:27,660
Greek symbols. Good luck. Well, when I finally

451
00:20:27,660 --> 00:20:29,799
understood how this worked, it completely blew

452
00:20:29,799 --> 00:20:32,680
my mind. It is exactly like the hierarchy of

453
00:20:32,680 --> 00:20:36,019
blame in a massive corporate bureaucracy. I am

454
00:20:36,019 --> 00:20:38,559
intrigued. Walk us through the corporate blame

455
00:20:38,559 --> 00:20:41,240
analogy. All right. Imagine the neural network

456
00:20:41,240 --> 00:20:44,099
is a massive multinational corporation. At the

457
00:20:44,099 --> 00:20:47,059
very top, you have the output layer. The total

458
00:20:47,059 --> 00:20:50,059
error is a furious CEO screaming that quarterly

459
00:20:50,059 --> 00:20:52,640
profits are drastically down. OK, I'm with you.

460
00:20:52,779 --> 00:20:54,799
Now, the CEO doesn't know why profits are down.

461
00:20:55,099 --> 00:20:57,200
They don't know how the factory floor operates.

462
00:20:57,579 --> 00:20:59,420
They just know there is a massive gap between

463
00:20:59,420 --> 00:21:02,460
the desired profits and actual profits. So who

464
00:21:02,460 --> 00:21:05,539
does the CEO yell at? The vice presidents. Exactly.

465
00:21:05,740 --> 00:21:09,059
The VPs are the output units. The CEO screams

466
00:21:09,059 --> 00:21:11,779
at the VPs. But a VP knows they don't actually

467
00:21:11,779 --> 00:21:14,359
build the products. They just process what the

468
00:21:14,359 --> 00:21:17,329
regional managers send them. Right. So the VP

469
00:21:17,329 --> 00:21:20,029
translates the CEO's rage into an understanding

470
00:21:20,029 --> 00:21:22,390
of how much their specific department's total

471
00:21:22,390 --> 00:21:25,349
input is to blame. They calculate their department's

472
00:21:25,349 --> 00:21:28,130
share of the massive error. Which, mathematically,

473
00:21:28,430 --> 00:21:30,390
is the equivalent of taking the derivative of

474
00:21:30,390 --> 00:21:33,210
the error with respect to the output and multiplying

475
00:21:33,210 --> 00:21:35,589
it by the derivative of that squashing function

476
00:21:35,589 --> 00:21:38,230
we talked about earlier. Right. The VP figures

477
00:21:38,230 --> 00:21:41,170
out their exact percentage of the blame. Then,

478
00:21:41,349 --> 00:21:43,470
the VP looks down at all the regional managers

479
00:21:43,470 --> 00:21:46,140
reporting to them. The managers are the signals

480
00:21:46,140 --> 00:21:48,400
coming up from the hidden layer. And the connections

481
00:21:48,400 --> 00:21:51,119
between them are the weights. Right, the performance

482
00:21:51,119 --> 00:21:53,420
bonuses. The VP looks at the incoming signals

483
00:21:53,420 --> 00:21:56,720
and says, OK, manager A sent me a huge shipment

484
00:21:56,720 --> 00:21:58,880
of terrible data, so they are getting 80 % of

485
00:21:58,880 --> 00:22:02,640
my blame. Manager B sent me almost nothing, so

486
00:22:02,640 --> 00:22:04,819
changing their bonus won't fix anything. They

487
00:22:04,819 --> 00:22:07,259
get 2 % of the blame. They are distributing the

488
00:22:07,259 --> 00:22:10,079
error backward, proportional to the strength

489
00:22:10,079 --> 00:22:14,170
of the forward signal. Exactly. But here is where

490
00:22:14,170 --> 00:22:16,390
the absolute genius of backpropagation kicks

491
00:22:16,390 --> 00:22:18,509
in. Because we aren't just adjusting the final

492
00:22:18,509 --> 00:22:21,069
layer, we have to go deeper. What happens at

493
00:22:21,069 --> 00:22:23,630
the regional manager level? Well, a regional

494
00:22:23,630 --> 00:22:26,769
manager, a hidden unit, doesn't have a desired

495
00:22:26,769 --> 00:22:30,049
target. The CEO didn't give them a quota directly.

496
00:22:30,309 --> 00:22:32,809
Right. And in this corporation, a regional manager

497
00:22:32,809 --> 00:22:35,849
doesn't just report to one VP. Their work is

498
00:22:35,849 --> 00:22:38,529
distributed. They might report to three different

499
00:22:38,529 --> 00:22:41,690
VPs. So how does this middle manager figure out

500
00:22:41,690 --> 00:22:44,759
how badly they messed up. They have to sum up

501
00:22:44,759 --> 00:22:47,559
the blame. Yes. The regional manager listens

502
00:22:47,559 --> 00:22:50,319
to all the VPs they report to. VP number one

503
00:22:50,319 --> 00:22:52,839
is screaming at them. VP number two is giving

504
00:22:52,839 --> 00:22:55,799
them a mild reprimand. VP number three is actually

505
00:22:55,799 --> 00:22:58,640
praising them. So they aggregate it. The manager

506
00:22:58,640 --> 00:23:00,740
mathematically adds up all those complaints and

507
00:23:00,740 --> 00:23:03,019
praises weighted by how important their connection

508
00:23:03,019 --> 00:23:06,859
to that specific VP is. Once a manager calculates

509
00:23:06,859 --> 00:23:09,359
their total accumulated blame from above, they

510
00:23:09,359 --> 00:23:12,769
act just like a VP. The cycle continues. They

511
00:23:12,769 --> 00:23:14,750
figure out their department's total failure,

512
00:23:15,089 --> 00:23:16,930
they look down at the interns reporting to them

513
00:23:16,930 --> 00:23:18,990
on the ground floor, and they pass the blame

514
00:23:18,990 --> 00:23:22,329
down again using the exact same math. It cascades.

515
00:23:23,069 --> 00:23:25,730
You repeat this entire process, passing the derivative

516
00:23:25,730 --> 00:23:29,170
of the error backward, layer by layer, multiplying

517
00:23:29,170 --> 00:23:32,230
and summing all the way down to the bottom. That's

518
00:23:32,230 --> 00:23:35,369
incredible. It is an incredibly precise mechanism

519
00:23:35,369 --> 00:23:38,450
for assigning microscopic slivers of responsibility

520
00:23:38,750 --> 00:23:41,829
for a macroscopic failure across a massively

521
00:23:41,829 --> 00:23:44,609
interconnected system. It's beautiful, really.

522
00:23:45,109 --> 00:23:46,990
Because no individual unit can see the whole

523
00:23:46,990 --> 00:23:49,849
picture, but through this reverse chain of command,

524
00:23:50,130 --> 00:23:52,309
every single volume knob in the entire network

525
00:23:52,309 --> 00:23:54,450
gets a specific number attached to it that basically

526
00:23:54,450 --> 00:23:57,569
says, if you want to lower the CEO's anger, turn

527
00:23:57,569 --> 00:24:00,710
me exactly this much to the left. And that brings

528
00:24:00,710 --> 00:24:03,369
us to how we actually update the weights. We

529
00:24:03,369 --> 00:24:05,809
have the instructions, but when do we make the

530
00:24:05,809 --> 00:24:09,049
change? The paper gives us two schemes. Scheme

531
00:24:09,049 --> 00:24:11,970
1 seems like the most obvious approach. You show

532
00:24:11,970 --> 00:24:15,309
the network an image, it makes a guess, you calculate

533
00:24:15,309 --> 00:24:17,970
the CEO's anger, you pass the blame down, and

534
00:24:17,970 --> 00:24:20,390
you immediately tweak all the volume knobs. Then

535
00:24:20,390 --> 00:24:22,509
you move to the next image. Which is efficient

536
00:24:22,509 --> 00:24:25,250
in terms of computer memory. You calculate the

537
00:24:25,250 --> 00:24:28,549
gradient, use it, and discard it. But the authors

538
00:24:28,549 --> 00:24:30,609
actually chose Scheme 2 for their core experiments.

539
00:24:31,170 --> 00:24:34,609
And Scheme 2 is wild. You don't change anything

540
00:24:34,609 --> 00:24:36,710
immediately. No. You show the network an image

541
00:24:36,710 --> 00:24:39,500
of a dog. calculate the massive blame matrix,

542
00:24:39,940 --> 00:24:41,619
and then you just write those numbers down on

543
00:24:41,619 --> 00:24:43,559
a clipboard. You don't touch the volume knobs.

544
00:24:43,880 --> 00:24:46,400
Then you show it a cat, calculate the new blame,

545
00:24:46,660 --> 00:24:48,940
and add it to the clipboard. You accumulate the

546
00:24:48,940 --> 00:24:51,180
gradient over every single image in your entire

547
00:24:51,180 --> 00:24:53,900
training set before you finally make a single

548
00:24:53,900 --> 00:24:56,359
combined adjustment to the weights. Why do you

549
00:24:56,359 --> 00:24:58,619
think they did that? Well, let's go back to your

550
00:24:58,619 --> 00:25:01,200
blindfolded person on the mountain. If you take

551
00:25:01,200 --> 00:25:04,180
a step after every single image, you might just

552
00:25:04,180 --> 00:25:06,440
be reacting to a weirdly shaped rock under your

553
00:25:06,440 --> 00:25:09,509
foot. Oh, like a local anomaly. Exactly. You

554
00:25:09,509 --> 00:25:12,029
might take a step north for the dog, and then

555
00:25:12,029 --> 00:25:15,150
a step south for the cat, and just vibrate in

556
00:25:15,150 --> 00:25:18,529
place. But by accumulating the gradient, you

557
00:25:18,529 --> 00:25:20,630
are essentially feeling the average slope of

558
00:25:20,630 --> 00:25:23,049
the entire mountain range at once. Ignoring the

559
00:25:23,049 --> 00:25:26,549
local pebbles. Right. And taking one confident

560
00:25:26,549 --> 00:25:29,809
step in the truest downward direction. That makes

561
00:25:29,809 --> 00:25:32,190
a lot of sense. It provides a much smoother,

562
00:25:32,369 --> 00:25:34,789
more stable descent toward the minimum error.

563
00:25:34,990 --> 00:25:37,250
And when they finally take that step, the size

564
00:25:37,250 --> 00:25:38,970
of the step is determined by something called

565
00:25:38,970 --> 00:25:41,960
the learning rate. A small fraction that prevents

566
00:25:41,960 --> 00:25:45,059
the network from taking a wildly huge leap and

567
00:25:45,059 --> 00:25:46,940
accidentally overshooting the valley entirely.

568
00:25:47,500 --> 00:25:50,099
But the authors realized that a tiny cautious

569
00:25:50,099 --> 00:25:53,279
learning rate can be agonizingly slow. So they

570
00:25:53,279 --> 00:25:55,339
introduced a brilliant workaround to speed things

571
00:25:55,339 --> 00:25:59,819
up. Momentum. Momentum in weight space. It's

572
00:25:59,819 --> 00:26:02,220
a really fascinating concept. I love the physics

573
00:26:02,220 --> 00:26:05,259
analogy here. If you are a ball rolling down

574
00:26:05,259 --> 00:26:08,079
a physical hill, you don't just calculate your

575
00:26:08,079 --> 00:26:10,339
speed based on the exact slope directly under

576
00:26:10,339 --> 00:26:12,339
you at that specific millisecond. Right, because

577
00:26:12,339 --> 00:26:15,019
you have mass. You have momentum. You remember

578
00:26:15,019 --> 00:26:16,740
that you've been rolling downhill for the last

579
00:26:16,740 --> 00:26:19,420
10 seconds, so you build up speed. And the authors

580
00:26:19,420 --> 00:26:21,900
translate this literally into the math. When

581
00:26:21,900 --> 00:26:23,900
calculating the new weight change, they don't

582
00:26:23,900 --> 00:26:26,720
just use the current gradient. They add a fraction

583
00:26:26,720 --> 00:26:29,019
of the previous weight change. So if the current

584
00:26:29,019 --> 00:26:32,410
slope says, go left, and the previous slope also

585
00:26:32,410 --> 00:26:35,450
said go left, the network says, great, we are

586
00:26:35,450 --> 00:26:37,950
definitely going left. Let's accelerate. And

587
00:26:37,950 --> 00:26:40,289
crucially, what happens when it hits a speed

588
00:26:40,289 --> 00:26:42,890
bump? This is the best part. If you are rolling

589
00:26:42,890 --> 00:26:45,410
fast down a long hill to the right and you hit

590
00:26:45,410 --> 00:26:47,950
a tiny rut where the local slope technically

591
00:26:47,950 --> 00:26:51,109
tilts left, without momentum, the algorithm would

592
00:26:51,109 --> 00:26:53,650
just stop and turn around. It would get stuck

593
00:26:53,650 --> 00:26:56,519
in the rut. But with momentum, With momentum,

594
00:26:56,779 --> 00:26:59,240
the massive historical force of rolling right

595
00:26:59,240 --> 00:27:02,359
completely overpowers the tiny local signal of

596
00:27:02,359 --> 00:27:04,920
tilt left. And the network just powers straight

597
00:27:04,920 --> 00:27:06,740
through the speed bump and continues down the

598
00:27:06,740 --> 00:27:09,099
mountain. It allows for incredibly fast learning

599
00:27:09,099 --> 00:27:11,240
rates without the network getting jittery or

600
00:27:11,240 --> 00:27:14,140
trapped in tiny surface variations. OK, we have

601
00:27:14,140 --> 00:27:16,559
built the engine. We understand the assembly

602
00:27:16,559 --> 00:27:20,000
line forward pass and the corporate blame backward

603
00:27:20,000 --> 00:27:23,140
pass. We have the theory. It makes logical sense

604
00:27:23,140 --> 00:27:26,769
on paper, but... As we know, the perceptron made

605
00:27:26,769 --> 00:27:29,230
sense on paper, too, right up until Minsky and

606
00:27:29,230 --> 00:27:32,619
Papert proved it was a dead end. Does backpropagation

607
00:27:32,619 --> 00:27:35,599
actually work on a problem that those rigid perceptrons

608
00:27:35,599 --> 00:27:37,819
fundamentally couldn't solve? To prove it does,

609
00:27:37,920 --> 00:27:39,619
the authors devised a series of experiments.

610
00:27:40,420 --> 00:27:42,880
And the first one is a classic, elegant logic

611
00:27:42,880 --> 00:27:46,289
puzzle, the detection of symmetry. Let's set

612
00:27:46,289 --> 00:27:49,230
the stage for this. The task seems absurdly simple

613
00:27:49,230 --> 00:27:51,609
for a human. You have a one -dimensional array,

614
00:27:51,690 --> 00:27:54,470
just a straight line, of binary inputs. a line

615
00:27:54,470 --> 00:27:57,650
of 6 ones and zeros. Okay. The network has to

616
00:27:57,650 --> 00:27:59,890
look at this line and answer one simple question.

617
00:28:00,690 --> 00:28:02,650
Is the pattern symmetrical around the center

618
00:28:02,650 --> 00:28:08,309
point? Yes or no? So if the line is 101101, you

619
00:28:08,309 --> 00:28:09,970
draw a line down the middle, the left mirrors

620
00:28:09,970 --> 00:28:13,250
the right, the answer is yes. Right. And if it's

621
00:28:13,250 --> 00:28:16,430
111000, it doesn't mirror at all, the answer

622
00:28:16,430 --> 00:28:19,250
is no. This takes a human a fraction of a second.

623
00:28:19,579 --> 00:28:22,559
Why was this considered a famously impossible

624
00:28:22,559 --> 00:28:25,339
problem for earlier AI? Because of the limitations

625
00:28:25,339 --> 00:28:28,099
we discussed earlier. A perceptron tries to solve

626
00:28:28,099 --> 00:28:30,660
problems by looking at individual pieces of evidence

627
00:28:30,660 --> 00:28:33,539
and adding them up. But symmetry cannot be solved

628
00:28:33,539 --> 00:28:37,259
that way. Why not? The formal proof by Minsky

629
00:28:37,259 --> 00:28:40,140
and Papert demonstrated that looking at an individual

630
00:28:40,140 --> 00:28:43,119
input unit provides absolutely zero evidence

631
00:28:43,119 --> 00:28:45,910
about the symmetry of the entire vector. Right,

632
00:28:46,029 --> 00:28:48,369
because if I tell you, hey, the very first digit

633
00:28:48,369 --> 00:28:50,650
on the far left is a one, that tells you nothing.

634
00:28:50,849 --> 00:28:53,430
It could be symmetrical if the far right digit

635
00:28:53,430 --> 00:28:56,309
is also a one, but it's asymmetrical if the far

636
00:28:56,309 --> 00:28:58,809
right is a zero. The individual data point is

637
00:28:58,809 --> 00:29:01,369
totally meaningless in isolation. Furthermore,

638
00:29:01,609 --> 00:29:03,309
you can't even look at the sum of the halves.

639
00:29:03,710 --> 00:29:06,450
If the left side has two ones and the right side

640
00:29:06,450 --> 00:29:08,589
has two ones, they can be in completely different

641
00:29:08,589 --> 00:29:12,490
non -mirroring positions. Oh, true. You absolutely

642
00:29:12,490 --> 00:29:15,450
must be able to compare specific paired relationships

643
00:29:15,450 --> 00:29:18,250
across the centerline. You must have a hidden

644
00:29:18,250 --> 00:29:21,230
layer capable of abstracting those exact relationships.

645
00:29:22,230 --> 00:29:24,390
So Hinton and his team set up their back propagation

646
00:29:24,390 --> 00:29:27,410
network. They created an input layer of six units,

647
00:29:27,890 --> 00:29:30,470
giving them 64 possible combinations of ones

648
00:29:30,470 --> 00:29:33,170
and zeros. And they sandwiched a hidden layer

649
00:29:33,170 --> 00:29:35,950
between the input and the output. But before

650
00:29:35,950 --> 00:29:37,869
they started training, they did something crucial.

651
00:29:38,250 --> 00:29:41,349
They initialized all the weights with small random

652
00:29:41,349 --> 00:29:44,789
numbers. Why random? Why not start everything

653
00:29:44,789 --> 00:29:47,990
at a clean zero? Think about your corporate analogy.

654
00:29:49,339 --> 00:29:51,539
If you hire five identical regional managers,

655
00:29:52,380 --> 00:29:54,140
give them the exact same starting instructions,

656
00:29:54,660 --> 00:29:57,180
give them the exact same data, and yell at them

657
00:29:57,180 --> 00:29:59,960
the exact same way when they fail, they will

658
00:29:59,960 --> 00:30:02,720
all make the exact same adjustments. They will

659
00:30:02,720 --> 00:30:05,279
remain identical clones forever. And you don't

660
00:30:05,279 --> 00:30:07,880
need five clums doing the exact same job. You

661
00:30:07,880 --> 00:30:10,980
need them to specialize. Exactly. By starting

662
00:30:10,980 --> 00:30:12,880
the weights with random noise, some slightly

663
00:30:12,880 --> 00:30:15,480
positive, some slightly negative, you basically

664
00:30:15,480 --> 00:30:18,640
break the symmetry. When the CEO yells, the slightly

665
00:30:18,640 --> 00:30:20,500
different starting weights cause the managers

666
00:30:20,500 --> 00:30:23,000
to absorb the blame differently. Which causes

667
00:30:23,000 --> 00:30:25,319
them to adjust their weights differently. They

668
00:30:25,319 --> 00:30:27,319
begin to carve out unique areas of expertise.

669
00:30:27,880 --> 00:30:30,259
So they initialize the random noise, feed in

670
00:30:30,259 --> 00:30:33,640
the 64 patterns, and let the network run. And

671
00:30:33,640 --> 00:30:36,700
in 1986, this was computationally heavy. It took

672
00:30:36,700 --> 00:30:41,400
1 ,425 complete sweeps through all 64 patterns.

673
00:30:41,920 --> 00:30:45,660
That's 1 ,425 times the CEO screamed at the VPs

674
00:30:45,660 --> 00:30:48,799
before the error finally plummeted to zero. But

675
00:30:48,799 --> 00:30:51,640
when they cracked open the black box to see what

676
00:30:51,640 --> 00:30:54,119
the hidden layer had learned, the authors call

677
00:30:54,119 --> 00:30:57,380
it an elegant solution, and it truly is. It's

678
00:30:57,380 --> 00:31:00,000
a master class in autonomous mathematical balancing.

679
00:31:00,740 --> 00:31:03,380
Let's look at how just one of those hidden units

680
00:31:03,380 --> 00:31:06,559
organized its volume knobs. I am going to walk

681
00:31:06,559 --> 00:31:08,900
through this slowly because visualizing this

682
00:31:08,900 --> 00:31:11,500
invisible math is, for me, the coolest part of

683
00:31:11,500 --> 00:31:14,160
the entire paper. The network arranged the weights

684
00:31:14,160 --> 00:31:16,500
connecting the inputs to the specific hidden

685
00:31:16,500 --> 00:31:20,960
unit in a perfect mirrored balance. How so? The

686
00:31:20,960 --> 00:31:22,500
weights on the left side of the array became

687
00:31:22,500 --> 00:31:25,279
equal in magnitude but completely opposite in

688
00:31:25,279 --> 00:31:27,480
sign to the corresponding weights on the right

689
00:31:27,480 --> 00:31:29,779
side. Let's anchor that with numbers to make

690
00:31:29,779 --> 00:31:32,160
it clear. If the first input unit on the far

691
00:31:32,160 --> 00:31:34,519
left learned a connection weight of plus 10,

692
00:31:34,980 --> 00:31:37,559
the input unit on the far right learned a weight

693
00:31:37,559 --> 00:31:41,059
of minus 10. Exactly. If the second from the

694
00:31:41,059 --> 00:31:43,240
left was minus 4, the second from the right was

695
00:31:43,240 --> 00:31:45,980
plus 4. So trace what happens when a perfectly

696
00:31:45,980 --> 00:31:48,660
symmetrical pattern is fed in. Let's say the

697
00:31:48,660 --> 00:31:51,019
two outer edges are both 1 and the middle is

698
00:31:51,019 --> 00:31:54,259
all zeros. The hidden unit receives a plus 10

699
00:31:54,259 --> 00:31:56,819
signal from the left and a minus 10 signal from

700
00:31:56,819 --> 00:31:59,279
the right. And because we are just doing a linear

701
00:31:59,279 --> 00:32:02,299
sum, they perfectly cancel each other out. The

702
00:32:02,299 --> 00:32:04,779
net total input from the array is exactly zero.

703
00:32:05,539 --> 00:32:07,460
This holds true for any symmetrical pattern.

704
00:32:07,940 --> 00:32:09,799
The positive and negative weights will always

705
00:32:09,799 --> 00:32:11,740
perfectly annihilate each other across the mirror

706
00:32:11,740 --> 00:32:14,339
line. Here is where the bias trick comes in.

707
00:32:14,819 --> 00:32:16,660
The network learned to give this hidden unit

708
00:32:16,660 --> 00:32:20,440
a massive negative bias. Let's say negative five.

709
00:32:21,430 --> 00:32:24,640
So the inputs cancel out to zero. but the bias

710
00:32:24,640 --> 00:32:27,140
is constantly dragging the unit down to negative

711
00:32:27,140 --> 00:32:30,559
5. And because of that squashing S -curve, a

712
00:32:30,559 --> 00:32:32,960
massive negative input means the hidden unit

713
00:32:32,960 --> 00:32:35,960
is effectively turned off. Its output is crushed

714
00:32:35,960 --> 00:32:39,759
down to near zero. OK, so for a symmetrical pattern,

715
00:32:40,279 --> 00:32:43,309
the hidden unit stays silent. Now look at the

716
00:32:43,309 --> 00:32:45,990
final output unit in the penthouse. The network

717
00:32:45,990 --> 00:32:48,630
learned to give the output unit a positive bias.

718
00:32:49,549 --> 00:32:52,630
So if the hidden unit is silent, The output unit

719
00:32:52,630 --> 00:32:55,869
just feels its own positive bias, and it happily

720
00:32:55,869 --> 00:32:59,630
turns on, shouting, yes, symmetrical. But what

721
00:32:59,630 --> 00:33:02,869
happens if the pattern is asymmetrical? What

722
00:33:02,869 --> 00:33:06,329
if the left edge is 1 with a weight of plus 10,

723
00:33:06,529 --> 00:33:10,289
and the right edge is 0 with a weight of 0? The

724
00:33:10,289 --> 00:33:13,470
sum is no longer 0. The left side overpowers

725
00:33:13,470 --> 00:33:16,410
the right. The hidden unit gets a massive jolt

726
00:33:16,410 --> 00:33:19,269
of plus 10, which easily overcomes its negative

727
00:33:19,269 --> 00:33:23,140
bias of negative 5. The hidden unit wakes up.

728
00:33:23,299 --> 00:33:25,819
It turns on and fires a massive signal up to

729
00:33:25,819 --> 00:33:28,220
the penthouse. And the final piece of the trap.

730
00:33:29,000 --> 00:33:31,519
The connection from the hidden unit to the penthouse

731
00:33:31,519 --> 00:33:35,400
has an incredibly strong negative weight. Right.

732
00:33:35,599 --> 00:33:38,220
So when the hidden unit wakes up, it blasts the

733
00:33:38,220 --> 00:33:40,920
output unit with this huge negative signal, completely

734
00:33:40,920 --> 00:33:43,259
suppressing the output unit's positive bias,

735
00:33:43,599 --> 00:33:47,119
forcing it to turn off, signaling, no, not symmetrical.

736
00:33:47,660 --> 00:33:50,789
It is remarkably clever. But there is a meticulous

737
00:33:50,789 --> 00:33:53,589
detail in the text that shows just how mathematically

738
00:33:53,589 --> 00:33:56,640
precise this gradient descent can be. The weights

739
00:33:56,640 --> 00:33:58,640
on each side of the midpoint aren't just random

740
00:33:58,640 --> 00:34:01,440
opposing numbers. No, they aren't. The paper

741
00:34:01,440 --> 00:34:04,519
notes they fell into a ratio of 1 to 2 to 4.

742
00:34:04,819 --> 00:34:07,400
Like binary code. Exactly. Why do you think the

743
00:34:07,400 --> 00:34:09,559
network settled on that specific exponential

744
00:34:09,559 --> 00:34:12,579
ratio? I had to think about this and it really

745
00:34:12,579 --> 00:34:15,570
blew my mind. If you use a 1 to 2 to 4 ratio,

746
00:34:15,829 --> 00:34:18,369
you guarantee that every single possible combination

747
00:34:18,369 --> 00:34:21,670
of 1s and 0s on the left side will generate a

748
00:34:21,670 --> 00:34:25,110
totally unique mathematical sum. Yes. Because

749
00:34:25,110 --> 00:34:29,070
if it was 1, 2, 3, then 1 plus 2 equals 3. You'd

750
00:34:29,070 --> 00:34:33,250
have overlapping sums. Right. So by using 1 to

751
00:34:33,250 --> 00:34:36,190
2 to 4, there is absolutely no way a random pattern

752
00:34:36,190 --> 00:34:38,690
on the right side can accidentally equal the

753
00:34:38,690 --> 00:34:40,909
sum of a completely different pattern on the

754
00:34:40,909 --> 00:34:44,110
left side. It's impossible. The only way the

755
00:34:44,110 --> 00:34:46,690
sum from the right side can exactly balance the

756
00:34:46,690 --> 00:34:50,070
unique sum from the left side is if it is a perfect,

757
00:34:50,170 --> 00:34:54,269
exact mirror image. It is an airtight mathematical

758
00:34:54,269 --> 00:34:57,030
proof against false positives, constructed entirely

759
00:34:57,030 --> 00:34:59,489
by a blind algorithm following the chain rule

760
00:34:59,489 --> 00:35:02,130
of errors. The researchers didn't program the

761
00:35:02,130 --> 00:35:05,570
1 .2 .4 ratio. They didn't program the equal

762
00:35:05,570 --> 00:35:08,110
and opposite signs. They didn't tell it to use

763
00:35:08,110 --> 00:35:10,670
negative biases to create a default off state.

764
00:35:11,079 --> 00:35:14,119
The network invented a perfectly balanced mathematical

765
00:35:14,119 --> 00:35:17,400
scale out of random noise. It perfectly validates

766
00:35:17,400 --> 00:35:20,079
their thesis. Back propagation can construct

767
00:35:20,079 --> 00:35:23,699
appropriate internal representations. But detecting

768
00:35:23,699 --> 00:35:26,119
symmetry is still, you know, a mathematical parlor

769
00:35:26,119 --> 00:35:29,360
trick. It's numbers and logic. A truly revolutionary

770
00:35:29,360 --> 00:35:32,139
AI needs to be able to handle messy, abstract

771
00:35:32,139 --> 00:35:35,139
human concepts. Things like semantics, categories,

772
00:35:35,599 --> 00:35:38,480
sociology. Which brings us to the most profound

773
00:35:38,480 --> 00:35:41,159
experiment in the paper. Experiment two. Family

774
00:35:41,159 --> 00:35:44,519
tree. If the algorithm is truly general purpose,

775
00:35:44,519 --> 00:35:48,389
how do we force it to learn sociology? The researchers

776
00:35:48,389 --> 00:35:50,429
introduce a completely different kind of task.

777
00:35:50,690 --> 00:35:52,989
They present two family trees, and they are what

778
00:35:52,989 --> 00:35:56,530
the text calls isomorphic family trees. Which

779
00:35:56,530 --> 00:35:59,110
simply means they have the exact same structural

780
00:35:59,110 --> 00:36:01,449
shape, but completely different labels. Right.

781
00:36:01,670 --> 00:36:04,050
One tree is an English family. Christopher and

782
00:36:04,050 --> 00:36:05,989
Penelope are married. They have a son named Colin.

783
00:36:06,349 --> 00:36:08,949
Colin has daughters, etc. The second tree is

784
00:36:08,949 --> 00:36:11,659
an Italian family. Roberto and Maria are married.

785
00:36:11,780 --> 00:36:14,079
They have a son named Alfonso. So the internal

786
00:36:14,079 --> 00:36:16,559
relationships, who is the patriarch, who is the

787
00:36:16,559 --> 00:36:19,099
aunt, who is the grandson, are structurally identical

788
00:36:19,099 --> 00:36:21,059
between the two families, even though the names

789
00:36:21,059 --> 00:36:24,119
share absolutely no similarities. The data is

790
00:36:24,119 --> 00:36:26,280
fed to the network as a set of logical triples,

791
00:36:26,900 --> 00:36:30,280
a formula. Person one plus relationship equals

792
00:36:30,280 --> 00:36:33,059
person two. For example, the input would be the

793
00:36:33,059 --> 00:36:36,539
name Colin and the relationship has aunt. The

794
00:36:36,539 --> 00:36:38,219
desired output from the network would be the

795
00:36:38,219 --> 00:36:40,590
correct answer. Victoria and Jennifer because

796
00:36:40,590 --> 00:36:43,489
Colin has two aunts. Now this feels radically

797
00:36:43,489 --> 00:36:45,469
different from looking at a string of ones and

798
00:36:45,469 --> 00:36:47,949
zeros. The network doesn't know what an aunt

799
00:36:47,949 --> 00:36:51,449
is. It doesn't know Colin is a man. How do they

800
00:36:51,449 --> 00:36:54,610
build an architecture to process this? They built

801
00:36:54,610 --> 00:36:58,210
a massive, highly constrained five layer architecture.

802
00:36:58,670 --> 00:37:00,929
Let's visualize this because the shape of the

803
00:37:00,929 --> 00:37:04,050
network totally dictates how it learns. On the

804
00:37:04,050 --> 00:37:06,710
ground floor, you have two totally separate groups

805
00:37:06,710 --> 00:37:10,219
of inputs. On the left, you have 24 distinct

806
00:37:10,219 --> 00:37:13,079
units, each representing one specific person

807
00:37:13,079 --> 00:37:16,369
in the two family trees. When we want to input

808
00:37:16,369 --> 00:37:20,610
colon, we turn the colon switch to 1, and the

809
00:37:20,610 --> 00:37:24,880
other 23 switches to 0. And on the right? On

810
00:37:24,880 --> 00:37:26,559
the right side of the ground floor, you have

811
00:37:26,559 --> 00:37:29,739
12 input units representing the 12 possible relationships,

812
00:37:30,380 --> 00:37:34,719
father, mother, husband, aunt, etc. Now move

813
00:37:34,719 --> 00:37:37,000
to the second floor. This is where the magic

814
00:37:37,000 --> 00:37:40,239
happens. The 24 person inputs do not connect

815
00:37:40,239 --> 00:37:43,559
to a massive, open, hidden layer. They are strictly

816
00:37:43,559 --> 00:37:46,699
routed into an isolated group of just six hidden

817
00:37:46,699 --> 00:37:49,960
units. This is a massive, deliberate bottleneck.

818
00:37:50,630 --> 00:37:53,849
You are taking the data of 24 distinct, unique

819
00:37:53,849 --> 00:37:56,750
individuals and violently compressing it down

820
00:37:56,750 --> 00:38:00,480
into just six volume knobs. Why do this? Well,

821
00:38:00,480 --> 00:38:02,980
because if you gave it 24 hidden units, the network

822
00:38:02,980 --> 00:38:05,820
would just lazily dedicate one unit to each person.

823
00:38:06,340 --> 00:38:08,199
It would just memorize the data like a flash

824
00:38:08,199 --> 00:38:10,559
card. There's no learning there. By forcing it

825
00:38:10,559 --> 00:38:12,519
through a bottleneck of six units, the network

826
00:38:12,519 --> 00:38:15,900
literally cannot afford to memorize. It has to

827
00:38:15,900 --> 00:38:17,639
find a more efficient way to encode the data.

828
00:38:18,000 --> 00:38:21,000
It has to discover the underlying features, the

829
00:38:21,000 --> 00:38:23,280
invisible attributes that group people together.

830
00:38:23,900 --> 00:38:27,059
The text calls this learning a distributed representation.

831
00:38:27,480 --> 00:38:30,139
It's forcing the network to distribute the abstract

832
00:38:30,139 --> 00:38:33,579
concept of a person across a pattern of activity

833
00:38:33,579 --> 00:38:36,340
in those six units. And side note for anyone

834
00:38:36,340 --> 00:38:39,519
listening who follows modern AI, this exact concept

835
00:38:39,519 --> 00:38:42,360
compressing sparse data into a dense bottleneck

836
00:38:42,360 --> 00:38:45,179
to find underlying meaning is the exact foundation

837
00:38:45,179 --> 00:38:48,019
of embeddings used in modern large language models

838
00:38:48,019 --> 00:38:49,920
to understand the relationship between words.

839
00:38:50,320 --> 00:38:54,000
Yes. It's the same math from 1986. It is the

840
00:38:54,000 --> 00:38:56,840
conceptual ancestor of the modern AI revolution.

841
00:38:57,599 --> 00:39:00,159
So moving up the architecture, those two groups

842
00:39:00,159 --> 00:39:03,079
of six hidden units, one for the person, one

843
00:39:03,079 --> 00:39:05,619
for the relationship, finally mix their signals

844
00:39:05,619 --> 00:39:08,800
together in a central third floor of 12 units.

845
00:39:09,300 --> 00:39:11,320
This is where the person meets the relationship.

846
00:39:11,880 --> 00:39:14,000
That connects to a fourth floor of six units,

847
00:39:14,219 --> 00:39:17,000
which finally connects to the penthouse, an output

848
00:39:17,000 --> 00:39:20,619
layer of 24 units, identical to the input names

849
00:39:20,619 --> 00:39:23,789
representing the final answer. It basically looks

850
00:39:23,789 --> 00:39:26,829
like an hourglass, wide at the ends, heavily

851
00:39:26,829 --> 00:39:29,389
squeezed in the middle. The training regiment

852
00:39:29,389 --> 00:39:33,530
for this beast was grueling. There are 104 true

853
00:39:33,530 --> 00:39:36,710
relational facts in these family trees. And the

854
00:39:36,710 --> 00:39:38,989
researchers trained the network on 100 of them,

855
00:39:39,230 --> 00:39:41,590
purposefully holding four back to test it later.

856
00:39:41,869 --> 00:39:45,489
And they ran it for 1500 sweeps. And they actively

857
00:39:45,489 --> 00:39:47,670
manipulated the learning environment as it trained.

858
00:39:48,050 --> 00:39:49,909
They started with a very small learning rate

859
00:39:49,909 --> 00:39:52,409
to ease it into the problem, then sped it up.

860
00:39:52,730 --> 00:39:55,190
But the most crucial intervention they made was

861
00:39:55,190 --> 00:39:58,869
introducing a concept called weight decay. I

862
00:39:58,869 --> 00:40:00,849
circled this in my notes because it feels entirely

863
00:40:00,849 --> 00:40:03,909
counterintuitive. The text says every single

864
00:40:03,909 --> 00:40:06,690
weight in the entire network is actively decreased,

865
00:40:07,070 --> 00:40:12,050
decayed, by a flat 0 .2 % after every single

866
00:40:12,050 --> 00:40:14,309
weight change. It's like building a muscle but

867
00:40:14,309 --> 00:40:16,909
intentionally starving it at the same time. Exactly.

868
00:40:17,530 --> 00:40:19,889
Why purposefully shrink the exact connections

869
00:40:19,889 --> 00:40:22,800
you are trying so hard to build up? It is a brilliant

870
00:40:22,800 --> 00:40:25,539
technique to force ruthless efficiency. Without

871
00:40:25,539 --> 00:40:28,239
weight decay, if a network finds a sloppy solution

872
00:40:28,239 --> 00:40:31,059
that kind of works, it might just keep increasing

873
00:40:31,059 --> 00:40:33,199
those specific weights to infinity to drive the

874
00:40:33,199 --> 00:40:35,960
outputs closer and closer to perfect ones and

875
00:40:35,960 --> 00:40:39,280
zeros. You end up with massive, unwieldy, fragile

876
00:40:39,280 --> 00:40:42,239
numbers. And it probably overfits the data. It

877
00:40:42,239 --> 00:40:44,659
just memorizes the exact training set rather

878
00:40:44,659 --> 00:40:47,179
than learning the flexible underlying rules of

879
00:40:47,179 --> 00:40:50,750
the family tree. Right. By introducing a constant

880
00:40:50,750 --> 00:40:53,849
0 .2 % decay, the network is fighting a constant

881
00:40:53,849 --> 00:40:56,469
headwind. The only way a weight can remain large

882
00:40:56,469 --> 00:40:59,429
and powerful is if the error gradient is constantly

883
00:40:59,429 --> 00:41:01,349
pushing it up harder than the decay is pulling

884
00:41:01,349 --> 00:41:04,010
it down. It's survival of the fittest for the

885
00:41:04,010 --> 00:41:07,210
synapses. Only the most universally vital connections

886
00:41:07,210 --> 00:41:10,349
survive the decay. The lazy memorization pathways

887
00:41:10,349 --> 00:41:14,170
wither away. Exactly. So after 1 ,500 sweeps

888
00:41:14,170 --> 00:41:16,650
of this grueling, constantly decaying training

889
00:41:16,650 --> 00:41:19,099
process, What did the network actually learn

890
00:41:19,099 --> 00:41:22,139
in that tight six -unit bottleneck? This is the

891
00:41:22,139 --> 00:41:25,079
defining aha moment of the entire paper. The

892
00:41:25,079 --> 00:41:27,579
researchers looked inside the receptive fields

893
00:41:27,579 --> 00:41:30,260
of those six hidden units that were processing

894
00:41:30,260 --> 00:41:33,119
the 24 people. They looked at the final volume

895
00:41:33,119 --> 00:41:35,639
knobs connecting the ground floor names to the

896
00:41:35,639 --> 00:41:38,079
second floor bottleneck, and they found something

897
00:41:38,079 --> 00:41:42,099
astonishing. Let's examine unit one in that bottleneck.

898
00:41:42,380 --> 00:41:44,440
They found that the weights connecting the 12

899
00:41:44,440 --> 00:41:47,460
English people to Unit 1 were all strongly positive,

900
00:41:48,039 --> 00:41:50,260
but the weights connecting the 12 Italian people

901
00:41:50,260 --> 00:41:53,760
to Unit 1 were all strongly negative. Unit 1

902
00:41:53,760 --> 00:41:56,239
organically learned to distinguish English from

903
00:41:56,239 --> 00:41:59,360
Italian. It invented the concept of nationality

904
00:41:59,360 --> 00:42:02,380
as a mathematical sorting mechanism. Now look

905
00:42:02,380 --> 00:42:05,219
at Unit 2. It had strong positive weights for

906
00:42:05,219 --> 00:42:07,860
Christopher, Penelope, Roberto, and Maria. The

907
00:42:07,860 --> 00:42:10,909
grandparents. Right. It had middle ground weights

908
00:42:10,909 --> 00:42:14,630
for Colin, Victoria, Alfonso, the parents, and

909
00:42:14,630 --> 00:42:16,809
it had strong negative weights for the children

910
00:42:16,809 --> 00:42:19,550
at the bottom of the tree. Unit 2 invented the

911
00:42:19,550 --> 00:42:22,250
concept of generation. It learned to sort people

912
00:42:22,250 --> 00:42:24,210
by their vertical height in the family tree.

913
00:42:25,090 --> 00:42:28,469
And Unit 6. Unit 6 had strong weights for the

914
00:42:28,469 --> 00:42:31,210
entire left side of the tree, the Christopher

915
00:42:31,210 --> 00:42:33,789
Arthur branch and the Roberto Emilio branch.

916
00:42:34,210 --> 00:42:36,829
It had opposing weights for the right side of

917
00:42:36,829 --> 00:42:39,460
the tree. It learned which specific branch of

918
00:42:39,460 --> 00:42:41,659
the family someone comes from. I want to make

919
00:42:41,659 --> 00:42:43,940
sure the profoundness of this is really landing

920
00:42:43,940 --> 00:42:46,219
for you listening. I have to extrapolate this

921
00:42:46,219 --> 00:42:48,900
because it blows my mind. The network was never

922
00:42:48,900 --> 00:42:52,320
told what an Italian is. It was never told what

923
00:42:52,320 --> 00:42:54,960
a generation is or what a branch of a family

924
00:42:54,960 --> 00:42:58,260
is. The input was literally just 24 independent

925
00:42:58,260 --> 00:43:01,380
switches representing arbitrary names. There

926
00:43:01,380 --> 00:43:04,440
was zero structure in the input data, just names.

927
00:43:05,000 --> 00:43:07,969
But to minimize the CEO's anger, to mathematically

928
00:43:07,969 --> 00:43:11,230
figure out who Colin's aunt is, the network had

929
00:43:11,230 --> 00:43:14,849
to find structural patterns. It essentially reinvented

930
00:43:14,849 --> 00:43:17,369
human sociology from scratch. It really did.

931
00:43:17,690 --> 00:43:20,690
It discovered that categorizing these 24 arbitrary

932
00:43:20,690 --> 00:43:23,849
names by generation, branch, and nationality

933
00:43:23,849 --> 00:43:26,610
was the most mathematically efficient way to

934
00:43:26,610 --> 00:43:29,590
predict human relationships. It built an internal

935
00:43:29,590 --> 00:43:33,130
conceptual model of a family tree just by multiplying

936
00:43:33,130 --> 00:43:35,619
numbers and descending a gradient. And here is

937
00:43:35,619 --> 00:43:38,199
the ultimate payoff of forcing the network to

938
00:43:38,199 --> 00:43:40,880
learn this distributed representation. Because

939
00:43:40,880 --> 00:43:43,900
Unit 1 exclusively handles the English -Italian

940
00:43:43,900 --> 00:43:46,500
split, the rest of the units don't have to worry

941
00:43:46,500 --> 00:43:49,179
about it. As the text states, the representation

942
00:43:49,179 --> 00:43:52,420
of an English person is very similar to the representation

943
00:43:52,420 --> 00:43:55,260
of their Italian equivalent. So Colin's pattern

944
00:43:55,260 --> 00:43:58,539
of activity across units 2 through 6 is nearly

945
00:43:58,539 --> 00:44:01,420
mathematically identical to Alfonso's pattern

946
00:44:01,420 --> 00:44:03,980
of activity because their structural position

947
00:44:03,980 --> 00:44:07,239
as the father on the left branch is identical.

948
00:44:07,659 --> 00:44:10,159
The network discovered the isomorphism on its

949
00:44:10,159 --> 00:44:13,159
own. Which brings us to the holy grail of machine

950
00:44:13,159 --> 00:44:17,800
learning. Generalization. Remember those four

951
00:44:17,800 --> 00:44:19,519
relational facts they held back during training?

952
00:44:20,159 --> 00:44:22,980
The ones the network had never seen? Right. it

953
00:44:22,980 --> 00:44:26,360
was never explicitly taught the answer. But when

954
00:44:26,360 --> 00:44:28,940
tested on them, because it had discovered the

955
00:44:28,940 --> 00:44:31,280
true underlying structure of how these families

956
00:44:31,280 --> 00:44:34,480
operated, it generalized sensibly. It got them

957
00:44:34,480 --> 00:44:37,179
right. It didn't memorize 100 isolated facts.

958
00:44:37,280 --> 00:44:39,840
It learned the universal rules of the system.

959
00:44:40,599 --> 00:44:43,699
This experiment definitively proved that feedforward

960
00:44:43,699 --> 00:44:46,679
layered networks trained with backpropagation

961
00:44:46,679 --> 00:44:50,039
could extract profound, meaningful semantic features

962
00:44:50,039 --> 00:44:53,849
from entirely unstructured data. It is a staggering

963
00:44:53,849 --> 00:44:56,670
triumph. But as we move into the final major

964
00:44:56,670 --> 00:44:58,769
section of the paper, the authors acknowledge

965
00:44:58,769 --> 00:45:01,610
that these feed -forward networks, as impressive

966
00:45:01,610 --> 00:45:04,150
as they are, have a massive limitation. Yes,

967
00:45:04,150 --> 00:45:07,730
they do. They only process a static, frozen input.

968
00:45:08,250 --> 00:45:10,309
You put a single picture in, you get a single

969
00:45:10,309 --> 00:45:14,050
answer out. But human existence and many of the

970
00:45:14,050 --> 00:45:17,639
hardest problems in AI unfold over time. speech

971
00:45:17,639 --> 00:45:20,880
recognition, video analysis, logic puzzles that

972
00:45:20,880 --> 00:45:23,639
require multiple sequential steps. The paper

973
00:45:23,639 --> 00:45:26,760
addresses this too, introducing the concept of

974
00:45:26,760 --> 00:45:29,780
iterative networks. Right. The authors show that

975
00:45:29,780 --> 00:45:32,320
the mathematics of backpropagation aren't limited

976
00:45:32,320 --> 00:45:35,699
to spatial layered architectures. They discuss

977
00:45:35,699 --> 00:45:38,420
synchronous iterative networks, which we now

978
00:45:38,420 --> 00:45:41,280
more commonly refer to as recurrent neural networks.

979
00:45:41,739 --> 00:45:43,559
This is where the network architecture gets a

980
00:45:43,559 --> 00:45:46,590
little trippy. In a recurrent network, the units

981
00:45:46,590 --> 00:45:49,250
don't just send signals upward. They have connections

982
00:45:49,250 --> 00:45:51,590
that loop back on themselves or loop back to

983
00:45:51,590 --> 00:45:54,510
units in the same layer. The signal cycles around

984
00:45:54,510 --> 00:45:56,750
and around. But how do you train a network that

985
00:45:56,750 --> 00:45:59,909
is constantly feeding back into itself? The error

986
00:45:59,909 --> 00:46:02,489
would just chase its own tail. The authors present

987
00:46:02,489 --> 00:46:04,829
a beautifully simple conceptual equivalence.

988
00:46:05,489 --> 00:46:08,309
An iterative network running over time can be

989
00:46:08,309 --> 00:46:11,010
mapped directly onto a static feed -forward network

990
00:46:11,010 --> 00:46:13,829
running over space. Okay, help me visualize that.

991
00:46:13,840 --> 00:46:17,380
Unrolling time? Exactly. Imagine a recurrent

992
00:46:17,380 --> 00:46:19,460
network running for three distinct ticks of a

993
00:46:19,460 --> 00:46:22,460
clock. Time step one, time step two, and time

994
00:46:22,460 --> 00:46:24,820
step three. You can physically visualize that

995
00:46:24,820 --> 00:46:27,300
process as a three -story feed -forward network.

996
00:46:27,679 --> 00:46:29,500
So the state of the recurrent network at time

997
00:46:29,500 --> 00:46:32,559
step one is the ground floor? Yes. And the connections

998
00:46:32,559 --> 00:46:35,079
looping back on themselves in time are simply

999
00:46:35,079 --> 00:46:37,380
represented as standard wires pointing upward

1000
00:46:37,380 --> 00:46:40,159
to the next floor. So time step two is the middle

1001
00:46:40,159 --> 00:46:44,789
floor, time step three is the penthouse. So we

1002
00:46:44,789 --> 00:46:47,329
just pretend time is space. And if we do that,

1003
00:46:47,489 --> 00:46:49,869
the exact same backpropagation math we just spent

1004
00:46:49,869 --> 00:46:52,110
an hour learning still perfectly applies. We

1005
00:46:52,110 --> 00:46:54,170
just passed the CEO's blame down through the

1006
00:46:54,170 --> 00:46:56,230
floors, which means we are technically passing

1007
00:46:56,230 --> 00:46:59,750
the error backward through time. That is precisely

1008
00:46:59,750 --> 00:47:02,210
what it is, backpropagation through time. But

1009
00:47:02,210 --> 00:47:04,610
the text notes two major logistical complications

1010
00:47:04,610 --> 00:47:06,630
when you try to actually program this into a

1011
00:47:06,630 --> 00:47:09,679
computer. Right. Because in a standard spatial

1012
00:47:09,679 --> 00:47:12,440
network, we said you need to remember the output

1013
00:47:12,440 --> 00:47:14,960
of the hidden units from the forward pass to

1014
00:47:14,960 --> 00:47:17,019
calculate the chain rule on the backward pass.

1015
00:47:17,320 --> 00:47:19,400
The first complication is memory constraint.

1016
00:47:19,780 --> 00:47:22,280
In an iterative network running over time, the

1017
00:47:22,280 --> 00:47:25,039
hidden units are just the network itself in the

1018
00:47:25,039 --> 00:47:27,500
past. You can't just let the network override

1019
00:47:27,500 --> 00:47:30,059
its state at each tick of the clock. You must

1020
00:47:30,059 --> 00:47:32,199
build a massive memory bank to store the entire

1021
00:47:32,199 --> 00:47:34,920
history of the output states of every single

1022
00:47:34,920 --> 00:47:38,119
unit at every single time step during the forward

1023
00:47:38,119 --> 00:47:41,300
run. So you have them preserved and waiting when

1024
00:47:41,300 --> 00:47:44,019
you run the error backward through time. Which,

1025
00:47:44,179 --> 00:47:47,940
in 1986, was a crippling amount of computer memory

1026
00:47:47,940 --> 00:47:51,019
for complex tasks. What's the second complication?

1027
00:47:51,679 --> 00:47:54,079
The second complication involves the volume knobs.

1028
00:47:54,619 --> 00:47:57,199
If our three -story spatial network is really

1029
00:47:57,199 --> 00:47:59,599
just one recurrent network unrolled over three

1030
00:47:59,599 --> 00:48:02,420
time steps, then the wires connecting floor one

1031
00:48:02,420 --> 00:48:05,940
to floor two must be perfectly identical to the

1032
00:48:05,940 --> 00:48:08,880
wires connecting floor 2 to floor 3. Because

1033
00:48:08,880 --> 00:48:11,519
they aren't actually different wires. They are

1034
00:48:11,519 --> 00:48:14,019
the exact same physical connections in the recurrent

1035
00:48:14,019 --> 00:48:16,539
network just operating at a different moment

1036
00:48:16,539 --> 00:48:20,239
in time. Right. But backpropagation calculates

1037
00:48:20,239 --> 00:48:23,300
a unique, highly specific error derivative for

1038
00:48:23,300 --> 00:48:25,559
every single physical connection based on its

1039
00:48:25,559 --> 00:48:28,539
local position. The calculated blame for floor

1040
00:48:28,539 --> 00:48:31,500
1 will naturally be different from the calculated

1041
00:48:31,500 --> 00:48:34,880
blame for floor 2. If we just update them independently

1042
00:48:34,880 --> 00:48:37,239
like a normal network, they won't match anymore.

1043
00:48:37,780 --> 00:48:42,380
The illusion of unrolled time shatters. So how

1044
00:48:42,380 --> 00:48:44,320
do you force them to stay identical while still

1045
00:48:44,320 --> 00:48:47,559
learning? The authors introduce a simple mathematical

1046
00:48:47,559 --> 00:48:50,699
proviso. You calculate the gradient for all the

1047
00:48:50,699 --> 00:48:52,559
corresponding weights across the different time

1048
00:48:52,559 --> 00:48:55,500
steps. Then you simply average those gradients

1049
00:48:55,500 --> 00:48:58,920
together. Finally, you change all the corresponding

1050
00:48:58,920 --> 00:49:01,280
weights simultaneously by an amount proportional

1051
00:49:01,280 --> 00:49:04,340
to that single average gradient. They are forced

1052
00:49:04,340 --> 00:49:07,619
to move together in lockstep. That is a brilliantly

1053
00:49:07,619 --> 00:49:10,769
practical solution. So with those two tweaks,

1054
00:49:11,630 --> 00:49:13,909
massive history storage and gradient averaging,

1055
00:49:14,650 --> 00:49:17,130
backpropagation can train networks to learn sequential

1056
00:49:17,130 --> 00:49:20,190
structures. It opened up entirely new worlds

1057
00:49:20,190 --> 00:49:23,070
of capability for AI. But before they conclude

1058
00:49:23,070 --> 00:49:25,710
the paper, the authors have to tackle the elephant

1059
00:49:25,710 --> 00:49:29,010
in the room. They have to address the major theoretical

1060
00:49:29,010 --> 00:49:31,630
criticism of gradient descent algorithms that

1061
00:49:31,630 --> 00:49:34,530
had haunted the field for decades. The nightmare

1062
00:49:34,530 --> 00:49:39,309
of local minima. Ah, yes. Let's return to our

1063
00:49:39,309 --> 00:49:42,429
mountain landscape analogy one last time. The

1064
00:49:42,429 --> 00:49:46,449
total error is the altitude. We want our blindfolded

1065
00:49:46,449 --> 00:49:48,510
algorithm to step its way down to the lowest

1066
00:49:48,510 --> 00:49:51,250
possible valley on the entire planet, the global

1067
00:49:51,250 --> 00:49:54,099
minimum. And the obvious glaring drawback to

1068
00:49:54,099 --> 00:49:56,380
this approach, which critics constantly point

1069
00:49:56,380 --> 00:49:59,760
out, is that the error surface of a complex problem

1070
00:49:59,760 --> 00:50:02,940
is not a simple smooth bowl with a single bottom.

1071
00:50:03,340 --> 00:50:07,239
It is rugged. It is highly non -linear. It contains

1072
00:50:07,239 --> 00:50:11,179
local minima. Local minima are like small shallow

1073
00:50:11,179 --> 00:50:13,719
valleys or craters perched high up on the side

1074
00:50:13,719 --> 00:50:16,699
of the mountain. If our blindfolded algorithm

1075
00:50:16,699 --> 00:50:19,400
wanders into one of these shallow craters, it

1076
00:50:19,400 --> 00:50:21,730
reaches the bottom of the crater. It feels the

1077
00:50:21,730 --> 00:50:23,829
ground with its foot, and every direction it

1078
00:50:23,829 --> 00:50:27,030
touches goes up. The math tells it, you are at

1079
00:50:27,030 --> 00:50:29,809
the bottom. Stop moving. The algorithm stops.

1080
00:50:30,489 --> 00:50:32,630
It thinks it's solved the problem, but it's actually

1081
00:50:32,630 --> 00:50:34,949
stuck in a poor local minimum with a massive

1082
00:50:34,949 --> 00:50:37,530
error rate. And it will never ever find the true

1083
00:50:37,530 --> 00:50:39,389
global minimum at the bottom of the mountain.

1084
00:50:40,250 --> 00:50:42,670
Theoretically, mathematically, this is a fatal

1085
00:50:42,670 --> 00:50:45,130
flaw. It suggests that gradient descent shouldn't

1086
00:50:45,130 --> 00:50:48,269
work on anything but the simplest problems. But

1087
00:50:48,269 --> 00:50:50,949
the authors rebuttal to this theoretical fear

1088
00:50:50,949 --> 00:50:54,150
is incredibly pragmatic. What did they actually

1089
00:50:54,150 --> 00:50:56,670
find when they ran these complex experiments?

1090
00:50:57,010 --> 00:50:59,769
They relay their empirical findings, which directly

1091
00:50:59,769 --> 00:51:02,650
contradict the theoretical panic. They state,

1092
00:51:02,829 --> 00:51:05,769
simply, experience with many tasks shows that

1093
00:51:05,769 --> 00:51:08,730
the network very rarely gets stuck in poor local

1094
00:51:08,730 --> 00:51:11,010
minima that are significantly worse than the

1095
00:51:11,010 --> 00:51:14,130
global minimum. In practice, it just works. But

1096
00:51:14,130 --> 00:51:17,070
why? Do they offer an explanation for how the

1097
00:51:17,070 --> 00:51:19,769
blindfolded algorithm magically avoids these

1098
00:51:19,769 --> 00:51:22,829
craters? They offer a fascinating geometrical

1099
00:51:22,829 --> 00:51:25,849
explanation. They observe that this getting stuck

1100
00:51:25,849 --> 00:51:28,190
behavior almost exclusively happens in networks

1101
00:51:28,190 --> 00:51:30,269
that have just enough connections to perform

1102
00:51:30,269 --> 00:51:34,130
the task. A network perfectly minimally sized

1103
00:51:34,130 --> 00:51:36,489
for the complexity of the problem. But what happens

1104
00:51:36,489 --> 00:51:38,730
if you make the network way bigger than it strictly

1105
00:51:38,730 --> 00:51:41,170
needs to be? What if you give it way too many

1106
00:51:41,170 --> 00:51:44,809
hidden units? That is the key. They state Adding

1107
00:51:44,809 --> 00:51:47,210
a few more connections creates extra dimensions

1108
00:51:47,210 --> 00:51:51,090
in weight space. I love this concept, the dimensional

1109
00:51:51,090 --> 00:51:53,789
escape route. Let me try to visualize this for

1110
00:51:53,789 --> 00:51:56,510
the listener. Imagine you are literally trapped

1111
00:51:56,510 --> 00:51:58,710
in a three -dimensional valley. There are walls

1112
00:51:58,710 --> 00:52:02,389
on the north, south, east, and west. You cannot

1113
00:52:02,389 --> 00:52:04,989
move without going up. You are stuck in a local

1114
00:52:04,989 --> 00:52:08,070
minimum. But then, you add a new connection to

1115
00:52:08,070 --> 00:52:11,250
the network. Mathematically, you just added a

1116
00:52:11,250 --> 00:52:13,090
fourth dimension of space to your landscape.

1117
00:52:13,239 --> 00:52:15,400
And in a four -dimensional space, what looked

1118
00:52:15,400 --> 00:52:19,260
like an impenetrable solid wall in 3D might not

1119
00:52:19,260 --> 00:52:21,500
be a wall at all. It might just be an open door.

1120
00:52:21,880 --> 00:52:25,219
Yes. The text literally says, these dimensions

1121
00:52:25,219 --> 00:52:28,079
provide paths around the barriers that create

1122
00:52:28,079 --> 00:52:31,300
poor local minima in the lower dimensional subspaces.

1123
00:52:32,380 --> 00:52:35,119
It's like finding a secret hyperdimensional tunnel

1124
00:52:35,119 --> 00:52:38,050
out of the crater. By over -parameterizing the

1125
00:52:38,050 --> 00:52:40,090
network, giving it way more connections than

1126
00:52:40,090 --> 00:52:42,550
it needs, you give it the spatial freedom to

1127
00:52:42,550 --> 00:52:45,170
flow like water around obstacles and find its

1128
00:52:45,170 --> 00:52:48,260
way to the bottom of the mountain. It is a profound,

1129
00:52:48,619 --> 00:52:50,840
counter -intuitive insight that remains a cornerstone

1130
00:52:50,840 --> 00:52:53,480
of how we design massive neural networks today.

1131
00:52:54,079 --> 00:52:56,659
We make them astronomically huge, partially to

1132
00:52:56,659 --> 00:52:58,980
ensure they have enough dimensional escape routes

1133
00:52:58,980 --> 00:53:03,179
to never get trapped. We have covered so much

1134
00:53:03,179 --> 00:53:05,929
incredible ground today. The assembly line forward

1135
00:53:05,929 --> 00:53:08,949
pass, the corporate blame backward pass, the

1136
00:53:08,949 --> 00:53:10,889
mathematical symmetry detector, the invention

1137
00:53:10,889 --> 00:53:14,050
of sociology and the family tree, unrolling time,

1138
00:53:14,289 --> 00:53:17,849
escaping 40 valleys. We are at the end of our

1139
00:53:17,849 --> 00:53:20,530
deep dive. And I want to circle back to something

1140
00:53:20,530 --> 00:53:23,050
that might seem very surprising after spending

1141
00:53:23,050 --> 00:53:26,429
an hour marveling at how smart, how organic this

1142
00:53:26,429 --> 00:53:30,579
system seems. The biological caveat. Yes. the

1143
00:53:30,579 --> 00:53:32,659
authors themselves in the concluding paragraph

1144
00:53:32,659 --> 00:53:36,000
of the paper explicitly state a major humbling

1145
00:53:36,000 --> 00:53:38,789
caveat. The learning procedure in its current

1146
00:53:38,789 --> 00:53:41,409
form is not a plausible model of learning in

1147
00:53:41,409 --> 00:53:43,849
brains. They are being rigorously scientific.

1148
00:53:44,190 --> 00:53:47,809
Despite the inspiration taken from neurons, biological

1149
00:53:47,809 --> 00:53:50,909
brains do not appear to perform the flawless,

1150
00:53:51,789 --> 00:53:55,170
synchronous backward pass of precise error calculus

1151
00:53:55,170 --> 00:53:58,230
that the chain rule requires. A brain doesn't

1152
00:53:58,230 --> 00:54:01,289
have a secondary set of perfectly reversed pathways

1153
00:54:01,289 --> 00:54:04,389
designed specifically to transmit fractional

1154
00:54:04,389 --> 00:54:07,550
derivatives of blame back to individual synapses.

1155
00:54:08,070 --> 00:54:09,989
So if our biological brains don't actually do

1156
00:54:09,989 --> 00:54:12,869
this, what is the ultimate value of this massive

1157
00:54:12,869 --> 00:54:14,949
paper? Why did organizations like the System

1158
00:54:14,949 --> 00:54:16,869
Development Foundation and the Office of Naval

1159
00:54:16,869 --> 00:54:19,429
Research fund this leap into artificial math?

1160
00:54:20,130 --> 00:54:22,610
The value is theoretical proof of existence.

1161
00:54:23,150 --> 00:54:27,309
Before 1986, it was widely doubted, almost dogma.

1162
00:54:27,530 --> 00:54:30,750
that any local decentralized mathematical rule

1163
00:54:30,750 --> 00:54:33,789
could construct complex internal representations,

1164
00:54:34,650 --> 00:54:37,630
like the abstract concept of a generation or

1165
00:54:37,630 --> 00:54:40,650
a family branch. By proving that gradient descent

1166
00:54:40,650 --> 00:54:44,030
in weight space can achieve this, the paper shattered

1167
00:54:44,030 --> 00:54:47,030
that dogma. It proved the mathematical viability

1168
00:54:47,030 --> 00:54:49,599
of the approach. And as the authors conclude,

1169
00:54:50,059 --> 00:54:52,460
this suggests that it is worth looking for more

1170
00:54:52,460 --> 00:54:54,739
biologically plausible ways of doing gradient

1171
00:54:54,739 --> 00:54:57,420
descent in neural networks. It gave neuroscience

1172
00:54:57,420 --> 00:55:00,559
and AI a true north. It proved that the murky

1173
00:55:00,559 --> 00:55:03,960
diagnostic muddy waters of a complex system actually

1174
00:55:03,960 --> 00:55:07,219
hold understandable structural truth if you can

1175
00:55:07,219 --> 00:55:09,800
just figure out how to mathematically descend

1176
00:55:09,800 --> 00:55:12,019
the gradient to find it. Which leaves me with

1177
00:55:12,019 --> 00:55:13,920
a final provocative thought for you to mull over

1178
00:55:13,920 --> 00:55:17,000
as we sign off. If our actual biological brains

1179
00:55:17,000 --> 00:55:19,539
don't use exact backpropagation, if we don't

1180
00:55:19,539 --> 00:55:21,639
have this flawless internal accounting department

1181
00:55:21,639 --> 00:55:24,340
passing exact error derivatives perfectly backward

1182
00:55:24,340 --> 00:55:26,860
down the chain of command, yet we somehow still

1183
00:55:26,860 --> 00:55:29,599
learn incredibly complex abstract representations

1184
00:55:29,599 --> 00:55:32,480
of the world, how are we doing it? That's the

1185
00:55:32,480 --> 00:55:35,079
million dollar question. Is there an entirely

1186
00:55:35,079 --> 00:55:37,559
different undiscovered mathematical principle

1187
00:55:37,559 --> 00:55:40,840
governing human learning? Or, perhaps more intriguingly,

1188
00:55:41,119 --> 00:55:43,420
is the wet, squishy brain somehow approximating

1189
00:55:43,420 --> 00:55:46,300
this exact calculus in a messy, noisy, organic

1190
00:55:46,300 --> 00:55:49,739
way? If backpropagation is the clean, perfectly

1191
00:55:49,739 --> 00:55:52,119
engineered X -ray we talked about at the start,

1192
00:55:52,539 --> 00:55:54,460
is the brain achieving the exact same result

1193
00:55:54,460 --> 00:55:56,940
through the murky, distributed fog of biology?

1194
00:55:57,519 --> 00:55:59,659
What does it mean for the future of artificial

1195
00:55:59,659 --> 00:56:02,440
intelligence if our most successful, world -changing

1196
00:56:02,440 --> 00:56:05,719
algorithms are fundamentally alien to the biology

1197
00:56:05,719 --> 00:56:08,119
that originally inspired them? It's a lot to

1198
00:56:08,119 --> 00:56:10,159
think about. Keep digging into those questions.

1199
00:56:10,360 --> 00:56:12,179
The answers are still out there, waiting to be

1200
00:56:12,179 --> 00:56:14,019
discovered. Thanks for joining us on this deep

1201
00:56:14,019 --> 00:56:14,239
dive.