1
00:00:00,000 --> 00:00:01,540
You know, it's funny. If you really look at a

2
00:00:01,540 --> 00:00:05,320
modern microprocessor, it is at its core just

3
00:00:05,320 --> 00:00:07,620
a piece of rock. Right. A very highly processed

4
00:00:07,620 --> 00:00:09,640
rock, but yes. Yeah, like we literally took some

5
00:00:09,640 --> 00:00:12,259
silicon, flattened it out, carved these microscopic

6
00:00:12,259 --> 00:00:15,099
lines into it, and essentially tricked it into

7
00:00:15,099 --> 00:00:17,190
doing math by... you know, trapping lightning

8
00:00:17,190 --> 00:00:20,429
inside. That is a, well, a very poetic and slightly

9
00:00:20,429 --> 00:00:23,649
terrifying way to describe modern computing hardware.

10
00:00:24,129 --> 00:00:26,570
But you're right. Fundamentally, at the deepest

11
00:00:26,570 --> 00:00:29,269
physical layer, a computer is just a really,

12
00:00:29,289 --> 00:00:32,770
really fast calculator. It only understands high

13
00:00:32,770 --> 00:00:36,649
voltage and low voltage, ones and zeros. It operates

14
00:00:36,649 --> 00:00:39,390
entirely on binary logic. Which makes total sense

15
00:00:39,390 --> 00:00:41,810
for math, right? Like getting that rock to add

16
00:00:41,810 --> 00:00:43,869
two numbers together seems completely logical

17
00:00:43,869 --> 00:00:45,890
to me. Sure. And even getting it to render a

18
00:00:45,890 --> 00:00:48,409
massive 3D video game, which is obviously way

19
00:00:48,409 --> 00:00:50,469
harder and requires millions more calculations,

20
00:00:50,869 --> 00:00:52,890
that still feels like it lives in the realm of

21
00:00:52,890 --> 00:00:55,270
numbers. But then I look at my phone. Where are

22
00:00:55,270 --> 00:00:56,950
we going with this? Well, I type a text message

23
00:00:56,950 --> 00:01:00,450
and I use some weird, highly nuanced sarcasm

24
00:01:00,450 --> 00:01:03,990
and my keyboard instantly suggests like a laughing,

25
00:01:04,010 --> 00:01:06,849
crying emoji. Ah, I see. And that just feels

26
00:01:06,849 --> 00:01:09,709
like we crossed some kind of boundary. You know,

27
00:01:09,709 --> 00:01:12,629
like we took this hyper logical rock and somehow

28
00:01:12,629 --> 00:01:15,670
taught it human empathy. Or at least the illusion

29
00:01:15,670 --> 00:01:18,250
of it. It honestly feels like magic. It does

30
00:01:18,250 --> 00:01:20,329
feel like magic, yeah. Because human language

31
00:01:20,329 --> 00:01:23,150
is really the complete antithesis of binary logic.

32
00:01:23,370 --> 00:01:26,510
It's messy. It's ambiguous. It constantly evolves.

33
00:01:26,909 --> 00:01:29,709
Messy. Right. And it relies on this immense amount

34
00:01:29,709 --> 00:01:32,469
of shared cultural context that we never explicitly

35
00:01:32,469 --> 00:01:35,670
say out loud. We leave so much unsaid. And yet

36
00:01:35,670 --> 00:01:38,170
over the last few decades, computer scientists

37
00:01:38,170 --> 00:01:40,409
have built these just astonishing architectural

38
00:01:40,409 --> 00:01:43,810
bridges between the rigidity of math and the

39
00:01:43,810 --> 00:01:46,109
fluidity of human thought. So it's not magic.

40
00:01:46,290 --> 00:01:49,430
There is zero magic. It is just incredibly complex,

41
00:01:49,650 --> 00:01:52,290
deeply fascinating architecture. Well, today

42
00:01:52,290 --> 00:01:54,670
we are taking a sledgehammer to that magic trick.

43
00:01:55,150 --> 00:01:58,069
Welcome to a brand new Deep Dive. Glad to be

44
00:01:58,069 --> 00:02:00,189
here. We have our hands on the absolute definitive

45
00:02:00,189 --> 00:02:02,609
text in the field today. It's Daniel Jarafsky

46
00:02:02,609 --> 00:02:04,870
and James H. Martin's speech and language processing.

47
00:02:05,230 --> 00:02:07,750
We've got an enormous stack of excerpts from

48
00:02:07,750 --> 00:02:10,129
their third edition draft, and our mission today

49
00:02:10,129 --> 00:02:12,889
is to just tear this subject down to the studs.

50
00:02:12,909 --> 00:02:15,590
It's a fantastic resource. If you are listening

51
00:02:15,590 --> 00:02:17,969
to this right now and you've ever wondered how

52
00:02:17,969 --> 00:02:22,449
Siri... parses your voice, or how Google Translate

53
00:02:22,449 --> 00:02:26,050
flips Japanese into English, or how a large language

54
00:02:26,050 --> 00:02:28,389
model actually generates a paragraph of text,

55
00:02:28,590 --> 00:02:31,629
we are going to meticulously unpack the mechanics

56
00:02:31,629 --> 00:02:35,030
behind every bit of it. And it is a massive subject.

57
00:02:35,189 --> 00:02:37,330
So you should really consider this part one of

58
00:02:37,330 --> 00:02:39,949
our journey. We are going to go concept by concept,

59
00:02:40,129 --> 00:02:42,409
starting from the very dawn of natural language

60
00:02:42,409 --> 00:02:44,979
processing. all the way through chapter 14 of

61
00:02:44,979 --> 00:02:47,080
the text. A lot of ground to cover. Yeah. We'll

62
00:02:47,080 --> 00:02:48,939
start with how a machine even looks at a raw

63
00:02:48,939 --> 00:02:51,479
letter and then move into how it calculates the

64
00:02:51,479 --> 00:02:54,860
probability of your next keystroke. We'll explore

65
00:02:54,860 --> 00:02:57,139
how researchers figured out a way to turn the

66
00:02:57,139 --> 00:03:00,919
actual meaning of a word into a geometric coordinate.

67
00:03:01,219 --> 00:03:04,199
That part is wild. It really is. And eventually

68
00:03:04,199 --> 00:03:06,000
we'll look at the deep neural networks that hold

69
00:03:06,000 --> 00:03:08,590
entire conversations in their memory. Now, I

70
00:03:08,590 --> 00:03:10,770
know it sounds intimidating when we start throwing

71
00:03:10,770 --> 00:03:12,889
around terms like, you know, neural architectures

72
00:03:12,889 --> 00:03:15,110
and vector semantics. It feels like this huge

73
00:03:15,110 --> 00:03:18,530
wall of jargon. It definitely can. But reading

74
00:03:18,530 --> 00:03:20,250
through these sources, I realized something.

75
00:03:20,530 --> 00:03:23,569
Every single one of these complex systems was

76
00:03:23,569 --> 00:03:27,009
built simply to answer a few beautifully basic

77
00:03:27,009 --> 00:03:30,199
questions like. What is a word? How does one

78
00:03:30,199 --> 00:03:32,780
word relate to another word? And how do we use

79
00:03:32,780 --> 00:03:35,340
the past to predict the future? Right. If we

80
00:03:35,340 --> 00:03:37,460
just hold onto those questions, the math just

81
00:03:37,460 --> 00:03:39,430
becomes a tool to answer them. That's a really

82
00:03:39,430 --> 00:03:41,909
perfect way to frame it. And the desire to answer

83
00:03:41,909 --> 00:03:44,110
those questions actually goes back way further

84
00:03:44,110 --> 00:03:47,810
than many people realize. In the text, Juraski

85
00:03:47,810 --> 00:03:50,090
and Martin take us all the way back to 1966.

86
00:03:50,430 --> 00:03:54,550
Yes. To MIT, to a chatbot named Eliza. This was

87
00:03:54,550 --> 00:03:57,490
totally mind -blowing to me. 1966, they built

88
00:03:57,490 --> 00:04:00,210
a system that simulated a Rogerian psychotherapist.

89
00:04:00,210 --> 00:04:02,189
And the transcript of people talking to this

90
00:04:02,189 --> 00:04:04,560
computer is just... wild he's famous for a reason

91
00:04:04,560 --> 00:04:06,840
yeah like the user types i need some help and

92
00:04:06,840 --> 00:04:08,560
eliza responds what would it mean to you if you

93
00:04:08,560 --> 00:04:11,219
got some help and then the user types perhaps

94
00:04:11,219 --> 00:04:13,419
i could learn to get along with my mother and

95
00:04:13,419 --> 00:04:15,819
eliza says tell me more about your family all

96
00:04:15,819 --> 00:04:18,360
in caps too right and people were stunned by

97
00:04:18,360 --> 00:04:21,139
this the text says some users genuinely believe

98
00:04:21,139 --> 00:04:23,220
eliza possessed deep emotional understanding

99
00:04:23,959 --> 00:04:26,079
Well, the irony there is that Rogerian therapy,

100
00:04:26,459 --> 00:04:28,319
which is a real framework where the therapist

101
00:04:28,319 --> 00:04:31,000
basically reflects the patient's statements back

102
00:04:31,000 --> 00:04:33,459
at them to encourage them to keep talking, is

103
00:04:33,459 --> 00:04:36,180
the absolute perfect persona for a machine that

104
00:04:36,180 --> 00:04:38,680
knows literally nothing about the world. Right,

105
00:04:38,759 --> 00:04:40,319
because it doesn't have to offer any original

106
00:04:40,319 --> 00:04:43,959
insight. Exactly. Eliza did not possess a single

107
00:04:43,959 --> 00:04:47,379
shred of actual comprehension. It was purely

108
00:04:47,379 --> 00:04:49,980
a parlor trick. But the mechanism of that trick

109
00:04:49,980 --> 00:04:52,860
is something we still use every single day. It

110
00:04:52,860 --> 00:04:54,860
relied on something called regular expressions.

111
00:04:56,290 --> 00:04:59,910
Yes, regular expressions form the absolute bedrock

112
00:04:59,910 --> 00:05:03,290
of string processing in computer science. A regular

113
00:05:03,290 --> 00:05:05,889
expression is essentially an algebraic notation

114
00:05:05,889 --> 00:05:08,910
for characterizing a set of strings. You are

115
00:05:08,910 --> 00:05:11,529
handing the computer a highly, highly structured

116
00:05:11,529 --> 00:05:13,649
search pattern. Okay, let's bring this down to

117
00:05:13,649 --> 00:05:15,430
earth for a second. Because reading the reject

118
00:05:15,430 --> 00:05:17,889
section, it felt exactly like giving instructions

119
00:05:17,889 --> 00:05:22,009
to a bouncer at a comically exclusive nightclub.

120
00:05:22,149 --> 00:05:24,410
I like that analogy. You can't just tell the

121
00:05:24,410 --> 00:05:26,620
computer bouncer, hey don't let anyone wearing

122
00:05:26,620 --> 00:05:28,319
sneakers in you have to give it a hyper specific

123
00:05:28,319 --> 00:05:31,740
list of physical parameters you have to say only

124
00:05:31,740 --> 00:05:33,920
admit a person if their name begins with a capital

125
00:05:33,920 --> 00:05:36,959
letter they're wearing exactly three rings and

126
00:05:36,959 --> 00:05:39,060
their shoe size is between eight and ten point

127
00:05:39,060 --> 00:05:41,939
one and if a guest deviates by one millimeter

128
00:05:41,939 --> 00:05:44,759
they are rejected that captures the literal minded

129
00:05:44,759 --> 00:05:47,139
nature of the machine perfectly because if you

130
00:05:47,139 --> 00:05:49,240
ask a computer to search a document for the word

131
00:05:49,240 --> 00:05:52,129
woodchuck all lowercase it will scan the text

132
00:05:52,129 --> 00:05:54,269
but if it encounters woodchuck at the beginning

133
00:05:54,269 --> 00:05:57,370
of a sentence with a capital w it just ignores

134
00:05:57,370 --> 00:05:59,569
it it doesn't even see it as the same word not

135
00:05:59,569 --> 00:06:03,089
at all to the machine a lowercase w and an uppercase

136
00:06:03,089 --> 00:06:05,910
w are entirely different entities their underlying

137
00:06:05,910 --> 00:06:08,110
binary codes are completely different so to get

138
00:06:08,110 --> 00:06:10,850
around that we have to use regex syntax we basically

139
00:06:10,850 --> 00:06:13,120
give the bouncer options. The book shows how

140
00:06:13,120 --> 00:06:15,360
we use square brackets for this. You type an

141
00:06:15,360 --> 00:06:18,060
opening bracket, a lowercase w, an undercase

142
00:06:18,060 --> 00:06:21,220
w, a closing bracket, and then oodchuck. So it

143
00:06:21,220 --> 00:06:23,660
looks like bracket w, capital dash w, bracket

144
00:06:23,660 --> 00:06:26,060
oodchuck. Right. And those brackets define what's

145
00:06:26,060 --> 00:06:29,089
called a disjunction of characters. you are telling

146
00:06:29,089 --> 00:06:32,149
the system match any single character that exists

147
00:06:32,149 --> 00:06:34,209
inside these brackets. So it makes your search

148
00:06:34,209 --> 00:06:36,350
just slightly more flexible. And you can set

149
00:06:36,350 --> 00:06:38,829
up ranges too, which is super useful. Instead

150
00:06:38,829 --> 00:06:41,170
of typing out every single letter of the alphabet,

151
00:06:41,370 --> 00:06:44,529
you just type bracket, a dash Z bracket, and

152
00:06:44,529 --> 00:06:46,670
the computer knows you mean literally any lowercase

153
00:06:46,670 --> 00:06:49,569
letter. Or zero through nine for any single digit.

154
00:06:49,769 --> 00:06:52,050
Yeah. I also saw you can build negative rules.

155
00:06:52,189 --> 00:06:54,839
If you put a caret symbol. you know, the little

156
00:06:54,839 --> 00:06:57,660
upward pointing arrow inside the brackets, it

157
00:06:57,660 --> 00:07:00,160
acts as a negation. So if I write caret A -Z

158
00:07:00,160 --> 00:07:01,980
inside the brackets, that tells the bouncer,

159
00:07:02,120 --> 00:07:04,459
let this character pass as long as it is not

160
00:07:04,459 --> 00:07:07,540
an uppercase letter. The real power of regex,

161
00:07:07,579 --> 00:07:10,019
though, emerges when we introduce counters and

162
00:07:10,019 --> 00:07:12,180
optionality, because human language is just full

163
00:07:12,180 --> 00:07:14,600
of variations in spelling or emphasis. Oh, for

164
00:07:14,600 --> 00:07:16,779
sure. Consider the word color. In America, it

165
00:07:16,779 --> 00:07:19,899
is C -O -L -O -R, but in Britain, it's C -O -L

166
00:07:19,899 --> 00:07:22,829
-O -U -R. Now, if you are searching a global

167
00:07:22,829 --> 00:07:25,949
database, you want to catch both spellings. In

168
00:07:25,949 --> 00:07:28,329
Rejects, you would place a question mark after

169
00:07:28,329 --> 00:07:31,129
the U, and the question mark dictates that the

170
00:07:31,129 --> 00:07:33,689
preceding character is optional, meaning it can

171
00:07:33,689 --> 00:07:37,170
appear zero times or exactly one time. Oh, so

172
00:07:37,170 --> 00:07:39,550
it catches both automatically. The textbook uses

173
00:07:39,550 --> 00:07:42,230
a really great example of a sheep bleeding to

174
00:07:42,230 --> 00:07:44,350
explain the other counters. The sheep example

175
00:07:44,350 --> 00:07:46,790
is a classic. Yeah, because a sheep could say

176
00:07:46,790 --> 00:07:51,250
bah with two A's or bah with three. or ba with

177
00:07:51,250 --> 00:07:54,310
five. And if I want the computer to catch that

178
00:07:54,310 --> 00:07:56,410
sheep noise, regardless of how long the vowel

179
00:07:56,410 --> 00:07:59,910
is dragged out, I use the clean star, the asterisk.

180
00:08:00,009 --> 00:08:02,990
So if I write ba star exclamation point, the

181
00:08:02,990 --> 00:08:05,910
star means the preceding of 2a can occur zero

182
00:08:05,910 --> 00:08:08,610
times, or basically an infinite number of times.

183
00:08:08,769 --> 00:08:10,949
Alternatively, you might use the clean plus,

184
00:08:11,029 --> 00:08:13,110
which demands that the character appears at least

185
00:08:13,110 --> 00:08:15,370
one time, but then allows for infinite repetitions

186
00:08:15,370 --> 00:08:17,509
after that. And we combine all these counters

187
00:08:17,509 --> 00:08:19,899
with anchors. Anchors. Yeah, so the carrot symbol,

188
00:08:20,100 --> 00:08:22,480
when it's placed outside of the brackets, means

189
00:08:22,480 --> 00:08:24,399
the pattern must start at the very beginning

190
00:08:24,399 --> 00:08:27,240
of a line. And a dollar sign means the pattern

191
00:08:27,240 --> 00:08:29,860
must hit the very end of the line. Okay, so these

192
00:08:29,860 --> 00:08:32,639
are the basic tools. But how did the creator

193
00:08:32,639 --> 00:08:37,879
of Eliza actually use these tools to fake emotional

194
00:08:37,879 --> 00:08:40,019
intelligence? Like, how did it literally flip

195
00:08:40,019 --> 00:08:43,340
I need help with my mother into tell me more

196
00:08:43,340 --> 00:08:45,879
about your family? It utilized a feature of RageX

197
00:08:45,879 --> 00:08:48,279
called Capture Groups. When you place parentheses

198
00:08:48,279 --> 00:08:50,679
around a specific part of your regex pattern,

199
00:08:50,940 --> 00:08:53,080
you instruct the computer to not only find a

200
00:08:53,080 --> 00:08:55,539
match, but to actually save whatever text it

201
00:08:55,539 --> 00:08:57,720
matched into its active memory. Oh, it grabs

202
00:08:57,720 --> 00:08:59,940
it. Yes. We store it in what's called a numbered

203
00:08:59,940 --> 00:09:01,779
register. I really want to walk through the exact

204
00:09:01,779 --> 00:09:03,879
code the book provides here because it is just

205
00:09:03,879 --> 00:09:07,539
the perfect illusion. The code is s slash dot

206
00:09:07,539 --> 00:09:17,990
star space im. Let's demystify that whole sequence.

207
00:09:18,129 --> 00:09:21,049
Please do. So, the is followed by a slash simply

208
00:09:21,049 --> 00:09:24,470
stands for substitute. It initiates a basic find

209
00:09:24,470 --> 00:09:27,840
and replace operation. The dot star at the very

210
00:09:27,840 --> 00:09:30,799
beginning means match any character any number

211
00:09:30,799 --> 00:09:34,289
of times. In plain English, it means... Ignore

212
00:09:34,289 --> 00:09:36,350
everything the user typed at the start of their

213
00:09:36,350 --> 00:09:38,470
sentence. Got it. So it's just scanning right

214
00:09:38,470 --> 00:09:40,870
past all the throat clearing. Like if I say,

215
00:09:40,950 --> 00:09:44,210
well, today I feel, it ignores that. Exactly.

216
00:09:44,830 --> 00:09:47,629
Next, it looks for the explicit string I am with

217
00:09:47,629 --> 00:09:49,830
spaces around it. Then we hit the parentheses,

218
00:09:49,950 --> 00:09:53,190
the capture group. Inside is depressed pipe sad.

219
00:09:53,509 --> 00:09:55,649
The vertical pipe is a disjunction operator.

220
00:09:55,889 --> 00:09:59,190
It means O -R. Okay. So the machine is hunting

221
00:09:59,190 --> 00:10:02,419
for the word depressed or the word sad. And because

222
00:10:02,419 --> 00:10:04,399
those words are locked inside the parentheses,

223
00:10:04,440 --> 00:10:06,500
the exact moment the computer spots one of them,

224
00:10:06,539 --> 00:10:08,299
it snatches the word and holds it in its pocket.

225
00:10:08,379 --> 00:10:10,360
Right. Then there's another dot star to ignore

226
00:10:10,360 --> 00:10:12,179
whatever else the user typed at the end of the

227
00:10:12,179 --> 00:10:14,299
sentence. And now we reach the second half of

228
00:10:14,299 --> 00:10:17,879
the substitution string. The computer prints,

229
00:10:18,100 --> 00:10:20,659
I'm sorry to hear you are. And then it encounters

230
00:10:20,659 --> 00:10:23,059
the slash one. Right, the slash one. That slash

231
00:10:23,059 --> 00:10:25,759
one is the explicit command to reach into its

232
00:10:25,759 --> 00:10:28,559
pocket, pull out the exact word it captured in

233
00:10:28,559 --> 00:10:30,720
the first set of parentheses, and drop it directly

234
00:10:30,720 --> 00:10:33,740
into the sentence. It is so brilliant in its

235
00:10:33,740 --> 00:10:37,649
simplicity. I type, I think I'm depressed today.

236
00:10:38,259 --> 00:10:40,899
Eliza ignores, I think, finds I am, captures

237
00:10:40,899 --> 00:10:44,440
depressed, ignores today, and spits out, I am

238
00:10:44,440 --> 00:10:46,240
sorry to hear you were depressed. And it feels

239
00:10:46,240 --> 00:10:48,840
like active listening. But it is just a microscopic

240
00:10:48,840 --> 00:10:51,259
find -and -replace script running in milliseconds.

241
00:10:51,659 --> 00:10:53,879
It runs through a whole cascading hierarchy of

242
00:10:53,879 --> 00:10:56,279
these scripts, actually. It swaps my for your,

243
00:10:56,379 --> 00:10:59,120
I for you. But constructing these regex patterns

244
00:10:59,120 --> 00:11:01,220
highlights the fundamental struggle of early

245
00:11:01,220 --> 00:11:03,590
natural language processing. Which is what? When

246
00:11:03,590 --> 00:11:05,610
you define a pattern, you are always engaging

247
00:11:05,610 --> 00:11:07,850
in a tug of war between two opposing metrics,

248
00:11:08,169 --> 00:11:11,110
precision and recall. Yes, I saw this mentioned

249
00:11:11,110 --> 00:11:12,950
constantly in the text. Let's break those down

250
00:11:12,950 --> 00:11:15,710
for the listener. Precision is a measure of exactness.

251
00:11:15,870 --> 00:11:19,029
It is about minimizing false positives. Let's

252
00:11:19,029 --> 00:11:21,049
say you write a simple regex to find the word

253
00:11:21,049 --> 00:11:25,009
the in a text, T -H -E. If your pattern accidentally

254
00:11:25,009 --> 00:11:27,649
highlights the letters T -H -E inside the word

255
00:11:27,649 --> 00:11:30,970
theology or other, your precision plummets. Because

256
00:11:30,970 --> 00:11:33,529
you are catching garbage you didn't want. Exactly.

257
00:11:33,769 --> 00:11:36,370
So to fix that, I tighten up my bouncer's rules.

258
00:11:36,470 --> 00:11:40,269
I say, okay, only find T -H -E if it has a space

259
00:11:40,269 --> 00:11:42,509
before it and a space after it. Which is a very

260
00:11:42,509 --> 00:11:45,850
logical step. But by tightening the rules to

261
00:11:45,850 --> 00:11:47,690
increase your precision, you often damage your

262
00:11:47,690 --> 00:11:50,429
recall. Recall is a measure of completeness.

263
00:11:50,450 --> 00:11:54,070
It is about minimizing false negatives. If you

264
00:11:54,070 --> 00:11:56,690
demand a space before the word the, you will

265
00:11:56,690 --> 00:11:59,799
suddenly miss the word the. It starts a paragraph

266
00:11:59,799 --> 00:12:02,019
because there is no space before it. You will

267
00:12:02,019 --> 00:12:04,980
miss the if it is immediately followed by a comma.

268
00:12:05,299 --> 00:12:07,519
You failed to recall all the valid instances.

269
00:12:07,860 --> 00:12:09,740
So building these systems is just this constant

270
00:12:09,740 --> 00:12:12,080
agonizing adjustment of the dial. You widen the

271
00:12:12,080 --> 00:12:14,019
net to catch more fish, but you start pulling

272
00:12:14,019 --> 00:12:16,220
up old boots. Right. Then you shrink the net

273
00:12:16,220 --> 00:12:18,139
to only kept salmon and half the salmon just

274
00:12:18,139 --> 00:12:20,500
swim right by. That tension never really goes

275
00:12:20,500 --> 00:12:23,919
away, even in modern systems. But regex is only

276
00:12:23,919 --> 00:12:26,299
useful if you know exactly what you are trying

277
00:12:26,299 --> 00:12:29,769
to find. And that leads us to a much deeper philosophical

278
00:12:29,769 --> 00:12:32,909
and technical hurdle. Before we can search for

279
00:12:32,909 --> 00:12:35,490
a word or count a word or translate a word, we

280
00:12:35,490 --> 00:12:37,950
have to figure out how to tell a machine what

281
00:12:37,950 --> 00:12:40,990
a word actually is. Which sounds ridiculously

282
00:12:40,990 --> 00:12:43,409
simple until you actually try to explain it to

283
00:12:43,409 --> 00:12:45,629
a rock with lightning in it. Exactly. To teach

284
00:12:45,629 --> 00:12:48,070
a machine language, we have to feed it text.

285
00:12:48,730 --> 00:12:51,490
Massive amounts of it. The textbook calls a large

286
00:12:51,490 --> 00:12:54,590
collection of text a corpus or corpora for plural.

287
00:12:55,110 --> 00:12:56,789
And the first thing that jumped out at me is

288
00:12:56,789 --> 00:12:59,610
that, Not all corpora are created equal, like

289
00:12:59,610 --> 00:13:02,870
at all. They vary wildly. The text contrasts

290
00:13:02,870 --> 00:13:04,730
the brown corpus with the switchboard corpus.

291
00:13:04,889 --> 00:13:07,529
The brown corpus was assembled in the early 1960s.

292
00:13:07,549 --> 00:13:10,389
It contains one million words of printed American

293
00:13:10,389 --> 00:13:13,629
English, so newspapers, books, government documents.

294
00:13:13,870 --> 00:13:16,330
Very formal. Extremely. The sentences are structured,

295
00:13:16,549 --> 00:13:18,450
grammatically formal, and neatly punctuated.

296
00:13:18,919 --> 00:13:20,580
And then you look at the switchboard corpus,

297
00:13:20,720 --> 00:13:23,539
which is from the 1990s, and it's three million

298
00:13:23,539 --> 00:13:25,879
words of transcribed telephone conversations.

299
00:13:26,299 --> 00:13:29,059
And it looks like absolute chaos. Because people

300
00:13:29,059 --> 00:13:32,139
don't speak in neat sentences. No, they stumble.

301
00:13:32,240 --> 00:13:37,080
They say and they interrupt each other constantly.

302
00:13:37,120 --> 00:13:39,659
They abandon sentences halfway through. There

303
00:13:39,659 --> 00:13:42,940
are no periods or commas in raw speech. Furthermore,

304
00:13:43,100 --> 00:13:45,200
the demographics of the speakers dictate the

305
00:13:45,200 --> 00:13:47,860
linguistic rules. The authors highlight African

306
00:13:47,860 --> 00:13:51,240
American language, or AAL, particularly as it

307
00:13:51,240 --> 00:13:53,399
appears on social media platforms like Twitter.

308
00:13:53,750 --> 00:13:55,610
Yeah, that was a fascinating section. A tweet

309
00:13:55,610 --> 00:13:58,389
might contain the string Iant instead of I don't

310
00:13:58,389 --> 00:14:00,730
or Talmud out instead of talking about. And to

311
00:14:00,730 --> 00:14:02,889
a machine trained only on, say, the Wall Street

312
00:14:02,889 --> 00:14:05,309
Journal, Iant looks like a catastrophic spelling

313
00:14:05,309 --> 00:14:07,450
error. It doesn't know how to process it. But

314
00:14:07,450 --> 00:14:09,610
the text makes a really strong point here. These

315
00:14:09,610 --> 00:14:13,210
aren't typos. They are systematic, rule -governed

316
00:14:13,210 --> 00:14:16,350
features of a specific dialect. This is a crucial

317
00:14:16,350 --> 00:14:18,909
data provenance issue in natural language processing.

318
00:14:19,610 --> 00:14:21,850
The textbook heavily stresses the concept of

319
00:14:21,850 --> 00:14:24,990
data sheets for data sets. When you use a corpus

320
00:14:24,990 --> 00:14:28,070
to train a system, you absolutely must document

321
00:14:28,070 --> 00:14:30,269
the technical and demographic parameters of that

322
00:14:30,269 --> 00:14:33,230
data. Like an ingredients label. Precisely. Who

323
00:14:33,230 --> 00:14:35,909
generated this text? Under what conditions? For

324
00:14:35,909 --> 00:14:38,830
what purpose? If you train a customer service

325
00:14:38,830 --> 00:14:41,570
chatbot exclusively on formal written documents

326
00:14:41,570 --> 00:14:44,190
from the 1980s and then deploy it to process

327
00:14:44,190 --> 00:14:46,970
modern text messages from teenagers, it will

328
00:14:46,970 --> 00:14:49,389
fail technically. Because the underlying data

329
00:14:49,389 --> 00:14:52,350
does not match the deployment reality. The system

330
00:14:52,350 --> 00:14:55,210
is brittle. Exactly. So we have this massive,

331
00:14:55,250 --> 00:14:57,850
messy pile of text. Before the machine can do

332
00:14:57,850 --> 00:14:59,529
anything intelligent with it, it has to clean

333
00:14:59,529 --> 00:15:01,549
it up. It undergoes text normalization. And the

334
00:15:01,549 --> 00:15:03,830
first major step is tokenization. We have to

335
00:15:03,830 --> 00:15:05,950
take a long, continuous string of characters

336
00:15:05,950 --> 00:15:08,049
and basically chop it up into individual tokens,

337
00:15:08,250 --> 00:15:10,509
which usually represent words. The intuitive

338
00:15:10,509 --> 00:15:12,950
human approach is just to split the text wherever

339
00:15:12,950 --> 00:15:15,330
there is a white space. But white space is a

340
00:15:15,330 --> 00:15:18,789
trap. Right. What about a comma? If I write apples,

341
00:15:18,889 --> 00:15:20,909
oranges, the comma is attached directly to the

342
00:15:20,909 --> 00:15:23,470
word apples. I don't want a token that is apples

343
00:15:23,470 --> 00:15:26,330
comma. I want apples and I want the comma to

344
00:15:26,330 --> 00:15:28,710
be its own separate token. So you tell the machine,

345
00:15:28,889 --> 00:15:31,610
okay, separate all punctuation. But then you

346
00:15:31,610 --> 00:15:35,250
encounter an abbreviation like PhD or a price

347
00:15:35,250 --> 00:15:38,370
tag like $45 to 55 cents. If you aggressively

348
00:15:38,370 --> 00:15:40,950
separate all punctuation, you shatter the price

349
00:15:40,950 --> 00:15:43,929
into 45 and 55 and you completely destroy the

350
00:15:43,929 --> 00:15:45,509
semantic meaning of the number. And then you

351
00:15:45,509 --> 00:15:48,100
have multi -word expressions. New York is two

352
00:15:48,100 --> 00:15:51,039
words separated by a space, but it represents

353
00:15:51,039 --> 00:15:53,740
one single geographic concept. Treating it as

354
00:15:53,740 --> 00:15:56,700
two independent tokens, new in York, dilutes

355
00:15:56,700 --> 00:15:58,460
the meaning. Oh, and then the book points out

356
00:15:58,460 --> 00:16:00,659
that languages like Japanese and Chinese don't

357
00:16:00,659 --> 00:16:03,200
even use spaces between characters. So the entire

358
00:16:03,200 --> 00:16:05,980
concept of relying on white space just collapses

359
00:16:05,980 --> 00:16:07,919
entirely. For standard English, organizations

360
00:16:07,919 --> 00:16:10,379
have developed really extensive rule sets like

361
00:16:10,379 --> 00:16:13,139
the Penn Tree Bank tokenization standard. It

362
00:16:13,139 --> 00:16:15,720
has very specific protocols such as how to handle

363
00:16:15,720 --> 00:16:18,419
clitics. Remind me what a clitic is. A word like

364
00:16:18,419 --> 00:16:21,480
doesn't is actually split into two distinct tokens

365
00:16:21,480 --> 00:16:25,490
under this standard. Does and int. Because linguistically,

366
00:16:25,629 --> 00:16:27,990
those are the two functional components. Okay,

367
00:16:28,070 --> 00:16:30,370
that makes sense for words the system actually

368
00:16:30,370 --> 00:16:33,190
knows. But here is the thing that really fascinated

369
00:16:33,190 --> 00:16:35,809
me. No matter how big your training corpus is,

370
00:16:35,950 --> 00:16:38,769
eventually, out in the real world, the machine

371
00:16:38,769 --> 00:16:41,509
is going to encounter a word it has never seen

372
00:16:41,509 --> 00:16:43,909
before in its life. Inevitably. A totally novel

373
00:16:43,909 --> 00:16:46,649
string of characters. If it tries to tokenize

374
00:16:46,649 --> 00:16:49,110
a word that isn't in its vocabulary, doesn't

375
00:16:49,110 --> 00:16:51,710
it just crash? It used to. This was known as

376
00:16:51,710 --> 00:16:54,059
the out -of -vocabulary problem. But modern systems

377
00:16:54,059 --> 00:16:57,080
use an incredibly elegant solution called subword

378
00:16:57,080 --> 00:17:00,100
tokenization. The most famous algorithm for this

379
00:17:00,100 --> 00:17:03,580
is byte pair encoding, or BPE. I honestly read

380
00:17:03,580 --> 00:17:05,599
this section three times because it is so clever.

381
00:17:05,799 --> 00:17:08,500
How does BPE actually figure out what a subword

382
00:17:08,500 --> 00:17:11,039
is? It is entirely data -driven. It does not

383
00:17:11,039 --> 00:17:12,960
rely on linguistic rules or human dictionaries

384
00:17:12,960 --> 00:17:15,460
at all. Instead, it looks at the raw character

385
00:17:15,460 --> 00:17:17,960
sequences in the training data. Initially, its

386
00:17:17,960 --> 00:17:20,200
vocabulary is literally just individual letters.

387
00:17:20,460 --> 00:17:25,589
A, B, C, D. Then it scans the entire corpus and

388
00:17:25,589 --> 00:17:28,150
finds the most frequently occurring adjacent

389
00:17:28,150 --> 00:17:31,730
pair of characters. Let's say E and R appear

390
00:17:31,730 --> 00:17:34,269
next to each other millions of times. Like in

391
00:17:34,269 --> 00:17:37,210
the, or her. Right. The algorithm merges them

392
00:17:37,210 --> 00:17:40,289
and adds er to its vocabulary as a brand new

393
00:17:40,289 --> 00:17:43,170
single token. Okay, so now I have individual

394
00:17:43,170 --> 00:17:46,440
letters. plus this new error block. Then it scans

395
00:17:46,440 --> 00:17:48,920
again. Maybe it notices that L and O appear together

396
00:17:48,920 --> 00:17:51,200
frequently. It merges them into low. It repeats

397
00:17:51,200 --> 00:17:54,000
this process thousands of times. Eventually,

398
00:17:54,039 --> 00:17:56,119
it starts merging the merged blocks. Oh, wow.

399
00:17:56,339 --> 00:17:58,940
It might see low and W frequently adjacent, so

400
00:17:58,940 --> 00:18:01,039
it creates a low token. It might see low and

401
00:18:01,039 --> 00:18:02,720
error adjacent, and it creates a lower token.

402
00:18:03,019 --> 00:18:04,799
It's exactly like building words out of Lido

403
00:18:04,799 --> 00:18:07,220
bricks. It starts with the tiny single studs,

404
00:18:07,220 --> 00:18:09,440
clicks them together into bigger pieces, and

405
00:18:09,440 --> 00:18:11,200
just keeps those really common chunks in its

406
00:18:11,200 --> 00:18:13,829
toolbox. That's exactly it. The brilliance of

407
00:18:13,829 --> 00:18:17,190
BPE is how it handles the unknown. If the machine

408
00:18:17,190 --> 00:18:19,890
has seen the word unbelievable millions of times

409
00:18:19,890 --> 00:18:22,410
during training, it probably merged all those

410
00:18:22,410 --> 00:18:25,109
characters into one massive token. Unbelievable.

411
00:18:25,809 --> 00:18:28,670
But what if a user types a completely made -up

412
00:18:28,670 --> 00:18:32,109
word, like unflappable? The machine has no token

413
00:18:32,109 --> 00:18:35,049
for that full word. But it doesn't crash. It

414
00:18:35,049 --> 00:18:37,410
just breaks the unknown word down into the biggest

415
00:18:37,410 --> 00:18:40,049
Lego bricks it does recognize. Exactly. It might

416
00:18:40,049 --> 00:18:42,920
look at it and say, well... I don't know unflappable,

417
00:18:43,079 --> 00:18:45,319
but I have a brick for un, a brick for flap,

418
00:18:45,500 --> 00:18:47,920
and a brick for able. So I will just process

419
00:18:47,920 --> 00:18:50,759
it as those three tokens. The machine is never

420
00:18:50,759 --> 00:18:53,240
truly stumped. It creates a perfectly robust

421
00:18:53,240 --> 00:18:56,880
system. Now, alongside tokenization, normalization

422
00:18:56,880 --> 00:18:59,700
also involves case folding. Right, which is usually

423
00:18:59,700 --> 00:19:02,539
just converting all text to lowercase so the

424
00:19:02,539 --> 00:19:05,140
machine can match words regardless of capitalization.

425
00:19:05,440 --> 00:19:07,240
Though, as the text notes, this can actually

426
00:19:07,240 --> 00:19:10,099
damage certain tasks. How so? Well, in sentiment

427
00:19:10,099 --> 00:19:13,400
analysis, a user typing in all cappy S is conveying

428
00:19:13,400 --> 00:19:16,400
anger or intense excitement. If you lowercase

429
00:19:16,400 --> 00:19:19,279
everything indiscriminately, you white out that

430
00:19:19,279 --> 00:19:21,420
emotional data. That's a great point. You also

431
00:19:21,420 --> 00:19:23,839
have to deal with word variations. Like if I

432
00:19:23,839 --> 00:19:26,400
have sang, sung, and sings in a document, I probably

433
00:19:26,400 --> 00:19:28,200
want the computer to realize those are all just

434
00:19:28,200 --> 00:19:30,140
versions of the same core concept. There are

435
00:19:30,140 --> 00:19:33,180
two distinct methods for doing that. Lemmatization

436
00:19:33,180 --> 00:19:36,079
is the careful, linguistically accurate method.

437
00:19:36,509 --> 00:19:39,710
It uses a complex morphological analyzer and

438
00:19:39,710 --> 00:19:42,809
a dictionary to map a word back to its true dictionary

439
00:19:42,809 --> 00:19:46,269
root, or lemma. It knows that the lemma of better

440
00:19:46,269 --> 00:19:50,190
is actually good. Which I imagine takes a lot

441
00:19:50,190 --> 00:19:52,309
of time and processing power. It does, which

442
00:19:52,309 --> 00:19:54,690
is why we also have stemming. Stemming algorithms,

443
00:19:54,869 --> 00:19:57,670
like the widely used Porter stemmer, are basically

444
00:19:57,670 --> 00:20:00,759
blunt instruments. they rely on a cascade of

445
00:20:00,759 --> 00:20:03,380
simple regex style rules that aggressively chop

446
00:20:03,380 --> 00:20:06,039
suffixes off the ends of words just hacking the

447
00:20:06,039 --> 00:20:08,720
ends off yes a stemmer doesn't care if the resulting

448
00:20:08,720 --> 00:20:10,960
word is a real english word it might chop the

449
00:20:10,960 --> 00:20:13,220
word ponies down to pony with an eye and it might

450
00:20:13,220 --> 00:20:15,299
chop pony down to pony as well so it literally

451
00:20:15,299 --> 00:20:17,819
creates gibberish it does create gibberish but

452
00:20:17,819 --> 00:20:20,220
as long as both variations map to the exact same

453
00:20:20,220 --> 00:20:22,680
gibberish string the machine successfully groups

454
00:20:22,680 --> 00:20:24,839
them together for search purposes it is fast

455
00:20:24,839 --> 00:20:27,440
dirty and highly effective for things like search

456
00:20:27,440 --> 00:20:29,980
engines okay so We've run our text through the

457
00:20:29,980 --> 00:20:32,940
car wash. We've tokenized the clitics. We've

458
00:20:32,940 --> 00:20:35,420
stemmed the suffixes. We finally have clean data.

459
00:20:35,819 --> 00:20:38,519
But humans are the ones supplying the data, which

460
00:20:38,519 --> 00:20:41,380
means there are typos. Always. If I am typing

461
00:20:41,380 --> 00:20:44,839
fast on my phone and I spell execution as E -X

462
00:20:44,839 --> 00:20:48,500
-E -C -U -T -O -N, omitting the I, my phone auto

463
00:20:48,500 --> 00:20:51,250
-corrects it in a microsecond. How does it know

464
00:20:51,250 --> 00:20:53,109
what I meant to type? It relies on a mathematical

465
00:20:53,109 --> 00:20:56,109
concept detailed in Chapter 2. Minimum edit distance.

466
00:20:56,450 --> 00:20:58,970
Minimum edit distance is a metric that quantifies

467
00:20:58,970 --> 00:21:01,589
how structurally similar two strings of text

468
00:21:01,589 --> 00:21:04,509
are. It calculates the absolute minimum number

469
00:21:04,509 --> 00:21:06,390
of editing operations required to physically

470
00:21:06,390 --> 00:21:08,930
transform the misspelled string into the target

471
00:21:08,930 --> 00:21:10,990
string. And the operations it is allowed to use

472
00:21:10,990 --> 00:21:13,329
are insertions, deletions, and substitutions.

473
00:21:13,490 --> 00:21:15,609
Correct. The book walks through a specific, really

474
00:21:15,609 --> 00:21:18,109
tricky example. Transforming the word intention,

475
00:21:18,430 --> 00:21:21,089
spelled with a T, into execution with an X and

476
00:21:21,089 --> 00:21:23,529
a C. Let's trace how the machine actually thinks

477
00:21:23,529 --> 00:21:25,569
about this. Let's do it. We start with I -N -T

478
00:21:25,569 --> 00:21:29,190
-E -N -T -I -O -N. The target is E -X -E -C -U

479
00:21:29,190 --> 00:21:31,970
-T -I -O -N. Okay, so looking at the first letter,

480
00:21:32,049 --> 00:21:34,329
I have an I, I want an E, I can just substitute

481
00:21:34,329 --> 00:21:36,950
the I for an E. You could, but let's look at

482
00:21:36,950 --> 00:21:39,789
the length. Intention has nine letters. Execution

483
00:21:39,789 --> 00:21:42,690
has nine letters. I -N -T -E -N -T -I -O -N is

484
00:21:42,690 --> 00:21:46,519
nine. E -X -E -C -U -T -I -O -N is nine. Oh.

485
00:21:46,859 --> 00:21:49,480
They are the exact same length. Okay, so substituting

486
00:21:49,480 --> 00:21:52,240
the first I for an E makes total sense. Actually,

487
00:21:52,240 --> 00:21:54,500
the optimal path the algorithm finds is slightly

488
00:21:54,500 --> 00:21:56,079
different because it looks at the whole board

489
00:21:56,079 --> 00:21:58,519
at once. The cheapest mathematical route is to

490
00:21:58,519 --> 00:22:01,039
delete the initial I entirely. Wait, really?

491
00:22:01,200 --> 00:22:03,380
Why would it do that? Because of how the subsequent

492
00:22:03,380 --> 00:22:06,180
letters align. If you delete the I, the N shifts

493
00:22:06,180 --> 00:22:09,299
over. You substitute... n for e then you substitute

494
00:22:09,299 --> 00:22:12,339
the t for x and then you leave the e alone because

495
00:22:12,339 --> 00:22:15,400
execution has an e in that exact spot oh i see

496
00:22:15,400 --> 00:22:17,619
by shifting everything over with that deletion

497
00:22:17,619 --> 00:22:20,200
it saved itself an operation down the line because

498
00:22:20,200 --> 00:22:22,759
that middle e suddenly lined up perfectly exactly

499
00:22:22,759 --> 00:22:25,529
after that free e You are looking at the second

500
00:22:25,529 --> 00:22:27,890
N. You substitute it with a C. You substitute

501
00:22:27,890 --> 00:22:30,329
the next T with a U. And then the rest of the

502
00:22:30,329 --> 00:22:32,910
word T -I -O -N lines up perfectly and requires

503
00:22:32,910 --> 00:22:35,509
zero operations. That is fascinating. So the

504
00:22:35,509 --> 00:22:37,450
operations were delete, substitute, substitute,

505
00:22:37,589 --> 00:22:39,990
substitute, substitute. Five operations in total.

506
00:22:40,230 --> 00:22:42,990
Right. And if we assign a cost of one to each

507
00:22:42,990 --> 00:22:45,950
operation, the minimum edit distance, specifically

508
00:22:45,950 --> 00:22:47,809
what's called the Levenstein distance between

509
00:22:47,809 --> 00:22:50,269
those two words, is five. But wait a second.

510
00:22:50,549 --> 00:22:52,789
We just work that out by eyeballing it and jumping

511
00:22:52,789 --> 00:22:55,549
back and forth like humans. A computer can't

512
00:22:55,549 --> 00:22:58,109
eyeball things. No, it cannot. For a computer

513
00:22:58,109 --> 00:23:01,009
to be absolutely certain that five is the mathematical

514
00:23:01,009 --> 00:23:03,750
minimum, wouldn't it have to calculate every

515
00:23:03,750 --> 00:23:07,210
single possible combination of insertions and

516
00:23:07,210 --> 00:23:09,670
deletions to prove there isn't a secret four

517
00:23:09,670 --> 00:23:12,329
-step path? That would be millions of calculations

518
00:23:12,329 --> 00:23:15,390
for long words. If it used a naive brute force

519
00:23:15,390 --> 00:23:18,660
approach, yes. It would face an exponential explosion

520
00:23:18,660 --> 00:23:20,819
of possibilities, and it would take an eternity.

521
00:23:21,359 --> 00:23:24,059
But computer scientists solved this using a paradigm

522
00:23:24,059 --> 00:23:26,519
called dynamic programming, specifically the

523
00:23:26,519 --> 00:23:28,460
Wagner -Fisher algorithm. How does that work?

524
00:23:28,599 --> 00:23:30,380
The philosophical core of dynamic programming

525
00:23:30,380 --> 00:23:33,440
is this. If you have a massive, complex problem,

526
00:23:33,680 --> 00:23:36,480
you break it down into tiny, overlapping subproblems.

527
00:23:36,480 --> 00:23:38,839
You solve the tiny problems, and crucially, you

528
00:23:38,839 --> 00:23:40,640
write the answers down so you never, ever have

529
00:23:40,640 --> 00:23:42,460
to solve them again. I was trying to picture

530
00:23:42,460 --> 00:23:44,619
this while reading. It felt like trying to navigate

531
00:23:44,619 --> 00:23:46,579
a massive hedge maze. If you take a left turn

532
00:23:46,579 --> 00:23:48,880
and hit a dead end, you don't walk all the way

533
00:23:48,880 --> 00:23:50,579
back to the entrance of the maze to try again.

534
00:23:50,799 --> 00:23:52,700
You just walk back to the last intersection,

535
00:23:53,019 --> 00:23:55,039
leave a little breadcrumb there that says dead

536
00:23:55,039 --> 00:23:57,619
end, and take the right turn instead. You are

537
00:23:57,619 --> 00:24:00,500
saving your intermediate progress. That is a

538
00:24:00,500 --> 00:24:03,200
very apt analogy. The Wagner -Fisher algorithm

539
00:24:03,200 --> 00:24:06,700
physically builds a grid, a matrix. The misspelled

540
00:24:06,700 --> 00:24:08,980
word runs down the left -hand column. The target

541
00:24:08,980 --> 00:24:11,420
word runs across the top row. The cell in the

542
00:24:11,420 --> 00:24:14,079
very top left corner represents the cost of transforming

543
00:24:14,079 --> 00:24:16,599
an empty string into an empty string, which is

544
00:24:16,599 --> 00:24:19,480
obviously zero. Okay, I have my grid. How does

545
00:24:19,480 --> 00:24:21,900
the computer actually fill it out? It moves systematically,

546
00:24:22,200 --> 00:24:25,640
left to right, top to bottom. To fill in any

547
00:24:25,640 --> 00:24:27,900
given empty cell, the algorithm only looks at

548
00:24:27,900 --> 00:24:30,880
three cells that it has already filled out. The

549
00:24:30,880 --> 00:24:32,680
cell immediately above it, the cell immediately

550
00:24:32,680 --> 00:24:35,440
to its left, and the cell diagonally up and left.

551
00:24:35,619 --> 00:24:37,279
Because those three cells represent the three

552
00:24:37,279 --> 00:24:39,960
operations, the cell above represents a deletion.

553
00:24:40,319 --> 00:24:42,180
The cell to the left represents an insertion,

554
00:24:42,200 --> 00:24:45,019
and the diagonal cell represents a substitution.

555
00:24:45,599 --> 00:24:48,140
Precisely. The algorithm looks at those three

556
00:24:48,140 --> 00:24:50,279
neighboring numbers, calculates what it would

557
00:24:50,279 --> 00:24:52,460
cost to take each of those three local steps,

558
00:24:52,759 --> 00:24:55,420
picks the cheapest option, adds it to the running

559
00:24:55,420 --> 00:24:58,460
total, and writes that new number in the empty

560
00:24:58,460 --> 00:25:01,099
cell. Oh, wow. It doesn't look at the whole word.

561
00:25:01,140 --> 00:25:03,839
It only ever makes a tiny local decision based

562
00:25:03,839 --> 00:25:06,519
on the three squares right next to it. And because

563
00:25:06,519 --> 00:25:09,059
it does this row by row, by the time it reaches

564
00:25:09,059 --> 00:25:11,400
the very bottom right corner of the entire grid,

565
00:25:11,640 --> 00:25:14,160
that final cell contains the absolute minimum

566
00:25:14,160 --> 00:25:17,000
cost for the entire word. It solved a massive

567
00:25:17,000 --> 00:25:19,740
problem just by making thousands of tiny localized

568
00:25:19,740 --> 00:25:22,819
choices. And it does one more vital thing. Every

569
00:25:22,819 --> 00:25:25,140
time it writes a number in a cell, it leaves

570
00:25:25,140 --> 00:25:28,660
a breadcrumb. It draws a tiny pointer, a back

571
00:25:28,660 --> 00:25:30,799
pointer, aimed at whichever of the three adjacent

572
00:25:30,799 --> 00:25:33,539
cells it chose as the cheapest path. Once it

573
00:25:33,539 --> 00:25:35,900
reaches the bottom right corner, it simply follows

574
00:25:35,900 --> 00:25:38,339
those pointers backward through the grid all

575
00:25:38,339 --> 00:25:41,000
the way to the top left. That back trace gives

576
00:25:41,000 --> 00:25:43,539
the computer the exact step -by -step alignment

577
00:25:43,539 --> 00:25:46,299
of the two words. It is such an elegant piece

578
00:25:46,299 --> 00:25:49,099
of logic. And what really struck me is that the

579
00:25:49,099 --> 00:25:51,319
textbook points out this isn't just for autocorrect

580
00:25:51,319 --> 00:25:55,200
on my phone. This exact same matrix math is the

581
00:25:55,200 --> 00:25:57,779
foundation for huge swaths of computer science.

582
00:25:58,059 --> 00:26:00,980
It is truly ubiquitous. If you're training a

583
00:26:00,980 --> 00:26:03,339
machine translation model... You use this algorithm

584
00:26:03,339 --> 00:26:06,279
to align sentences in parallel texts. In speech

585
00:26:06,279 --> 00:26:08,700
recognition, if a user speaks a sentence and

586
00:26:08,700 --> 00:26:10,640
the computer generates a transcript, how do we

587
00:26:10,640 --> 00:26:12,920
grade the computer? We use minimum edit distance

588
00:26:12,920 --> 00:26:15,279
to calculate the word error rate against a human

589
00:26:15,279 --> 00:26:18,460
transcript. Exactly. It is even used in bioinformatics

590
00:26:18,460 --> 00:26:21,119
to align sequences of human DNA to find genetic

591
00:26:21,119 --> 00:26:23,859
mutations. The math just scales beautifully.

592
00:26:24,259 --> 00:26:27,240
Okay, so we can clean up text and we can mathematically

593
00:26:27,240 --> 00:26:30,230
fix errors from the past. But what if we want

594
00:26:30,230 --> 00:26:32,170
to predict the future? If I pull out my phone

595
00:26:32,170 --> 00:26:34,950
right now and type, I'm going to... My keyboard

596
00:26:34,950 --> 00:26:38,309
instantly suggests three words. Store, park,

597
00:26:38,529 --> 00:26:42,009
and gym is guessing my intent. How do we move

598
00:26:42,009 --> 00:26:44,750
from fixing typos to actually anticipating thought?

599
00:26:45,029 --> 00:26:47,210
This represents the major leap from processing

600
00:26:47,210 --> 00:26:49,869
text to modeling language. We are entering the

601
00:26:49,869 --> 00:26:52,369
domain of Chapter 3, Language Models. Let's dive

602
00:26:52,369 --> 00:26:55,119
in. The fundamental objective of an Engegram

603
00:26:55,119 --> 00:26:57,819
language model is to calculate the probability

604
00:26:57,819 --> 00:27:01,319
of a specific word appearing given the history

605
00:27:01,319 --> 00:27:03,980
of the words that came before it. Mathematically,

606
00:27:04,180 --> 00:27:07,319
what is the probability of word W given history

607
00:27:07,319 --> 00:27:10,819
H? Correct. Let's say your history is the phrase.

608
00:27:11,700 --> 00:27:14,960
Its water is so transparent. What is the probability

609
00:27:14,960 --> 00:27:18,480
that the very next word is that? Okay. To calculate

610
00:27:18,480 --> 00:27:20,759
this perfectly, strictly according to the laws

611
00:27:20,759 --> 00:27:23,319
of probability, you would use what's called the

612
00:27:23,319 --> 00:27:25,599
chain rule. You would calculate the probability

613
00:27:25,599 --> 00:27:28,539
of the word its, multiply it by the probability

614
00:27:28,539 --> 00:27:31,740
of water given its, multiply that by the probability

615
00:27:31,740 --> 00:27:34,819
of a is given its water and continue chaining

616
00:27:34,819 --> 00:27:36,779
those conditional probabilities until you reach

617
00:27:36,779 --> 00:27:38,839
the end of the sentence. But the textbook makes

618
00:27:38,839 --> 00:27:41,500
it very clear that doing that in reality is literally

619
00:27:41,500 --> 00:27:43,619
impossible. It is impossible because language

620
00:27:43,619 --> 00:27:46,559
is infinitely creative. Humans constantly generate

621
00:27:46,559 --> 00:27:48,400
sentences that have never been uttered before

622
00:27:48,400 --> 00:27:50,099
in the history of the universe. I mean, we're

623
00:27:50,099 --> 00:27:52,660
probably doing it right now. Very likely. So

624
00:27:52,660 --> 00:27:54,819
if your machine is trying to predict the next

625
00:27:54,819 --> 00:27:57,900
word by searching its corpus for an exact match

626
00:27:57,900 --> 00:28:01,710
of a unique 20 -word history, it will never find

627
00:28:01,710 --> 00:28:04,769
it. The count will be zero. You cannot calculate

628
00:28:04,769 --> 00:28:06,970
meaningful statistics from an event that has

629
00:28:06,970 --> 00:28:09,589
never occurred. So we can't look at the whole

630
00:28:09,589 --> 00:28:12,329
history. We have to make a compromise. We essentially

631
00:28:12,329 --> 00:28:15,049
have to cheat. We use a simplifying heuristic

632
00:28:15,049 --> 00:28:17,829
called the Markov assumption. We assume that

633
00:28:17,829 --> 00:28:19,750
the probability of a word does not depend on

634
00:28:19,750 --> 00:28:21,869
the entire history of the universe, but only

635
00:28:21,869 --> 00:28:24,410
on the local context. We restrict our vision.

636
00:28:24,609 --> 00:28:27,109
An n -gram model looks n words into the past.

637
00:28:27,369 --> 00:28:30,960
So a bingram model where n is 2 only looks at

638
00:28:30,960 --> 00:28:33,119
the single word immediately preceding the target.

639
00:28:33,220 --> 00:28:35,240
It ignores everything else. If the sentence is,

640
00:28:35,279 --> 00:28:37,680
its water is so transparent, a bigogram model

641
00:28:37,680 --> 00:28:39,819
trying to predict the next word completely ignores

642
00:28:39,819 --> 00:28:43,039
its water is so. It only asks, what is the probability

643
00:28:43,039 --> 00:28:45,500
of the word that, given just the word transparent?

644
00:28:45,839 --> 00:28:48,559
It seems reductive, and it is, but it makes the

645
00:28:48,559 --> 00:28:50,960
math actually computable. How does the model

646
00:28:50,960 --> 00:28:53,500
calculate that probability? It uses maximum likelihood

647
00:28:53,500 --> 00:28:56,339
estimation, or MLE. MLE sounds complex, but it

648
00:28:56,339 --> 00:28:58,559
is literally just counting and dividing. It's

649
00:28:58,559 --> 00:29:00,410
just... Building giant lookup tables, right?

650
00:29:00,410 --> 00:29:02,789
That is all it is. The machine scans the entire

651
00:29:02,789 --> 00:29:05,369
training corpus. It counts the total number of

652
00:29:05,369 --> 00:29:07,809
times the single word transparent appears. Let's

653
00:29:07,809 --> 00:29:11,269
say it appears 1 ,000 times. Then it counts how

654
00:29:11,269 --> 00:29:13,769
many times the specific two -word sequence transparent

655
00:29:13,769 --> 00:29:16,369
that appears. Let's say it finds it 500 times.

656
00:29:16,569 --> 00:29:19,589
To get the probability, you divide 500 by 1 ,000.

657
00:29:19,630 --> 00:29:23,480
The big round probability is 0 .5, 50%. So the

658
00:29:23,480 --> 00:29:25,720
model just has billions of these little fractions

659
00:29:25,720 --> 00:29:27,539
stored in memory. And if I want to find the probability

660
00:29:27,539 --> 00:29:30,279
of a whole sentence, I just multiply the bibliogram

661
00:29:30,279 --> 00:29:32,039
probability of every step together. Correct.

662
00:29:32,299 --> 00:29:34,519
But when I read that, I immediately thought of

663
00:29:34,519 --> 00:29:36,920
a massive hardware problem. Probabilities are

664
00:29:36,920 --> 00:29:40,309
decimals between 0 and 1. If I multiply 0 .5

665
00:29:40,309 --> 00:29:44,650
times 0 .1 times 0 .05 times 0 .01, the number

666
00:29:44,650 --> 00:29:47,329
shrinks incredibly fast. It shrinks exponentially.

667
00:29:47,930 --> 00:29:50,509
In computers, hardware processors only have a

668
00:29:50,509 --> 00:29:53,049
finite amount of memory allocated to store the

669
00:29:53,049 --> 00:29:55,670
decimal places of a floating point number. If

670
00:29:55,670 --> 00:29:57,829
you multiply enough small probabilities together,

671
00:29:58,109 --> 00:30:00,250
the resulting decimal becomes so microscopic

672
00:30:00,250 --> 00:30:02,390
that the computer physically cannot represent

673
00:30:02,390 --> 00:30:05,069
it anymore. It just gives up. It simply rounds

674
00:30:05,069 --> 00:30:08,690
the number down to absolute zero. Which destroys

675
00:30:08,690 --> 00:30:11,769
the entire calculation. The book calls this mathematical

676
00:30:11,769 --> 00:30:15,190
underflow. Yes. To bypass hardware limitations,

677
00:30:15,609 --> 00:30:18,250
language models do not multiply raw probabilities.

678
00:30:18,569 --> 00:30:21,569
They convert every probability into its logarithm.

679
00:30:21,670 --> 00:30:24,029
Okay, log probabilities. In mathematics, if you

680
00:30:24,029 --> 00:30:26,829
add the logarithms of two numbers, it is functionally

681
00:30:26,829 --> 00:30:28,910
equivalent to multiplying the original numbers.

682
00:30:29,130 --> 00:30:31,450
By adding log probabilities together instead,

683
00:30:31,789 --> 00:30:34,150
the numbers stay in a manageable range, usually

684
00:30:34,150 --> 00:30:37,180
as negative numbers. Underflow is avoided entirely.

685
00:30:37,519 --> 00:30:39,880
If you need the raw probability at the very end,

686
00:30:39,900 --> 00:30:42,559
you just exponentiate the final sum. Okay, so

687
00:30:42,559 --> 00:30:44,839
we've dodged the hardware limits. We've built

688
00:30:44,839 --> 00:30:47,700
an Enneagram model. It has tables of log probabilities.

689
00:30:48,240 --> 00:30:50,700
How do we actually know if the model is smart?

690
00:30:50,900 --> 00:30:53,400
Like, how do we grade it? We need an evaluation

691
00:30:53,400 --> 00:30:56,380
metric. The standard intrinsic metric for language

692
00:30:56,380 --> 00:30:59,109
models is perplexity. Perplexity basically measures

693
00:30:59,109 --> 00:31:01,490
how surprised a model is when it encounters real

694
00:31:01,490 --> 00:31:04,390
human language. I really love that framing. We

695
00:31:04,390 --> 00:31:07,569
want a model that is very hard to surprise. Precisely.

696
00:31:07,670 --> 00:31:10,630
You take a test, set a document the model was

697
00:31:10,630 --> 00:31:13,430
not allowed to look at during training. You feed

698
00:31:13,430 --> 00:31:15,450
this text into the model and ask it to calculate

699
00:31:15,450 --> 00:31:18,890
the probability of the text occurring. Perplexity

700
00:31:18,890 --> 00:31:21,490
is mathematically defined as the inverse probability

701
00:31:21,490 --> 00:31:24,410
of the test set normalized by the total number

702
00:31:24,410 --> 00:31:27,349
of words. Okay, inverse probability. Meaning,

703
00:31:27,470 --> 00:31:30,170
if the model assigns a very high probability

704
00:31:30,170 --> 00:31:32,670
to the text, if it thinks the text is totally

705
00:31:32,670 --> 00:31:35,210
normal and expected, the perplexity score will

706
00:31:35,210 --> 00:31:38,309
be very low. Yes. A lower perplexity score indicates

707
00:31:38,309 --> 00:31:41,269
a much better model. It means the model's internal

708
00:31:41,269 --> 00:31:43,529
statistical distribution closely matches the

709
00:31:43,529 --> 00:31:46,109
actual distribution of human language. But to

710
00:31:46,109 --> 00:31:48,390
truly grasp what an n -gram model has learned,

711
00:31:48,630 --> 00:31:50,670
the best method is to actually run it in reverse,

712
00:31:50,950 --> 00:31:53,329
to make it hallucinate. This was my absolute

713
00:31:53,329 --> 00:31:55,690
favorite visual in the chapter, the Shannon visualization

714
00:31:55,690 --> 00:31:58,130
from the 1950s. You basically use the probability

715
00:31:58,130 --> 00:32:00,450
tables to generate text. It is like a statistical

716
00:32:00,450 --> 00:32:02,769
game of Mad Libs. Let's walk through it. Imagine

717
00:32:02,769 --> 00:32:05,109
you have a bigogram model. You need to pick a

718
00:32:05,109 --> 00:32:07,250
starting word. You look at your unigram tables,

719
00:32:07,549 --> 00:32:09,970
the raw frequency of all words, and you treat

720
00:32:09,970 --> 00:32:12,849
it like a weighted roulette wheel. The word the

721
00:32:12,849 --> 00:32:16,309
occupies a huge chunk of the wheel. The word

722
00:32:16,309 --> 00:32:19,869
xylophone occupies a microscopic sliver. You

723
00:32:19,869 --> 00:32:22,890
spin the wheel. It lands on the. So you print

724
00:32:22,890 --> 00:32:25,930
the word they. Now you shift to your bigger gram

725
00:32:25,930 --> 00:32:28,529
tables. You pull up the specific table for the

726
00:32:28,529 --> 00:32:30,930
word the. This is a brand new roulette wheel.

727
00:32:31,029 --> 00:32:33,390
It only contains words that followed the in the

728
00:32:33,390 --> 00:32:36,029
training data. Dog might have a 10 % chance.

729
00:32:36,170 --> 00:32:38,569
Man might have an 8 % chance. You spin this new

730
00:32:38,569 --> 00:32:41,630
wheel. It lands on dog. You print dog. And then

731
00:32:41,630 --> 00:32:43,569
because it is a bigger gram model, you completely

732
00:32:43,569 --> 00:32:45,990
throw away the word the. The model forgets it

733
00:32:45,990 --> 00:32:47,970
ever existed. You pull up the roulette wheel

734
00:32:47,970 --> 00:32:51,160
for dog, spin it, and maybe get barked. If you

735
00:32:51,160 --> 00:32:53,559
generate text using only a unigram model spinning

736
00:32:53,559 --> 00:32:55,980
the overall frequency wheel with zero context,

737
00:32:56,079 --> 00:32:58,299
the output is absolute gibberish. The textbook

738
00:32:58,299 --> 00:33:02,000
shows an example to him. Boyan. It's just statistical

739
00:33:02,000 --> 00:33:04,279
noise. Right. But if you step up to a bigogram

740
00:33:04,279 --> 00:33:07,319
model, suddenly local phrases start to make sense.

741
00:33:07,500 --> 00:33:09,579
He stepped out into the hall. It feels like syntax.

742
00:33:09,920 --> 00:33:13,039
And if you step up to a trigram or a fourgram

743
00:33:13,039 --> 00:33:15,799
model, the generated sentences start to look

744
00:33:15,799 --> 00:33:18,859
eerily fluent. They sound like real English.

745
00:33:19,430 --> 00:33:21,869
However, if you read a full paragraph of it,

746
00:33:21,950 --> 00:33:24,390
you realize it lacks any overarching logical

747
00:33:24,390 --> 00:33:28,089
coherence. It is just wandering aimlessly. It

748
00:33:28,089 --> 00:33:29,930
sounds fluent because the statistics enforce

749
00:33:29,930 --> 00:33:32,930
local grammatical rules, but the Markov assumption

750
00:33:32,930 --> 00:33:35,109
prevents it from holding on to a global thought.

751
00:33:35,470 --> 00:33:38,250
But there is a fatal flaw in this entire counting

752
00:33:38,250 --> 00:33:40,150
method, isn't there? The MLE counting method.

753
00:33:40,289 --> 00:33:42,450
We touched on it briefly before. Well, zero problem.

754
00:33:42,569 --> 00:33:45,410
It is the bane of all Enneagram models. Let's

755
00:33:45,410 --> 00:33:47,509
say I train my bigger model on the entire...

756
00:33:47,670 --> 00:33:49,910
Complete works of William Shakespeare. Millions

757
00:33:49,910 --> 00:33:52,210
of words. And then I ask it to evaluate the probability

758
00:33:52,210 --> 00:33:53,990
of the sentence, I want to eat Chinese food.

759
00:33:54,170 --> 00:33:56,309
Okay. The model looks at its tables. It finds

760
00:33:56,309 --> 00:33:59,460
the bigger M, eat Chinese. But Shakespeare never

761
00:33:59,460 --> 00:34:01,700
wrote the word Chinese. So the count in the training

762
00:34:01,700 --> 00:34:04,200
corpus is exactly zero. Which means the probability

763
00:34:04,200 --> 00:34:07,180
the model assigns to eat Chinese is zero. And

764
00:34:07,180 --> 00:34:08,940
because you are calculating the probability of

765
00:34:08,940 --> 00:34:11,360
the entire sentence by chaining these local probabilities

766
00:34:11,360 --> 00:34:14,500
together. If you multiply anything by zero, the

767
00:34:14,500 --> 00:34:18,239
entire equation drops to absolute zero. The model

768
00:34:18,239 --> 00:34:21,199
effectively declares, this sentence is a mathematical

769
00:34:21,199 --> 00:34:23,940
impossibility in the English language, which

770
00:34:23,940 --> 00:34:26,519
is absurd. Just because you haven't seen a specific

771
00:34:26,519 --> 00:34:28,719
combination of words doesn't mean it's literally

772
00:34:28,719 --> 00:34:31,380
impossible. To mitigate this, computer scientists

773
00:34:31,380 --> 00:34:33,699
developed techniques covered in Chapter 3 under

774
00:34:33,699 --> 00:34:36,480
the umbrella of smoothing. The goal of smoothing

775
00:34:36,480 --> 00:34:38,820
is to eliminate zeros without completely destroying

776
00:34:38,820 --> 00:34:41,380
the integrity of the real statistics. The most

777
00:34:41,380 --> 00:34:44,420
obvious solution is Laplace smoothing, also known

778
00:34:44,420 --> 00:34:46,940
as add -one smoothing. If you have a zero in

779
00:34:46,940 --> 00:34:48,980
your table, just artificially add the number

780
00:34:48,980 --> 00:34:51,539
one to it. Problem solved. But you can't just

781
00:34:51,539 --> 00:34:54,460
add one to the specific zeros you happen to encounter

782
00:34:54,460 --> 00:34:56,920
on the test. To keep the math valid, you have

783
00:34:56,920 --> 00:34:59,159
to add one to the count of every single possible

784
00:34:59,159 --> 00:35:01,800
big rem in your entire vocabulary matrix. Oh,

785
00:35:01,800 --> 00:35:04,300
and if my vocabulary is 20 ,000 words, that means

786
00:35:04,300 --> 00:35:06,480
my matrix of every possible two -word combination

787
00:35:06,480 --> 00:35:10,110
has 400 million slots. And the vast, vast majority

788
00:35:10,110 --> 00:35:12,530
of those combinations are absolute nonsense like

789
00:35:12,530 --> 00:35:16,630
the Tomodoro apple runs. If I add 1 to 399 million

790
00:35:16,630 --> 00:35:19,750
nonsense B -grams... You generate a massive ocean

791
00:35:19,750 --> 00:35:22,809
of fake probability mass. But the total probability

792
00:35:22,809 --> 00:35:26,570
of all events still has to sum to exactly 1 .0.

793
00:35:26,710 --> 00:35:28,989
So where does that fake mass come from? You have

794
00:35:28,989 --> 00:35:30,730
to steal it from the bigograms that actually

795
00:35:30,730 --> 00:35:32,889
did occur in the text. It's like a Robin Hood

796
00:35:32,889 --> 00:35:35,170
algorithm, but it is disastrously aggressive.

797
00:35:35,550 --> 00:35:38,449
It steals probability mass from the rich, frequent

798
00:35:38,449 --> 00:35:41,349
words and gives it to the millions of poor, unseen

799
00:35:41,349 --> 00:35:43,429
words until the frequent words are virtually

800
00:35:43,429 --> 00:35:46,349
bankrupt. The statistics get completely distorted.

801
00:35:46,809 --> 00:35:49,309
Which is why Laplace is rarely used for engrams

802
00:35:49,309 --> 00:35:51,570
in practice. We need more nuanced approaches.

803
00:35:52,170 --> 00:35:55,070
The text introduces interpolation. What's that?

804
00:35:55,309 --> 00:35:58,010
Interpolation is highly pragmatic. If you are

805
00:35:58,010 --> 00:36:00,289
using a trigram model to predict a word, but

806
00:36:00,289 --> 00:36:01,989
the specific three -word history has a count

807
00:36:01,989 --> 00:36:04,789
of zero, why throw your hands up? Why not also

808
00:36:04,789 --> 00:36:06,969
consult the bigogram and the unigram models?

809
00:36:07,070 --> 00:36:09,869
You just mix them all together? Yes. You calculate

810
00:36:09,869 --> 00:36:12,389
the trigram, the b -gram, and the unigram probabilities,

811
00:36:12,710 --> 00:36:15,150
and you combine them using specific weights,

812
00:36:15,329 --> 00:36:17,210
usually represented by the Greek letter lambda.

813
00:36:18,010 --> 00:36:20,389
You might give the trigram 60 % of the weight,

814
00:36:20,469 --> 00:36:24,389
the b -gram 30%, and the unigram 10%. You determine

815
00:36:24,389 --> 00:36:26,550
the optimal weights by testing them on a separate

816
00:36:26,550 --> 00:36:29,030
held -out dataset until you find the mix that

817
00:36:29,030 --> 00:36:31,590
yields the lowest perplexity. So even if the

818
00:36:31,590 --> 00:36:34,570
trigram count is a hard zero, the b -gram or

819
00:36:34,570 --> 00:36:36,849
unigram will almost certainly contribute some

820
00:36:36,849 --> 00:36:39,300
decimal. keeping the overall equation alive.

821
00:36:39,539 --> 00:36:42,059
Exactly. The other method is backoff, which is

822
00:36:42,059 --> 00:36:44,199
similar, but instead of mixing them constantly,

823
00:36:44,420 --> 00:36:46,679
you only back off to the b -gram if the trigram

824
00:36:46,679 --> 00:36:49,179
is exactly zero. You only consult the lower order

825
00:36:49,179 --> 00:36:51,940
model in an emergency. But backoff introduces

826
00:36:51,940 --> 00:36:55,019
a mathematical complication. If you simply sub

827
00:36:55,019 --> 00:36:57,360
in the b -gram probability when the trigram fails,

828
00:36:57,619 --> 00:36:59,920
you are artificially inflating the total probability

829
00:36:59,920 --> 00:37:02,380
space. The numbers will add up to more than 1

830
00:37:02,380 --> 00:37:05,300
.0. You must hold back some probability mass

831
00:37:05,300 --> 00:37:07,599
specifically to fund these back -off emergencies.

832
00:37:07,980 --> 00:37:10,119
You must discount the counts of the n -grams

833
00:37:10,119 --> 00:37:12,800
you did see. This leads to my absolute favorite

834
00:37:12,800 --> 00:37:15,139
piece of statistical trickery in the whole book.

835
00:37:15,599 --> 00:37:18,679
Good Turing discounting. Specifically the realization

836
00:37:18,679 --> 00:37:21,340
by Church and Gale in 1991. That's very clever.

837
00:37:21,519 --> 00:37:23,099
They were trying to figure out how much to discount.

838
00:37:23,579 --> 00:37:25,920
They looked at a massive training corpus and

839
00:37:25,920 --> 00:37:28,949
asked a simple question. If a specific bigorum

840
00:37:28,949 --> 00:37:31,469
appears exactly four times in this training text,

841
00:37:31,750 --> 00:37:33,949
how many times should we expect it to appear

842
00:37:33,949 --> 00:37:36,769
in a brand new text of the exact same size? The

843
00:37:36,769 --> 00:37:39,289
intuitive human answer is four. If it happened

844
00:37:39,289 --> 00:37:41,289
four times here, it should happen four times

845
00:37:41,289 --> 00:37:44,010
there. But it doesn't. They empirically tested

846
00:37:44,010 --> 00:37:47,449
it and found the answer was 3 .23. Because of

847
00:37:47,449 --> 00:37:49,570
the mathematical distribution of rare words and

848
00:37:49,570 --> 00:37:51,730
language, things that appear four times in a

849
00:37:51,730 --> 00:37:54,130
training set are statistically slightly less

850
00:37:54,130 --> 00:37:56,730
likely to appear in a new text. It is a profound

851
00:37:56,730 --> 00:37:59,210
insight into the geometry of language distribution.

852
00:37:59,590 --> 00:38:01,590
So Church and Gale use that insight. If they

853
00:38:01,590 --> 00:38:03,510
see a count of four, they artificially lower

854
00:38:03,510 --> 00:38:07,050
it to 3 .23. They take that leftover 0 .77 of

855
00:38:07,050 --> 00:38:09,150
probability mass and toss it into a savings account.

856
00:38:09,349 --> 00:38:11,449
And they use that savings account to fund all

857
00:38:11,449 --> 00:38:14,389
the zero -count n -grams. It is a much more accurate,

858
00:38:14,550 --> 00:38:17,369
empirically grounded Robin Hood. But the undisputed

859
00:38:17,369 --> 00:38:19,449
heavyweight champion of smoothing algorithms

860
00:38:19,449 --> 00:38:23,030
is Messernay. And then Messernay introduces a

861
00:38:23,030 --> 00:38:27,000
concept that... genuinely blew my mind, the probability

862
00:38:27,000 --> 00:38:30,039
of a word being a novel continuation. I really

863
00:38:30,039 --> 00:38:32,179
need to walk through this slowly to make sure

864
00:38:32,179 --> 00:38:34,699
I grasp it. Let's set up a scenario. You are

865
00:38:34,699 --> 00:38:36,739
building a biogram model. You have the history

866
00:38:36,739 --> 00:38:39,460
phrase I went to, and you want to predict the

867
00:38:39,460 --> 00:38:41,440
next word. Let's evaluate the word Francisco.

868
00:38:42,019 --> 00:38:43,920
Okay, if I look at standard unigram frequencies.

869
00:38:44,500 --> 00:38:46,440
The word Francisco is actually incredibly common

870
00:38:46,440 --> 00:38:48,679
in an English corpus. San Francisco is a major

871
00:38:48,679 --> 00:38:51,380
city. It's in the news constantly. So a standard

872
00:38:51,380 --> 00:38:54,320
back -off model might look at its high overall

873
00:38:54,320 --> 00:38:56,440
frequency and guess, sure, Francisco has a pretty

874
00:38:56,440 --> 00:38:59,019
good chance of following I went to. But we know

875
00:38:59,019 --> 00:39:02,119
intuitively that is wrong. Why? Because Francisco

876
00:39:02,119 --> 00:39:04,400
essentially has only one friend in the English

877
00:39:04,400 --> 00:39:07,340
language. It almost exclusively appears immediately

878
00:39:07,340 --> 00:39:10,519
after the word San. It does not freely mingle.

879
00:39:10,619 --> 00:39:13,900
It never appears after the or a or two. It is

880
00:39:13,900 --> 00:39:17,019
not a versatile word. Exactly. Now contrast that

881
00:39:17,019 --> 00:39:19,300
with the word glasses. Glasses might have the

882
00:39:19,300 --> 00:39:22,260
exact same total unigram frequency as Francisco

883
00:39:22,260 --> 00:39:25,099
in the corpus. But glasses appears after dozens

884
00:39:25,099 --> 00:39:27,699
of different words. Reading glasses, water glasses,

885
00:39:28,099 --> 00:39:30,780
sunglasses, my glasses. It has a very high continuation

886
00:39:30,780 --> 00:39:33,019
probability. It is promiscuous. It hangs out

887
00:39:33,019 --> 00:39:35,349
with everybody. Kineserne smoothing fundamentally

888
00:39:35,349 --> 00:39:38,570
alters the back -off calculation. When the model

889
00:39:38,570 --> 00:39:41,650
drops down to a lower order distribution to evaluate

890
00:39:41,650 --> 00:39:44,489
a word, it does not ask how many times did this

891
00:39:44,489 --> 00:39:47,469
word appear in total. Instead, it asks how many

892
00:39:47,469 --> 00:39:49,710
different unique words did this word follow in

893
00:39:49,710 --> 00:39:52,210
the training data. It measures the word's versatility.

894
00:39:52,489 --> 00:39:55,090
It knows that Francisco is a terrible novel continuation,

895
00:39:55,210 --> 00:39:57,949
but Glasses is a great one. It drastically improves

896
00:39:57,949 --> 00:40:00,570
the accuracy of the predictions. It is so incredibly

897
00:40:00,570 --> 00:40:02,889
elegant. We've pushed counting to its absolute

898
00:40:02,889 --> 00:40:05,849
mathematical limit here. But the textbook makes

899
00:40:05,849 --> 00:40:08,449
a hard pivot here. Even with Nezerne, even if

900
00:40:08,449 --> 00:40:11,170
we train an engram model on the entire internet,

901
00:40:11,489 --> 00:40:14,349
engrams have a fetal, insurmountable ceiling.

902
00:40:14,550 --> 00:40:16,889
They do. Because engrams possess absolutely zero

903
00:40:16,889 --> 00:40:18,949
understanding of semantics, they do not know

904
00:40:18,949 --> 00:40:21,360
what words actually mean. To an Enneagram model,

905
00:40:21,559 --> 00:40:24,880
the string of characters C -A -T and the string

906
00:40:24,880 --> 00:40:27,940
D -O -G are as completely unrelated as C -A -T

907
00:40:27,940 --> 00:40:31,199
and R -E -F -R -G -E -R -A -T -O -R. Exactly.

908
00:40:31,320 --> 00:40:33,639
They are just distinct variables. The model does

909
00:40:33,639 --> 00:40:35,880
not realize that cats and dogs share properties.

910
00:40:36,340 --> 00:40:37,960
They are both pets. They both have fur. They

911
00:40:37,960 --> 00:40:40,219
both eat food. Because it cannot generalize,

912
00:40:40,219 --> 00:40:42,559
it is horribly inefficient. Right. If the model

913
00:40:42,559 --> 00:40:44,420
sees the sentence, the cat is fed a thousand

914
00:40:44,420 --> 00:40:46,500
times in training, it learns that fed follows

915
00:40:46,500 --> 00:40:49,679
cat. But if it never explicitly sees the phrase

916
00:40:49,679 --> 00:40:52,980
the dog is fed, it has no idea what to do. It

917
00:40:52,980 --> 00:40:55,659
cannot infer that dogs might also get fed. It

918
00:40:55,659 --> 00:40:58,559
treats every string as an isolated island of

919
00:40:58,559 --> 00:41:00,880
data. Which brings us to the real revolution.

920
00:41:01,280 --> 00:41:03,500
How do you teach a rock to understand the meaning

921
00:41:03,500 --> 00:41:06,119
of a word when the rock only understands numbers?

922
00:41:06,280 --> 00:41:09,320
You turn the word into a location in space. We

923
00:41:09,320 --> 00:41:12,119
are diving into Chapter 6, Vector Semantics and

924
00:41:12,119 --> 00:41:14,400
Embeddings. The foundational linguistic philosophy

925
00:41:14,400 --> 00:41:16,739
behind this actually dates back to the 1950s,

926
00:41:16,739 --> 00:41:19,340
to structural linguists like J .R. Firth. Firth

927
00:41:19,340 --> 00:41:21,599
famously declared, you shall know a word by the

928
00:41:21,599 --> 00:41:23,940
company it keeps. I love that quote. The core

929
00:41:23,940 --> 00:41:26,400
theory is that the meaning of a word is not defined

930
00:41:26,400 --> 00:41:29,400
by a dictionary entry. It is defined by its distribution

931
00:41:29,400 --> 00:41:32,260
in language. It is defined by the neighbors that

932
00:41:32,260 --> 00:41:35,400
constantly surround it. The textbook uses a brilliant

933
00:41:35,400 --> 00:41:38,840
example to prove this. The word ongchoy. I had

934
00:41:38,840 --> 00:41:41,480
absolutely no idea what ongchoy was. But the

935
00:41:41,480 --> 00:41:43,960
authors say, imagine you read a corpus and you

936
00:41:43,960 --> 00:41:46,579
notice that the word ong choy frequently appears

937
00:41:46,579 --> 00:41:49,000
in sentences alongside words like rice, garlic,

938
00:41:49,119 --> 00:41:52,059
delicious, woke, and salty. What is your brain

939
00:41:52,059 --> 00:41:54,639
doing when you read that? I am instantly cross

940
00:41:54,639 --> 00:41:57,099
-referencing. I ask my brain, what other words

941
00:41:57,099 --> 00:41:59,639
do I know that hang out with rice, garlic, and

942
00:41:59,639 --> 00:42:02,679
woke? And my brain pulls up spinach, chard, and

943
00:42:02,679 --> 00:42:05,059
collard greens. Suddenly, without ever looking

944
00:42:05,059 --> 00:42:07,260
at a dictionary, I know with near certainty that

945
00:42:07,260 --> 00:42:09,659
ong choy must be a leafy green vegetable used

946
00:42:09,659 --> 00:42:12,230
in Asian cooking. I inferred its meaning purely

947
00:42:12,230 --> 00:42:14,389
from its neighborhood. That human intuition is

948
00:42:14,389 --> 00:42:17,010
the engine of vector semantics. We can represent

949
00:42:17,010 --> 00:42:19,889
a word as a vector, an array of numbers. In the

950
00:42:19,889 --> 00:42:22,590
early iterations, researchers used sparse vectors,

951
00:42:22,710 --> 00:42:26,150
specifically TF -IDF, which stands for Term Frequency

952
00:42:26,150 --> 00:42:28,550
Inverse Document Frequency. How does a sparse

953
00:42:28,550 --> 00:42:31,650
vector work? Imagine an array with 50 ,000 slots.

954
00:42:31,969 --> 00:42:35,130
One slot for every word in the English language.

955
00:42:35,530 --> 00:42:38,329
To build the vector for Ong Choi, you scan the

956
00:42:38,329 --> 00:42:41,599
corpus. Every time ang shui appears within 10

957
00:42:41,599 --> 00:42:44,039
words of garlic, you add a 1 to the garlic slot.

958
00:42:44,440 --> 00:42:46,900
Every time it appears near rice, you add a 1

959
00:42:46,900 --> 00:42:49,300
to the rice slot. But because ang shui doesn't

960
00:42:49,300 --> 00:42:51,579
appear near refrigerator or spaceship, those

961
00:42:51,579 --> 00:42:54,760
slots just stay at zero. Exactly. You end up

962
00:42:54,760 --> 00:42:58,219
with a vector of 50 ,000 numbers, and 49 ,900

963
00:42:58,219 --> 00:43:01,519
of them are zeros. It is incredibly sparse. But

964
00:43:01,519 --> 00:43:03,599
even with all those zeros, we suddenly have a

965
00:43:03,599 --> 00:43:06,860
mathematical way to compare meanings. If we plot

966
00:43:06,860 --> 00:43:09,239
these vectors in a 50 ,000 -dimensional space,

967
00:43:09,559 --> 00:43:12,159
which is obviously impossible to picture but

968
00:43:12,159 --> 00:43:15,059
mathematically trivial, we can measure the distance

969
00:43:15,059 --> 00:43:17,320
between them. We measure the similarity using

970
00:43:17,320 --> 00:43:19,800
a metric called cosine similarity. You treat

971
00:43:19,800 --> 00:43:22,079
the two vectors as arrows originating from the

972
00:43:22,079 --> 00:43:24,460
same central origin point. You calculate the

973
00:43:24,460 --> 00:43:26,780
geometric angle between the two arrows. If the

974
00:43:26,780 --> 00:43:29,079
angle is very small, if the arrows point in roughly

975
00:43:29,079 --> 00:43:31,159
the same direction, it means their vectors share

976
00:43:31,159 --> 00:43:33,610
high numbers in the same slots. It means they

977
00:43:33,610 --> 00:43:35,530
share the same neighbors. Therefore, they have

978
00:43:35,530 --> 00:43:37,610
similar meanings. And the brilliant thing about

979
00:43:37,610 --> 00:43:40,329
cosine similarity is that it measures the angle,

980
00:43:40,429 --> 00:43:43,329
not the length of the arrow. So if the word spinach

981
00:43:43,329 --> 00:43:46,309
appears 10 million times in the corpus and encoin

982
00:43:46,309 --> 00:43:48,489
only appears 100 times, their arrows will be

983
00:43:48,489 --> 00:43:50,670
vastly different lengths. But because they point

984
00:43:50,670 --> 00:43:52,869
in the same direction, the computer still knows

985
00:43:52,869 --> 00:43:55,050
they mean the same thing. The frequency doesn't

986
00:43:55,050 --> 00:43:57,750
distort the semantic comparison. Cosine similarity

987
00:43:57,750 --> 00:44:00,809
remains a vital tool. But the real explosion

988
00:44:00,809 --> 00:44:03,250
in AI capabilities occurred when researchers

989
00:44:03,250 --> 00:44:06,289
abandoned those massive, sparse vectors filled

990
00:44:06,289 --> 00:44:09,889
with zeros and moved to dense embeddings. Models

991
00:44:09,889 --> 00:44:13,050
like Word2Vec and GloVe. Instead of 50 ,000 slots

992
00:44:13,050 --> 00:44:15,389
of mostly zeros, we compress the vector down

993
00:44:15,389 --> 00:44:18,489
to maybe 300 slots. But every slot contains a

994
00:44:18,489 --> 00:44:22,090
dense, continuous decimal number. No zeros. But

995
00:44:22,090 --> 00:44:23,989
how on earth do you figure out what those 300

996
00:44:23,989 --> 00:44:26,780
decimals should be? Word2Vec uses an architecture

997
00:44:26,780 --> 00:44:29,099
called Skipgram, and reading how it trains itself

998
00:44:29,099 --> 00:44:31,019
is just a revelation. It feels like a magic trick.

999
00:44:31,199 --> 00:44:33,360
Well, it is a remarkable piece of indirect engineering.

1000
00:44:33,940 --> 00:44:36,179
Skipgram does not try to explicitly calculate

1001
00:44:36,179 --> 00:44:38,719
the meaning of a word. Instead, it sets up a

1002
00:44:38,719 --> 00:44:41,400
fake localized task. It builds a very simple

1003
00:44:41,400 --> 00:44:44,039
logistic regression classifier. Okay, let's slow

1004
00:44:44,039 --> 00:44:47,619
down. A logistic regression classifier. What

1005
00:44:47,619 --> 00:44:50,559
is that actually doing in plain English? It is

1006
00:44:50,559 --> 00:44:53,090
simply a binary sorting machine. You feed it

1007
00:44:53,090 --> 00:44:55,750
two words, and it outputs a probability between

1008
00:44:55,750 --> 00:44:58,730
0 and 1, attempting to answer a simple yes -no

1009
00:44:58,730 --> 00:45:01,789
question. Are these two words likely to be neighbors

1010
00:45:01,789 --> 00:45:04,670
in a real sentence? Okay, so how do we train

1011
00:45:04,670 --> 00:45:07,469
this sorting machine? We use the corpus. We take

1012
00:45:07,469 --> 00:45:09,829
a target word, say apricot, we look at the actual

1013
00:45:09,829 --> 00:45:12,190
sentence it appears in, and we grab a real neighboring

1014
00:45:12,190 --> 00:45:15,090
word, say jam. We feed apricot and jam into the

1015
00:45:15,090 --> 00:45:17,869
classifier, and we mathematically force its weights

1016
00:45:17,869 --> 00:45:19,550
to adjust so it outputs a higher probability.

1017
00:45:19,929 --> 00:45:22,369
We tell it, yes, these belong together. That

1018
00:45:22,369 --> 00:45:24,590
makes sense. It's learning true neighbors. But

1019
00:45:24,590 --> 00:45:26,429
to prevent the classifier from just guessing

1020
00:45:26,429 --> 00:45:28,949
yes all the time, we introduce negative sampling.

1021
00:45:29,230 --> 00:45:31,469
We take the target word apricot, but instead

1022
00:45:31,469 --> 00:45:33,449
of a real neighbor, we reach into the dictionary

1023
00:45:33,449 --> 00:45:36,630
and grab a completely random word, aardvark.

1024
00:45:37,090 --> 00:45:39,869
We feed apricot and aardvark into the classifier

1025
00:45:39,869 --> 00:45:42,510
and we force its weights to output a low probability.

1026
00:45:42,789 --> 00:45:45,449
We tell it, no, these do not belong together.

1027
00:45:45,730 --> 00:45:48,690
So the machine is just endlessly practicing this

1028
00:45:48,690 --> 00:45:52,659
tiny specific task millions and millions of times.

1029
00:45:53,280 --> 00:45:56,500
Is apricot in your jam? Yes. Is apricot in your

1030
00:45:56,500 --> 00:46:00,449
air fork? No. It slowly tweaks its internal numbers

1031
00:46:00,449 --> 00:46:02,650
to get better and better at this game. And here's

1032
00:46:02,650 --> 00:46:04,989
the magic trick. Once the classifier becomes

1033
00:46:04,989 --> 00:46:07,530
highly accurate at distinguishing true neighbors

1034
00:46:07,530 --> 00:46:11,170
from random noise, we completely throw the classifier

1035
00:46:11,170 --> 00:46:13,309
away. We delete the sorting machine. We never

1036
00:46:13,309 --> 00:46:15,170
actually cared about predicting neighbors. We

1037
00:46:15,170 --> 00:46:17,050
only cared about the internal weights the network

1038
00:46:17,050 --> 00:46:19,789
developed to win the game. To successfully predict

1039
00:46:19,789 --> 00:46:22,110
that apricot and jam go together, the network

1040
00:46:22,110 --> 00:46:23,889
had to compress the distributional properties

1041
00:46:23,889 --> 00:46:27,570
of those words into 300 continuous numbers. Those

1042
00:46:27,570 --> 00:46:30,809
300 numbers are the dense embedding. The network

1043
00:46:30,809 --> 00:46:32,769
inadvertently learned semantics by trying to

1044
00:46:32,769 --> 00:46:35,170
win a proximity guessing game. And the resulting

1045
00:46:35,170 --> 00:46:37,989
geometry is so robust that it captures abstract

1046
00:46:37,989 --> 00:46:40,909
relational concepts. This leads to the most famous

1047
00:46:40,909 --> 00:46:44,429
analogy in NLP, the parallelogram. Ah, the classic

1048
00:46:44,429 --> 00:46:47,150
king and queen example. It is so cool. If you

1049
00:46:47,150 --> 00:46:49,190
take the 300 -dimensional vector for the word

1050
00:46:49,190 --> 00:46:51,969
king, and you mathematically subtract the vector

1051
00:46:51,969 --> 00:46:54,289
for man, and then you add the vector for woman,

1052
00:46:54,840 --> 00:46:57,260
The resulting coordinate in that 300 -dimensional

1053
00:46:57,260 --> 00:47:00,619
space is incredibly close to the vector for queen.

1054
00:47:00,800 --> 00:47:02,900
It's like the vector space is a physical map

1055
00:47:02,900 --> 00:47:05,400
of a city, and the concept of gender is just

1056
00:47:05,400 --> 00:47:08,460
a specific street heading perfectly north. Royalty

1057
00:47:08,460 --> 00:47:11,199
is a street heading east. You can navigate concepts

1058
00:47:11,199 --> 00:47:14,159
like a cab driver. It is a stunning conceptual

1059
00:47:14,159 --> 00:47:17,079
visualization. However, I must interject with

1060
00:47:17,079 --> 00:47:19,559
the textbook's reality check on the parallelogram.

1061
00:47:20,090 --> 00:47:22,750
It is heavily cited in pop science, but it is

1062
00:47:22,750 --> 00:47:25,550
deeply flawed in practice. Wait, really? It doesn't

1063
00:47:25,550 --> 00:47:27,889
work? Often, if you actually run that specific

1064
00:47:27,889 --> 00:47:30,530
vector arithmetic, the algorithm just returns

1065
00:47:30,530 --> 00:47:33,110
one of the input words, like king, or perhaps

1066
00:47:33,110 --> 00:47:35,889
a pluralized version. The geometry is not always

1067
00:47:35,889 --> 00:47:38,409
that perfectly delineated. Researchers have demonstrated

1068
00:47:38,409 --> 00:47:41,070
that this simple additive analogy task fails

1069
00:47:41,070 --> 00:47:43,750
on many complex relational categories, and it

1070
00:47:43,750 --> 00:47:46,349
is likely far too simplistic to accurately model

1071
00:47:46,349 --> 00:47:49,559
human cognitive analogies. Oh, man. It's a little

1072
00:47:49,559 --> 00:47:52,000
heartbreaking. But do not lose the forest for

1073
00:47:52,000 --> 00:47:55,500
the trees. While the parallelogram might be exaggerated,

1074
00:47:55,880 --> 00:47:58,599
the fundamental underlying truth that dense,

1075
00:47:58,699 --> 00:48:01,679
continuous geometry can capture rich semantic

1076
00:48:01,679 --> 00:48:05,099
relationships completely revolutionized natural

1077
00:48:05,099 --> 00:48:08,179
language processing. Embeddings are the fuel

1078
00:48:08,179 --> 00:48:10,539
for everything that follows. Because once we

1079
00:48:10,539 --> 00:48:12,719
have these dense, meaningful vectors, we can

1080
00:48:12,719 --> 00:48:15,239
finally feed them into neural networks that process

1081
00:48:15,239 --> 00:48:17,860
whole sequences over time. We are crossing the

1082
00:48:17,860 --> 00:48:20,800
threshold into chapters 7 and 9. Deep learning

1083
00:48:20,800 --> 00:48:24,199
for sequences. The first major step past n -grams

1084
00:48:24,199 --> 00:48:26,039
was the feed -forward neural language model.

1085
00:48:26,400 --> 00:48:28,539
Instead of looking at exact strings of words,

1086
00:48:28,739 --> 00:48:31,619
it uses a sliding window. It takes the embeddings

1087
00:48:31,619 --> 00:48:34,659
of the last, say, three words, concatenates them,

1088
00:48:34,760 --> 00:48:36,780
passes them through a hidden neural layer, and

1089
00:48:36,780 --> 00:48:38,739
attempts to predict the embedding of the next

1090
00:48:38,739 --> 00:48:41,400
word. And this instantly solves the sparsity

1091
00:48:41,400 --> 00:48:44,260
problem that plagued engrams. Exactly. How? Because

1092
00:48:44,260 --> 00:48:46,500
it operates on geometry, not specific spellings.

1093
00:48:46,699 --> 00:48:48,820
If the network is trained on the sentence the

1094
00:48:48,820 --> 00:48:51,940
dog gets fed, it adjusts its weights. Later...

1095
00:48:52,139 --> 00:48:54,739
It encounters the incomplete phrase the cat gets.

1096
00:48:54,820 --> 00:48:57,139
An engram model would freeze if it had never

1097
00:48:57,139 --> 00:48:59,440
seen that exact phrase. But the neural network

1098
00:48:59,440 --> 00:49:03,400
looks at the vector for cat, realizes it is physically

1099
00:49:03,400 --> 00:49:06,260
sitting right next to the vector for dog in embedding

1100
00:49:06,260 --> 00:49:08,440
space, and says, oh, these are practically the

1101
00:49:08,440 --> 00:49:12,059
same thing. I bet the next word is fed. It generalizes

1102
00:49:12,059 --> 00:49:14,840
to unseen data. It is a massive leap forward.

1103
00:49:14,980 --> 00:49:17,619
However, the feedforward network still suffers

1104
00:49:17,619 --> 00:49:20,519
from a critical architectural flaw. The sliding

1105
00:49:20,519 --> 00:49:22,880
window is a fixed size. It is still essentially

1106
00:49:22,880 --> 00:49:25,420
making the Markov assumption. It can only see

1107
00:49:25,420 --> 00:49:27,579
a few words into the past. Which is a huge problem

1108
00:49:27,579 --> 00:49:29,719
because human language has long -distance dependencies.

1109
00:49:30,440 --> 00:49:32,840
If I say, the keys that I left on the kitchen

1110
00:49:32,840 --> 00:49:34,719
counter next to the overflowing fruit bowl are

1111
00:49:34,719 --> 00:49:37,519
missing, the verb are is plural because it relies

1112
00:49:37,519 --> 00:49:39,440
on the word keys, which is like 14 words away.

1113
00:49:39,619 --> 00:49:42,360
A fixed window of five words would only see the

1114
00:49:42,360 --> 00:49:44,599
overflowing fruit bowl, and you might incorrectly

1115
00:49:44,599 --> 00:49:47,679
guess the singular verb is. To solve this, researchers

1116
00:49:47,679 --> 00:49:50,519
developed recurrent neural networks, or RNNs.

1117
00:49:50,539 --> 00:49:54,599
An RNN abandons the fixed window. Instead, it

1118
00:49:54,599 --> 00:49:57,039
maintains a hidden state, a vector of numbers

1119
00:49:57,039 --> 00:50:00,019
that acts as a continuous memory. It processes

1120
00:50:00,019 --> 00:50:02,860
the sentence sequentially. I try to picture this

1121
00:50:02,860 --> 00:50:05,800
as a game of telephone. The network looks at

1122
00:50:05,800 --> 00:50:08,840
the first word, processes its embedding, and

1123
00:50:08,840 --> 00:50:10,519
writes a little summary in its hidden state.

1124
00:50:10,840 --> 00:50:13,039
Then it moves to the second word, mixes that

1125
00:50:13,039 --> 00:50:14,820
new word with the summary of the first word,

1126
00:50:14,900 --> 00:50:17,820
and generates a new combined summary. It passes

1127
00:50:17,820 --> 00:50:20,019
this evolving hidden state forward through time,

1128
00:50:20,099 --> 00:50:21,980
step by step, until the end of the sentence.

1129
00:50:22,139 --> 00:50:25,159
A very functional visualization. And to train

1130
00:50:25,159 --> 00:50:27,219
these temporal networks, we use a technique called

1131
00:50:27,219 --> 00:50:29,920
teacher forcing. How does that work? At time

1132
00:50:29,920 --> 00:50:32,239
step one, the network tries to guess time step

1133
00:50:32,239 --> 00:50:35,429
two. We calculate its error using a metric called

1134
00:50:35,429 --> 00:50:37,829
cross -entropy loss, which essentially measures

1135
00:50:37,829 --> 00:50:40,590
how far its predicted probability distribution

1136
00:50:40,590 --> 00:50:43,929
is from the actual truth. Now, normally, a network

1137
00:50:43,929 --> 00:50:46,250
might feed its own prediction into the next step.

1138
00:50:46,389 --> 00:50:49,289
But if its prediction was wrong, that error compounds

1139
00:50:49,289 --> 00:50:51,630
and the network spirals into hallucinations.

1140
00:50:51,809 --> 00:50:55,030
So instead, we force the true correct word from

1141
00:50:55,030 --> 00:50:57,469
the training corpus into the next step. We keep

1142
00:50:57,469 --> 00:50:59,789
it on the rails while it learns. But even with

1143
00:50:59,789 --> 00:51:02,250
teacher forcing, the telephone game analogy reveals

1144
00:51:02,250 --> 00:51:05,710
the fatal weakness of a basic RNN. If the sentence

1145
00:51:05,710 --> 00:51:08,150
is 50 words long, by the time the hidden state

1146
00:51:08,150 --> 00:51:10,769
reaches the final word, it has been mixed and

1147
00:51:10,769 --> 00:51:13,610
remixed 50 times. The original signal from the

1148
00:51:13,610 --> 00:51:15,889
first word is completely diluted. The network

1149
00:51:15,889 --> 00:51:18,150
just forgets the beginning of the sentence. Mathematically,

1150
00:51:18,309 --> 00:51:20,630
this manifests as the vanishing gradient problem.

1151
00:51:20,909 --> 00:51:23,210
Walk me through that. What is vanishing? When

1152
00:51:23,210 --> 00:51:25,010
a neural network makes a mistake at the end of

1153
00:51:25,010 --> 00:51:27,570
a sentence, it has to send an error signal backwards

1154
00:51:27,570 --> 00:51:29,670
through time to update the weights of the earlier

1155
00:51:29,670 --> 00:51:32,789
steps. This process involves multiplying gradients,

1156
00:51:32,809 --> 00:51:34,389
which are essentially fractions together over

1157
00:51:34,389 --> 00:51:37,730
and over. If you multiply 0 .5 by 0 .5, you get

1158
00:51:37,730 --> 00:51:41,690
0 .25. Multiply it again, 0 .125. Over 50 time

1159
00:51:41,690 --> 00:51:44,389
steps, that error signal becomes so microscopically

1160
00:51:44,389 --> 00:51:46,980
small that it vanishes. The network literally

1161
00:51:46,980 --> 00:51:49,380
cannot send a learning signal across long temporal

1162
00:51:49,380 --> 00:51:52,079
distances. It cannot learn long -term dependencies.

1163
00:51:52,539 --> 00:51:54,900
So to fix the telephone game, engineers added

1164
00:51:54,900 --> 00:51:58,179
gates. We get LSTMs, which stands for long short

1165
00:51:58,179 --> 00:52:01,420
-term memory networks, and GRUs, gated recurrent

1166
00:52:01,420 --> 00:52:04,820
units. An LSTM is essentially an RNN with a highly

1167
00:52:04,820 --> 00:52:07,400
sophisticated control mechanism. Instead of just

1168
00:52:07,400 --> 00:52:09,599
blindly mixing the new word with the old memory,

1169
00:52:09,820 --> 00:52:12,019
it has specific neural gates that learn what

1170
00:52:12,019 --> 00:52:14,400
to keep and what to throw away. It's like the

1171
00:52:14,400 --> 00:52:16,920
person playing the telephone game, has a notepad

1172
00:52:16,920 --> 00:52:19,900
and an eraser. The forget gate decides if the

1173
00:52:19,900 --> 00:52:22,719
old context is obsolete. The input gate decides

1174
00:52:22,719 --> 00:52:24,460
if the new word is important enough to write

1175
00:52:24,460 --> 00:52:26,420
down, so it can choose to write down the plural

1176
00:52:26,420 --> 00:52:29,420
subject keys, stick it in its pocket, completely

1177
00:52:29,420 --> 00:52:32,219
ignore the 14 words about the fruit bowl, and

1178
00:52:32,219 --> 00:52:34,219
pull keys back out when it reaches the verb.

1179
00:52:35,219 --> 00:52:38,460
LSTMs and GRUs dominated natural language processing

1180
00:52:38,460 --> 00:52:41,849
for years. They were the state of the art. But

1181
00:52:41,849 --> 00:52:44,469
they still possessed a fundamental bottleneck.

1182
00:52:44,530 --> 00:52:47,250
They were rigidly sequential. You physically

1183
00:52:47,250 --> 00:52:49,909
could not process Word 50 until you had finished

1184
00:52:49,909 --> 00:52:52,989
processing Word 49. You couldn't parallelize

1185
00:52:52,989 --> 00:52:55,530
the training on modern GPUs efficiently. Which

1186
00:52:55,530 --> 00:52:58,110
sets the stage for the absolute monumental breakthrough

1187
00:52:58,110 --> 00:53:00,289
of Chapter 9, the architecture that currently

1188
00:53:00,289 --> 00:53:03,889
powers ChatGPT, Claude, BERT, everything. The

1189
00:53:03,889 --> 00:53:06,420
Transformer. The transformer abandons the sequential

1190
00:53:06,420 --> 00:53:08,599
telephone game entirely. It does not process

1191
00:53:08,599 --> 00:53:11,039
language left to right. It uses a mechanism called

1192
00:53:11,039 --> 00:53:13,539
self -attention. Break down self -attention for

1193
00:53:13,539 --> 00:53:15,400
me, because this really feels like the core of

1194
00:53:15,400 --> 00:53:18,559
modern AI. In a transformer, every single word

1195
00:53:18,559 --> 00:53:21,139
in the input sequence looks at every other word

1196
00:53:21,139 --> 00:53:23,820
in the sequence simultaneously. It computes a

1197
00:53:23,820 --> 00:53:26,099
mathematical compatibility score between itself

1198
00:53:26,099 --> 00:53:28,239
and all its neighbors, regardless of distance.

1199
00:53:28,559 --> 00:53:30,780
So if the network is processing the word bank,

1200
00:53:31,059 --> 00:53:34,059
it instantly casts a net across the entire sentence.

1201
00:53:34,590 --> 00:53:37,329
If it sees the word river 12 words away, the

1202
00:53:37,329 --> 00:53:39,489
attention mechanism calculates a high score between

1203
00:53:39,489 --> 00:53:42,449
them, and the network dynamically updates the

1204
00:53:42,449 --> 00:53:44,610
vector for bank to incorporate the river context.

1205
00:53:45,289 --> 00:53:48,010
It instantly knows we are talking about a muddy

1206
00:53:48,010 --> 00:53:51,010
bank, not a financial institution. Exactly. It

1207
00:53:51,010 --> 00:53:54,130
creates a web of relationships. I like your previous

1208
00:53:54,130 --> 00:53:56,929
analogy of a crowded room. An RNN is a bucket

1209
00:53:56,929 --> 00:53:59,610
brigade passing a whisper down a line. A transformer

1210
00:53:59,610 --> 00:54:01,670
is standing in the center of a crowded room where

1211
00:54:01,670 --> 00:54:03,829
everyone is shouting at once. But through the

1212
00:54:03,829 --> 00:54:06,050
mathematical weights of self -attention, the

1213
00:54:06,050 --> 00:54:08,510
network instinctively knows exactly whose voice

1214
00:54:08,510 --> 00:54:11,190
to tune into to get the necessary context. And

1215
00:54:11,190 --> 00:54:14,269
it doesn't just do this once. It uses multi -hit

1216
00:54:14,269 --> 00:54:16,630
attention. It's doing this dozens of times in

1217
00:54:16,630 --> 00:54:19,130
parallel. One attention head might be specifically

1218
00:54:19,130 --> 00:54:21,110
trained to look for subject -verb relationships.

1219
00:54:21,630 --> 00:54:24,030
Another head might be looking for semantic clusters.

1220
00:54:24,449 --> 00:54:26,730
It parses all these different dimensions of the

1221
00:54:26,730 --> 00:54:29,300
sentence simultaneously. The shift from Markov

1222
00:54:29,300 --> 00:54:32,840
chains to LSTMs to transformers is profound.

1223
00:54:33,300 --> 00:54:35,820
It isn't merely an engineering trick to speed

1224
00:54:35,820 --> 00:54:38,840
up training. It represents a fundamental leap

1225
00:54:38,840 --> 00:54:41,820
in capturing the non -linear hierarchical web

1226
00:54:41,820 --> 00:54:44,739
of human thought. A sentence is not a ticker

1227
00:54:44,739 --> 00:54:47,880
tape of isolated events. It is a complex structure

1228
00:54:47,880 --> 00:54:50,380
of dependencies. The transformer architecture

1229
00:54:50,380 --> 00:54:53,750
naturally maps to that reality. perfectly sets

1230
00:54:53,750 --> 00:54:56,190
up the ultimate test. We have this architecture

1231
00:54:56,190 --> 00:54:58,690
capable of holding a massive web of context in

1232
00:54:58,690 --> 00:55:00,849
its head all at once. What happens if we force

1233
00:55:00,849 --> 00:55:03,030
it to spit that context out in an entirely different

1234
00:55:03,030 --> 00:55:05,690
language? We are tackling chapter 11, machine

1235
00:55:05,690 --> 00:55:07,710
translation. Translation is perhaps the most

1236
00:55:07,710 --> 00:55:10,570
difficult task in NLP because of language divergences.

1237
00:55:11,070 --> 00:55:13,369
Typology, the linguistic study of how languages

1238
00:55:13,369 --> 00:55:16,010
differ structurally, outlines the massive hurdles.

1239
00:55:16,230 --> 00:55:18,050
Give me an example of a structural hurdle. The

1240
00:55:18,050 --> 00:55:20,570
most common is word order. English is predominantly

1241
00:55:20,570 --> 00:55:24,800
a subject -verb -object language. or SVO, the

1242
00:55:24,800 --> 00:55:28,199
cat ate the mouse. Japanese, however, is SOV,

1243
00:55:28,280 --> 00:55:31,699
subject -object verb, the cat, the mouse ate.

1244
00:55:31,980 --> 00:55:34,739
So you literally cannot do a word -for -word

1245
00:55:34,739 --> 00:55:36,920
translation. The machine has to read the whole

1246
00:55:36,920 --> 00:55:39,099
English sentence, hold the noun mouse in its

1247
00:55:39,099 --> 00:55:41,579
memory, wait for the verb ate, and then flip

1248
00:55:41,579 --> 00:55:43,320
their positions when generating the Japanese.

1249
00:55:43,679 --> 00:55:45,880
And that is a simple divergence. The textbook

1250
00:55:45,880 --> 00:55:48,239
also discusses pro -drop languages like Spanish

1251
00:55:48,239 --> 00:55:51,559
or Italian. In English, you must explicitly state

1252
00:55:51,559 --> 00:55:54,369
the pronoun. The green witch arrived, she said.

1253
00:55:54,570 --> 00:55:58,150
But in Spanish, you might say, And the next sentence

1254
00:55:58,150 --> 00:55:59,989
might just be the verb dropping the she entirely

1255
00:55:59,989 --> 00:56:02,389
because the verb conjugation implies the subject.

1256
00:56:02,670 --> 00:56:04,710
But if the machine is translating that Spanish

1257
00:56:04,710 --> 00:56:07,550
text back into English, it has to magically generate

1258
00:56:07,550 --> 00:56:10,010
the word she out of thin air. It has to look

1259
00:56:10,010 --> 00:56:12,750
backward, realize a female witch was the subject

1260
00:56:12,750 --> 00:56:15,150
of the previous sentence, and synthesize the

1261
00:56:15,150 --> 00:56:17,409
correct English pronoun. To solve these immense

1262
00:56:17,409 --> 00:56:20,179
structural puzzles, We use the encoder -decoder

1263
00:56:20,179 --> 00:56:22,639
architecture, also known as sequence -to -sequence

1264
00:56:22,639 --> 00:56:24,980
models. How does the division of labor work there?

1265
00:56:25,179 --> 00:56:27,480
It is two separate networks working in tandem.

1266
00:56:27,760 --> 00:56:30,460
The encoder network reads the entire source sentence,

1267
00:56:30,500 --> 00:56:33,769
say, the Spanish text. It uses self -attention

1268
00:56:33,769 --> 00:56:36,670
to process all the complex relationships, and

1269
00:56:36,670 --> 00:56:39,090
it compresses all of that semantic and syntactic

1270
00:56:39,090 --> 00:56:41,590
information into a dense mathematical representation.

1271
00:56:42,150 --> 00:56:45,750
It creates a pure concept vector. It strips away

1272
00:56:45,750 --> 00:56:47,690
the Spanish and just leaves the meaning. Essentially.

1273
00:56:47,769 --> 00:56:50,809
Then it hands that concept over to the decoder

1274
00:56:50,809 --> 00:56:53,679
network. The decoder's sole job is to take that

1275
00:56:53,679 --> 00:56:56,840
pure concept and begin generating the target

1276
00:56:56,840 --> 00:56:59,960
language, say English word by word, conditioned

1277
00:56:59,960 --> 00:57:03,139
on that context. But the textbook notes a flaw

1278
00:57:03,139 --> 00:57:05,800
with early models. If you just hand the decoder

1279
00:57:05,800 --> 00:57:08,099
one single summary vector at the very beginning

1280
00:57:08,099 --> 00:57:10,960
and ask it to generate a 30 -word English sentence,

1281
00:57:11,219 --> 00:57:13,639
it forgets the details of the Spanish sentence

1282
00:57:13,639 --> 00:57:16,139
halfway through. It's the bottleneck problem

1283
00:57:16,139 --> 00:57:18,280
again. Which is why we introduced the attention

1284
00:57:18,280 --> 00:57:20,719
mechanism specifically into the translation process.

1285
00:57:21,179 --> 00:57:23,860
The decoder does not rely on a single static

1286
00:57:23,860 --> 00:57:26,639
summary. At every single step of generating the

1287
00:57:26,639 --> 00:57:28,760
English sentence, the decoder gets to look back

1288
00:57:28,760 --> 00:57:30,900
at all the hidden states of the Spanish encoder.

1289
00:57:31,019 --> 00:57:33,300
It's like it had the original document open on

1290
00:57:33,300 --> 00:57:35,960
the desk next to it. While it is trying to figure

1291
00:57:35,960 --> 00:57:38,420
out the fifth English word, it asks the encoder,

1292
00:57:38,519 --> 00:57:41,099
hey, which of the Spanish words are most heavily

1293
00:57:41,099 --> 00:57:44,239
weighted for what I'm doing right now? It dynamically

1294
00:57:44,239 --> 00:57:47,219
aligns the two languages. Yes, soft alignment.

1295
00:57:47,639 --> 00:57:50,440
But once the decoder knows what it wants to say,

1296
00:57:50,659 --> 00:57:53,579
how does it actually choose the words? The most

1297
00:57:53,579 --> 00:57:56,420
straightforward method is greedy decoding. At

1298
00:57:56,420 --> 00:57:58,679
each time step, the model looks at its probability

1299
00:57:58,679 --> 00:58:01,519
distribution and simply picks the single most

1300
00:58:01,519 --> 00:58:04,619
likely next word. That sounds logical, but the

1301
00:58:04,619 --> 00:58:07,400
text says it's a trap. Why? Because of dead ends.

1302
00:58:07,639 --> 00:58:10,539
Imagine navigating a maze by always taking the

1303
00:58:10,539 --> 00:58:13,219
widest hallway at every intersection. It seems

1304
00:58:13,219 --> 00:58:15,139
like a good strategy locally, but it might lead

1305
00:58:15,139 --> 00:58:17,340
you straight into a brick wall. If the model

1306
00:58:17,340 --> 00:58:19,619
makes a slightly suboptimal word choice early

1307
00:58:19,619 --> 00:58:21,579
in the sentence, it cannot go back and fix it.

1308
00:58:21,739 --> 00:58:23,960
That early error might force the rest of the

1309
00:58:23,960 --> 00:58:25,920
sentence into a grammatical corner that ruins

1310
00:58:25,920 --> 00:58:28,099
the translation. So instead of being greedy,

1311
00:58:28,280 --> 00:58:31,260
we use beam search. Beam search is like exploring

1312
00:58:31,260 --> 00:58:33,739
the multiverse. Instead of committing to one

1313
00:58:33,739 --> 00:58:36,219
single path, the computer keeps co -possible

1314
00:58:36,219 --> 00:58:39,300
paths open simultaneously. Let's trace it. Say

1315
00:58:39,300 --> 00:58:42,829
your beam width, k, is 3. At the first step,

1316
00:58:42,929 --> 00:58:45,630
the decoder doesn't just pick the top word. It

1317
00:58:45,630 --> 00:58:47,690
generates the top three most likely starting

1318
00:58:47,690 --> 00:58:50,690
words. Path A, Path B, Path C. Then, for each

1319
00:58:50,690 --> 00:58:53,170
of those three paths, it looks ahead to the second

1320
00:58:53,170 --> 00:58:55,510
word and generates the top three options. You

1321
00:58:55,510 --> 00:58:58,489
now have nine branching paths. It calculates

1322
00:58:58,489 --> 00:59:01,130
the total probability for all nine paths. It

1323
00:59:01,130 --> 00:59:03,289
throws away the worst six and keeps the top three

1324
00:59:03,289 --> 00:59:06,269
survivors going. It is constantly pruning the

1325
00:59:06,269 --> 00:59:08,769
multiverse tree, ensuring it finds a sequence

1326
00:59:08,769 --> 00:59:12,000
that is globally optimal, not just locally optimal.

1327
00:59:12,199 --> 00:59:14,199
It is a profoundly effective search strategy.

1328
00:59:14,400 --> 00:59:17,000
However, the true bottleneck in modern machine

1329
00:59:17,000 --> 00:59:19,340
translation isn't the architecture, it's the

1330
00:59:19,340 --> 00:59:22,179
data. To train these massive encoder -decoder

1331
00:59:22,179 --> 00:59:25,320
models, you need a bitext, a parallel corpus

1332
00:59:25,320 --> 00:59:28,139
containing millions of sentences flawlessly translated

1333
00:59:28,139 --> 00:59:30,659
by human experts in both languages. Which is

1334
00:59:30,659 --> 00:59:32,500
easy to find for English and French. The Canadian

1335
00:59:32,500 --> 00:59:34,300
parliament has been translating everything for

1336
00:59:34,300 --> 00:59:36,380
decades. But if you want to translate English

1337
00:59:36,380 --> 00:59:38,869
to a low resource language, you don't have millions

1338
00:59:38,869 --> 00:59:41,409
of human translated sentences. This scarcity

1339
00:59:41,409 --> 00:59:43,610
led to one of the most ingenious hats detailed

1340
00:59:43,610 --> 00:59:46,389
in the textbook, back translation. I read this

1341
00:59:46,389 --> 00:59:49,130
and thought it was a joke at first. Okay, I want

1342
00:59:49,130 --> 00:59:51,090
to build an incredible English or French translator.

1343
00:59:51,690 --> 00:59:54,210
But I am totally out of human -translated English

1344
00:59:54,210 --> 00:59:56,769
-French data. You are telling me I can take a

1345
00:59:56,769 --> 00:59:59,809
massive pile of monolingual French text, just

1346
00:59:59,809 --> 01:00:02,650
random French Wikipedia articles, and I run them

1347
01:00:02,650 --> 01:00:04,769
backwards through a terrible, mediocre French

1348
01:00:04,769 --> 01:00:06,630
-to -English translator. Or re -reverse model,

1349
01:00:06,809 --> 01:00:08,949
yes. And it generates completely garbage English.

1350
01:00:09,190 --> 01:00:11,750
Like, the apple of the earth is good. Then I

1351
01:00:11,750 --> 01:00:14,289
pair that garbage, synthetic English with a perfect

1352
01:00:14,289 --> 01:00:17,389
French Wikipedia text, and I train my real...

1353
01:00:17,659 --> 01:00:20,239
forward -facing English to French model on that

1354
01:00:20,239 --> 01:00:22,760
synthetic pairing. How does training on garbage

1355
01:00:22,760 --> 01:00:25,059
data make the model better? It seems entirely

1356
01:00:25,059 --> 01:00:27,659
counterintuitive, but consider what the model

1357
01:00:27,659 --> 01:00:30,340
is actually learning during training. The encoder

1358
01:00:30,340 --> 01:00:32,760
processes the garbage English. It struggles,

1359
01:00:32,820 --> 01:00:35,679
but it extracts a fuzzy concept. But the decoder,

1360
01:00:35,739 --> 01:00:37,780
the part of the network generating the final

1361
01:00:37,780 --> 01:00:40,280
French output, is being trained to match the

1362
01:00:40,280 --> 01:00:43,059
perfect French Wikipedia text. Back translation

1363
01:00:43,059 --> 01:00:45,679
acts as a massive language modeling exercise

1364
01:00:45,679 --> 01:00:48,889
for the decoder side. It forces the decoder to

1365
01:00:48,889 --> 01:00:51,750
practice generating exceptionally fluent, grammatically

1366
01:00:51,750 --> 01:00:54,289
perfect target language, even when the input

1367
01:00:54,289 --> 01:00:57,250
is noisy. Fluency on the target side dramatically

1368
01:00:57,250 --> 01:00:59,590
improves the human perception of the translation

1369
01:00:59,590 --> 01:01:03,210
quality. That is deviously clever. But speaking

1370
01:01:03,210 --> 01:01:05,829
of human perception, how do we grade these machines?

1371
01:01:06,070 --> 01:01:08,590
We can't have humans sit and grade a million

1372
01:01:08,590 --> 01:01:11,800
translated sentences. Historically, the industry

1373
01:01:11,800 --> 01:01:15,500
standard was a metric called BLEU. BLEU is essentially

1374
01:01:15,500 --> 01:01:18,760
an n -gram overlap counter. It compares the machine's

1375
01:01:18,760 --> 01:01:21,239
generated translation to a human reference translation,

1376
01:01:21,559 --> 01:01:23,940
and it counts how many exact word sequences match.

1377
01:01:24,469 --> 01:01:26,369
But that has a glaring flaw, doesn't it? If the

1378
01:01:26,369 --> 01:01:28,969
human translator wrote the fast car arrived and

1379
01:01:28,969 --> 01:01:31,630
the machine translated it as the quick car arrived,

1380
01:01:31,809 --> 01:01:34,929
BLEU penalizes the machine because the string

1381
01:01:34,929 --> 01:01:36,989
quick does not exactly match the string fast.

1382
01:01:37,269 --> 01:01:39,309
It treats them as failures, even though the semantic

1383
01:01:39,309 --> 01:01:42,130
translation is perfect. To rectify this, modern

1384
01:01:42,130 --> 01:01:44,289
evaluation utilizes metrics like BERT score.

1385
01:01:44,550 --> 01:01:47,519
Which relies on embeddings. Exactly. BERT score

1386
01:01:47,519 --> 01:01:50,659
uses dense contextual embeddings. It calculates

1387
01:01:50,659 --> 01:01:53,039
the cosine similarity between the vector of the

1388
01:01:53,039 --> 01:01:55,320
machine's word and the vector of the human's

1389
01:01:55,320 --> 01:01:58,480
word. In vector space, quick and fast have a

1390
01:01:58,480 --> 01:02:01,320
very high similarity score. BERT score provides

1391
01:02:01,320 --> 01:02:04,000
a soft match, granting the machine credit for

1392
01:02:04,000 --> 01:02:06,239
capturing the true meaning regardless of the

1393
01:02:06,239 --> 01:02:09,179
explicit spelling. This entire evolution in translation

1394
01:02:09,179 --> 01:02:12,519
is staggering. The textbook outlines how, historically,

1395
01:02:12,880 --> 01:02:15,539
linguists used the Volquois triangle. You had

1396
01:02:15,539 --> 01:02:18,000
to parse the source text, write explicit transfer

1397
01:02:18,000 --> 01:02:20,320
rules to map the syntax to the target language,

1398
01:02:20,460 --> 01:02:23,860
and then generate words. It was hundreds of thousands

1399
01:02:23,860 --> 01:02:26,420
of lines of hand -coded grammatical logic. And

1400
01:02:26,420 --> 01:02:28,960
neural networks bypass almost all of it. They

1401
01:02:28,960 --> 01:02:31,440
learn to translate purely from statistical associations

1402
01:02:31,440 --> 01:02:34,219
in parallel corpora, mapping strings to vectors

1403
01:02:34,219 --> 01:02:52,599
and back to strings. But that raises a ma - When

1404
01:02:52,599 --> 01:02:55,139
linguists attempt to formally model the syntax

1405
01:02:55,139 --> 01:02:57,679
of human language, they predominantly rely on

1406
01:02:57,679 --> 01:03:00,340
two distinct frameworks. The first, covered in

1407
01:03:00,340 --> 01:03:03,019
Chapter 12, is context -free grammars, or CFGs.

1408
01:03:03,280 --> 01:03:06,219
They are also known as phrase structure grammars.

1409
01:03:06,570 --> 01:03:09,730
How does a CFG model language? By relying on

1410
01:03:09,730 --> 01:03:13,230
abstract hierarchical rules. A CFG assumes that

1411
01:03:13,230 --> 01:03:16,030
words naturally group into larger abstract structural

1412
01:03:16,030 --> 01:03:19,250
units called constituents. For example, a core

1413
01:03:19,250 --> 01:03:21,150
rule might state that a sentence consists of

1414
01:03:21,150 --> 01:03:23,909
a noun phrase followed by a verb phrase. Another

1415
01:03:23,909 --> 01:03:25,670
rule might state that a noun phrase consists

1416
01:03:25,670 --> 01:03:28,250
of a determiner followed by a noun. The textbook

1417
01:03:28,250 --> 01:03:31,030
uses the ISS domain as an ongoing example. It

1418
01:03:31,030 --> 01:03:33,349
was an early 1990s system for booking airline

1419
01:03:33,349 --> 01:03:36,510
flights. You would build CFG rules to parse a

1420
01:03:36,510 --> 01:03:38,750
user saying, I'd like to fly to Atlanta. Right.

1421
01:03:38,829 --> 01:03:40,969
A parser would analyze that sentence and build

1422
01:03:40,969 --> 01:03:43,429
an invisible hierarchical tree above the words.

1423
01:03:43,610 --> 01:03:46,050
It groups to Atlanta into a prepositional phrase.

1424
01:03:46,409 --> 01:03:48,670
It groups that phrase with the verb fly into

1425
01:03:48,670 --> 01:03:50,889
a massive verb phrase. It connects that to the

1426
01:03:50,889 --> 01:03:53,289
subject I to form the overarching sentence root.

1427
01:03:53,739 --> 01:03:55,760
It reminds me of the biological taxonomy system,

1428
01:03:55,900 --> 01:03:58,219
classifying animals. You have the kingdom, the

1429
01:03:58,219 --> 01:04:00,619
phylum, the class, the order, down to the genus

1430
01:04:00,619 --> 01:04:03,639
and species. You are placing specific instances,

1431
01:04:03,860 --> 01:04:06,960
the words, into increasingly abstract overarching

1432
01:04:06,960 --> 01:04:10,059
categories. CFGs are incredibly rigid, which

1433
01:04:10,059 --> 01:04:12,260
makes them highly structured and computationally

1434
01:04:12,260 --> 01:04:14,739
tractable. They are absolutely essential for

1435
01:04:14,739 --> 01:04:17,800
tasks like deep semantic parsing. If you are

1436
01:04:17,800 --> 01:04:19,639
building a system that takes a natural language

1437
01:04:19,639 --> 01:04:22,210
request like Book me a flight to Atlanta under

1438
01:04:22,210 --> 01:04:24,989
$300 and needs to convert it into a flawless

1439
01:04:24,989 --> 01:04:28,050
SQL database query to actually execute the transaction,

1440
01:04:28,389 --> 01:04:31,409
the rigid hierarchical tree of a CFG guarantees

1441
01:04:31,409 --> 01:04:34,170
the structure is mapped correctly. But they have

1442
01:04:34,170 --> 01:04:36,989
a weakness. A significant one. CFGs struggle

1443
01:04:36,989 --> 01:04:39,869
immensely with free word order languages. In

1444
01:04:39,869 --> 01:04:41,949
languages where the subject, verb, and object

1445
01:04:41,949 --> 01:04:44,190
can freely move around the sentence for emphasis,

1446
01:04:44,489 --> 01:04:47,329
trying to define strict structural rules for

1447
01:04:47,329 --> 01:04:50,530
image. Noun phrase followed by verb phrase becomes

1448
01:04:50,530 --> 01:04:53,269
a nightmare. The invisible trees get impossibly

1449
01:04:53,269 --> 01:04:55,750
tangled. Which brings us to chapter 14 and the

1450
01:04:55,750 --> 01:04:58,630
second major linguistic framework. Dependency

1451
01:04:58,630 --> 01:05:01,369
grammars. I found this visual so much more intuitive.

1452
01:05:01,650 --> 01:05:04,070
Instead of building an invisible abstract tree

1453
01:05:04,070 --> 01:05:06,789
of categories above the words, dependency parsing

1454
01:05:06,789 --> 01:05:09,989
just draws direct arrows between the actual words

1455
01:05:09,989 --> 01:05:12,329
physically on the page. It strips away the abstraction.

1456
01:05:12,940 --> 01:05:15,280
Dependency grammar argues that syntactic structure

1457
01:05:15,280 --> 01:05:18,280
is defined by binary, asymmetrical relations

1458
01:05:18,280 --> 01:05:21,039
between the words themselves. You identify a

1459
01:05:21,039 --> 01:05:23,000
head word, usually the main verb of the sentence,

1460
01:05:23,219 --> 01:05:25,260
and you draw directed lines to its dependence.

1461
01:05:25,699 --> 01:05:28,360
So in I'd Like to Fly to Atlanta, the word fly

1462
01:05:28,360 --> 01:05:31,039
is the head. It draws an arrow directly to I

1463
01:05:31,039 --> 01:05:34,079
and labels it the nominal subject. It draws an

1464
01:05:34,079 --> 01:05:35,900
arrow to Atlanta and labels it the destination.

1465
01:05:36,699 --> 01:05:39,730
There are no invisible noun phrases. Just words

1466
01:05:39,730 --> 01:05:41,889
axing on other words. I believe you had an analogy

1467
01:05:41,889 --> 01:05:45,050
for this as well. Yes. If CFGs are the abstract

1468
01:05:45,050 --> 01:05:47,809
biological classification of animals, dependency

1469
01:05:47,809 --> 01:05:50,190
grammars are a food web. You don't say a hawk

1470
01:05:50,190 --> 01:05:52,170
is an avian, which is a vertebrate. You just

1471
01:05:52,170 --> 01:05:54,630
draw an arrow that says the hawk eats the mouse.

1472
01:05:54,809 --> 01:05:57,610
It maps direct functional relationships. It shows

1473
01:05:57,610 --> 01:06:00,199
exactly who is doing what to whom. That is exactly

1474
01:06:00,199 --> 01:06:02,679
what it does. And machines parse these dependencies

1475
01:06:02,679 --> 01:06:05,219
using a few different algorithms. Transition

1476
01:06:05,219 --> 01:06:07,599
-based parsing processes the sentence left to

1477
01:06:07,599 --> 01:06:09,860
right, maintaining a stack of words and making

1478
01:06:09,860 --> 01:06:12,360
a rapid sequence of decisions. Do I link the

1479
01:06:12,360 --> 01:06:14,199
word I'm looking at now to the word I just put

1480
01:06:14,199 --> 01:06:16,440
in the stack? It is incredibly fast. And the

1481
01:06:16,440 --> 01:06:19,500
alternative? Graph -based parsing. It evaluates

1482
01:06:19,500 --> 01:06:21,920
the entire sentence at once, scores every possible

1483
01:06:21,920 --> 01:06:24,900
combination of dependency arrows, and uses algorithms

1484
01:06:24,900 --> 01:06:28,389
to search for the globally optimal tree. Regardless

1485
01:06:28,389 --> 01:06:30,769
of the method, dependency grammars are incredibly

1486
01:06:30,769 --> 01:06:33,829
robust for those free word order languages. Because

1487
01:06:33,829 --> 01:06:35,889
the arrows just stretch directly from word to

1488
01:06:35,889 --> 01:06:38,090
word, the structure remains clear no matter how

1489
01:06:38,090 --> 01:06:39,929
the words are physically scrambled in the sentence.

1490
01:06:40,230 --> 01:06:43,030
Okay, let's take a collective breath here. Because

1491
01:06:43,030 --> 01:06:44,809
the journey we just took through the first half

1492
01:06:44,809 --> 01:06:47,969
of Jarafsky and Martin is staggering. We started

1493
01:06:47,969 --> 01:06:52,530
in 1966 with Eliza. a machine blindly swapping

1494
01:06:52,530 --> 01:06:56,170
out words using rigid regex bouncer rules, completely

1495
01:06:56,170 --> 01:06:59,070
devoid of understanding. We moved from that to

1496
01:06:59,070 --> 01:07:01,530
defining the boundaries of a word through tokenization

1497
01:07:01,530 --> 01:07:04,670
and byte pair encoding. We learned to quantify

1498
01:07:04,670 --> 01:07:07,469
typos with the dynamic programming matrix of

1499
01:07:07,469 --> 01:07:10,369
minimum edit distance. We built n -gram probability

1500
01:07:10,369 --> 01:07:12,969
tables to guess the next word based on historical

1501
01:07:12,969 --> 01:07:15,769
counting, and we implemented Nezerne smoothing

1502
01:07:15,769 --> 01:07:17,650
to ensure the zeros didn't destroy our math.

1503
01:07:17,980 --> 01:07:20,460
Then we abandoned mere counting for geometry.

1504
01:07:20,780 --> 01:07:24,179
We mapped meaning into vector space with Word2Vec.

1505
01:07:24,280 --> 01:07:26,940
We upgraded our temporal processing from the

1506
01:07:26,940 --> 01:07:29,400
sliding windows of feedforward networks to the

1507
01:07:29,400 --> 01:07:33,280
gated memory of LSTMs. And finally to the transformers,

1508
01:07:33,340 --> 01:07:36,239
using self -attention to ingest entire context

1509
01:07:36,239 --> 01:07:38,980
simultaneously, allowing us to build encoder

1510
01:07:38,980 --> 01:07:41,559
-decoders that translate languages while we map

1511
01:07:41,559 --> 01:07:44,219
the fundamental rules of human syntax using CFG

1512
01:07:44,219 --> 01:07:46,940
trees and dependency arrows. It's a phenomenal...

1513
01:07:47,070 --> 01:07:49,630
awe -inspiring amount of engineering just to

1514
01:07:49,630 --> 01:07:51,530
get a rock to read a text message. It is the

1515
01:07:51,530 --> 01:07:53,989
culmination of decades of human ingenuity. But

1516
01:07:53,989 --> 01:07:57,130
as we close out part one of this deep dive, I

1517
01:07:57,130 --> 01:07:59,369
want to pose a provocative thought. It hit me

1518
01:07:59,369 --> 01:08:01,030
while we were discussing those final grammar

1519
01:08:01,030 --> 01:08:03,829
chapters. We spent all that time looking at CFGs

1520
01:08:03,829 --> 01:08:05,730
and dependency trees, the human linguistic way

1521
01:08:05,730 --> 01:08:08,969
of mapping syntax. We spent centuries going back

1522
01:08:08,969 --> 01:08:11,510
to Panini in ancient India, writing down these

1523
01:08:11,510 --> 01:08:14,849
formal rules of grammar, but... Modern transformers,

1524
01:08:14,849 --> 01:08:16,829
like the ones doing flawless machine translation,

1525
01:08:17,149 --> 01:08:20,189
aren't explicitly taught these CFG rules. They

1526
01:08:20,189 --> 01:08:22,369
are never shown a dependency tree. They are not.

1527
01:08:22,470 --> 01:08:25,130
They ingest raw text. They learn to generate

1528
01:08:25,130 --> 01:08:27,409
perfect syntax purely through the mathematical

1529
01:08:27,409 --> 01:08:29,630
weights of self -attention and the objective

1530
01:08:29,630 --> 01:08:32,090
of predicting the next word in a massive corpus.

1531
01:08:32,289 --> 01:08:34,710
Which forces a very uncomfortable question. Does

1532
01:08:34,710 --> 01:08:36,489
this mean the formal grammar rules we've spent

1533
01:08:36,489 --> 01:08:38,989
centuries obsessing over are actually just a

1534
01:08:38,989 --> 01:08:41,689
human illusion? Are CFGs and dependency arrows

1535
01:08:41,689 --> 01:08:44,960
just... crude, low -resolution maps of a statistical

1536
01:08:44,960 --> 01:08:47,460
territory that the machine, with its billions

1537
01:08:47,460 --> 01:08:49,800
of dense vectors, actually understands far better

1538
01:08:49,800 --> 01:08:52,340
than we do? Are we just projecting our own biological

1539
01:08:52,340 --> 01:08:55,579
need for neat, visible rules onto a system that

1540
01:08:55,579 --> 01:08:58,340
is fundamentally fluid? It is a profound philosophical

1541
01:08:58,340 --> 01:09:02,039
question. Have we discovered the true structure

1542
01:09:02,039 --> 01:09:04,979
of language, or merely invented a useful fiction

1543
01:09:04,979 --> 01:09:07,640
to explain the statistical patterns our brains

1544
01:09:07,640 --> 01:09:11,029
recognize intuitively? The success of deep learning

1545
01:09:11,029 --> 01:09:13,529
suggests the underlying reality of language is

1546
01:09:13,529 --> 01:09:16,609
far more geometric and continuous than discrete

1547
01:09:16,609 --> 01:09:19,069
and rule -based. It brings us right back to our

1548
01:09:19,069 --> 01:09:21,430
opening. We tried to find the jagged white line

1549
01:09:21,430 --> 01:09:23,689
on an X -ray of language. We wanted strict bones.

1550
01:09:23,930 --> 01:09:26,470
But it turns out language isn't bone. It's a

1551
01:09:26,470 --> 01:09:29,710
cloud, a massive multidimensional cloud of probabilities.

1552
01:09:30,470 --> 01:09:32,449
And that is something for you to mull over until

1553
01:09:32,449 --> 01:09:34,229
next time. Thank you for taking this deep dive

1554
01:09:34,229 --> 01:09:36,189
with us. The next time you look at your phone's

1555
01:09:36,189 --> 01:09:38,289
autocomplete or use a translation app today,

1556
01:09:38,430 --> 01:09:40,890
take a second to look past the screen. Appreciate

1557
01:09:40,890 --> 01:09:43,430
the invisible magnificent math making it happen.

1558
01:09:43,630 --> 01:09:44,989
We will see you for part two.