1
00:00:00,000 --> 00:00:03,540
Welcome, the learner, to the Deep Dive. We are,

2
00:00:03,660 --> 00:00:05,240
well, we're really thrilled to have you here

3
00:00:05,240 --> 00:00:07,459
with us today. You know, usually when we talk

4
00:00:07,459 --> 00:00:10,900
about a physical system, there's this expectation

5
00:00:10,900 --> 00:00:14,720
of smooth continuity. Think about, like, engineering

6
00:00:14,720 --> 00:00:17,420
or physics. If you drop a tennis ball off a roof,

7
00:00:17,839 --> 00:00:20,379
gravity pulls it down. and its trajectory is

8
00:00:20,379 --> 00:00:23,300
this continuous, predictable arc. Right, yeah.

9
00:00:23,579 --> 00:00:25,120
You could hypothetically measure its position

10
00:00:25,120 --> 00:00:27,579
at, I mean, any infinitesimally small fraction

11
00:00:27,579 --> 00:00:29,620
of a second, and you would find it moving along

12
00:00:29,620 --> 00:00:32,479
a perfectly smooth mathematical curve. It is,

13
00:00:32,500 --> 00:00:35,119
you know, a very comforting way to view the universe.

14
00:00:35,219 --> 00:00:37,619
Physical laws just lend themselves so beautifully

15
00:00:37,619 --> 00:00:40,600
to continuous mathematics. The equations of motion

16
00:00:40,600 --> 00:00:43,700
allow for infinite gradations, which makes modeling

17
00:00:43,700 --> 00:00:45,759
them computationally, well, pretty straightforward.

18
00:00:46,329 --> 00:00:48,049
But then you step into the world of human language,

19
00:00:48,210 --> 00:00:51,649
and suddenly that smooth, continuous curve just

20
00:00:51,649 --> 00:00:54,030
completely shatters. For those of you listening,

21
00:00:54,130 --> 00:00:57,210
imagine trying to plot a conversation on a graph.

22
00:00:57,390 --> 00:00:59,590
We're looking at a landscape that is fundamentally

23
00:00:59,590 --> 00:01:02,329
discrete, incredibly messy, and honestly, entirely

24
00:01:02,329 --> 00:01:05,060
murky. Yeah, that's the core issue. Language

25
00:01:05,060 --> 00:01:08,260
operates in discrete combinatorial chunks. You

26
00:01:08,260 --> 00:01:10,700
cannot utter a sound that is mathematically like

27
00:01:10,700 --> 00:01:13,379
halfway between the word cat and the word dog

28
00:01:13,379 --> 00:01:15,780
to convey some sort of hybrid animal. Right,

29
00:01:15,840 --> 00:01:18,900
a dat, a cog, doesn't work. Exactly. You either

30
00:01:18,900 --> 00:01:21,180
select the discrete symbol for cat or you select

31
00:01:21,180 --> 00:01:23,500
the discrete symbol for dog. There is no smooth

32
00:01:23,500 --> 00:01:25,659
gradient there. And that fundamental friction.

33
00:01:26,239 --> 00:01:29,180
Between the continuous math that computers absolutely

34
00:01:29,180 --> 00:01:32,319
love and the discrete, chaotic building blocks

35
00:01:32,319 --> 00:01:36,519
of human language is our mission for today. We

36
00:01:36,519 --> 00:01:38,939
are going to meticulously unpack a truly foundational

37
00:01:38,939 --> 00:01:41,500
text in the realm of artificial intelligence.

38
00:01:41,760 --> 00:01:43,739
I'm talking about Jacob Eisenstein's introduction

39
00:01:43,739 --> 00:01:46,150
to natural language processing. It's an incredible

40
00:01:46,150 --> 00:01:48,489
text. It really is. And our goal for you today

41
00:01:48,489 --> 00:01:51,409
is to explore the specific mathematical and computational

42
00:01:51,409 --> 00:01:54,489
toolkit that engineers use to translate our messy

43
00:01:54,489 --> 00:01:56,989
communication into something a machine can process,

44
00:01:57,090 --> 00:02:00,689
learn from, and ultimately generate on its own.

45
00:02:00,890 --> 00:02:03,430
Bridging that gap really is the central challenge

46
00:02:03,430 --> 00:02:06,079
of the entire field because Beyond just being

47
00:02:06,079 --> 00:02:09,639
discrete, language is wildly compositional. We

48
00:02:09,639 --> 00:02:12,180
take these distinct symbols words and we snap

49
00:02:12,180 --> 00:02:14,659
them together to form phrases, which then snap

50
00:02:14,659 --> 00:02:17,159
together to form sentences. Like Legos, basically.

51
00:02:17,379 --> 00:02:19,759
Exactly like Legos. The number of possible valid

52
00:02:19,759 --> 00:02:22,379
sentences you can create is virtually infinite.

53
00:02:22,639 --> 00:02:25,580
Yet historically, computational hardware is built

54
00:02:25,580 --> 00:02:29,539
to multiply matrices of continuous numbers. Reconciling

55
00:02:29,539 --> 00:02:31,620
those two completely different realities requires

56
00:02:31,620 --> 00:02:34,629
some... Well, some extraordinary mathematical

57
00:02:34,629 --> 00:02:37,389
ingenuity. Yeah, and before we get into the heavy

58
00:02:37,389 --> 00:02:40,090
machinery of how computers actually read, I think

59
00:02:40,090 --> 00:02:42,740
we need to set a baseline here. Eisenstein draws

60
00:02:42,740 --> 00:02:45,099
a really important distinction early on between

61
00:02:45,099 --> 00:02:47,780
computational linguistics and natural language

62
00:02:47,780 --> 00:02:49,780
processing. Because, you know, on the surface,

63
00:02:49,879 --> 00:02:51,740
they sound like the exact same discipline. They

64
00:02:51,740 --> 00:02:54,099
do, and they frequently share academic spaces,

65
00:02:54,199 --> 00:02:56,520
but their underlying philosophies are entirely

66
00:02:56,520 --> 00:02:59,099
distinct. Computational linguistics is primarily

67
00:02:59,099 --> 00:03:02,860
a scientific endeavor. The ultimate goal is to

68
00:03:02,860 --> 00:03:05,419
understand the cognitive and structural realities

69
00:03:05,419 --> 00:03:08,129
of human language. Okay, so it's about the humans,

70
00:03:08,210 --> 00:03:11,210
not the computers. Right. A computational linguist

71
00:03:11,210 --> 00:03:14,129
uses the computer as a microscope, a tool to

72
00:03:14,129 --> 00:03:17,030
test hypotheses about how the human brain processes

73
00:03:17,030 --> 00:03:20,409
grammar or acquires vocabulary. So the computer

74
00:03:20,409 --> 00:03:23,129
is just serving the science of linguistics. But

75
00:03:23,129 --> 00:03:25,430
NLP natural language processing is completely

76
00:03:25,430 --> 00:03:28,090
different, right? NLP is an engineering discipline.

77
00:03:28,860 --> 00:03:31,400
The goal is to build computational algorithms

78
00:03:31,400 --> 00:03:34,520
to actually do a specific task. Like if you're

79
00:03:34,520 --> 00:03:36,259
building a spam filter, a machine translation

80
00:03:36,259 --> 00:03:39,199
app, or a virtual assistant, your goal is pure

81
00:03:39,199 --> 00:03:42,550
utility. Yes, utility is the perfect word. If

82
00:03:42,550 --> 00:03:45,090
the machine translation system successfully translates

83
00:03:45,090 --> 00:03:47,750
a highly complex medical document from Mandarin

84
00:03:47,750 --> 00:03:50,389
to English, that is a massive triumph of NLP.

85
00:03:50,590 --> 00:03:52,830
It really does not matter if the algorithm used

86
00:03:52,830 --> 00:03:55,189
to achieve that translation bears absolutely

87
00:03:55,189 --> 00:03:57,629
no resemblance to how a bilingual human being

88
00:03:57,629 --> 00:03:59,830
actually translates language in their head. The

89
00:03:59,830 --> 00:04:01,750
machine doesn't have to think like us, it just

90
00:04:01,750 --> 00:04:04,990
has to perform for us. Exactly. NLP cares about

91
00:04:04,990 --> 00:04:07,110
the functional output, not necessarily cognitive

92
00:04:07,110 --> 00:04:10,400
plausibility. That makes total sense. And to

93
00:04:10,400 --> 00:04:13,060
build these highly functional systems, Eisenstein

94
00:04:13,060 --> 00:04:15,580
introduces a core motif that we are going to

95
00:04:15,580 --> 00:04:18,860
see over and over today. It's this duality, search

96
00:04:18,860 --> 00:04:21,610
versus learning. Yeah, any robust NLP system

97
00:04:21,610 --> 00:04:24,410
has to balance those two forces. To understand

98
00:04:24,410 --> 00:04:26,230
the distinction, we really need to look at how

99
00:04:26,230 --> 00:04:28,449
a machine approaches a problem space. Okay, let

100
00:04:28,449 --> 00:04:30,170
me try out an analogy here for the listener.

101
00:04:30,569 --> 00:04:33,870
Imagine learning is like figuring out the rules

102
00:04:33,870 --> 00:04:36,910
of a new, incredibly complex board game just

103
00:04:36,910 --> 00:04:38,750
by watching people play it over and over again.

104
00:04:39,129 --> 00:04:41,569
You observe thousands of matches. Eventually,

105
00:04:41,610 --> 00:04:43,970
you synthesize a model in your head, like, aha,

106
00:04:44,170 --> 00:04:46,269
the night piece moves in an L shape. I like that,

107
00:04:46,350 --> 00:04:48,329
yeah. So that's the learning part, building your

108
00:04:48,329 --> 00:04:50,449
internal model of how the universe of the game

109
00:04:50,449 --> 00:04:53,790
works. But search is the act of actually sitting

110
00:04:53,790 --> 00:04:56,230
down to play. It's looking multiple moves ahead,

111
00:04:56,430 --> 00:04:58,689
evaluating all the possible branching paths on

112
00:04:58,689 --> 00:05:00,910
the board, and trying to find the absolute winning

113
00:05:00,910 --> 00:05:03,370
sequence of moves. That captures the dynamic

114
00:05:03,370 --> 00:05:06,189
perfectly, I mean, really well. In modern natural

115
00:05:06,189 --> 00:05:08,350
language systems, the learning component relies

116
00:05:08,350 --> 00:05:11,480
on numerical optimization. We feed the machine

117
00:05:11,480 --> 00:05:14,740
vast data sets of text, and it adjusts millions

118
00:05:14,740 --> 00:05:18,120
of internal mathematical weights to build a probabilistic

119
00:05:18,120 --> 00:05:21,420
model of the language. That model dictates the

120
00:05:21,420 --> 00:05:23,699
rules of the game. Right, so the machine is staring

121
00:05:23,699 --> 00:05:26,319
at the board game, absorbing the patterns. Precisely.

122
00:05:26,379 --> 00:05:30,000
Then comes the search. When you hand that trained

123
00:05:30,000 --> 00:05:32,819
model a brand new, never -before -seen sentence

124
00:05:32,819 --> 00:05:35,980
and ask it to parse the grammar, the model faces

125
00:05:35,980 --> 00:05:39,040
an exponentially large space of possible interpretations.

126
00:05:39,769 --> 00:05:41,949
Because language is ambiguous. Highly ambiguous.

127
00:05:42,029 --> 00:05:44,449
A complex sentence might have thousands of theoretically

128
00:05:44,449 --> 00:05:47,230
valid grammatical structures. The search algorithm

129
00:05:47,230 --> 00:05:49,209
is the engine that efficiently navigates that

130
00:05:49,209 --> 00:05:52,750
massive combinatorial explosion to find the single

131
00:05:52,750 --> 00:05:55,829
most probable interpretation based entirely on

132
00:05:55,829 --> 00:05:57,569
the probabilistic rules it established during

133
00:05:57,569 --> 00:06:00,519
the learning phase. Which introduces a huge tension,

134
00:06:00,639 --> 00:06:02,399
right? Yeah. If your linguistic model is too

135
00:06:02,399 --> 00:06:04,959
simplistic, your search is fast, but your results

136
00:06:04,959 --> 00:06:07,839
are absolute garbage. But if your model captures

137
00:06:07,839 --> 00:06:10,540
all the beautiful, nuanced complexity of human

138
00:06:10,540 --> 00:06:13,319
language, the search algorithm might take until

139
00:06:13,319 --> 00:06:15,839
the heat death of the universe just to evaluate

140
00:06:15,839 --> 00:06:18,980
every possible interpretation. That is the defining

141
00:06:18,980 --> 00:06:21,980
struggle of the field. Balancing the expressiveness

142
00:06:21,980 --> 00:06:24,399
of the model with the computational tractability

143
00:06:24,399 --> 00:06:26,839
of the search algorithm, you are always trading

144
00:06:26,839 --> 00:06:30,230
one for the other. So how do we even begin? We

145
00:06:30,230 --> 00:06:32,430
need to get a computer to look at a single word.

146
00:06:32,629 --> 00:06:35,209
Let's step into the foundational methods of learning,

147
00:06:35,310 --> 00:06:37,730
starting with chapter two. The absolute simplest

148
00:06:37,730 --> 00:06:40,610
way a computer can process a document is something

149
00:06:40,610 --> 00:06:43,689
called the bag of words representation. Ah, yes.

150
00:06:43,930 --> 00:06:46,269
The bag of words takes a complex piece of literature

151
00:06:46,269 --> 00:06:48,949
and reduces it to a sheer mathematical vector.

152
00:06:49,089 --> 00:06:51,209
It is literally just a simple list of frequencies.

153
00:06:51,509 --> 00:06:54,430
It discards grammar, it discards narrative structure,

154
00:06:54,629 --> 00:06:57,319
and it completely discards word order. For you

155
00:06:57,319 --> 00:06:59,819
listening, picture making a smoothie out of a

156
00:06:59,819 --> 00:07:03,199
classic novel. Let's say you take Moby Dick and

157
00:07:03,199 --> 00:07:06,259
throw the entire book into a blender. You know

158
00:07:06,259 --> 00:07:08,779
the exact inventory of your smoothie. You know

159
00:07:08,779 --> 00:07:11,420
the word whale went in 1 ,200 times. You know

160
00:07:11,420 --> 00:07:14,699
Ahab went in 500 times. But you have... utterly

161
00:07:14,699 --> 00:07:17,480
lost the shape and structure of the ingredients.

162
00:07:17,620 --> 00:07:20,420
The sequence is gone. Exactly. The sequence of

163
00:07:20,420 --> 00:07:23,540
events, the looming dread of the prose, it is

164
00:07:23,540 --> 00:07:26,079
completely obliterated. You just have a sludge

165
00:07:26,079 --> 00:07:28,800
of word counts. It sounds incredibly reductive,

166
00:07:28,819 --> 00:07:32,199
and honestly, it is. However, this representation

167
00:07:32,199 --> 00:07:35,100
is astonishingly effective for tasks like linear

168
00:07:35,100 --> 00:07:38,399
text classification. Consider a spam filter.

169
00:07:38,740 --> 00:07:40,300
Oh, right, because the structure doesn't matter

170
00:07:40,300 --> 00:07:42,810
there. Exactly. To determine if an incoming email

171
00:07:42,810 --> 00:07:46,009
is spam, the system rarely needs to parse the

172
00:07:46,009 --> 00:07:48,689
complex syntactic dependencies of the sentences.

173
00:07:49,269 --> 00:07:51,790
It primarily needs to check if the frequencies

174
00:07:51,790 --> 00:07:55,089
of specific words like urgent, discount, wire,

175
00:07:55,250 --> 00:07:57,889
or lottery are abnormally high compared to a

176
00:07:57,889 --> 00:07:59,889
normal email. The machine doesn't care how you

177
00:07:59,889 --> 00:08:01,670
use the word lottery. It just cares that the

178
00:08:01,670 --> 00:08:03,709
lottery ingredient is dominating the smoothie.

179
00:08:03,910 --> 00:08:06,509
And to classify these texts, Eisenstein brings

180
00:08:06,509 --> 00:08:10,509
up the Naive Days classifier. I really need some

181
00:08:10,509 --> 00:08:12,810
help here, because the way this algorithm works

182
00:08:12,810 --> 00:08:15,709
feels totally backward to me. It relies on something

183
00:08:15,709 --> 00:08:18,290
called a generative process. I know, it is a

184
00:08:18,290 --> 00:08:21,870
bit counterintuitive. Naive Bayes solves a classification

185
00:08:21,870 --> 00:08:24,850
problem by inventing a probabilistic fiction

186
00:08:24,850 --> 00:08:27,029
about how the text was written in the first place.

187
00:08:27,209 --> 00:08:30,430
A fiction? Yes. Imagine the computer assumes

188
00:08:30,430 --> 00:08:33,529
that before you type an email, you randomly pull

189
00:08:33,529 --> 00:08:36,250
a hidden label out of a hat. Let's say you pull

190
00:08:36,250 --> 00:08:39,240
out the label spam. Okay, I am a spammer. Holding

191
00:08:39,240 --> 00:08:41,320
my label. Based solely on the fact that you hold

192
00:08:41,320 --> 00:08:43,860
that S -spam label, the computer imagines you

193
00:08:43,860 --> 00:08:46,200
walk over to a giant physical bag filled with

194
00:08:46,200 --> 00:08:48,960
spam -related vocabulary. You reach in, pull

195
00:08:48,960 --> 00:08:50,559
out a word, write it down, put it back. Then

196
00:08:50,559 --> 00:08:52,360
you reach in, pull out another word, write it

197
00:08:52,360 --> 00:08:54,539
down. You repeat this until your email is finished.

198
00:08:54,740 --> 00:08:56,580
Wait, I have a massive problem with that. Humans

199
00:08:56,580 --> 00:08:59,360
do not vomit independent words onto a page like

200
00:08:59,360 --> 00:09:02,200
that. If I write the word sand, the very next

201
00:09:02,200 --> 00:09:04,799
word I write is overwhelmingly likely to be Francisco.

202
00:09:05,220 --> 00:09:07,720
The words are intimately dependent on each other.

203
00:09:08,000 --> 00:09:10,639
your objection is entirely valid which is exactly

204
00:09:10,639 --> 00:09:13,440
why the algorithm explicitly includes the word

205
00:09:13,440 --> 00:09:16,220
naive in its name it makes the mathematically

206
00:09:16,220 --> 00:09:19,000
convenient though linguistically absurd assumption

207
00:09:19,000 --> 00:09:22,100
that every single word in the document is generated

208
00:09:22,100 --> 00:09:24,399
completely independently of every other word

209
00:09:24,399 --> 00:09:27,139
conditional only on the label if it's so absurd

210
00:09:27,139 --> 00:09:30,799
why build a system around it why force the computer

211
00:09:30,799 --> 00:09:33,370
to make such a dumb assumption We make that assumption

212
00:09:33,370 --> 00:09:36,389
to avoid computational paralysis. If you do not

213
00:09:36,389 --> 00:09:38,750
assume independence the probability mass becomes

214
00:09:38,750 --> 00:09:42,019
intractable. To calculate the exact probability

215
00:09:42,019 --> 00:09:44,480
of a 20 -word sentence without the independence

216
00:09:44,480 --> 00:09:46,940
assumption, you would need to calculate the joint

217
00:09:46,940 --> 00:09:49,820
probability of all 20 words appearing in that

218
00:09:49,820 --> 00:09:52,220
exact sequence. It's just rare. Extremely rare.

219
00:09:52,320 --> 00:09:54,039
You would never have enough training data to

220
00:09:54,039 --> 00:09:56,320
reliably estimate that. By assuming independence,

221
00:09:56,759 --> 00:09:59,620
the math simplifies drastically. You simply calculate

222
00:09:59,620 --> 00:10:02,440
the individual probability of each word occurring

223
00:10:02,440 --> 00:10:05,100
in a spam email and you multiply them all together.

224
00:10:05,279 --> 00:10:08,860
Wow. Okay. It turns a massive impossible probability

225
00:10:08,860 --> 00:10:12,080
puzzle into just a long string of simple multiplication.

226
00:10:12,460 --> 00:10:14,720
Furthermore, it acts as a linear classifier.

227
00:10:14,860 --> 00:10:17,519
In a high -dimensional space where every word

228
00:10:17,519 --> 00:10:19,299
in the English language represents a different

229
00:10:19,299 --> 00:10:21,980
geometric dimension, Naive Bayes essentially

230
00:10:21,980 --> 00:10:25,039
draws a single flat hyperplane, like a straight

231
00:10:25,039 --> 00:10:27,679
sheet of glass, that separates the spam vectors

232
00:10:27,679 --> 00:10:30,240
on one side from the non -spam vectors on the

233
00:10:30,240 --> 00:10:32,470
other. That brings up an interesting historical

234
00:10:32,470 --> 00:10:35,629
quirk in Chapter 3. We hear so much about deep

235
00:10:35,629 --> 00:10:38,450
learning and neural networks today, but NLP relied

236
00:10:38,450 --> 00:10:41,929
on these simple linear models like naive Bayes

237
00:10:41,929 --> 00:10:44,750
or support vector machines for a really, really

238
00:10:44,750 --> 00:10:48,529
long time. Computer vision adopted deep, nonlinear

239
00:10:48,529 --> 00:10:51,090
neural networks way before text processing did.

240
00:10:51,289 --> 00:10:53,629
Why was language so late to the deep learning

241
00:10:53,629 --> 00:10:56,269
revolution? That delay really stems from the

242
00:10:56,269 --> 00:10:58,289
fundamental difference between an image pixel

243
00:10:58,289 --> 00:11:01,429
and a word. In computer vision, a single pixel

244
00:11:01,429 --> 00:11:04,549
carries almost zero semantic meaning. A lone

245
00:11:04,549 --> 00:11:06,649
brown pixel does not tell you if you are looking

246
00:11:06,649 --> 00:11:09,490
at a dog, a tree, or, you know, a cup of coffee.

247
00:11:09,629 --> 00:11:12,190
Right, it's just brown. Exactly. You must use

248
00:11:12,190 --> 00:11:14,590
nonlinear neural networks to combine pixels into

249
00:11:14,590 --> 00:11:17,110
edges, combine edges into textures, and combine

250
00:11:17,110 --> 00:11:19,549
textures into recognizable shapes. You have to

251
00:11:19,549 --> 00:11:21,669
build the meaning up from nothing. But a word

252
00:11:21,669 --> 00:11:24,809
is different. A single word carries immense semantic

253
00:11:24,809 --> 00:11:27,450
weight right out of the gate. If a movie review

254
00:11:27,450 --> 00:11:30,690
contains the word horrendous, that single feature

255
00:11:30,690 --> 00:11:33,070
provides a massive signal about the sentiment

256
00:11:33,070 --> 00:11:35,870
of the text, regardless of the surrounding words.

257
00:11:36,129 --> 00:11:38,750
Because text represents such a high dimensional

258
00:11:38,750 --> 00:11:41,929
space with hundreds of thousands of discrete

259
00:11:41,929 --> 00:11:44,350
vocabulary words acting as independent features,

260
00:11:44,769 --> 00:11:47,570
linear models were highly effective. They didn't

261
00:11:47,570 --> 00:11:49,509
need the deep layers to find the meaning. No,

262
00:11:49,529 --> 00:11:52,230
they didn't. In fact, simpler linear models are

263
00:11:52,230 --> 00:11:54,909
naturally resistant to overfitting in those massive

264
00:11:54,909 --> 00:11:57,710
dimensional spaces. whereas early neural networks

265
00:11:57,710 --> 00:12:00,450
struggled immensely with text. But eventually,

266
00:12:00,669 --> 00:12:03,250
the sheer predictive power of nonlinear models

267
00:12:03,250 --> 00:12:05,549
like feedforward and convolutional neural networks

268
00:12:05,549 --> 00:12:09,269
became too massive to ignore. What did neural

269
00:12:09,269 --> 00:12:11,350
networks offer that the linear sheet of glass

270
00:12:11,350 --> 00:12:13,950
couldn't? They offered the ability to learn intermediate

271
00:12:13,950 --> 00:12:16,870
representations. A linear model maps the input

272
00:12:16,870 --> 00:12:19,549
directly to the output. A neural network passes

273
00:12:19,549 --> 00:12:21,429
the input through hidden layers, mathematically

274
00:12:21,429 --> 00:12:24,129
transforming the data into entirely new geometric

275
00:12:24,129 --> 00:12:27,549
spaces. This allows the network to capture complex

276
00:12:27,549 --> 00:12:30,009
interactions between words. Give me an example

277
00:12:30,009 --> 00:12:32,450
of that. Okay, a linear model sees the word good

278
00:12:32,450 --> 00:12:35,070
and adds points to the positive sentiment score.

279
00:12:35,289 --> 00:12:38,210
It sees the word not and adds a tiny bit to the

280
00:12:38,210 --> 00:12:40,490
negative score, just adds them up. But a neural

281
00:12:40,490 --> 00:12:42,389
network can learn that the specific combination

282
00:12:42,389 --> 00:12:45,870
not good operates as a single conceptual unit

283
00:12:45,870 --> 00:12:48,429
that wildly flips the sentiment, capturing a

284
00:12:48,429 --> 00:12:51,750
nonlinear interaction. Ah, I see. And building

285
00:12:51,750 --> 00:12:53,889
these networks introduces the concept of computation

286
00:12:53,889 --> 00:12:57,139
graphs. Eisenstein makes a huge point about the

287
00:12:57,139 --> 00:12:59,019
difference between static graphs and dynamic

288
00:12:59,019 --> 00:13:01,600
graphs, which seems to be a uniquely painful

289
00:13:01,600 --> 00:13:04,000
problem for language processing. It was a massive

290
00:13:04,000 --> 00:13:06,220
headache. A computation graph is essentially

291
00:13:06,220 --> 00:13:08,299
the architectural blueprint of the mathematical

292
00:13:08,299 --> 00:13:11,320
operations the neural network will perform. With

293
00:13:11,320 --> 00:13:13,879
a static graph, you define that blueprint once.

294
00:13:13,960 --> 00:13:16,519
You compile it and you push thousands of examples

295
00:13:16,519 --> 00:13:19,110
through it. This works flawlessly for computer

296
00:13:19,110 --> 00:13:21,610
vision because every image can be resized to

297
00:13:21,610 --> 00:13:26,330
exactly 256 by 256 pixels. The input size never

298
00:13:26,330 --> 00:13:29,190
changes. But sentences refuse to conform to a

299
00:13:29,190 --> 00:13:32,450
fixed size. If you are analyzing a book, one

300
00:13:32,450 --> 00:13:34,909
sentence might be three words long. The very

301
00:13:34,909 --> 00:13:37,769
next sentence might be 45 words long with multiple

302
00:13:37,769 --> 00:13:41,370
nested clauses. And that variability breaks static

303
00:13:41,370 --> 00:13:45,000
architectures completely. Consequently, NLP engineers

304
00:13:45,000 --> 00:13:48,240
had to pioneer dynamic computation graphs. The

305
00:13:48,240 --> 00:13:50,299
architecture of the neural network, the actual

306
00:13:50,299 --> 00:13:53,080
mathematical flowchart, must be constructed dynamically,

307
00:13:53,539 --> 00:13:57,139
on the fly, for every single input sentence to

308
00:13:57,139 --> 00:13:59,340
accommodate its unique length and grammatical

309
00:13:59,340 --> 00:14:02,240
structure. It requires significantly more complex

310
00:14:02,240 --> 00:14:04,399
engineering. Let's actually look at how these

311
00:14:04,399 --> 00:14:06,850
classifiers perform in the real world. In Chapter

312
00:14:06,850 --> 00:14:08,870
4, Eisenstein highlights a really fascinating

313
00:14:08,870 --> 00:14:11,370
study by Alm and colleagues regarding linguistic

314
00:14:11,370 --> 00:14:13,850
applications and emotion. They wanted to see

315
00:14:13,850 --> 00:14:15,590
if a machine could read a children's fairy tale,

316
00:14:15,710 --> 00:14:17,909
like a Grimm Brothers story, and map individual

317
00:14:17,909 --> 00:14:20,289
sentences to Ekman's six universal emotions.

318
00:14:20,549 --> 00:14:23,110
You know, anger, disgust, fear, joy, sadness,

319
00:14:23,250 --> 00:14:25,629
and surprise. Yes, the practical application

320
00:14:25,629 --> 00:14:28,009
of this research was actually text -to -speech

321
00:14:28,009 --> 00:14:30,870
generation. If an automated system is reading

322
00:14:30,870 --> 00:14:33,870
a story to a child, you want the synthesizer

323
00:14:33,870 --> 00:14:36,610
to inject the appropriate emotional prosody rather

324
00:14:36,610 --> 00:14:40,730
than droning on in a flat robotic monotone. I

325
00:14:40,730 --> 00:14:42,970
have to express some deep skepticism here, though.

326
00:14:43,210 --> 00:14:46,690
Human emotion is incredibly subjective. If you

327
00:14:46,690 --> 00:14:48,990
hand a paragraph of a grim fairy tale to three

328
00:14:48,990 --> 00:14:51,289
different human adults, they will likely disagree

329
00:14:51,289 --> 00:14:54,049
on the underlying emotion. One might say the

330
00:14:54,049 --> 00:14:56,269
villain's dialogue shows anger, another might

331
00:14:56,269 --> 00:14:58,509
say disgust, and a third might argue it's actually

332
00:14:58,509 --> 00:15:01,129
fear. That's very true. If the human beings providing

333
00:15:01,129 --> 00:15:03,789
the training data cannot even agree on the emotional

334
00:15:03,789 --> 00:15:06,830
ground truth, how can we possibly expect a machine

335
00:15:06,830 --> 00:15:09,450
to classify it accurately? You have identified

336
00:15:09,450 --> 00:15:11,610
the exact theoretical ceiling of machine learning

337
00:15:11,610 --> 00:15:14,389
on subjective tasks. Eisenstein highlights the

338
00:15:14,389 --> 00:15:16,590
study specifically to demonstrate that limitation.

339
00:15:17,289 --> 00:15:19,490
Even the most sophisticated classifiers in that

340
00:15:19,490 --> 00:15:22,190
study maxed out at roughly 60 to 70 % accuracy.

341
00:15:22,450 --> 00:15:25,049
When the training labels themselves are noisy

342
00:15:25,049 --> 00:15:27,590
and subjective, the algorithm's performance is

343
00:15:27,590 --> 00:15:30,769
permanently capped by that human ambiguity. Wow.

344
00:15:30,950 --> 00:15:33,830
You literally cannot optimize past the ambiguity

345
00:15:33,830 --> 00:15:36,309
of the human soul. A poetic way to frame it,

346
00:15:36,370 --> 00:15:39,360
but mathematically accurate. However... What

347
00:15:39,360 --> 00:15:41,460
makes the study remarkable is how the researchers

348
00:15:41,460 --> 00:15:45,440
managed to squeeze out that 70 % accuracy. They

349
00:15:45,440 --> 00:15:47,860
realized that the bag -of -words smoothie was

350
00:15:47,860 --> 00:15:50,860
totally insufficient. So they engineered new

351
00:15:50,860 --> 00:15:53,659
features based on metadata. Metadata like what?

352
00:15:53,759 --> 00:15:56,519
Structural context. For example, they tracked

353
00:15:56,519 --> 00:15:58,600
the physical position of the sentence within

354
00:15:58,600 --> 00:16:00,440
the overall story. Oh, that makes so much sense.

355
00:16:00,809 --> 00:16:03,450
The climax of a fairy tale, which is usually

356
00:16:03,450 --> 00:16:06,269
situated near the end of the text, is statistically

357
00:16:06,269 --> 00:16:09,149
far more likely to contain expressions of fear

358
00:16:09,149 --> 00:16:12,149
or surprise than the opening once upon a time

359
00:16:12,149 --> 00:16:15,269
paragraph. Precisely. By injecting structural

360
00:16:15,269 --> 00:16:17,769
narrative knowledge into the statistical learning

361
00:16:17,769 --> 00:16:20,230
algorithm, they provided the machine with hints

362
00:16:20,230 --> 00:16:23,049
that exist outside the raw vocabulary. OK, but

363
00:16:23,049 --> 00:16:26,000
everything we have discussed so far. Naive Bayes'

364
00:16:26,019 --> 00:16:29,200
neural networks sentiment analysis relies on

365
00:16:29,200 --> 00:16:31,720
human beings sitting in a room and manually labeling

366
00:16:31,720 --> 00:16:35,120
thousands of emails as spam or thousands of sentences

367
00:16:35,120 --> 00:16:39,539
as angry. This is supervised learning. But Chapter

368
00:16:39,539 --> 00:16:42,000
5 asks, what happens when we don't have labeled

369
00:16:42,000 --> 00:16:44,899
data? How does machine learn when we absolutely

370
00:16:44,899 --> 00:16:47,820
refuse to give it the answer key? That brings

371
00:16:47,820 --> 00:16:51,320
us to expectation maximization or the EM algorithm.

372
00:16:51,950 --> 00:16:53,490
I need you to break this down for me because

373
00:16:53,490 --> 00:16:55,690
reading the math for this felt like looking at

374
00:16:55,690 --> 00:16:58,450
a chicken and egg paradox. It is a bit mind bending

375
00:16:58,450 --> 00:17:01,110
at first. Expectation maximization is widely

376
00:17:01,110 --> 00:17:03,029
considered one of the most elegant algorithms

377
00:17:03,029 --> 00:17:05,430
in statistical machine learning. We use it when

378
00:17:05,430 --> 00:17:07,549
dealing with hidden variables. Let's assume you

379
00:17:07,549 --> 00:17:09,609
have 10 ,000 news articles and you want the machine

380
00:17:09,609 --> 00:17:12,210
to cluster them into distinct topics, say sports,

381
00:17:12,369 --> 00:17:15,009
politics and finance. But you do not have any

382
00:17:15,009 --> 00:17:17,349
labels and you do not know which words indicate

383
00:17:17,349 --> 00:17:20,859
which topics. That's the paradox. If I knew what

384
00:17:20,859 --> 00:17:23,140
the topics were, I could easily find the words

385
00:17:23,140 --> 00:17:25,759
that represent them. And if I knew which words

386
00:17:25,759 --> 00:17:28,000
represented the topics, I could easily sort the

387
00:17:28,000 --> 00:17:30,319
articles. But I have zero information on either

388
00:17:30,319 --> 00:17:32,859
side. To break that deadlock, the EM algorithm

389
00:17:32,859 --> 00:17:35,079
performs an iterative two -step mathematical

390
00:17:35,079 --> 00:17:38,920
dance. The first phase is the E -step, or expectation

391
00:17:38,920 --> 00:17:41,700
step. Because you lack the parameters of the

392
00:17:41,700 --> 00:17:44,819
clusters, you simply formulate a guess. You initialize

393
00:17:44,819 --> 00:17:47,700
the model with entirely random... terrible parameters.

394
00:17:48,079 --> 00:17:50,319
So the machine just randomly assumes that the

395
00:17:50,319 --> 00:17:53,259
word baseball belongs to the politics cluster.

396
00:17:53,480 --> 00:17:55,880
Yes. Initially, it is a completely uninformed

397
00:17:55,880 --> 00:17:58,869
guess. Using those terrible random guesses, the

398
00:17:58,869 --> 00:18:02,349
machine evaluates every single document. It calculates

399
00:18:02,349 --> 00:18:04,849
the probability of each hidden variable. It basically

400
00:18:04,849 --> 00:18:07,490
says, given my current flawed definition of what

401
00:18:07,490 --> 00:18:09,490
these topics look like, what is the probability

402
00:18:09,490 --> 00:18:12,630
that document A belongs to topic 1? It doesn't

403
00:18:12,630 --> 00:18:14,829
assign a hard label, though, right? It assigns

404
00:18:14,829 --> 00:18:16,750
a soft probability, like saying document A is

405
00:18:16,750 --> 00:18:20,849
60 % topic 1 and 40 % topic 2. Correct. That

406
00:18:20,849 --> 00:18:23,650
soft assignment completes the E -step. Then we

407
00:18:23,650 --> 00:18:25,809
transition to the M -step, the maximization step.

408
00:18:26,009 --> 00:18:28,710
The algorithm takes all of those newly generated

409
00:18:28,710 --> 00:18:31,450
soft probabilities and uses them to fundamentally

410
00:18:31,450 --> 00:18:34,170
update its parameters. It looks at the data and

411
00:18:34,170 --> 00:18:37,410
says, if document A is 60 % likely to be in topic

412
00:18:37,410 --> 00:18:40,730
1, let me look at the actual words inside document

413
00:18:40,730 --> 00:18:43,369
A and adjust my mathematical definition of topic

414
00:18:43,369 --> 00:18:45,930
1 to more closely resemble those words. And then

415
00:18:45,930 --> 00:18:48,569
it just feeds those updated definitions back

416
00:18:48,569 --> 00:18:51,559
into the beginning. It loops indefinitely. Armed

417
00:18:51,559 --> 00:18:53,519
with a slightly less terrible definition of the

418
00:18:53,519 --> 00:18:56,220
topics, it returns to the E -step and recalculates

419
00:18:56,220 --> 00:18:58,980
the soft probabilities for every document. Then

420
00:18:58,980 --> 00:19:01,359
back to the M -step to refine the topic definitions

421
00:19:01,359 --> 00:19:04,279
again. As this dance iterates, the mathematics

422
00:19:04,279 --> 00:19:06,480
guarantee that the model will continually climb

423
00:19:06,480 --> 00:19:09,259
upward, converging on a solution that maximizes

424
00:19:09,259 --> 00:19:11,759
the likelihood of the observed data. It discovers

425
00:19:11,759 --> 00:19:14,299
the hidden structure entirely on its own. It

426
00:19:14,299 --> 00:19:17,259
pulls itself up by its own statistical bootstraps.

427
00:19:17,299 --> 00:19:20,369
That is so cool. But here's the major pivot as

428
00:19:20,369 --> 00:19:22,789
we move into part two of the book. Everything

429
00:19:22,789 --> 00:19:25,150
we have explored so far has fundamentally treated

430
00:19:25,150 --> 00:19:28,210
language as our smoothie. But true language is

431
00:19:28,210 --> 00:19:30,289
not a blender full of independent words. The

432
00:19:30,289 --> 00:19:32,970
sequence of the ingredients is the entire point.

433
00:19:33,089 --> 00:19:36,450
If I say the dog bit the man. That represents

434
00:19:36,450 --> 00:19:38,730
a very different reality than the man bit the

435
00:19:38,730 --> 00:19:41,170
dog, even though the bag of words contains the

436
00:19:41,170 --> 00:19:43,450
exact same frequency of ingredients. That realization

437
00:19:43,450 --> 00:19:46,009
marks the critical threshold in computational

438
00:19:46,009 --> 00:19:48,990
linguistics to capture actual linguistic phenomena,

439
00:19:49,289 --> 00:19:52,549
syntax, grammar, meaning we absolutely must abandon

440
00:19:52,549 --> 00:19:54,789
the bag of words and construct mathematical models

441
00:19:54,789 --> 00:19:57,390
that respect sequences. Which takes us into the

442
00:19:57,390 --> 00:19:59,829
realm of language models and engrams in Chapter

443
00:19:59,829 --> 00:20:03,049
6. How does Eisenstein define a language model

444
00:20:03,049 --> 00:20:05,230
in this context? Because, you know, today we

445
00:20:05,230 --> 00:20:07,130
hear language model and immediately think of

446
00:20:07,130 --> 00:20:09,930
giant chatbots, but the mathematical definition

447
00:20:09,930 --> 00:20:12,589
is much narrower here. At its foundational level,

448
00:20:12,769 --> 00:20:15,190
a language model is simply a probability distribution

449
00:20:15,190 --> 00:20:18,650
over sequences of words. Its only job is to assign

450
00:20:18,650 --> 00:20:20,829
a probability score to a specific string of text

451
00:20:20,829 --> 00:20:23,829
answering the question, how likely is this exact

452
00:20:23,829 --> 00:20:25,990
sequence to occur in this specific language?

453
00:20:26,349 --> 00:20:28,630
If you give the model the sequence, I would like

454
00:20:28,630 --> 00:20:31,109
a cup of coffee, it assigns a high mathematical

455
00:20:31,109 --> 00:20:34,000
probability. If you give it coffee cup, like

456
00:20:34,000 --> 00:20:36,779
what an eye of, it assigns a probability approaching

457
00:20:36,779 --> 00:20:40,240
zero. And practically, it achieves this by predicting

458
00:20:40,240 --> 00:20:42,660
the very next word in a sequence, right? Like

459
00:20:42,660 --> 00:20:44,839
the autocomplete function on a smartphone. Yes,

460
00:20:44,880 --> 00:20:47,930
exactly. By chaining together conditional probabilities,

461
00:20:48,130 --> 00:20:50,329
predicting the next word given the history of

462
00:20:50,329 --> 00:20:52,809
the previous words, you can calculate the score

463
00:20:52,809 --> 00:20:55,289
for an entire sentence. Historically, the most

464
00:20:55,289 --> 00:20:57,990
common way to do this was using engrams. An engram

465
00:20:57,990 --> 00:21:00,990
is simply a contiguous sequence of n items. A

466
00:21:00,990 --> 00:21:03,210
bigogram is a sequence of two words. A trigram

467
00:21:03,210 --> 00:21:05,549
is a sequence of three words. So if I want to

468
00:21:05,549 --> 00:21:08,190
predict the next word using a bigogram model,

469
00:21:08,509 --> 00:21:11,250
the machine is only allowed to look at the single

470
00:21:11,250 --> 00:21:14,990
preceding word. If I use a trigram model, it

471
00:21:14,990 --> 00:21:18,650
looks... The two preceding words. It's like dragging

472
00:21:18,650 --> 00:21:21,009
a tiny restrictive window across a sentence.

473
00:21:21,230 --> 00:21:23,529
Why restrict the machine's vision so severely?

474
00:21:24,009 --> 00:21:26,190
Why not let it look at the previous 20 words

475
00:21:26,190 --> 00:21:28,670
to make a better guess? Because if you expand

476
00:21:28,670 --> 00:21:30,650
the window, you immediately crash into the problem

477
00:21:30,650 --> 00:21:35,180
of data sparsity. infinitely creative. If you

478
00:21:35,180 --> 00:21:37,019
want to calculate the probability of the word

479
00:21:37,019 --> 00:21:40,480
coffee based on the specific exact sequence of

480
00:21:40,480 --> 00:21:43,079
20 words that preceded it, you must search your

481
00:21:43,079 --> 00:21:45,079
training database to find how many times that

482
00:21:45,079 --> 00:21:47,599
exact 20 -word sequence occurred. And even if

483
00:21:47,599 --> 00:21:50,140
your database is the entire internet, you will

484
00:21:50,140 --> 00:21:52,339
almost certainly never find that exact 20 -word

485
00:21:52,339 --> 00:21:54,880
sequence. People invent new sentences every second.

486
00:21:55,180 --> 00:21:58,059
Exactly. The frequency count would be zero. That

487
00:21:58,059 --> 00:22:00,740
is the sparsity problem. So we intentionally

488
00:22:00,740 --> 00:22:03,940
restrict the context window to big grams or prigrams,

489
00:22:03,940 --> 00:22:06,140
ensuring we actually have historical data to

490
00:22:06,140 --> 00:22:09,539
count. But even with a trigram window, sparsity

491
00:22:09,539 --> 00:22:12,759
haunts us. What happens when your model encounters

492
00:22:12,759 --> 00:22:15,920
a valid three -word sequence it has never seen

493
00:22:15,920 --> 00:22:19,049
in its training data? Say the sequence walrus

494
00:22:19,049 --> 00:22:21,369
performing ballet. If it just counts frequencies,

495
00:22:21,569 --> 00:22:23,789
the count is zero. And in probability math, if

496
00:22:23,789 --> 00:22:25,750
you multiply a long chain of fractions and just

497
00:22:25,750 --> 00:22:27,910
one of them is zero, it wipes out the entire

498
00:22:27,910 --> 00:22:30,210
equation. The whole sentence gets a probability

499
00:22:30,210 --> 00:22:32,710
of absolute zero. To prevent that mathematical

500
00:22:32,710 --> 00:22:35,150
catastrophe, engineers utilize techniques known

501
00:22:35,150 --> 00:22:38,250
as smoothing and discounting. Smoothing conceptually

502
00:22:38,250 --> 00:22:40,809
involves taking a tiny fractional sliver of probability

503
00:22:40,809 --> 00:22:43,190
mass away from the common word sequences we have

504
00:22:43,190 --> 00:22:45,769
seen and redistributing it evenly across all

505
00:22:45,769 --> 00:22:47,609
the hypothetical word combinations we haven't

506
00:22:47,609 --> 00:22:50,349
seen. It ensures no sequence ever receives a

507
00:22:50,349 --> 00:22:52,809
hard zero. A more elegant solution discussed

508
00:22:52,809 --> 00:22:55,450
in the text is Katz -Bakoff. I really appreciate

509
00:22:55,450 --> 00:22:58,009
the intuition behind Katz -Bakoff. It operates

510
00:22:58,009 --> 00:23:00,430
the way a human might guess. The model says,

511
00:23:00,890 --> 00:23:03,329
I have never seen this specific three -word sequence

512
00:23:03,329 --> 00:23:06,809
before, so I have no data. Let me back off and

513
00:23:06,809 --> 00:23:08,529
just check my data for the two -word sequence.

514
00:23:08,750 --> 00:23:10,509
If I haven't seen that either, I'll back off

515
00:23:10,509 --> 00:23:12,990
again and just look at the raw frequency of the

516
00:23:12,990 --> 00:23:15,990
single word. It systematically retreats to shorter,

517
00:23:16,109 --> 00:23:18,549
more abundant histories when the longer history

518
00:23:18,549 --> 00:23:21,670
is too sparse to trust. It is highly effective.

519
00:23:22,250 --> 00:23:25,089
However, the paradigm fundamentally shifted with

520
00:23:25,089 --> 00:23:27,009
the introduction of recurrent neural networks,

521
00:23:27,170 --> 00:23:31,019
or RNNs. Ngram models, even with cats back off,

522
00:23:31,160 --> 00:23:33,380
are ultimately just playing a massive game of

523
00:23:33,380 --> 00:23:35,440
counting occurrences and dividing them into fractions.

524
00:23:35,960 --> 00:23:38,119
RNNs approach the problem entirely differently.

525
00:23:38,359 --> 00:23:40,519
They treat word prediction as discriminative

526
00:23:40,519 --> 00:23:42,400
learning. Walk me through what that actually

527
00:23:42,400 --> 00:23:44,420
means for the machine. Instead of looking up

528
00:23:44,420 --> 00:23:46,960
a fraction in a massive table of counts, the

529
00:23:46,960 --> 00:23:50,339
RNN processes a sequence of words one at a time.

530
00:23:50,730 --> 00:23:53,569
As it reads each word, it continually updates

531
00:23:53,569 --> 00:23:55,829
an internal mathematical vector called a hidden

532
00:23:55,829 --> 00:23:58,589
state. This hidden state acts as a compressed

533
00:23:58,589 --> 00:24:00,930
memory of everything the network has read up

534
00:24:00,930 --> 00:24:03,869
to that point. The network then uses that dense

535
00:24:03,869 --> 00:24:06,769
mathematical state to discriminate, to actively

536
00:24:06,769 --> 00:24:09,430
classify which word from the entire vocabulary

537
00:24:09,430 --> 00:24:12,230
should be generated next. So it's not just checking

538
00:24:12,230 --> 00:24:14,390
a lookup table anymore. Right. Because it is

539
00:24:14,390 --> 00:24:16,390
utilizing dense mathematical representations

540
00:24:16,390 --> 00:24:19,690
of context rather than rigid text matching, it

541
00:24:19,690 --> 00:24:22,029
can generalize to completely unseen combinations

542
00:24:22,029 --> 00:24:25,049
of words far more effectively than any n -gram

543
00:24:25,049 --> 00:24:27,690
model ever could. Okay, so language models predict

544
00:24:27,690 --> 00:24:29,869
the next word. But let's shift gears to Chapter

545
00:24:29,869 --> 00:24:32,730
7. We also need to be able to analyze the words

546
00:24:32,730 --> 00:24:35,180
we already have. That brings us to sequence labeling.

547
00:24:35,380 --> 00:24:37,900
We want to take a complete sentence and tag every

548
00:24:37,900 --> 00:24:40,359
single word. A classic example is part of speech

549
00:24:40,359 --> 00:24:42,740
tagging. We want the machine to label every word

550
00:24:42,740 --> 00:24:46,960
as a noun, verb, adjective, and so on. A foundational

551
00:24:46,960 --> 00:24:49,240
probabilistic framework for sequence labeling

552
00:24:49,240 --> 00:24:52,720
is the hidden Markov model, or HMM. It views

553
00:24:52,720 --> 00:24:54,819
language through a specific theoretical lens.

554
00:24:55,059 --> 00:24:57,240
The sequence of grammatical tags is the hidden

555
00:24:57,240 --> 00:24:59,799
process operating beneath the surface. You cannot

556
00:24:59,799 --> 00:25:02,059
physically see the tags. You can only see the

557
00:25:02,059 --> 00:25:04,799
emitted observations, which are the actual words

558
00:25:04,799 --> 00:25:07,180
printed on the page. So for you listening, imagine

559
00:25:07,180 --> 00:25:09,940
the computer is staring at the word fly. It has

560
00:25:09,940 --> 00:25:11,980
no idea if that word is functioning as a noun,

561
00:25:12,140 --> 00:25:15,680
like the annoying insect buzzing around. or a

562
00:25:15,680 --> 00:25:18,059
verb like boarding an airplane. To resolve that

563
00:25:18,059 --> 00:25:20,960
ambiguity, the HMM relies on two distinct sets

564
00:25:20,960 --> 00:25:23,599
of probabilities. The first are transition probabilities.

565
00:25:24,140 --> 00:25:26,940
Structurally, how likely is it that a noun immediately

566
00:25:26,940 --> 00:25:29,619
follows an adjective? The second are emission

567
00:25:29,619 --> 00:25:32,559
probabilities. If the hidden stay is currently

568
00:25:32,559 --> 00:25:35,579
noun, how likely is it to omit the specific word

569
00:25:35,579 --> 00:25:38,319
fly compared to omitting the word apple? The

570
00:25:38,319 --> 00:25:40,180
algorithm balances the structural transitions

571
00:25:40,180 --> 00:25:42,319
against the specific vocabulary emissions to

572
00:25:42,319 --> 00:25:44,670
find the optimal path. But hold on. The math

573
00:25:44,670 --> 00:25:46,849
here seems to spiral out of control immediately.

574
00:25:47,549 --> 00:25:49,910
Let's say we have a short 10 -word sentence,

575
00:25:50,109 --> 00:25:53,630
and we have 40 possible grammatical tags. The

576
00:25:53,630 --> 00:25:55,730
number of possible tag sequences for that one

577
00:25:55,730 --> 00:25:59,369
short sentence is 40 to the power of 10. That

578
00:25:59,369 --> 00:26:03,069
is trillions of possible combinations. We surely

579
00:26:03,069 --> 00:26:05,250
cannot ask the computer to calculate the probability

580
00:26:05,250 --> 00:26:08,430
for every single one of those trillions of combinations

581
00:26:08,430 --> 00:26:11,390
just to find the highest score. You are correct.

582
00:26:11,609 --> 00:26:13,950
Calculating every path would be computationally

583
00:26:13,950 --> 00:26:16,910
crippling. This is where we rely on an extraordinary

584
00:26:16,910 --> 00:26:19,250
piece of computer science known as the Viterbi

585
00:26:19,250 --> 00:26:21,950
algorithm. Viterbi employs a strategy called

586
00:26:21,950 --> 00:26:25,130
dynamic programming to bypass the combinatorial

587
00:26:25,130 --> 00:26:27,329
explosion. How does it navigate without checking

588
00:26:27,329 --> 00:26:29,390
everything? Instead of generating full sequences,

589
00:26:29,670 --> 00:26:31,690
Viterbi builds a mathematical structure called

590
00:26:31,690 --> 00:26:34,029
a trellis, moving from left to right across the

591
00:26:34,029 --> 00:26:36,130
sentence, one word at a time. I always pictured

592
00:26:36,130 --> 00:26:38,349
the trellis like a giant sports tournament bracket.

593
00:26:39,099 --> 00:26:41,640
That is an excellent visualization. Let's walk

594
00:26:41,640 --> 00:26:44,420
through it. At word number two, the algorithm

595
00:26:44,420 --> 00:26:46,940
calculates the most probable path to arrive at

596
00:26:46,940 --> 00:26:50,079
each possible tag from word number one. And here

597
00:26:50,079 --> 00:26:52,900
is the genius of dynamic programming. Once it

598
00:26:52,900 --> 00:26:55,380
finds the best path to a specific tag at word

599
00:26:55,380 --> 00:26:58,279
two, it completely and permanently deletes all

600
00:26:58,279 --> 00:27:01,220
the inferior paths that led there. It only leaves

601
00:27:01,220 --> 00:27:03,960
a small algorithmic breadcrumb pointing backward

602
00:27:03,960 --> 00:27:06,019
to the winning predecessor. So it aggressively

603
00:27:06,019 --> 00:27:08,819
prunes the tree at every single step, doesn't

604
00:27:08,819 --> 00:27:11,900
drag any dead weight forward. Exactly. When it

605
00:27:11,900 --> 00:27:14,440
moves to word three, it only evaluates connections

606
00:27:14,440 --> 00:27:17,279
from the surviving winners of word two. By the

607
00:27:17,279 --> 00:27:19,079
time it reaches the final word of the sentence,

608
00:27:19,299 --> 00:27:21,160
it simply selects the single highest scoring

609
00:27:21,160 --> 00:27:23,579
tag at the very end and follows the breadcrumb

610
00:27:23,579 --> 00:27:26,009
trail backward through the bracket. This yields

611
00:27:26,009 --> 00:27:28,410
the absolute most mathematically probable sequence

612
00:27:28,410 --> 00:27:30,789
for the entire sentence, while only computing

613
00:27:30,789 --> 00:27:33,089
a tiny fraction of the total possible paths.

614
00:27:33,349 --> 00:27:36,210
It's brilliant. But I have to push back on the

615
00:27:36,210 --> 00:27:39,289
fundamental logic of the HMM. If the Viterbi

616
00:27:39,289 --> 00:27:42,309
algorithm is only looking at adjacent tags, like

617
00:27:42,309 --> 00:27:46,029
checking if a noun follows an adjective, how

618
00:27:46,029 --> 00:27:47,910
does it handle long -distance relationships?

619
00:27:48,859 --> 00:27:51,859
In English, I might have a plural noun at the

620
00:27:51,859 --> 00:27:53,980
very beginning of a sentence, and due to a bunch

621
00:27:53,980 --> 00:27:56,599
of intervening clauses, the plural verb that

622
00:27:56,599 --> 00:28:00,359
matches it might be 15 words away. You have pinpointed

623
00:28:00,359 --> 00:28:03,079
the fatal flaw of the basic hidden Markov model.

624
00:28:03,480 --> 00:28:06,640
It is bound by the Markov assumption, which dictates

625
00:28:06,640 --> 00:28:08,440
that the probability of the current state depends

626
00:28:08,440 --> 00:28:10,819
exclusively on the immediately preceding state.

627
00:28:10,940 --> 00:28:13,660
It has absolute amnesia regarding anything further

628
00:28:13,660 --> 00:28:16,369
back. So it totally forgets the subject by the

629
00:28:16,369 --> 00:28:18,769
time it hits the verb. Yes. You can attempt to

630
00:28:18,769 --> 00:28:21,650
fix this by building higher -order HMMs, like

631
00:28:21,650 --> 00:28:23,650
a trigram tagger that looks at the two previous

632
00:28:23,650 --> 00:28:26,150
tags, but expanding the memory exponentially

633
00:28:26,150 --> 00:28:28,869
increases the computational complexity of the

634
00:28:28,869 --> 00:28:31,069
Viterbi trellis. The tournament bracket just

635
00:28:31,069 --> 00:28:34,230
gets too large to process. Yes. This limitation

636
00:28:34,230 --> 00:28:36,509
forced the field to evolve toward conditional

637
00:28:36,509 --> 00:28:39,670
random fields and neural sequence labelers. These

638
00:28:39,670 --> 00:28:42,250
advanced architectures evaluate the entire sequence

639
00:28:42,250 --> 00:28:44,930
of input words simultaneously when scoring the

640
00:28:44,930 --> 00:28:47,710
tags, shattering the rigid local -only constraints

641
00:28:47,710 --> 00:28:51,150
of the HMM, while still utilizing a Viterbi -like

642
00:28:51,150 --> 00:28:53,829
decoding process at the very end to ensure the

643
00:28:53,829 --> 00:28:56,730
final output sequence is coherent. Okay, so we

644
00:28:56,730 --> 00:28:59,190
have successfully tagged a flat sequence of words.

645
00:28:59,579 --> 00:29:02,220
But sentences are not just flat strings of pearls.

646
00:29:02,420 --> 00:29:05,720
They have deep nested architecture. We are transitioning

647
00:29:05,720 --> 00:29:08,559
now in chapters 9 and 10 from flat sequences

648
00:29:08,559 --> 00:29:11,880
to hierarchical trees, moving into formal language

649
00:29:11,880 --> 00:29:14,359
theory. Formal language theory sits at the intersection

650
00:29:14,359 --> 00:29:16,700
of pure mathematics, computer science, and linguistics.

651
00:29:16,880 --> 00:29:19,480
It defines a language strictly as a mathematically

652
00:29:19,480 --> 00:29:21,539
defined set of strings generated by specific

653
00:29:21,539 --> 00:29:24,619
rules. And there's a rigid hierarchy of complexity,

654
00:29:24,920 --> 00:29:27,470
often called the Chomsky hierarchy. The simplest

655
00:29:27,470 --> 00:29:30,009
level is regular languages. Regular languages

656
00:29:30,009 --> 00:29:32,309
are basically patterns you can capture with simple

657
00:29:32,309 --> 00:29:34,589
regular expressions, and Eisenstein highlights

658
00:29:34,589 --> 00:29:37,289
Spanish morphology as a perfect use case for

659
00:29:37,289 --> 00:29:39,589
this simple level of computing. Morphology studies

660
00:29:39,589 --> 00:29:41,930
how individual words are constructed from smaller

661
00:29:41,930 --> 00:29:45,390
units. Take the Spanish word canto. Depending

662
00:29:45,390 --> 00:29:48,349
on the context, it can translate to a song, operating

663
00:29:48,349 --> 00:29:51,359
as a masculine singular noun. Alternatively,

664
00:29:51,539 --> 00:29:53,619
it can function as the first person present indicative

665
00:29:53,619 --> 00:29:56,960
verb, I sing. A computational system known as

666
00:29:56,960 --> 00:30:00,880
a finite state transducer, or FST, can efficiently

667
00:30:00,880 --> 00:30:04,119
map the surface string canto to these two distinct

668
00:30:04,119 --> 00:30:06,599
grammatical realities. How does it do that? It

669
00:30:06,599 --> 00:30:08,400
does this by transitioning through a series of

670
00:30:08,400 --> 00:30:11,980
simple linear states based on the letters. FSTs

671
00:30:11,980 --> 00:30:14,460
are incredibly lightweight and blazingly fast

672
00:30:14,460 --> 00:30:16,940
for this kind of linear word building. They're

673
00:30:16,940 --> 00:30:20,359
great for building words. But they completely

674
00:30:20,359 --> 00:30:22,480
collapse when trying to understand English syntax.

675
00:30:23,140 --> 00:30:25,460
Eisenstein provides this wondrously brain -melting

676
00:30:25,460 --> 00:30:27,839
example of center embedding to prove exactly

677
00:30:27,839 --> 00:30:29,940
why a machine needs a more complex system than

678
00:30:29,940 --> 00:30:32,380
a regular language to read a book. Center embedding

679
00:30:32,380 --> 00:30:34,859
reveals the recursive nature of human grammar.

680
00:30:35,039 --> 00:30:38,720
A simple, linear FST can easily parse a sentence

681
00:30:38,720 --> 00:30:41,960
like, the dog chased the cat. It expects a noun,

682
00:30:42,119 --> 00:30:45,319
then a verb. Then a noun. But English allows

683
00:30:45,319 --> 00:30:47,700
you to embed a completely new relative clause

684
00:30:47,700 --> 00:30:50,200
right in the middle of that structure. The dog

685
00:30:50,200 --> 00:30:52,700
the cat kissed chased the goat. And theoretically,

686
00:30:52,920 --> 00:30:54,859
you can keep nesting them like Russian nesting

687
00:30:54,859 --> 00:30:58,059
dolls. The dog the cat goat chased kissed. It

688
00:30:58,059 --> 00:31:00,160
sounds like absolute nonsense to the human ear

689
00:31:00,160 --> 00:31:02,099
because our biological working memory fails.

690
00:31:02,599 --> 00:31:05,119
But grammatically, the syntax is perfectly valid.

691
00:31:05,279 --> 00:31:07,500
The nouns and verbs are mathematically nested

692
00:31:07,500 --> 00:31:09,500
inside one another. And that nesting destroys

693
00:31:09,500 --> 00:31:12,420
the finite state transducer. The mechanism driving

694
00:31:12,420 --> 00:31:15,200
regular languages has no internal memory. As

695
00:31:15,200 --> 00:31:18,240
it reads the sequence of nouns dog, cat, goat,

696
00:31:18,339 --> 00:31:20,640
it has no capacity to remember how many nouns

697
00:31:20,640 --> 00:31:23,160
it has seen, which means it has no idea how many

698
00:31:23,160 --> 00:31:25,519
corresponding verbs it needs to expect later

699
00:31:25,519 --> 00:31:27,759
in the sentence to resolve the grammar. To capture

700
00:31:27,759 --> 00:31:30,539
those nested Russian dolls, we must ascend the

701
00:31:30,539 --> 00:31:33,019
complexity hierarchy to context -free grammars,

702
00:31:33,019 --> 00:31:36,759
or CFGs. A context -free grammar relies on explicit

703
00:31:36,759 --> 00:31:39,750
substitution rules. It dictates that a sentence

704
00:31:39,750 --> 00:31:42,390
can be broken down into a noun phrase and a verb

705
00:31:42,390 --> 00:31:45,529
phrase. And crucially, a noun phrase can contain

706
00:31:45,529 --> 00:31:48,809
an entirely new sentence inside of it. That recursive

707
00:31:48,809 --> 00:31:50,869
loop is what allows the machine to build the

708
00:31:50,869 --> 00:31:54,009
hierarchical tree. To process these trees algorithmically,

709
00:31:54,109 --> 00:31:56,650
computer scientists often force the grammar into

710
00:31:56,650 --> 00:31:59,990
Chomsky normal form. This is a mathematical restriction

711
00:31:59,990 --> 00:32:03,009
where every grammatical rule must branch into

712
00:32:03,009 --> 00:32:06,329
exactly two non -terminal categories or terminate

713
00:32:06,329 --> 00:32:09,349
into exactly one vocabulary word. It forces the

714
00:32:09,349 --> 00:32:12,450
messy syntax into a pristine binary branching

715
00:32:12,450 --> 00:32:14,809
tree. But how does the computer actually build

716
00:32:14,809 --> 00:32:17,230
that tree from a raw sentence? We talked about

717
00:32:17,230 --> 00:32:20,049
Viterbi for flat sequences, but building a hierarchical

718
00:32:20,049 --> 00:32:22,829
tree requires the CKY algorithm. How does this

719
00:32:22,829 --> 00:32:24,910
algorithm navigate the geometry of a sentence?

720
00:32:25,319 --> 00:32:27,759
If Viterbi operates like a left -to -right tournament

721
00:32:27,759 --> 00:32:31,019
bracket, the Kot -Kasami -Younger or CKY algorithm

722
00:32:31,019 --> 00:32:34,240
operates as a bottom -up combinatorial grid search.

723
00:32:34,500 --> 00:32:36,920
It starts at the very bottom, looking only at

724
00:32:36,920 --> 00:32:39,359
the individual words. It asks the grammar rules,

725
00:32:39,599 --> 00:32:42,319
can the word the and the word dog combine to

726
00:32:42,319 --> 00:32:45,099
form a valid noun phrase? The rules say yes,

727
00:32:45,299 --> 00:32:48,460
so it builds a tiny subtree. Then it asks, can

728
00:32:48,460 --> 00:32:52,140
dog and chaste combine? The rules say no, so

729
00:32:52,140 --> 00:32:54,970
it ignores them. So it scans all the pairs, then

730
00:32:54,970 --> 00:32:57,029
it moves up a level and scans all the triplets

731
00:32:57,029 --> 00:32:59,150
of words, seeing if those underlying subtrees

732
00:32:59,150 --> 00:33:01,750
can be merged into bigger branches. Systematically.

733
00:33:01,849 --> 00:33:04,589
It fills a massive triangular matrix, combining

734
00:33:04,589 --> 00:33:07,190
increasingly larger spans of words until it reaches

735
00:33:07,190 --> 00:33:09,569
the final cell at the top of the grid, which

736
00:33:09,569 --> 00:33:11,170
represents the entire length of the sentence.

737
00:33:11,490 --> 00:33:13,730
If that final cell contains a valid sentence

738
00:33:13,730 --> 00:33:16,250
root node, the algorithm has successfully parsed

739
00:33:16,250 --> 00:33:18,710
the grammatical tree. It is an exhaustive search

740
00:33:18,710 --> 00:33:22,230
of the syntactic space. But in chapter 11, Eisenstein

741
00:33:22,230 --> 00:33:24,250
points out that building these massive nested

742
00:33:24,250 --> 00:33:26,950
boxes of phrases isn't the only way to map a

743
00:33:26,950 --> 00:33:30,130
sentence. We also have dependency parsing. Phrase

744
00:33:30,130 --> 00:33:33,039
structure trees. The binary trees CKY builds

745
00:33:33,039 --> 00:33:36,880
focus on abstract constituent boxes. Dependency

746
00:33:36,880 --> 00:33:39,759
parsing takes a much more direct approach. Instead

747
00:33:39,759 --> 00:33:42,480
of abstract nodes, it draws a directed mathematical

748
00:33:42,480 --> 00:33:44,920
graph of relationships connecting the actual

749
00:33:44,920 --> 00:33:47,660
words themselves. In this framework, verbs act

750
00:33:47,660 --> 00:33:49,559
as the structural center of gravity pointing

751
00:33:49,559 --> 00:33:52,579
outward to their subjects and objects. So instead

752
00:33:52,579 --> 00:33:55,119
of abstractly declaring that the dog is a noun

753
00:33:55,119 --> 00:33:58,059
phrase sitting in a larger box, dependency parsing

754
00:33:58,059 --> 00:34:00,660
just draws a literal arrow. originating from

755
00:34:00,660 --> 00:34:02,960
the verb chased, pointing directly to the word

756
00:34:02,960 --> 00:34:06,480
dog, and ladles that arrow subject. It feels

757
00:34:06,480 --> 00:34:09,139
vastly more intuitive. It maps directly to how

758
00:34:09,139 --> 00:34:10,800
we actually think about action, who did what

759
00:34:10,800 --> 00:34:13,739
to whom. It is significantly lighter computationally,

760
00:34:13,760 --> 00:34:15,940
and because it explicitly defines the relationships

761
00:34:15,940 --> 00:34:18,719
between entities, it is often far more useful

762
00:34:18,719 --> 00:34:21,000
for downstream engineering tasks, regardless

763
00:34:21,000 --> 00:34:22,800
of how the words are physically ordered in the

764
00:34:22,800 --> 00:34:25,679
sentence. Which brings us to a massive philosophical

765
00:34:25,679 --> 00:34:29,659
pivot in our deep dive in part three. A perfectly

766
00:34:29,659 --> 00:34:32,340
constructed sentence diagram, whether it is a

767
00:34:32,340 --> 00:34:35,219
beautifully nested CKY tree or an elegant dependency

768
00:34:35,219 --> 00:34:37,800
graph, is a staggering technical achievement

769
00:34:37,800 --> 00:34:40,679
of search and learning. But syntax is not semantics.

770
00:34:41,219 --> 00:34:43,219
Just because the computer knows that dog is the

771
00:34:43,219 --> 00:34:45,260
subject of the verb chased does not mean the

772
00:34:45,260 --> 00:34:47,019
computer has any idea what the sentence actually

773
00:34:47,019 --> 00:34:49,280
means, which leads us into the quest for meaning.

774
00:34:49,750 --> 00:34:52,110
Semantics forces the discipline to confront true

775
00:34:52,110 --> 00:34:54,530
comprehension. And the historical starting point

776
00:34:54,530 --> 00:34:56,949
for this in chapter 12 is logical semantics,

777
00:34:57,150 --> 00:34:59,730
which aims to translate the fluid ambiguity of

778
00:34:59,730 --> 00:35:02,269
human language into the rigid, unambiguous structure

779
00:35:02,269 --> 00:35:04,690
of formal mathematical logic. For those of you

780
00:35:04,690 --> 00:35:07,190
who code, I relate to this perfectly through

781
00:35:07,190 --> 00:35:09,550
programming. We want to translate English into

782
00:35:09,550 --> 00:35:12,670
something like Python or SQL. Eisenstein uses

783
00:35:12,670 --> 00:35:14,949
a great mathematical analogy. The text string

784
00:35:14,949 --> 00:35:18,130
5 plus 3 and the text string 4, 3, 3 plus 1 look

785
00:35:18,130 --> 00:35:20,329
completely different. But they have the exact

786
00:35:20,329 --> 00:35:23,130
same denotation. If you evaluate them, they both

787
00:35:23,130 --> 00:35:25,530
equal 8. We want to do the exact same thing with

788
00:35:25,530 --> 00:35:27,670
English sentences. We want the sentence, the

789
00:35:27,670 --> 00:35:29,929
canine chased the feline, and the sentence, the

790
00:35:29,929 --> 00:35:32,849
cat was pursued by the dog, to compile down into

791
00:35:32,849 --> 00:35:35,309
the exact same logical reality. To achieve that

792
00:35:35,309 --> 00:35:38,150
denotation, logical semantics relies heavily

793
00:35:38,150 --> 00:35:41,230
on the principle of compositionality. This is

794
00:35:41,230 --> 00:35:43,369
the axiom that the meaning of a complex expression

795
00:35:43,369 --> 00:35:45,369
is entirely determined by the meanings of its

796
00:35:45,369 --> 00:35:47,769
constituent parts and the grammatical rules used

797
00:35:47,769 --> 00:35:49,780
to combine them. But how do you mathematically

798
00:35:49,780 --> 00:35:52,320
combine the meaning of a noun and a verb? The

799
00:35:52,320 --> 00:35:54,400
primary mechanism detailed in the text is the

800
00:35:54,400 --> 00:35:57,239
lambda calculus. Let us consider the simple sentence,

801
00:35:57,400 --> 00:36:00,440
Alex likes brick. The entities are straightforward

802
00:36:00,440 --> 00:36:03,900
constants, Alex and Brit. The complexity lies

803
00:36:03,900 --> 00:36:07,159
in the verb likes. In the lambda calculus, the

804
00:36:07,159 --> 00:36:10,340
verb phrase likes Brit is not a static object.

805
00:36:10,440 --> 00:36:12,579
It is represented as a mathematical function

806
00:36:12,579 --> 00:36:15,050
waiting for an input. For the listener, imagine

807
00:36:15,050 --> 00:36:17,949
the verb phrase as a wooden puzzle piece with

808
00:36:17,949 --> 00:36:21,050
a very specifically shaped empty slot. The puzzle

809
00:36:21,050 --> 00:36:23,750
piece represents the concept blank likes Brit.

810
00:36:23,929 --> 00:36:26,469
That is an excellent visualization. Mathematically,

811
00:36:26,650 --> 00:36:28,610
it is a function that requires a single argument.

812
00:36:28,829 --> 00:36:31,110
When the syntactic tree combines the subject

813
00:36:31,110 --> 00:36:34,050
Alex with the verb phrase likes Brit, the math

814
00:36:34,050 --> 00:36:36,590
literally plugs the constant Alex into the empty

815
00:36:36,590 --> 00:36:39,090
slot of the function. The function executes,

816
00:36:39,130 --> 00:36:41,889
resulting in a final, unambiguous logical predicate.

817
00:36:42,269 --> 00:36:44,929
Likes Alex Britt. And suddenly you have a discrete,

818
00:36:45,170 --> 00:36:47,010
true or false statement about the state of the

819
00:36:47,010 --> 00:36:49,050
world that a database can query and understand.

820
00:36:49,329 --> 00:36:51,570
It provides absolute precision. It's incredibly

821
00:36:51,570 --> 00:36:54,010
elegant on a whiteboard. But practically, it

822
00:36:54,010 --> 00:36:56,409
seems incredibly brittle. Someone has to sit

823
00:36:56,409 --> 00:36:58,670
in the lab and manually write the lambda calculus

824
00:36:58,670 --> 00:37:01,630
functions for every single verb, preposition,

825
00:37:01,750 --> 00:37:04,599
and adjective in the English language. And what

826
00:37:04,599 --> 00:37:07,639
happens when a human uses sarcasm? What happens

827
00:37:07,639 --> 00:37:10,369
if a word has a double meaning? The rigid rules

828
00:37:10,369 --> 00:37:13,389
of formal logic must shatter upon contact with

829
00:37:13,389 --> 00:37:15,869
real -world conversational data. They absolutely

830
00:37:15,869 --> 00:37:18,889
do. Logical semantics struggles immensely with

831
00:37:18,889 --> 00:37:21,809
ambiguity and scale. Which brings us to Chapter

832
00:37:21,809 --> 00:37:24,349
14 and the counter -revolution in how machines

833
00:37:24,349 --> 00:37:27,510
process meaning, Distributional Semantics. If

834
00:37:27,510 --> 00:37:29,670
logical semantics is a rigid dictionary of rules,

835
00:37:29,949 --> 00:37:32,269
distributional semantics is fluid sociology.

836
00:37:32,650 --> 00:37:34,570
It operates on the distributional hypothesis,

837
00:37:35,050 --> 00:37:37,489
best summarized by the famous linguist John Rupert

838
00:37:37,489 --> 00:37:42,190
Firth, who stated, I find this concept deeply

839
00:37:42,190 --> 00:37:44,849
fascinating. Under this theory, you don't define

840
00:37:44,849 --> 00:37:46,869
a word by giving it a rigid logical definition.

841
00:37:47,150 --> 00:37:49,769
You define it entirely by observing the other

842
00:37:49,769 --> 00:37:52,829
words that surround it in natural text. If I

843
00:37:52,829 --> 00:37:54,409
introduce a mystery word to you, let's say the

844
00:37:54,409 --> 00:37:56,780
word is Barty. And I tell you that people frequently

845
00:37:56,780 --> 00:37:59,320
say, I drank a glass of Barty, the Barty was

846
00:37:59,320 --> 00:38:01,739
fermented in oak, or I bought a bottle of red

847
00:38:01,739 --> 00:38:04,719
Barty. You instantly know exactly what Barty

848
00:38:04,719 --> 00:38:07,579
means without me ever explicitly defining wine.

849
00:38:07,860 --> 00:38:11,400
The Conteld reveals the meaning. Computationally,

850
00:38:11,400 --> 00:38:14,219
this insight gave rise to neural word embeddings,

851
00:38:14,340 --> 00:38:18,190
most notably the word Tuvec models. Instead of

852
00:38:18,190 --> 00:38:20,769
treating words as discrete, isolated symbols,

853
00:38:21,130 --> 00:38:23,730
the neural network maps every single word into

854
00:38:23,730 --> 00:38:26,650
a dense, multidimensional geometric space. We

855
00:38:26,650 --> 00:38:28,469
are turning words into physical coordinates.

856
00:38:28,849 --> 00:38:31,460
Yes. And because the network trains by observing

857
00:38:31,460 --> 00:38:33,739
the surrounding context, words that frequently

858
00:38:33,739 --> 00:38:36,280
appear in similar contexts are physically pushed

859
00:38:36,280 --> 00:38:38,920
closer together in that geometric space. So the

860
00:38:38,920 --> 00:38:40,980
coordinate for coffee and the coordinate for

861
00:38:40,980 --> 00:38:43,800
tea end up geometrically right next to each other

862
00:38:43,800 --> 00:38:45,820
because they both frequently appear near words

863
00:38:45,820 --> 00:38:49,440
like cup, drink, and hot. Precisely. The text

864
00:38:49,440 --> 00:38:52,019
details the two primary neural architectures

865
00:38:52,019 --> 00:38:54,659
used to construct this space, continuous bag

866
00:38:54,659 --> 00:38:59,230
of words, or CBOW, and skip grams. In the CBOW

867
00:38:59,230 --> 00:39:01,250
architecture, the network looks at the surrounding

868
00:39:01,250 --> 00:39:03,909
context words and attempts to predict the missing

869
00:39:03,909 --> 00:39:06,809
target word in the middle. It operates like a

870
00:39:06,809 --> 00:39:09,309
massive fill -in -the -blank test. The Stipgram

871
00:39:09,309 --> 00:39:11,929
architecture inverts the process. You provide

872
00:39:11,929 --> 00:39:13,670
the network with the target word, and it must

873
00:39:13,670 --> 00:39:15,690
predict the context words that are statistically

874
00:39:15,690 --> 00:39:18,329
likely to surround it. But here is where we hit

875
00:39:18,329 --> 00:39:21,199
a massive computational wall. If you are running

876
00:39:21,199 --> 00:39:23,340
a skip gram model and you feed it the word whale,

877
00:39:23,539 --> 00:39:26,059
it has to calculate a probability distribution

878
00:39:26,059 --> 00:39:28,900
over the entire vocabulary of the English language

879
00:39:28,900 --> 00:39:32,079
to guess which words might be nearby. If your

880
00:39:32,079 --> 00:39:34,559
vocabulary contained 100 ,000 words computing

881
00:39:34,559 --> 00:39:36,659
the exact mathematical probability, specifically

882
00:39:36,659 --> 00:39:39,960
running the SoftBanks function, for all 100 ,000

883
00:39:39,960 --> 00:39:42,559
words, every single time you process a single

884
00:39:42,559 --> 00:39:45,099
word in your training data would grind any supercomputer

885
00:39:45,099 --> 00:39:48,610
to a halt. it is an immense computational bottleneck

886
00:39:48,610 --> 00:39:51,309
evaluating the denominator of that probability

887
00:39:51,309 --> 00:39:54,150
function across a massive vocabulary is mathematically

888
00:39:54,150 --> 00:39:57,210
grueling to solve this eisenstein highlights

889
00:39:57,210 --> 00:40:00,849
two highly ingenious engineering shortcuts the

890
00:40:00,849 --> 00:40:03,750
first is hierarchical softmax how does that bypass

891
00:40:03,750 --> 00:40:06,639
the math Instead of treating the 100 ,000 word

892
00:40:06,639 --> 00:40:09,239
vocabulary as a flat linear list that must be

893
00:40:09,239 --> 00:40:11,880
calculated individually, Hierarchical Softmax

894
00:40:11,880 --> 00:40:14,260
arranges all the words into the terminal leaves

895
00:40:14,260 --> 00:40:16,679
of a massive binary tree. For you listening.

896
00:40:17,059 --> 00:40:19,360
Imagine organizing the entire English language

897
00:40:19,360 --> 00:40:22,340
into a giant game of 20 questions. That perfectly

898
00:40:22,340 --> 00:40:24,639
describes the mechanism. To calculate the probability

899
00:40:24,639 --> 00:40:27,480
of the word blubber appearing near whale, the

900
00:40:27,480 --> 00:40:29,840
network no longer computes 100 ,000 separate

901
00:40:29,840 --> 00:40:32,239
probabilities. It simply navigates down the binary

902
00:40:32,239 --> 00:40:34,719
tree. Is the word on the left half or right half

903
00:40:34,719 --> 00:40:38,599
of the tree? Left, top half, or bottom? The network

904
00:40:38,599 --> 00:40:41,639
only has to make a short sequence of binary decisions.

905
00:40:41,860 --> 00:40:43,980
Oh, wow. Mathematically, the calculation time

906
00:40:43,980 --> 00:40:46,099
changes from scaling linearly with the size of

907
00:40:46,099 --> 00:40:49,179
the vocabulary to scaling logarithmically. It

908
00:40:49,179 --> 00:40:51,480
transforms an impossible calculation into something

909
00:40:51,480 --> 00:40:54,300
that runs in milliseconds. That is incredibly

910
00:40:54,300 --> 00:40:57,260
clever. But the second solution is even sneakier.

911
00:40:57,340 --> 00:40:59,769
It's called negative sampling. Negative sampling

912
00:40:59,769 --> 00:41:02,130
is a brilliant evasion of the probability calculation

913
00:41:02,130 --> 00:41:05,110
entirely. Instead of attempting to predict the

914
00:41:05,110 --> 00:41:07,510
exact probability distribution across the whole

915
00:41:07,510 --> 00:41:10,489
vocabulary, the network frames it as a simple

916
00:41:10,489 --> 00:41:13,489
binary classification task. It takes the true

917
00:41:13,489 --> 00:41:16,010
pair of words, for example, whale and blubber,

918
00:41:16,090 --> 00:41:18,909
and pairs it with a handful of randomly generated

919
00:41:18,909 --> 00:41:22,610
negative pairs, like whale and calculator or

920
00:41:22,610 --> 00:41:25,849
whale and spaghetti. It just invents fake data.

921
00:41:26,050 --> 00:41:28,889
Yes. And it trains the neural network on a very

922
00:41:28,889 --> 00:41:31,889
simple objective. Mathematically increase the

923
00:41:31,889 --> 00:41:34,369
score for the true pair and decrease the score

924
00:41:34,369 --> 00:41:36,849
for the handful of fake pairs. By optimizing

925
00:41:36,849 --> 00:41:39,250
against just 5 or 10 negative samples instead

926
00:41:39,250 --> 00:41:42,889
of 100 ,000 vocabulary words, it completely sidesteps

927
00:41:42,889 --> 00:41:45,530
the massive computational bottleneck while still

928
00:41:45,530 --> 00:41:47,530
effectively pushing the word vectors into their

929
00:41:47,530 --> 00:41:50,030
correct geometric alignment. So we have successfully

930
00:41:50,030 --> 00:41:53,190
mapped individual words into this beautiful continuous

931
00:41:53,190 --> 00:41:56,809
geometric space. But human comprehension isn't

932
00:41:56,809 --> 00:41:59,050
limited to isolated words or single sentences.

933
00:41:59,289 --> 00:42:02,150
We read entire novels. The machine has to learn

934
00:42:02,150 --> 00:42:04,670
how to keep track of a narrative over time. This

935
00:42:04,670 --> 00:42:07,989
introduces us to chapters 15 and 16 on discourse

936
00:42:07,989 --> 00:42:10,389
and reference resolution. Coreference resolution

937
00:42:10,389 --> 00:42:13,130
is the classic test of multi -sentence comprehension.

938
00:42:13,710 --> 00:42:16,650
If a text reads, Abigail dropped the fragile

939
00:42:16,650 --> 00:42:18,889
glass. She was deeply upset because it shattered.

940
00:42:19,400 --> 00:42:21,559
The machine must deduce that the pronoun she

941
00:42:21,559 --> 00:42:24,780
refers back to Abigail and the pronoun it refers

942
00:42:24,780 --> 00:42:27,440
to the glass. As humans, we resolve those pronouns

943
00:42:27,440 --> 00:42:30,119
flawlessly and instinctively without even noticing

944
00:42:30,119 --> 00:42:32,980
we are doing it. But for a machine, the text

945
00:42:32,980 --> 00:42:34,900
string Abigail, the string she, and the string

946
00:42:34,900 --> 00:42:37,059
her are completely different data points. How

947
00:42:37,059 --> 00:42:38,820
does it learn to mathematically link them together

948
00:42:38,820 --> 00:42:41,260
into a single identity? Eisenstein explains a

949
00:42:41,260 --> 00:42:43,360
fascinating structural approach called the Bell

950
00:42:43,360 --> 00:42:46,230
Tree. This model frames corference resolution

951
00:42:46,230 --> 00:42:49,909
not as static labeling, but as a dynamic sequential

952
00:42:49,909 --> 00:42:53,170
decision -making process. As the algorithm reads

953
00:42:53,170 --> 00:42:55,469
through a document left to right, every time

954
00:42:55,469 --> 00:42:57,889
it encounters a new mention, a noun or a pronoun,

955
00:42:58,190 --> 00:43:01,090
it faces a structural choice. Does it link this

956
00:43:01,090 --> 00:43:03,150
new mention to an existing cluster of entities

957
00:43:03,150 --> 00:43:05,590
it has already built, or does it branch off and

958
00:43:05,590 --> 00:43:08,130
establish a brand new cluster? So when it encounters

959
00:43:08,130 --> 00:43:10,840
the word she, It evaluates the Abigail cluster,

960
00:43:11,059 --> 00:43:13,679
the Glass cluster, and an empty new entity cluster

961
00:43:13,679 --> 00:43:16,619
and makes a choice. But how do you train a model

962
00:43:16,619 --> 00:43:19,099
to make a long chain of choices? You cannot easily

963
00:43:19,099 --> 00:43:21,480
train that with standard supervised learning.

964
00:43:21,760 --> 00:43:25,099
Instead, modern NLP relies on reinforcement learning,

965
00:43:25,300 --> 00:43:27,900
specifically techniques like policy gradients.

966
00:43:28,099 --> 00:43:31,079
The system is not evaluating isolated classification

967
00:43:31,079 --> 00:43:34,340
decisions. It is attempting to learn an overarching

968
00:43:34,340 --> 00:43:37,559
policy for navigating the bell tree. The model

969
00:43:37,559 --> 00:43:39,760
plays through the document. making a sequence

970
00:43:39,760 --> 00:43:42,519
of clustering decisions. If the final resulting

971
00:43:42,519 --> 00:43:44,460
clusters at the end of the document are highly

972
00:43:44,460 --> 00:43:46,639
accurate, the system receives a mathematical

973
00:43:46,639 --> 00:43:49,679
reward and the specific neural pathways that

974
00:43:49,679 --> 00:43:51,460
led to those correct decisions are strengthened.

975
00:43:51,679 --> 00:43:54,400
We are back to the board game analogy. The machine

976
00:43:54,400 --> 00:43:57,119
is playing a game of entity clustering, evaluating

977
00:43:57,119 --> 00:44:00,099
its moves, and optimizing its strategy to maximize

978
00:44:00,099 --> 00:44:02,800
its final score. Exactly. It learns a policy

979
00:44:02,800 --> 00:44:05,579
of comprehension through trial, error, and optimization.

980
00:44:06,059 --> 00:44:08,590
So if we step back, And look at the whole picture

981
00:44:08,590 --> 00:44:11,269
for a moment in part four. We have covered how

982
00:44:11,269 --> 00:44:14,829
a computer reduces text to numerical frequencies,

983
00:44:15,130 --> 00:44:18,309
how it discovers hidden structures, how it navigates

984
00:44:18,309 --> 00:44:20,690
the viterity bracket to assign grammar, how it

985
00:44:20,690 --> 00:44:23,329
builds massive syntactic trees, how it forces

986
00:44:23,329 --> 00:44:26,230
words into geometric spaces, and how it resolves

987
00:44:26,230 --> 00:44:28,969
narrative identities. What do we actually do

988
00:44:28,969 --> 00:44:30,989
with all this theoretical machinery? What are

989
00:44:30,989 --> 00:44:33,969
the grand applications? This is where the science

990
00:44:33,969 --> 00:44:36,440
transitions into applied engineering. One of

991
00:44:36,440 --> 00:44:38,980
the most vital applications is information extraction.

992
00:44:39,519 --> 00:44:43,059
The primary objective here is taking the unstructured,

993
00:44:43,099 --> 00:44:45,820
chaotic prose of the internet, medical journals

994
00:44:45,820 --> 00:44:48,340
or corporate archives, and distilling it into

995
00:44:48,340 --> 00:44:50,860
clean, highly structured relational databases.

996
00:44:51,239 --> 00:44:53,400
For example, turning a massive rambling Wikipedia

997
00:44:53,400 --> 00:44:56,039
article about a film into a crisp spreadsheet

998
00:44:56,039 --> 00:44:58,739
that explicitly lists the director, the lead

999
00:44:58,739 --> 00:45:01,119
actor, and the release year. To accomplish that,

1000
00:45:01,239 --> 00:45:04,059
the NLP pipeline executes multiple discrete steps.

1001
00:45:04,599 --> 00:45:07,619
First, it performs entity linking, ensuring that

1002
00:45:07,619 --> 00:45:10,340
the text string Washington in the document explicitly

1003
00:45:10,340 --> 00:45:13,219
maps to the database entry for George Washington

1004
00:45:13,219 --> 00:45:15,519
the person, rather than Washington the state.

1005
00:45:16,019 --> 00:45:19,360
Next, it performs relation extraction, predicting

1006
00:45:19,360 --> 00:45:21,719
the specific logical relationship connecting

1007
00:45:21,719 --> 00:45:24,389
two entities within a sentence. And how does

1008
00:45:24,389 --> 00:45:27,329
it train that? I recall the text mentioning pairwise

1009
00:45:27,329 --> 00:45:29,150
ranking loss functions, but that terminology

1010
00:45:29,150 --> 00:45:32,250
sounds incredibly opaque. How does the math actually

1011
00:45:32,250 --> 00:45:35,309
work to extract a fact? A pairwise ranking loss

1012
00:45:35,309 --> 00:45:37,610
function operates conceptually like a mathematical

1013
00:45:37,610 --> 00:45:40,369
scale. Imagine the machine is evaluating the

1014
00:45:40,369 --> 00:45:42,929
entities George Washington and Virginia. It must

1015
00:45:42,929 --> 00:45:45,590
determine their relationship. The model evaluates

1016
00:45:45,590 --> 00:45:48,190
the true relationship born in and assigns it

1017
00:45:48,190 --> 00:45:50,630
a score. It then evaluates a false relationship,

1018
00:45:50,829 --> 00:45:53,610
say, employed by, and assigns it a score. The

1019
00:45:53,610 --> 00:45:55,550
loss function simply demands that the score for

1020
00:45:55,550 --> 00:45:57,630
the true relationship must remain significantly

1021
00:45:57,630 --> 00:45:59,750
heavier ranked higher than the score for the

1022
00:45:59,750 --> 00:46:01,949
false relationship. Like tipping a scale. Yes.

1023
00:46:02,329 --> 00:46:04,670
By constantly adjusting its internal weights

1024
00:46:04,670 --> 00:46:06,869
to keep the true facts heavier on the scale,

1025
00:46:07,050 --> 00:46:10,150
the model learns to identify and extract verifiable

1026
00:46:10,150 --> 00:46:12,849
relationships directly from unstructured prose.

1027
00:46:13,320 --> 00:46:15,840
That is a phenomenal way to visualize it. Moving

1028
00:46:15,840 --> 00:46:17,900
to another massive application in Chapter 18,

1029
00:46:18,039 --> 00:46:20,719
we have machine translation. This has famously

1030
00:46:20,719 --> 00:46:22,860
been the holy grail of artificial intelligence

1031
00:46:22,860 --> 00:46:25,360
since the Cold War. It has driven massive amounts

1032
00:46:25,360 --> 00:46:28,280
of funding and research for decades. Eisenstein

1033
00:46:28,280 --> 00:46:30,500
roots the discussion of translation by examining

1034
00:46:30,500 --> 00:46:33,619
IBM Model 1, a truly foundational statistical

1035
00:46:33,619 --> 00:46:36,460
model. What makes this model so conceptually

1036
00:46:36,460 --> 00:46:38,739
important is how it utilizes the expectation

1037
00:46:38,739 --> 00:46:40,980
maximization algorithm we discussed earlier.

1038
00:46:41,369 --> 00:46:43,789
The soft probability dance. How does clustering

1039
00:46:43,789 --> 00:46:46,610
apply to translation? Imagine you possess a massive

1040
00:46:46,610 --> 00:46:49,369
corpus of parallel texts, like the Canadian Parliament

1041
00:46:49,369 --> 00:46:51,809
transcripts, where you have the exact same speeches

1042
00:46:51,809 --> 00:46:53,849
in English and French. You know that sentence

1043
00:46:53,849 --> 00:46:56,670
A in English corresponds to sentence B in French.

1044
00:46:56,929 --> 00:46:59,269
However, you do not possess a bilingual dictionary.

1045
00:46:59,570 --> 00:47:02,190
You have no idea which specific English word

1046
00:47:02,190 --> 00:47:04,789
aligns with which specific French word. It's

1047
00:47:04,789 --> 00:47:07,250
the chicken and egg paradox again. If I have

1048
00:47:07,250 --> 00:47:09,329
a dictionary, I can align the words perfectly.

1049
00:47:09,769 --> 00:47:11,889
If I have the words aligned perfectly, I can

1050
00:47:11,889 --> 00:47:14,630
build the dictionary. But I have neither. So

1051
00:47:14,630 --> 00:47:18,269
IBM Model 1 executes the EM dance. In the E step,

1052
00:47:18,469 --> 00:47:20,929
the model generates entirely random, terrible

1053
00:47:20,929 --> 00:47:23,909
guesses regarding word alignments. It might guess

1054
00:47:23,909 --> 00:47:27,510
that house translates to bonjour. In the M step,

1055
00:47:27,690 --> 00:47:29,650
it uses those terrible alignments to estimate

1056
00:47:29,650 --> 00:47:31,730
the actual mathematical translation probabilities

1057
00:47:31,730 --> 00:47:34,889
across the entire corpus. Then it iterates. And

1058
00:47:34,889 --> 00:47:37,679
it fixes itself over time. Yes. Without any human

1059
00:47:37,679 --> 00:47:40,119
supervision or hard -coded dictionaries, the

1060
00:47:40,119 --> 00:47:42,360
mathematics eventually converge on the undeniable

1061
00:47:42,360 --> 00:47:44,519
statistical reality that the English word house

1062
00:47:44,519 --> 00:47:47,340
and the French word maison co -occur in parallel

1063
00:47:47,340 --> 00:47:50,340
sentences with overwhelming frequency. The alignment

1064
00:47:50,340 --> 00:47:52,960
resolves itself purely through statistical maximization.

1065
00:47:53,199 --> 00:47:55,519
That is awe -inspiring. It just mathematically

1066
00:47:55,519 --> 00:47:58,079
inevitably discovers the translation. Finally,

1067
00:47:58,119 --> 00:48:00,260
the text pivots to the ultimate frontier in Chapter

1068
00:48:00,260 --> 00:48:03,519
19, text generation and dialogue. Everything

1069
00:48:03,519 --> 00:48:06,579
so far has been about ingesting data. Now we

1070
00:48:06,579 --> 00:48:08,639
are going from data back into natural language

1071
00:48:08,639 --> 00:48:11,599
output. If parsing syntax is like taking a car

1072
00:48:11,599 --> 00:48:13,780
engine apart to understand how it works, generation

1073
00:48:13,780 --> 00:48:16,139
is attempting to build a functioning engine from

1074
00:48:16,139 --> 00:48:19,239
a pile of spare parts. And dialogue systems represent

1075
00:48:19,239 --> 00:48:22,309
the absolute apex of that complexity. Because

1076
00:48:22,309 --> 00:48:24,449
in a dialogue system, you are not simply generating

1077
00:48:24,449 --> 00:48:27,909
text into a static void. You are actively interacting

1078
00:48:27,909 --> 00:48:30,949
with an unpredictable human being. To model this

1079
00:48:30,949 --> 00:48:33,010
mathematically, engineers frame the conversation

1080
00:48:33,010 --> 00:48:36,650
as a Markov decision process, or MDP. Another

1081
00:48:36,650 --> 00:48:38,730
Markov process. We talked about the hidden Markov

1082
00:48:38,730 --> 00:48:40,809
model earlier for tagging. But this is a Markov

1083
00:48:40,809 --> 00:48:43,050
decision process. How does the math differ here?

1084
00:48:43,250 --> 00:48:45,849
In an MDP, the machine is not merely labeling

1085
00:48:45,849 --> 00:48:49,190
text. It is an active agent. The system occupies

1086
00:48:49,190 --> 00:48:51,860
a specific state. For instance, it understands

1087
00:48:51,860 --> 00:48:54,119
the user wants to book a flight but does not

1088
00:48:54,119 --> 00:48:56,579
know the destination. The machine must choose

1089
00:48:56,579 --> 00:48:59,320
an action from its available repertoire. It can

1090
00:48:59,320 --> 00:49:01,659
generate text asking for the destination city,

1091
00:49:01,760 --> 00:49:04,280
or it can generate text prematurely asking for

1092
00:49:04,280 --> 00:49:06,739
payment. And how does it know which text to generate?

1093
00:49:06,940 --> 00:49:09,500
It calculates the path to maximize a mathematical

1094
00:49:09,500 --> 00:49:13,050
reward. The ultimate reward is successfully completing

1095
00:49:13,050 --> 00:49:15,289
the transaction booking the flight. The machine

1096
00:49:15,289 --> 00:49:17,829
uses transition probabilities to model the unpredictable

1097
00:49:17,829 --> 00:49:21,090
nature of human responses, and it uses discounting

1098
00:49:21,090 --> 00:49:23,309
to weigh immediate rewards against long -term

1099
00:49:23,309 --> 00:49:26,050
success. The text generation is not just stringing

1100
00:49:26,050 --> 00:49:28,809
words together. It is the execution of a mathematically

1101
00:49:28,809 --> 00:49:31,789
optimized policy. The machine is making sequential

1102
00:49:31,789 --> 00:49:34,829
strategic decisions to navigate the chaotic environment

1103
00:49:34,829 --> 00:49:37,369
of human conversation toward a successful resolution.

1104
00:49:37,909 --> 00:49:41,440
It is playing the board game to win. The Learner,

1105
00:49:41,460 --> 00:49:43,340
we have covered a staggering amount of ground

1106
00:49:43,340 --> 00:49:45,960
in this deep dive. We started by throwing classic

1107
00:49:45,960 --> 00:49:48,019
novels into a blender to make basic numerical

1108
00:49:48,019 --> 00:49:51,320
smoothies with a bag of words. We learned how

1109
00:49:51,320 --> 00:49:53,619
to discover hidden structures using the expectation

1110
00:49:53,619 --> 00:49:56,960
maximization dance. We watched the Viterbi algorithm

1111
00:49:56,960 --> 00:49:59,679
navigate its bracket to assign grammar and the

1112
00:49:59,679 --> 00:50:02,719
CKY algorithm build massive nested trees of syntax.

1113
00:50:03,690 --> 00:50:05,829
We embedded words into geometric space using

1114
00:50:05,829 --> 00:50:08,110
the game of 20 questions. And finally, we explored

1115
00:50:08,110 --> 00:50:10,349
systems that can automatically translate languages

1116
00:50:10,349 --> 00:50:12,909
and hold strategic, mathematically optimized

1117
00:50:12,909 --> 00:50:15,769
conversations. It is a remarkable conceptual

1118
00:50:15,769 --> 00:50:18,590
journey from discrete, isolated symbols through

1119
00:50:18,590 --> 00:50:21,289
complex, continuous mathematics and ultimately

1120
00:50:21,289 --> 00:50:23,769
back to the generation of meaningful human symbols.

1121
00:50:24,070 --> 00:50:25,969
But I want to leave you, the listener, with a

1122
00:50:25,969 --> 00:50:28,349
final thought to ponder, derived from the core

1123
00:50:28,349 --> 00:50:30,969
tension that permeates Eisenstein's entire text.

1124
00:50:31,719 --> 00:50:34,559
Throughout this exploration, we constantly contrasted

1125
00:50:34,559 --> 00:50:37,579
two opposing philosophies. On one side, we have

1126
00:50:37,579 --> 00:50:39,579
the structured, rule -based approach of formal

1127
00:50:39,579 --> 00:50:42,139
linguistics, like the pristine logic of the lambda

1128
00:50:42,139 --> 00:50:44,800
calculus and the rigid geometry of syntax trees.

1129
00:50:45,059 --> 00:50:47,679
On the other side, we have the sheer brute force

1130
00:50:47,679 --> 00:50:50,500
predictive power of modern statistical deep learning,

1131
00:50:50,699 --> 00:50:53,619
like the geometric spaces of Word2Vec and neural

1132
00:50:53,619 --> 00:50:56,219
sequence labelers. That dichotomy remains the

1133
00:50:56,219 --> 00:50:58,699
defining philosophical debate of modern natural

1134
00:50:58,699 --> 00:51:01,079
language processing. So I ask you to ponder this.

1135
00:51:01,480 --> 00:51:04,360
As our neural networks become unimaginably massive,

1136
00:51:04,699 --> 00:51:07,420
training on literally the entirety of the Internet's

1137
00:51:07,420 --> 00:51:10,380
text, they are becoming increasingly opaque and

1138
00:51:10,380 --> 00:51:13,099
uninterpretable. We cannot open up a trillion

1139
00:51:13,099 --> 00:51:15,480
parameter model and read its hidden states the

1140
00:51:15,480 --> 00:51:18,039
way we can cleanly read a binary syntax tree.

1141
00:51:18,340 --> 00:51:20,619
Are these statistical networks actually getting

1142
00:51:20,619 --> 00:51:23,800
closer to replicating true cognitive human comprehension?

1143
00:51:24,820 --> 00:51:27,619
Or are we simply building an unimaginably complex,

1144
00:51:28,019 --> 00:51:30,539
high -dimensional mathematical mirror of our

1145
00:51:30,539 --> 00:51:33,719
own statistical patterns? Does true meaning ultimately

1146
00:51:33,719 --> 00:51:36,619
require a logical rulebook? Or is meaning just

1147
00:51:36,619 --> 00:51:39,460
an emergent property of seeing enough data? We

1148
00:51:39,460 --> 00:51:41,400
started today by acknowledging that human language

1149
00:51:41,400 --> 00:51:43,219
isn't a smooth curve, it's messy and discrete.

1150
00:51:43,659 --> 00:51:45,880
But maybe in the end, if you pour enough data

1151
00:51:45,880 --> 00:51:48,260
into the machine, the muddy discrete waters of

1152
00:51:48,260 --> 00:51:50,139
human language finally start to look continuous

1153
00:51:50,139 --> 00:51:50,539
again.