1
00:00:00,000 --> 00:00:01,600
You know, usually when we think about artificial

2
00:00:01,600 --> 00:00:04,519
intelligence, there is this deeply ingrained

3
00:00:04,519 --> 00:00:07,559
expectation of just pristine mathematics. Oh,

4
00:00:07,580 --> 00:00:09,859
yeah, absolutely. Like we picture this flawless,

5
00:00:09,960 --> 00:00:12,339
frictionless assembly line. You feed in the raw

6
00:00:12,339 --> 00:00:15,439
data. The algorithm spins up in some sterile

7
00:00:15,439 --> 00:00:17,719
server room. The silicon gears turn and the machine

8
00:00:17,719 --> 00:00:20,059
just points at the world and says, I mean, here

9
00:00:20,059 --> 00:00:23,440
is the absolute objective answer. It is an incredibly

10
00:00:23,440 --> 00:00:25,739
seductive illusion. I mean, it feels binary,

11
00:00:25,839 --> 00:00:28,920
right? Solved or not solved. It gives us comfort

12
00:00:28,920 --> 00:00:30,719
to think that a sufficiently advanced algorithm

13
00:00:30,719 --> 00:00:34,280
can just figure out reality entirely on its own.

14
00:00:34,320 --> 00:00:36,700
Yeah, completely detached from the messiness

15
00:00:36,700 --> 00:00:40,079
of human interpretation. Exactly. But then you

16
00:00:40,079 --> 00:00:42,460
look at the reality of deploying AI out in the

17
00:00:42,460 --> 00:00:45,679
wild, and suddenly that pristine, frictionless

18
00:00:45,679 --> 00:00:48,359
assembly line completely grinds to a halt. Because

19
00:00:48,359 --> 00:00:51,000
the technological landscape we are actually living

20
00:00:51,000 --> 00:00:54,320
in right now is entirely dependent on human labor.

21
00:00:54,460 --> 00:00:56,759
It really is. It is the absolute definition of

22
00:00:56,759 --> 00:00:59,140
algorithmic muddy waters. And there is a story

23
00:00:59,140 --> 00:01:01,600
that perfectly captures this, which honestly

24
00:01:01,600 --> 00:01:04,359
absolutely blew my mind. It is a sobering reality

25
00:01:04,359 --> 00:01:06,379
check for anyone who thinks building the neural

26
00:01:06,379 --> 00:01:08,640
network architecture is like the whole game.

27
00:01:08,780 --> 00:01:11,640
Right. Exactly. So a major auto manufacturer

28
00:01:11,640 --> 00:01:14,500
decided they were going to definitively win the

29
00:01:14,500 --> 00:01:16,939
autonomous vehicle race. So what do they do?

30
00:01:17,040 --> 00:01:19,879
They go out and hire a massive team of brilliant

31
00:01:19,879 --> 00:01:22,560
PhDs. Like top of their class. Top of their class.

32
00:01:23,040 --> 00:01:24,859
Absolute wizards at building state -of -the -art

33
00:01:24,859 --> 00:01:27,319
algorithms, optimizing loss functions, tweaking

34
00:01:27,319 --> 00:01:29,859
learning rates, all of that. But a little while

35
00:01:29,859 --> 00:01:32,900
later, the company had to fire that entire team.

36
00:01:33,140 --> 00:01:35,269
Literally all of them. All of them. Because while

37
00:01:35,269 --> 00:01:37,989
these PhDs knew every mathematical proof there

38
00:01:37,989 --> 00:01:40,250
was to know about algorithms, they did not know

39
00:01:40,250 --> 00:01:43,969
data. They had zero idea how to scale their data

40
00:01:43,969 --> 00:01:46,189
annotation strategy. Which is a huge problem.

41
00:01:46,390 --> 00:01:48,269
Yeah. They literally could not teach the car

42
00:01:48,269 --> 00:01:50,909
what it was looking at fast enough or accurately

43
00:01:50,909 --> 00:01:53,489
enough to make the algorithms functional. We

44
00:01:53,489 --> 00:01:56,430
spend so much time in academia and industry obsessing

45
00:01:56,430 --> 00:01:58,150
over the structural engineering of the model.

46
00:01:58,459 --> 00:02:00,719
And we argue about how many layers the network

47
00:02:00,719 --> 00:02:03,239
should have or the optimization algorithms. Right,

48
00:02:03,319 --> 00:02:06,239
the flashy stuff. Exactly. But roughly 90 % of

49
00:02:06,239 --> 00:02:08,199
machine learning in production today relies on

50
00:02:08,199 --> 00:02:10,620
supervised learning. And supervised learning,

51
00:02:10,719 --> 00:02:13,099
by definition, requires human feedback. Right.

52
00:02:13,180 --> 00:02:15,699
It requires a human being to look at a piece

53
00:02:15,699 --> 00:02:18,620
of data and attach a label to it. If you do not

54
00:02:18,620 --> 00:02:20,740
know how to get that data, how to mathematically

55
00:02:20,740 --> 00:02:23,599
choose which specific data points to get, and

56
00:02:23,599 --> 00:02:25,659
how to build a psychological interface to connect

57
00:02:25,659 --> 00:02:29,139
the human to the machine, well, Your brilliant

58
00:02:29,139 --> 00:02:31,360
multi -million dollar algorithm is effectively

59
00:02:31,360 --> 00:02:34,479
a paperweight. Okay, so let us unpack this. For

60
00:02:34,479 --> 00:02:36,740
you listening, we are doing a massive deep dive

61
00:02:36,740 --> 00:02:39,340
today exploring the science of exactly that gap.

62
00:02:39,500 --> 00:02:42,439
It is called Human in the Loop Machine Learning,

63
00:02:42,520 --> 00:02:46,120
or HITL. And our mission today is to meticulously

64
00:02:46,120 --> 00:02:48,740
unpack a foundational text on this topic, which

65
00:02:48,740 --> 00:02:51,639
is Human in the Loop Machine Learning by Robert

66
00:02:51,639 --> 00:02:54,240
Monarch. It is such an essential text. It really

67
00:02:54,240 --> 00:02:56,990
is. And the whole field basically breaks down

68
00:02:56,990 --> 00:02:59,289
into three massive pillars. First, annotation.

69
00:02:59,729 --> 00:03:02,789
So how do we get the data? Second, active learning,

70
00:03:02,949 --> 00:03:05,590
which is how do we choose which data to get?

71
00:03:05,909 --> 00:03:08,750
And third, human -computer interaction. How do

72
00:03:08,750 --> 00:03:10,710
we design the bridge between the human brain

73
00:03:10,710 --> 00:03:12,909
and the silicon brain? Yeah, and to understand

74
00:03:12,909 --> 00:03:15,490
why this is a highly rigorous science and not

75
00:03:15,490 --> 00:03:17,210
just a matter of hiring a bunch of interns to

76
00:03:17,210 --> 00:03:19,469
click on pictures, we have to run the math on

77
00:03:19,469 --> 00:03:21,349
the scale of the problem. Oh man, the math is

78
00:03:21,349 --> 00:03:23,900
terrifying. The sheer volume is what paralyzes

79
00:03:23,900 --> 00:03:25,620
most engineering teams the first time they face

80
00:03:25,620 --> 00:03:28,180
it. Let us use that self -driving car example.

81
00:03:28,439 --> 00:03:31,000
So let us say you have a single video feed from

82
00:03:31,000 --> 00:03:33,419
a car dashboard, right? And you need a human

83
00:03:33,419 --> 00:03:37,020
to draw bounding boxes around 20 objects per

84
00:03:37,020 --> 00:03:39,659
frame. So pedestrians, stop signs, other cars,

85
00:03:39,800 --> 00:03:42,330
traffic lights. Exactly. Now, standard video

86
00:03:42,330 --> 00:03:46,349
is 30 frames a second. That is 36 ,000 bounding

87
00:03:46,349 --> 00:03:48,490
boxes for just one minute of video. Just one

88
00:03:48,490 --> 00:03:51,250
minute. One minute. Now, U .S. drivers log about

89
00:03:51,250 --> 00:03:54,969
95 billion hours a year. If you wanted to annotate

90
00:03:54,969 --> 00:03:57,270
just one year of U .S. driving to train your

91
00:03:57,270 --> 00:04:00,729
AI, it would take roughly 60 trillion hours of

92
00:04:00,729 --> 00:04:03,689
human labor. Which is just... There are literally

93
00:04:03,689 --> 00:04:06,370
not enough human beings alive on the planet to

94
00:04:06,370 --> 00:04:10,159
do that. We cannot. look at everything. We absolutely

95
00:04:10,159 --> 00:04:13,180
must sample the data. And that reality introduces

96
00:04:13,180 --> 00:04:16,699
the core tension we are exploring today. If we

97
00:04:16,699 --> 00:04:18,980
can only realistically look at a microscopic

98
00:04:18,980 --> 00:04:21,660
fraction of our data, how do we mathematically

99
00:04:21,660 --> 00:04:24,379
guarantee we are looking at the right fraction?

100
00:04:24,600 --> 00:04:26,959
Right. How do we find the precise data points

101
00:04:26,959 --> 00:04:29,139
that will fundamentally alter the machine's understanding

102
00:04:29,139 --> 00:04:32,620
rather than just confirming what it already knows?

103
00:04:33,230 --> 00:04:36,089
So to solve that 60 trillion hour problem, we

104
00:04:36,089 --> 00:04:38,170
need a mental model of what a machine learning

105
00:04:38,170 --> 00:04:41,589
model actually, quote unquote, knows at any given

106
00:04:41,589 --> 00:04:43,689
millisecond. Yeah, and if we connect this to

107
00:04:43,689 --> 00:04:45,910
the bigger picture, you can break down a model's

108
00:04:45,910 --> 00:04:48,449
knowledge state into four distinct quadrants.

109
00:04:48,910 --> 00:04:51,370
This framework from the text is incredibly useful

110
00:04:51,370 --> 00:04:54,209
for mapping out exactly what kind of algorithmic

111
00:04:54,209 --> 00:04:56,889
strategy you need to deploy. Okay, let us go

112
00:04:56,889 --> 00:04:59,050
through the quadrants. So the first quadrant

113
00:04:59,050 --> 00:05:02,959
is the known knowns. These are the highly confident,

114
00:05:03,060 --> 00:05:06,579
accurate predictions. The model looks at a perfectly

115
00:05:06,579 --> 00:05:09,879
lit stop sign on a sunny day and outputs, I am

116
00:05:09,879 --> 00:05:13,720
99 .9 % sure that is a stop sign. So this is

117
00:05:13,720 --> 00:05:16,040
the current state of your model functioning perfectly.

118
00:05:16,199 --> 00:05:18,459
We do not need humans for that quadrant. The

119
00:05:18,459 --> 00:05:21,540
machine has got it. Let it run. Exactly. But

120
00:05:21,540 --> 00:05:25,860
the second quadrant is the known unknowns. These

121
00:05:25,860 --> 00:05:28,540
are the non -confident predictions. So the model

122
00:05:28,540 --> 00:05:30,759
looks at a stop sign covered in heavy snow at

123
00:05:30,759 --> 00:05:34,560
midnight, and its internal math says, I am 50

124
00:05:34,560 --> 00:05:37,860
% sure this is a stop sign and 50 % sure it is

125
00:05:37,860 --> 00:05:40,959
a yield sign. It mathematically knows that it

126
00:05:40,959 --> 00:05:43,360
is confused. Right, it knows it does not know.

127
00:05:43,839 --> 00:05:46,339
We attack this quadrant using a suite of algorithms

128
00:05:46,339 --> 00:05:50,439
called uncertainty sampling. Okay, so known knowns

129
00:05:50,439 --> 00:05:53,680
and known unknowns. That covers what the model

130
00:05:53,680 --> 00:05:55,959
is looking at right now. Then we have the third

131
00:05:55,959 --> 00:05:59,139
quadrant, which is the unknown knowns. It sounds

132
00:05:59,139 --> 00:06:02,019
like a riddle, but it refers to latent structural

133
00:06:02,019 --> 00:06:04,699
information that exists in other related models

134
00:06:04,699 --> 00:06:07,279
or data sets that your current model has not

135
00:06:07,279 --> 00:06:10,149
tapped into yet. Like knowledge sitting right

136
00:06:10,149 --> 00:06:12,790
there waiting to be used. Exactly. We extract

137
00:06:12,790 --> 00:06:15,750
this knowledge using transfer learning. And finally,

138
00:06:15,850 --> 00:06:18,870
the most dangerous quadrant, the unknown unknowns.

139
00:06:19,410 --> 00:06:22,839
These are the terrifying blind spots. The things

140
00:06:22,839 --> 00:06:25,290
it does not even know it does not know. Right.

141
00:06:25,689 --> 00:06:27,889
The model looks at a person riding a motorized

142
00:06:27,889 --> 00:06:30,350
unicycle, which is something it has never seen

143
00:06:30,350 --> 00:06:33,110
in its life, and with 100 % confidence labels

144
00:06:33,110 --> 00:06:36,089
it a pedestrian. It doesn't even know that it

145
00:06:36,089 --> 00:06:37,930
lacks the conceptual framework to understand

146
00:06:37,930 --> 00:06:41,290
what it is looking at. Oh, wow. Yeah. We have

147
00:06:41,290 --> 00:06:44,709
to root these out using diversity sampling. Okay.

148
00:06:44,910 --> 00:06:47,550
Let us hover on that third quadrant for a second,

149
00:06:47,610 --> 00:06:50,470
the unknown knowns or transfer learning, because

150
00:06:50,470 --> 00:06:53,759
this feels like a massive shortcut. we do not

151
00:06:53,759 --> 00:06:56,180
actually have to start training in AI from complete

152
00:06:56,180 --> 00:06:58,259
scratch every time we build a system, right?

153
00:06:58,560 --> 00:07:01,480
Not at all. Building a model from a totally blank

154
00:07:01,480 --> 00:07:03,860
slate, initializing it with random mathematical

155
00:07:03,860 --> 00:07:06,839
weights, is astronomically expensive and time

156
00:07:06,839 --> 00:07:09,899
-consuming. Instead, we take a model trained

157
00:07:09,899 --> 00:07:12,600
on a massive general data set and physically

158
00:07:12,600 --> 00:07:15,079
adapt its neural architecture to our specific

159
00:07:15,079 --> 00:07:19,189
smaller task. Okay, how does that work in practice,

160
00:07:19,410 --> 00:07:22,319
like with images? Well, in computer vision, The

161
00:07:22,319 --> 00:07:25,060
classic paradigm is using the ImageNet data set.

162
00:07:25,740 --> 00:07:28,160
ImageNet consists of millions of images classified

163
00:07:28,160 --> 00:07:31,529
into thousands of broad categories. When a deep

164
00:07:31,529 --> 00:07:33,870
neural network trains on ImageNet, the data passes

165
00:07:33,870 --> 00:07:36,389
through multiple layers. Right, the neural layers.

166
00:07:36,629 --> 00:07:39,029
Yes. And the very first layers of the network

167
00:07:39,029 --> 00:07:42,170
do not learn what a dog or a car is. They just

168
00:07:42,170 --> 00:07:44,449
learn mathematical gradients. They learn what

169
00:07:44,449 --> 00:07:46,449
a vertical edge is. They learn what a horizontal

170
00:07:46,449 --> 00:07:49,279
edge is. Because, I mean, at the pixel level,

171
00:07:49,360 --> 00:07:51,680
an edge is just a sharp mathematical contrast

172
00:07:51,680 --> 00:07:54,220
between light and dark pixels, right? Precisely.

173
00:07:54,560 --> 00:07:56,800
The next layer combines those edges to learn

174
00:07:56,800 --> 00:07:59,740
textures and corners. The layer after that learns

175
00:07:59,740 --> 00:08:02,379
basic geometric shapes. It is only in the final

176
00:08:02,379 --> 00:08:05,160
layers that it combines shapes to say, ah, that

177
00:08:05,160 --> 00:08:07,579
is a wheel. Okay, I see where this is going.

178
00:08:07,920 --> 00:08:09,879
So if you want to build a highly specialized

179
00:08:09,879 --> 00:08:13,519
model to identify, say, subtle defects in agricultural

180
00:08:13,519 --> 00:08:16,720
tractor tires on an assembly line, You do not

181
00:08:16,720 --> 00:08:18,959
need to teach the AI what a visual edge is from

182
00:08:18,959 --> 00:08:21,220
scratch. You already have that from ImageNet.

183
00:08:21,339 --> 00:08:23,519
Right. You take the pre -trained ImageNet model,

184
00:08:23,639 --> 00:08:26,100
you freeze all those early layers that understand

185
00:08:26,100 --> 00:08:27,980
the physics of light, texture, and geometry,

186
00:08:28,199 --> 00:08:30,540
and you just retrain the very last layer on your

187
00:08:30,540 --> 00:08:33,320
specific tractor tire images. It is like hiring

188
00:08:33,320 --> 00:08:35,120
an experienced sous chef for your restaurant.

189
00:08:35,320 --> 00:08:37,679
You do not have to teach them the basic physics

190
00:08:37,679 --> 00:08:40,200
of how to hold a knife or how water boils. You

191
00:08:40,200 --> 00:08:42,679
just hand them your specific menu and teach them

192
00:08:42,679 --> 00:08:45,450
your unique recipes. Their foundational knowledge

193
00:08:45,450 --> 00:08:47,769
completely transfers over. That is a perfect

194
00:08:47,769 --> 00:08:50,210
analogy. But this mechanical transfer works for

195
00:08:50,210 --> 00:08:53,070
text too, right? Like in natural language processing

196
00:08:53,070 --> 00:08:56,549
or NLP. It does. But historically, human language

197
00:08:56,549 --> 00:08:59,259
was a much harder problem to crack. You cannot

198
00:08:59,259 --> 00:09:01,940
just map the texture or edge of a word the way

199
00:09:01,940 --> 00:09:03,779
you calculate the light gradient of a pixel.

200
00:09:04,080 --> 00:09:07,000
A word is an abstract symbol. Yeah, a collection

201
00:09:07,000 --> 00:09:09,620
of letters doesn't have physical geometry. Right.

202
00:09:09,970 --> 00:09:12,149
But there was a profound breakthrough in NLP

203
00:09:12,149 --> 00:09:14,529
transfer learning, and it is anchored by a quote

204
00:09:14,529 --> 00:09:18,629
from the linguist J .R. Firth from 1957. He said,

205
00:09:18,750 --> 00:09:21,289
you shall know a word by the company it keeps.

206
00:09:21,409 --> 00:09:23,789
You shall know a word by the company it keeps.

207
00:09:23,870 --> 00:09:27,429
I love the poetry of that. But how do you turn

208
00:09:27,429 --> 00:09:30,419
that poetry into math? That quote is literally

209
00:09:30,419 --> 00:09:33,200
the entire functional mathematical theory behind

210
00:09:33,200 --> 00:09:35,740
modern pre -trained language models. Instead

211
00:09:35,740 --> 00:09:38,059
of paying humans to explicitly label thousands

212
00:09:38,059 --> 00:09:40,700
of sentences for grammar and sentiment, we take

213
00:09:40,700 --> 00:09:44,379
a massive corpus of raw, unlabeled text, essentially

214
00:09:44,379 --> 00:09:47,409
the entire internet. Okay. We systematically

215
00:09:47,409 --> 00:09:50,549
hide certain words, and we force the AI's neural

216
00:09:50,549 --> 00:09:53,289
network to guess the missing word based purely

217
00:09:53,289 --> 00:09:55,730
on the surrounding context. Like an impossibly

218
00:09:55,730 --> 00:09:58,700
large game of fill in the blank. So... If the

219
00:09:58,700 --> 00:10:02,220
sentence is, the doctor picked up his blank to

220
00:10:02,220 --> 00:10:05,580
perform the surgery, the AI mathematically calculates

221
00:10:05,580 --> 00:10:07,379
that the missing word is highly likely to be

222
00:10:07,379 --> 00:10:10,820
scalpel. Exactly. And by forcing the model to

223
00:10:10,820 --> 00:10:14,019
do this billions of times, it starts to build

224
00:10:14,019 --> 00:10:17,360
a multidimensional geometric space where every

225
00:10:17,360 --> 00:10:20,919
word is a vector. Imagine an arrow pointing in

226
00:10:20,919 --> 00:10:23,179
a specific direction. Okay, so the words become

227
00:10:23,179 --> 00:10:26,850
geometry. Yes. The AI notices that the vector

228
00:10:26,850 --> 00:10:29,750
for doctor and the vector for surgeon consistently

229
00:10:29,750 --> 00:10:32,909
appear in the exact same contexts, surrounded

230
00:10:32,909 --> 00:10:36,230
by the vectors for hospital, patient, or scalpel.

231
00:10:36,330 --> 00:10:39,110
So they get pulled together in that space? Precisely.

232
00:10:39,519 --> 00:10:42,379
Without any human ever explicitly typing out

233
00:10:42,379 --> 00:10:45,259
a dictionary definition, the AI calculates that

234
00:10:45,259 --> 00:10:47,980
the geometric distance between doctor and surgeon

235
00:10:47,980 --> 00:10:51,259
is incredibly small. They mean almost the same

236
00:10:51,259 --> 00:10:54,690
thing. It learns grammar, syntax, and deep semantics

237
00:10:54,690 --> 00:10:57,169
strictly from the latent mathematical structure

238
00:10:57,169 --> 00:11:01,509
of the data itself. That is wild. That is the

239
00:11:01,509 --> 00:11:05,230
ultimate unknown known being unlocked. Okay,

240
00:11:05,269 --> 00:11:07,549
so we use transfer learning to establish this

241
00:11:07,549 --> 00:11:10,370
powerful baseline intelligence, but eventually

242
00:11:10,370 --> 00:11:13,909
we have to start building our specific minimum

243
00:11:13,909 --> 00:11:16,629
viable product, our MVP, for whatever custom

244
00:11:16,629 --> 00:11:18,629
task we are actually trying to solve. Right.

245
00:11:19,000 --> 00:11:20,720
There's an example in the book of building a

246
00:11:20,720 --> 00:11:24,159
first HITL system using a disaster response data

247
00:11:24,159 --> 00:11:26,639
set. So the goal is to classify incoming news

248
00:11:26,639 --> 00:11:30,100
headlines as either disaster related or not disaster

249
00:11:30,100 --> 00:11:32,580
related. So emergency responders can filter information.

250
00:11:32,879 --> 00:11:35,820
It's a great real world example. Now, if I am

251
00:11:35,820 --> 00:11:38,000
a project manager tasked with this, my immediate

252
00:11:38,000 --> 00:11:40,539
instinct is, well, why do we need complex machine

253
00:11:40,539 --> 00:11:43,299
learning at all? Why not just write a fast, simple

254
00:11:43,299 --> 00:11:45,379
script that searches the incoming headlines for

255
00:11:45,379 --> 00:11:48,789
keywords like earthquake? Flood or hurricane

256
00:11:48,789 --> 00:11:51,549
and just automatically labels all of those as

257
00:11:51,549 --> 00:11:53,850
disaster related. I mean, it's not that infinitely

258
00:11:53,850 --> 00:11:57,429
faster and cheaper. It is incredibly fast. It

259
00:11:57,429 --> 00:12:00,470
is very cheap. And it is a massive architectural

260
00:12:00,470 --> 00:12:05,009
trap. Wait, really? Why? That ad hoc keyword

261
00:12:05,009 --> 00:12:07,789
based approach is known as hacktive learning.

262
00:12:07,950 --> 00:12:10,990
It is entirely unprincipled. Think about what

263
00:12:10,990 --> 00:12:13,370
happens mathematically if you just query your

264
00:12:13,370 --> 00:12:15,870
database for the keyword earthquake to build

265
00:12:15,870 --> 00:12:18,919
your training set. Okay. I am thinking. You're

266
00:12:18,919 --> 00:12:21,259
only gathering data that perfectly matches what

267
00:12:21,259 --> 00:12:23,759
you already know. You're going to completely

268
00:12:23,759 --> 00:12:26,059
miss all the nuanced human headlines that say

269
00:12:26,059 --> 00:12:29,639
things like seismic event devastates region or

270
00:12:29,639 --> 00:12:32,879
tremors collapse local infrastructure. Oh, right,

271
00:12:32,980 --> 00:12:34,940
because those headlines do not contain the literal

272
00:12:34,940 --> 00:12:38,200
string of characters E -A -R -T -H -Q -U -A -K

273
00:12:38,200 --> 00:12:41,279
-E. Exactly. Furthermore, you create a completely

274
00:12:41,279 --> 00:12:44,240
non -representative mathematical sample. The

275
00:12:44,240 --> 00:12:46,419
machine learning model does not learn the complex

276
00:12:46,419 --> 00:12:49,159
linguistic structures of how disasters are reported.

277
00:12:49,379 --> 00:12:53,179
It just learns a brittle, lazy rule. Like a really

278
00:12:53,179 --> 00:12:55,899
dumb shortcut. Yes, it learns if the specific

279
00:12:55,899 --> 00:12:58,120
string of characters for earthquake is present,

280
00:12:58,240 --> 00:13:01,899
fire a positive prediction. If not, ignore. It

281
00:13:01,899 --> 00:13:04,139
becomes highly susceptible to semantic drift

282
00:13:04,139 --> 00:13:06,639
and completely useless in the real world. Okay,

283
00:13:06,700 --> 00:13:10,320
so we avoid hactive learning. We need to sample

284
00:13:10,320 --> 00:13:13,019
carefully and thoughtfully. But before we even

285
00:13:13,019 --> 00:13:15,080
start pulling data out of the pile to train our

286
00:13:15,080 --> 00:13:17,580
model, there is a golden rule that dictates the

287
00:13:17,580 --> 00:13:21,000
entire sequence of operations. The rule is always

288
00:13:21,000 --> 00:13:24,740
get your evaluation data first. Always. So why

289
00:13:24,740 --> 00:13:26,799
is the sequencing of getting your test data so

290
00:13:26,799 --> 00:13:29,820
absolutely critical? Because evaluation data,

291
00:13:30,000 --> 00:13:33,100
the test set, is the absolute truth of your system.

292
00:13:33,480 --> 00:13:36,139
It is a purely random subset of your data that

293
00:13:36,139 --> 00:13:38,700
you lock away in a vault. You never, ever let

294
00:13:38,700 --> 00:13:41,519
the model train on it. It is the final exam that

295
00:13:41,519 --> 00:13:43,720
mathematically proves whether your model actually

296
00:13:43,720 --> 00:13:46,259
generalizes to the real world. Okay, that makes

297
00:13:46,259 --> 00:13:49,299
sense. Now, if you deploy active learning, like

298
00:13:49,299 --> 00:13:51,600
actively searching your data pool for confusing

299
00:13:51,600 --> 00:13:54,879
items, before you lock away that test set, you

300
00:13:54,879 --> 00:13:57,139
permanently contaminate your evaluation environment.

301
00:13:57,600 --> 00:13:59,700
Walk me through the mechanics of that contamination.

302
00:14:00,019 --> 00:14:03,059
How does looking for confusing items ruin the

303
00:14:03,059 --> 00:14:06,500
test? Imagine you have a raw, unorganized database

304
00:14:06,500 --> 00:14:09,649
of 100 ,000 unlabeled headlines. You run your

305
00:14:09,649 --> 00:14:12,250
active learning algorithm to find the 10 ,000

306
00:14:12,250 --> 00:14:15,509
most complex edge case confusing headlines. Right.

307
00:14:15,950 --> 00:14:17,990
And you pull them out of the database to send

308
00:14:17,990 --> 00:14:20,840
to your human annotators for training. You then

309
00:14:20,840 --> 00:14:23,019
randomly take 5 ,000 of the remaining headlines

310
00:14:23,019 --> 00:14:27,269
to be your locked away evaluation test set. What

311
00:14:27,269 --> 00:14:29,970
is the fundamental flaw in that sequence? Oh,

312
00:14:30,009 --> 00:14:32,950
I see the math. The remaining pool of 90 ,000

313
00:14:32,950 --> 00:14:34,889
headlines has been artificially drained of all

314
00:14:34,889 --> 00:14:38,399
its confusing items. You literally skimmed all

315
00:14:38,399 --> 00:14:41,200
the hard stuff off the top. Exactly. So the pool

316
00:14:41,200 --> 00:14:43,379
that you draw your test set from only contains

317
00:14:43,379 --> 00:14:46,320
the incredibly easy, obvious stuff that the model

318
00:14:46,320 --> 00:14:49,440
is already confident about. Precisely. If you

319
00:14:49,440 --> 00:14:51,399
test your freshly trained model on that leftover

320
00:14:51,399 --> 00:14:54,519
pool, it is going to score 99 .9 % accuracy.

321
00:14:54,679 --> 00:14:56,840
You and your engineering team are going to pop

322
00:14:56,840 --> 00:14:58,840
champagne and think you have built a genius AI.

323
00:14:59,320 --> 00:15:01,659
But then you put it out in the world. Right.

324
00:15:01,980 --> 00:15:03,879
When you deploy it in the real world, which is

325
00:15:03,879 --> 00:15:06,960
chaotic, nuanced, and full of confusing borderline

326
00:15:06,960 --> 00:15:10,019
cases, the model will fail catastrophically.

327
00:15:10,320 --> 00:15:13,259
You've fundamentally biased your test set by

328
00:15:13,259 --> 00:15:15,960
removing the variants. It is like a teacher writing

329
00:15:15,960 --> 00:15:18,940
a comprehensive final exam for a student, but

330
00:15:18,940 --> 00:15:21,759
only using the specific easy flashcards they

331
00:15:21,759 --> 00:15:23,840
know the student has already memorized, rather

332
00:15:23,840 --> 00:15:26,440
than testing their actual robust knowledge of

333
00:15:26,440 --> 00:15:29,000
the whole subject. It creates a complete false

334
00:15:29,000 --> 00:15:31,159
sense of security. That is exactly what happens.

335
00:15:31,539 --> 00:15:34,539
So, golden rule, mathematically randomize and

336
00:15:34,539 --> 00:15:37,580
lock away a purely untouched test set before

337
00:15:37,580 --> 00:15:40,379
you run a single active learning algorithm. Once

338
00:15:40,379 --> 00:15:42,899
that pristine test set is secured in the vault,

339
00:15:43,039 --> 00:15:45,639
we can finally unleash our active learning strategies

340
00:15:45,639 --> 00:15:48,580
to intelligently build the training set. We attack

341
00:15:48,580 --> 00:15:51,340
the known unknowns. We need to find the data

342
00:15:51,340 --> 00:15:54,399
that genuinely confuses the model. But let us

343
00:15:54,399 --> 00:15:56,919
get under the hood here, because how does a model

344
00:15:56,919 --> 00:15:59,669
actually communicate its confusion? Like, if

345
00:15:59,669 --> 00:16:02,730
an AI looks at an image, it does not sweat, it

346
00:16:02,730 --> 00:16:05,230
does not furrow its brow. How mathematically

347
00:16:05,230 --> 00:16:07,850
does a neural network tell us, I have absolutely

348
00:16:07,850 --> 00:16:10,929
no idea what I am looking at? The mechanism almost

349
00:16:10,929 --> 00:16:13,309
always relies on a specific mathematical function

350
00:16:13,309 --> 00:16:18,990
called softmax. Softmax. Okay. Softmax acts as

351
00:16:18,990 --> 00:16:21,250
the final layer that translate a remote neural

352
00:16:21,250 --> 00:16:23,940
network classification architectures. When a

353
00:16:23,940 --> 00:16:26,720
deep neural network processes an input, say an

354
00:16:26,720 --> 00:16:29,779
image, its raw internal outputs are just arbitrary,

355
00:16:30,039 --> 00:16:33,320
unconstrained numbers called logits. Logits.

356
00:16:33,399 --> 00:16:36,179
What do those look like? A logit could be positive

357
00:16:36,179 --> 00:16:39,440
500, it could be negative 12, it could be .001.

358
00:16:39,940 --> 00:16:43,120
They represent the network's raw activations,

359
00:16:43,159 --> 00:16:45,500
but they do not mean anything intuitive to a

360
00:16:45,500 --> 00:16:48,870
human engineer. Softmax takes those raw, chaotic

361
00:16:48,870 --> 00:16:52,350
numbers and systematically squashes them into

362
00:16:52,350 --> 00:16:55,610
a clean, readable probability distribution that

363
00:16:55,610 --> 00:16:59,570
always sums exactly to 1 .0 or 100%. So how does

364
00:16:59,570 --> 00:17:02,210
it do that squashing? What is the actual mathematical

365
00:17:02,210 --> 00:17:04,670
operation? Why not just turn them directly into

366
00:17:04,670 --> 00:17:07,309
percentages? Well, you cannot just use basic

367
00:17:07,309 --> 00:17:09,589
percentages because logits can be negative numbers.

368
00:17:09,910 --> 00:17:13,470
And you cannot have a negative probability. Softmax

369
00:17:13,470 --> 00:17:16,569
solves this by using exponentials. Exponentials.

370
00:17:16,809 --> 00:17:20,250
Right. It takes Euler's number, E, and raises

371
00:17:20,250 --> 00:17:23,450
it to the power of each raw logit. Exponentiating

372
00:17:23,450 --> 00:17:26,529
does two things. First, it turns every single

373
00:17:26,529 --> 00:17:30,529
number positive. Second, it aggressively exaggerates

374
00:17:30,529 --> 00:17:32,730
the differences between the numbers. A slightly

375
00:17:32,730 --> 00:17:35,690
higher logit becomes a massively higher exponential

376
00:17:35,690 --> 00:17:38,390
value. Okay, and then how do we get the percentage?

377
00:17:39,170 --> 00:17:41,869
Then to get the final percentage... Softmax simply

378
00:17:41,869 --> 00:17:44,869
takes the exponential of one specific class and

379
00:17:44,869 --> 00:17:46,869
divides it by the sum of all the exponentials

380
00:17:46,869 --> 00:17:49,549
across every class. So if the model is looking

381
00:17:49,549 --> 00:17:51,630
at a picture and choosing between three classes,

382
00:17:51,809 --> 00:17:55,589
say dog, cat, and car, Softmax runs the exponentials,

383
00:17:55,589 --> 00:17:57,789
divides by the sum, and outputs a distribution

384
00:17:57,789 --> 00:18:02,349
where dog might be 0 .85, cat is 0 .10, and car

385
00:18:02,349 --> 00:18:06,890
is 0 .05. It all adds up to 1 .0. Exactly. You

386
00:18:06,890 --> 00:18:11,029
now have a clean probability distribution. But

387
00:18:11,029 --> 00:18:13,509
the real engineering challenge is, how do we

388
00:18:13,509 --> 00:18:15,750
use those probabilities to mathematically identify

389
00:18:15,750 --> 00:18:19,710
uncertainty? There are four main algorithms for

390
00:18:19,710 --> 00:18:21,890
calculating uncertainty on a decision boundary.

391
00:18:22,680 --> 00:18:24,940
Let us start with the most intuitive one, which

392
00:18:24,940 --> 00:18:28,099
is least confidence. How does least confidence

393
00:18:28,099 --> 00:18:32,000
work? It is exactly what it sounds like. You

394
00:18:32,000 --> 00:18:34,079
run a massive pool of unlabeled data through

395
00:18:34,079 --> 00:18:37,119
the model. The model generates a softmax probability

396
00:18:37,119 --> 00:18:40,539
distribution for every single item. You instruct

397
00:18:40,539 --> 00:18:43,220
your script to only look at the highest predicted

398
00:18:43,220 --> 00:18:46,319
probability for each item. Just the top pick.

399
00:18:46,819 --> 00:18:49,589
Right. You are searching the database for the

400
00:18:49,589 --> 00:18:52,170
item where the highest probability is the absolute

401
00:18:52,170 --> 00:18:55,230
lowest. So to use our dog -cat -car example,

402
00:18:55,390 --> 00:18:58,029
if image A goes through the network and the softmax

403
00:18:58,029 --> 00:19:02,890
output is dog at 90%, cat at 7%, car at 3%, the

404
00:19:02,890 --> 00:19:05,569
highest confidence is 90%. The model is pretty

405
00:19:05,569 --> 00:19:08,289
sure. Very sure. But if image B goes through

406
00:19:08,289 --> 00:19:12,140
and it gets... dog at 40%, cat at 35%, and car

407
00:19:12,140 --> 00:19:16,420
at 25%, the highest confidence is only 40%. Exactly.

408
00:19:16,859 --> 00:19:20,900
Image B has the lowest, highest confidence. The

409
00:19:20,900 --> 00:19:22,720
model is essentially shrugging its mathematical

410
00:19:22,720 --> 00:19:26,400
shoulders. That 40 % item is sitting right on

411
00:19:26,400 --> 00:19:29,160
the razor's edge of the decision boundary. Because

412
00:19:29,160 --> 00:19:31,859
it is basically guessing? Yeah. It is highly

413
00:19:31,859 --> 00:19:33,779
likely that the model is going to guess wrong.

414
00:19:34,309 --> 00:19:36,750
which makes it an incredibly valuable piece of

415
00:19:36,750 --> 00:19:38,970
data to pull out of the database and show to

416
00:19:38,970 --> 00:19:41,630
a human annotator to establish the correct ground

417
00:19:41,630 --> 00:19:45,089
truth. But least confidence has a pretty glaring

418
00:19:45,089 --> 00:19:47,150
blind spot, doesn't it? Because it completely

419
00:19:47,150 --> 00:19:50,009
ignores the runner -up. It only looks at the

420
00:19:50,009 --> 00:19:52,210
top pick and just throws away the rest of the

421
00:19:52,210 --> 00:19:54,849
distribution. We need a way to measure the gap

422
00:19:54,849 --> 00:19:57,710
between first and second place. Which brings

423
00:19:57,710 --> 00:20:00,490
us to the second algorithm, margin of confidence.

424
00:20:01,259 --> 00:20:03,859
Instead of just looking at the top score, margin

425
00:20:03,859 --> 00:20:06,700
of confidence explicitly calculates the difference,

426
00:20:06,920 --> 00:20:09,740
the mathematical gap, between the model's top

427
00:20:09,740 --> 00:20:12,119
prediction and its second highest prediction.

428
00:20:12,339 --> 00:20:14,880
Okay. You are searching your database for the

429
00:20:14,880 --> 00:20:17,700
items where that margin is the absolute smallest.

430
00:20:18,200 --> 00:20:21,279
Because a tiny margin means the model is agonizing

431
00:20:21,279 --> 00:20:24,339
over a very specific choice. Like, if an item

432
00:20:24,339 --> 00:20:29,019
is predicted as 48 % cat and 47 % dog, the margin

433
00:20:29,019 --> 00:20:32,400
is only 1%. The model is literally tearing itself

434
00:20:32,400 --> 00:20:34,900
apart trying to decide between those two animals.

435
00:20:35,140 --> 00:20:37,900
Exactly. Contrast that with an item that is 48

436
00:20:37,900 --> 00:20:41,400
% cat, but the next highest is car at only 20%.

437
00:20:41,400 --> 00:20:44,420
The top confidence is exactly the same, 48%,

438
00:20:44,420 --> 00:20:47,359
so the least confidence algorithm would treat

439
00:20:47,359 --> 00:20:49,940
those two images equally. But they are completely

440
00:20:49,940 --> 00:20:52,980
different states of confusion. Right. Margin

441
00:20:52,980 --> 00:20:55,099
of confidence recognizes that the first scenario

442
00:20:55,099 --> 00:20:58,440
is a much tighter, more volatile race and therefore

443
00:20:58,440 --> 00:21:00,680
represents a more critical point of uncertainty

444
00:21:00,680 --> 00:21:03,880
on the decision boundary. Precisely. Margin gives

445
00:21:03,880 --> 00:21:06,079
you a much higher resolution view of the confusion.

446
00:21:06,359 --> 00:21:09,140
But subtraction can get mathematically skewed,

447
00:21:09,140 --> 00:21:11,380
right? Especially if you have a data set with

448
00:21:11,380 --> 00:21:14,240
thousands of classes. The raw probability numbers

449
00:21:14,240 --> 00:21:17,779
might become vanishingly small. A 1 % margin

450
00:21:17,779 --> 00:21:20,210
means something very... different when the top

451
00:21:20,210 --> 00:21:22,670
scores are in the 90s versus when the top scores

452
00:21:22,670 --> 00:21:25,309
are in the single digits. Is there a way to compare

453
00:21:25,309 --> 00:21:28,490
the top two choices proportionally? Yes, and

454
00:21:28,490 --> 00:21:31,470
that is the third strategy, ratio of confidence.

455
00:21:31,789 --> 00:21:34,630
It is structurally related to margin, but instead

456
00:21:34,630 --> 00:21:36,869
of subtracting the second highest from the highest,

457
00:21:37,109 --> 00:21:40,329
you divide them. You take the highest probability

458
00:21:40,329 --> 00:21:43,849
and divide it by the second highest. Okay. You

459
00:21:43,849 --> 00:21:46,089
are mathematically searching for the ratio closest

460
00:21:46,089 --> 00:21:50,319
to 1 .0. which would signify a perfect tie. What

461
00:21:50,319 --> 00:21:52,460
is the practical advantage of dividing instead

462
00:21:52,460 --> 00:21:55,079
of subtracting, though? Aside from scaling better

463
00:21:55,079 --> 00:21:58,160
across massive numbers of classes, where ratio

464
00:21:58,160 --> 00:22:01,480
of confidence really shines is in human -computer

465
00:22:01,480 --> 00:22:04,759
interface design. Oh, really? How so? If you

466
00:22:04,759 --> 00:22:07,559
are building a software tool that suggests autocomplete

467
00:22:07,559 --> 00:22:10,700
options or labels for a human worker, ratio provides

468
00:22:10,700 --> 00:22:13,000
a brilliant threshold for deciding when to show

469
00:22:13,000 --> 00:22:15,720
the human two options instead of one. That makes

470
00:22:15,720 --> 00:22:18,140
sense. Yeah, if the ratio of the top two predictions

471
00:22:18,140 --> 00:22:21,119
drops below a certain threshold, meaning they're

472
00:22:21,119 --> 00:22:24,599
incredibly close proportionally, the UI can automatically

473
00:22:24,599 --> 00:22:27,279
pop up both options and ask the human to click

474
00:22:27,279 --> 00:22:30,079
one because the model acknowledges it is genuinely

475
00:22:30,079 --> 00:22:34,119
tied. Okay, so least confidence looks at one

476
00:22:34,119 --> 00:22:37,579
prediction. Margin and ratio look at... two predictions.

477
00:22:37,759 --> 00:22:40,400
But what if our model has 10 possible labels

478
00:22:40,400 --> 00:22:43,019
or a thousand? Just looking at the top two options

479
00:22:43,019 --> 00:22:45,460
misses the holistic picture of how confused the

480
00:22:45,460 --> 00:22:48,180
model might be across the entire board. And that

481
00:22:48,180 --> 00:22:50,839
requires the fourth algorithm, which relies on

482
00:22:50,839 --> 00:22:53,359
information theory. And it's called entropy.

483
00:22:53,660 --> 00:22:57,539
Entropy? Like in physics? Similar concept. Yeah.

484
00:22:57,859 --> 00:23:00,480
Entropy is a mathematical measure of the total

485
00:23:00,480 --> 00:23:03,519
chaos or dispersion of probability across every

486
00:23:03,519 --> 00:23:06,920
single possible label in the distribution. How

487
00:23:06,920 --> 00:23:09,319
does calculating the spread change our understanding

488
00:23:09,319 --> 00:23:11,900
of the model's confusion? Let us visualize a

489
00:23:11,900 --> 00:23:15,400
model with 10 possible labels. Image A goes through,

490
00:23:15,480 --> 00:23:19,559
and the softmax output is 30 % for label 1. The

491
00:23:19,559 --> 00:23:22,539
remaining 70 % of the probability is perfectly

492
00:23:22,539 --> 00:23:25,660
evenly distributed across the other nine labels,

493
00:23:25,759 --> 00:23:31,299
so about 7 .7 % each. Now consider image B. It

494
00:23:31,299 --> 00:23:35,079
also gets 30 % for label 1. but it gets 70 %

495
00:23:35,079 --> 00:23:38,579
for label 2 and exactly 0 % for the remaining

496
00:23:38,579 --> 00:23:41,440
eight labels. So if we run our previous algorithms,

497
00:23:41,720 --> 00:23:44,299
least confidence and margin of confidence might

498
00:23:44,299 --> 00:23:47,059
struggle to see the true difference in the model's

499
00:23:47,059 --> 00:23:49,279
internal state because the top numbers are kind

500
00:23:49,279 --> 00:23:52,579
of similar? Exactly. But entropy calculates the

501
00:23:52,579 --> 00:23:56,039
total system chaos using a logarithmic function,

502
00:23:56,220 --> 00:23:59,400
specifically the sum of each probability multiplied

503
00:23:59,400 --> 00:24:02,000
by its logarithm. The math gets heavy there.

504
00:24:02,380 --> 00:24:05,440
It does, but when you run that math, image A

505
00:24:05,440 --> 00:24:08,339
has massive entropy because the probability is

506
00:24:08,339 --> 00:24:10,859
scattered everywhere. The model has absolutely

507
00:24:10,859 --> 00:24:13,220
no idea what is going on. It is just throwing

508
00:24:13,220 --> 00:24:16,880
darts at the board. And image B? Image B has

509
00:24:16,880 --> 00:24:19,559
much lower entropy because while it is confused,

510
00:24:19,799 --> 00:24:22,279
that confusion is highly concentrated into just

511
00:24:22,279 --> 00:24:25,619
two specific options. The total chaos is lower.

512
00:24:26,250 --> 00:24:28,930
Entropy provides the ultimate holistic view of

513
00:24:28,930 --> 00:24:31,849
the model's brain fog. Okay, those four algorithms,

514
00:24:32,069 --> 00:24:34,450
least confidence, margin, ratio, and entropy,

515
00:24:34,609 --> 00:24:36,890
all operate beautifully on a single probability

516
00:24:36,890 --> 00:24:39,789
distribution from a single model. But what if

517
00:24:39,789 --> 00:24:41,750
we aren't relying on a single model? What if

518
00:24:41,750 --> 00:24:44,329
we are running an ensemble? Which is super common

519
00:24:44,329 --> 00:24:46,569
in production. Right. You train five slightly

520
00:24:46,569 --> 00:24:48,630
different neural networks on the exact same data

521
00:24:48,630 --> 00:24:50,710
to get a more robust, stabilized prediction.

522
00:24:51,230 --> 00:24:54,069
How do you measure uncertainty across a committee

523
00:24:54,069 --> 00:24:58,029
of distinct AI models? The mathematical mechanism

524
00:24:58,029 --> 00:25:01,309
is wonderfully democratic. It is called query

525
00:25:01,309 --> 00:25:04,630
by committee. You take your massive pool of unlabeled

526
00:25:04,630 --> 00:25:07,190
data, and you feed every single item to all five

527
00:25:07,190 --> 00:25:10,170
models independently. Okay. If all five models

528
00:25:10,170 --> 00:25:12,809
vote that an image is a cat, system certainty

529
00:25:12,809 --> 00:25:15,750
is incredibly high. But if two models vote cat,

530
00:25:15,910 --> 00:25:19,029
two vote dog, and one votes car, you have high

531
00:25:19,029 --> 00:25:21,750
committee disagreement. You mathematically calculate

532
00:25:21,750 --> 00:25:23,829
the variance across the committee's predictions,

533
00:25:24,029 --> 00:25:26,269
and the items that generate the highest statistical

534
00:25:26,269 --> 00:25:28,369
variance are your most uncertain data points.

535
00:25:28,880 --> 00:25:31,900
That makes intuitive sense. It does. It is like

536
00:25:31,900 --> 00:25:34,539
asking a board of directors for a vote. But training

537
00:25:34,539 --> 00:25:37,220
and maintaining five entirely separate deep neural

538
00:25:37,220 --> 00:25:40,400
networks sounds computationally exhausting and

539
00:25:40,400 --> 00:25:43,559
incredibly expensive. It is. Very expensive.

540
00:25:43,940 --> 00:25:46,359
There is a technique mentioned in the book to

541
00:25:46,359 --> 00:25:48,779
simulate a committee without the overhead called

542
00:25:48,779 --> 00:25:52,480
dropouts. Explain how dropouts work. So in neural

543
00:25:52,480 --> 00:25:55,119
network architecture, a dropout is a mathematical

544
00:25:55,119 --> 00:25:57,559
technique typically used during the initial training

545
00:25:57,559 --> 00:26:00,799
phase. To prevent the model from simply memorizing

546
00:26:00,799 --> 00:26:04,259
the data at a basic level, it forces the network

547
00:26:04,259 --> 00:26:07,200
to randomly turn off a certain percentage of

548
00:26:07,200 --> 00:26:10,640
its, quote unquote, neurons or connections during

549
00:26:10,640 --> 00:26:13,420
each forward pass of the data. So if we use a

550
00:26:13,420 --> 00:26:16,019
corporate analogy, it is like a company making

551
00:26:16,019 --> 00:26:19,579
a critical strategic decision. If the CEO is

552
00:26:19,579 --> 00:26:22,420
unexpectedly absent, can the vice president make

553
00:26:22,420 --> 00:26:25,950
the same call? If you randomly send 10 % of the

554
00:26:25,950 --> 00:26:28,529
executive staff home every single day, does the

555
00:26:28,529 --> 00:26:31,029
company still arrive at the identical robust

556
00:26:31,029 --> 00:26:33,650
conclusion using different internal communication

557
00:26:33,650 --> 00:26:37,869
pathways? That is the perfect analogy. The network

558
00:26:37,869 --> 00:26:40,130
is forced to learn redundant pathways to the

559
00:26:40,130 --> 00:26:42,849
correct answer. Now, usually when you deploy

560
00:26:42,849 --> 00:26:45,349
the trained model into the real world for actual

561
00:26:45,349 --> 00:26:48,230
use, you turn dropouts off. You want the whole

562
00:26:48,230 --> 00:26:50,670
company working at 100 % capacity. Makes sense.

563
00:26:50,910 --> 00:26:53,730
But for active learning, You can intentionally

564
00:26:53,730 --> 00:26:56,089
leave dropouts on during the prediction phase.

565
00:26:56,690 --> 00:26:59,609
If you pass the exact same unlabeled image through

566
00:26:59,609 --> 00:27:02,529
the single network five different times, and

567
00:27:02,529 --> 00:27:04,950
the algorithm randomly turns off 10 % of the

568
00:27:04,950 --> 00:27:07,829
neurons differently each time, you essentially

569
00:27:07,829 --> 00:27:10,349
generate five distinct predictions from a single

570
00:27:10,349 --> 00:27:13,309
model. You are simulating a committee by randomly

571
00:27:13,309 --> 00:27:16,900
handicapping the stuff. Exactly. If the company's

572
00:27:16,900 --> 00:27:19,579
decision wildly swings every time a few different

573
00:27:19,579 --> 00:27:21,960
middle managers are out of the office, the model's

574
00:27:21,960 --> 00:27:24,839
underlying logic is fundamentally unstable. It

575
00:27:24,839 --> 00:27:28,200
is uncertain. But if it stubbornly outputs cat,

576
00:27:28,500 --> 00:27:31,220
no matter which neurons you randomly drop, the

577
00:27:31,220 --> 00:27:33,200
internal representation is deeply confident.

578
00:27:33,440 --> 00:27:36,619
Precisely. Dropouts are an incredibly elegant,

579
00:27:36,839 --> 00:27:39,440
computationally cheap way to expose the model's

580
00:27:39,440 --> 00:27:42,160
internal stability. Wow. But whether you use

581
00:27:42,160 --> 00:27:44,980
dropouts or entropy or margin, we have to recognize

582
00:27:44,980 --> 00:27:47,240
a fundamental limitation about the nature of

583
00:27:47,240 --> 00:27:50,500
uncertainty itself. What's fascinating here is

584
00:27:50,500 --> 00:27:52,440
there are two distinct types of uncertainty that

585
00:27:52,440 --> 00:27:55,579
these math equations uncover. Aleatoric uncertainty

586
00:27:55,579 --> 00:27:59,079
and epistemic uncertainty. Unpack the difference

587
00:27:59,079 --> 00:28:00,960
between those two because it dictates how we

588
00:28:00,960 --> 00:28:03,359
spend our engineering resources, right? Absolutely.

589
00:28:04,220 --> 00:28:06,779
Aleatoric uncertainty is the inherent, mathematically

590
00:28:06,779 --> 00:28:09,980
unavoidable noise in the data itself. It is a

591
00:28:09,980 --> 00:28:13,099
property of the world. Not the model. Okay, give

592
00:28:13,099 --> 00:28:16,759
me an example. Imagine you are training in audio

593
00:28:16,759 --> 00:28:19,400
transcription AI. You feed it a recording of

594
00:28:19,400 --> 00:28:21,759
a person speaking, but there is a blaring fire

595
00:28:21,759 --> 00:28:24,460
engine siren directly in the background. The

596
00:28:24,460 --> 00:28:26,720
model's entropy spikes. It is highly uncertain

597
00:28:26,720 --> 00:28:29,460
about what the person is saying. But a human

598
00:28:29,460 --> 00:28:32,079
annotator would be uncertain too. Right. The

599
00:28:32,079 --> 00:28:33,960
acoustic signal is literally destroyed by the

600
00:28:33,960 --> 00:28:36,609
siren. No amount of human labeling, no matter

601
00:28:36,609 --> 00:28:38,950
how many times you sample it, will miraculously

602
00:28:38,950 --> 00:28:42,309
reconstruct that audio. It is inherently, permanently

603
00:28:42,309 --> 00:28:46,430
ambiguous. So aleatoric uncertainty is just messy

604
00:28:46,430 --> 00:28:48,930
reality. You cannot engineer your way out of

605
00:28:48,930 --> 00:28:51,789
it. Right. If you spend all your human annotation

606
00:28:51,789 --> 00:28:54,250
budget having workers squint at blurry photos

607
00:28:54,250 --> 00:28:56,869
or listen to sirens, you are wasting millions

608
00:28:56,869 --> 00:29:00,109
of dollars. What we want to target with our algorithms

609
00:29:00,109 --> 00:29:03,329
is epistemic uncertainty. Epistemic uncertainty.

610
00:29:04,099 --> 00:29:06,779
Yes. Epistemic uncertainty is a property of the

611
00:29:06,779 --> 00:29:09,900
model. It occurs simply because the neural network

612
00:29:09,900 --> 00:29:12,920
lacks the required training data to form a mathematical

613
00:29:12,920 --> 00:29:15,900
understanding. A crystal clear, high definition

614
00:29:15,900 --> 00:29:19,259
photo of an electric unicycle has zero aleatoric

615
00:29:19,259 --> 00:29:22,339
noise. It is a perfect image. But the model's

616
00:29:22,339 --> 00:29:25,539
entropy spikes because of an epistemic gap. Its

617
00:29:25,539 --> 00:29:27,500
matrix of weights has never been optimized to

618
00:29:27,500 --> 00:29:30,609
recognize a unicycle. Exactly. Active learning

619
00:29:30,609 --> 00:29:32,809
algorithms are designed to hunt down and resolve

620
00:29:32,809 --> 00:29:36,109
epistemic uncertainty. So uncertainty sampling

621
00:29:36,109 --> 00:29:38,450
fixes the borders of our knowledge. It finds

622
00:29:38,450 --> 00:29:40,769
the unicycles sitting right on the edge of the

623
00:29:40,769 --> 00:29:43,470
decision boundary. But what happens if the model

624
00:29:43,470 --> 00:29:46,190
is perfectly confident but completely wrong about

625
00:29:46,190 --> 00:29:48,490
something it has never seen before? How do we

626
00:29:48,490 --> 00:29:50,930
find that? Because uncertainty sampling has a

627
00:29:50,930 --> 00:29:53,569
massive blind spot here. It does. Uncertainty

628
00:29:53,569 --> 00:29:56,220
sampling is... fantastic at refining the microscopic

629
00:29:56,220 --> 00:29:59,539
details of the decision boundary. But if you

630
00:29:59,539 --> 00:30:02,480
only use uncertainty algorithms, you fall into

631
00:30:02,480 --> 00:30:05,279
the peril of isolation. My mental analogy for

632
00:30:05,279 --> 00:30:07,960
this is a map. If uncertainty sampling is a team

633
00:30:07,960 --> 00:30:10,740
of surveyors staring intensely with magnifying

634
00:30:10,740 --> 00:30:13,640
glasses at the exact mathematical border between

635
00:30:13,640 --> 00:30:16,000
two countries, trying to get the line perfectly

636
00:30:16,000 --> 00:30:18,880
straight down to the millimeter, diversity sampling

637
00:30:18,880 --> 00:30:22,400
is panning the camera way, way out. to make sure

638
00:30:22,400 --> 00:30:24,980
you did not miss entire continents that are not

639
00:30:24,980 --> 00:30:27,759
even on the map yet. That is exactly the dynamic.

640
00:30:28,019 --> 00:30:30,079
You might get a perfectly sharp decision boundary

641
00:30:30,079 --> 00:30:33,259
for the data you currently have, but you remain

642
00:30:33,259 --> 00:30:35,880
mathematically blind to the data distributions

643
00:30:35,880 --> 00:30:38,900
you do not have. But here is where the logic

644
00:30:38,900 --> 00:30:41,859
feels like a paradox. If the model outputs a

645
00:30:41,859 --> 00:30:45,539
softmax probability of 99 .9%, confident in its

646
00:30:45,539 --> 00:30:48,119
completely wrong answer, and it has no mathematical

647
00:30:48,119 --> 00:30:51,559
idea that the missing continent exists, how on

648
00:30:51,559 --> 00:30:53,519
earth do we write a script to ask it to find

649
00:30:53,519 --> 00:30:55,960
what it does not know it is missing? To solve

650
00:30:55,960 --> 00:30:58,619
that paradox, we have to completely ignore the

651
00:30:58,619 --> 00:31:01,579
final softmax probability output. We have to

652
00:31:01,579 --> 00:31:03,460
look deeper into the architecture of the neural

653
00:31:03,460 --> 00:31:06,400
network itself. This brings us to the core mechanisms

654
00:31:06,400 --> 00:31:09,160
of diversity sampling. Okay. The first strategy

655
00:31:09,160 --> 00:31:12,230
is finding model -based outliers. To find these,

656
00:31:12,250 --> 00:31:14,230
we extract the data from the hidden layers of

657
00:31:14,230 --> 00:31:16,549
the network. How does looking at the hidden layers

658
00:31:16,549 --> 00:31:19,829
help us find the unknown unknowns? Remember how

659
00:31:19,829 --> 00:31:22,190
we discussed ImageNet and how the data passes

660
00:31:22,190 --> 00:31:24,809
sequentially through layers that detect edges,

661
00:31:24,910 --> 00:31:27,769
then textures, then shapes? Yeah. In a modern

662
00:31:27,769 --> 00:31:30,130
framework like PyTorch, you can actually pause

663
00:31:30,130 --> 00:31:32,970
the network and extract the mathematical activation

664
00:31:32,970 --> 00:31:36,269
values of every single neuron at every single

665
00:31:36,269 --> 00:31:39,619
layer as an image passes through. If we feed

666
00:31:39,619 --> 00:31:42,200
the network an image of a dog, the specific neurons

667
00:31:42,200 --> 00:31:45,259
optimized to detect fur and snouts light up.

668
00:31:45,299 --> 00:31:48,019
Their activation values spike numerically. Okay.

669
00:31:48,200 --> 00:31:50,779
But what if we feed it an image of a massive

670
00:31:50,779 --> 00:31:53,720
radar dish? The network's weights have never

671
00:31:53,720 --> 00:31:55,940
been trained to recognize the geometric structure

672
00:31:55,940 --> 00:31:59,220
of a radar dish. Exactly. So the radar dish image

673
00:31:59,220 --> 00:32:01,640
passes through the hidden layers and none of

674
00:32:01,640 --> 00:32:03,960
the feature detectors light up. The internal

675
00:32:03,960 --> 00:32:06,500
activation values remain incredibly low across

676
00:32:06,500 --> 00:32:09,400
the entire matrix. Now, because of the math of

677
00:32:09,400 --> 00:32:11,819
the final layer, the network might arbitrarily

678
00:32:11,819 --> 00:32:15,259
force a prediction and say, 99 % confident it

679
00:32:15,259 --> 00:32:18,079
is a stop sign, just because it has to output

680
00:32:18,079 --> 00:32:21,640
something that sums to 1 .0. Wow, so it is basically

681
00:32:21,640 --> 00:32:25,559
lying. Mathematically, yes. But if we analyze

682
00:32:25,559 --> 00:32:28,839
the hidden layers, we see the silence. The model

683
00:32:28,839 --> 00:32:31,720
is quietly experiencing a total failure of feature

684
00:32:31,720 --> 00:32:34,470
recognition. By writing a script to identify

685
00:32:34,470 --> 00:32:36,890
the unlabeled items that cause the lowest overall

686
00:32:36,890 --> 00:32:39,769
hidden layer activations, we discover our model

687
00:32:39,769 --> 00:32:42,150
-based outliers. These are the missing continents.

688
00:32:42,730 --> 00:32:46,130
That is incredibly elegant engineering. You ignore

689
00:32:46,130 --> 00:32:47,910
the confident shout at the end of the pipeline

690
00:32:47,910 --> 00:32:50,589
and you measure the silence in the internal processing.

691
00:32:50,869 --> 00:32:53,910
Okay, so that relies on the model. What if we

692
00:32:53,910 --> 00:32:56,269
want to bypass the model entirely and just look

693
00:32:56,269 --> 00:32:59,210
at the raw data? Then we use cluster -based sampling.

694
00:32:59,809 --> 00:33:02,309
This strategy relies entirely on the raw mathematical

695
00:33:02,309 --> 00:33:05,049
features of the data itself. We take our massive

696
00:33:05,049 --> 00:33:07,990
database of a million unlabeled text documents,

697
00:33:08,130 --> 00:33:10,289
and we group them using an unsupervised algorithm

698
00:33:10,289 --> 00:33:13,150
like k -means clustering. Unpack the math of

699
00:33:13,150 --> 00:33:15,369
k -means clustering for me. How does a computer

700
00:33:15,369 --> 00:33:18,130
know that two documents are similar without actually

701
00:33:18,130 --> 00:33:20,710
reading them? It translates the documents into

702
00:33:20,710 --> 00:33:23,430
geometry. Every document is converted into a

703
00:33:23,430 --> 00:33:26,109
feature vector. Imagine an arrow pointing in

704
00:33:26,109 --> 00:33:28,269
a multidimensional space where each dimension

705
00:33:28,269 --> 00:33:31,369
represents a specific word or concept. Okay,

706
00:33:31,430 --> 00:33:34,109
I am picturing the arrows. To find out if two

707
00:33:34,109 --> 00:33:36,950
documents are similar, K -Means calculates the

708
00:33:36,950 --> 00:33:39,789
cosine similarity between their vectors. What

709
00:33:39,789 --> 00:33:42,890
does cosine similarity actually measure? It measures

710
00:33:42,890 --> 00:33:46,130
the angle between the two arrows. It calculates

711
00:33:46,130 --> 00:33:48,289
the dot product of the vectors and divides it

712
00:33:48,289 --> 00:33:51,240
by the product of their magnitudes. If two documents

713
00:33:51,240 --> 00:33:53,680
use the exact same vocabulary to discuss financial

714
00:33:53,680 --> 00:33:56,039
markets, their arrows point in the exact same

715
00:33:56,039 --> 00:33:58,900
direction in that multidimensional space. The

716
00:33:58,900 --> 00:34:01,960
angle between them is zero degrees. Their cosine

717
00:34:01,960 --> 00:34:05,839
similarity is one. Got it. If one document is

718
00:34:05,839 --> 00:34:07,720
about finance and the other is about cooking,

719
00:34:08,019 --> 00:34:10,219
their arrows point in completely different directions.

720
00:34:10,280 --> 00:34:13,400
They are orthogonal. K -Means uses these angle

721
00:34:13,400 --> 00:34:16,199
calculations to mathematically pull all the similar

722
00:34:16,199 --> 00:34:18,980
documents together into K number of distinct

723
00:34:18,980 --> 00:34:22,250
clusters. Okay, so k -means gives us a cluster

724
00:34:22,250 --> 00:34:24,789
of financial news, a cluster of sports, a cluster

725
00:34:24,789 --> 00:34:28,010
of politics, all based purely on geometric angles.

726
00:34:28,329 --> 00:34:30,590
Once we have those clusters, how do we sample

727
00:34:30,590 --> 00:34:32,889
them to guarantee diversity? We systematically

728
00:34:32,889 --> 00:34:35,250
sample specific regions of the cluster geometry.

729
00:34:36,010 --> 00:34:38,469
First, you calculate and sample the centroid

730
00:34:38,469 --> 00:34:41,289
of each cluster. The centroid is the dead center,

731
00:34:41,429 --> 00:34:43,909
the mathematical average of all the vectors in

732
00:34:43,909 --> 00:34:46,570
that group. By sampling the centroids, you guarantee

733
00:34:46,570 --> 00:34:49,159
that you have at least one training item. representing

734
00:34:49,159 --> 00:34:51,920
every major thematic continent in your dataset.

735
00:34:52,489 --> 00:34:55,849
Okay, that covers the basics. Second, you sample

736
00:34:55,849 --> 00:34:58,289
the outliers of each cluster, the vectors whose

737
00:34:58,289 --> 00:35:00,889
angles place them furthest away from the centroid.

738
00:35:00,989 --> 00:35:03,789
These are the weird hybrid edge cases that bridge

739
00:35:03,789 --> 00:35:06,849
different topics. And finally, you sample random

740
00:35:06,849 --> 00:35:09,230
vectors from within the cluster volume to fill

741
00:35:09,230 --> 00:35:11,980
out the distribution. It is like a topological

742
00:35:11,980 --> 00:35:15,619
survey of the entire dataset. Okay, so we have

743
00:35:15,619 --> 00:35:18,559
model -based outliers looking at internal activations

744
00:35:18,559 --> 00:35:21,480
and cluster -based sampling looking at vector

745
00:35:21,480 --> 00:35:24,659
geometry. What is the third strategy for diversity?

746
00:35:25,340 --> 00:35:28,079
Representative sampling. This becomes mathematically

747
00:35:28,079 --> 00:35:30,559
critical when there is a phenomenon called domain

748
00:35:30,559 --> 00:35:33,280
shift. This is when the environment where your

749
00:35:33,280 --> 00:35:35,239
model was trained looks statistically different

750
00:35:35,239 --> 00:35:37,119
from the environment where it will be deployed.

751
00:35:37,500 --> 00:35:40,139
Give me a practical engineering example of domain

752
00:35:40,139 --> 00:35:42,949
shift. Let us say you collected millions of images

753
00:35:42,949 --> 00:35:44,809
and trained your autonomous vehicle strictly

754
00:35:44,809 --> 00:35:47,570
in the sunny, sprawling, flat suburbs of Phoenix,

755
00:35:47,690 --> 00:35:51,110
Arizona. The model's parameters are perfectly

756
00:35:51,110 --> 00:35:54,329
optimized for that environment. Sure. But now

757
00:35:54,329 --> 00:35:56,389
the company wants to deploy the car in the rainy,

758
00:35:56,550 --> 00:35:59,409
dense, vertical urban streets of Seattle, Washington.

759
00:35:59,789 --> 00:36:02,389
If you just take your Seattle video feed and

760
00:36:02,389 --> 00:36:05,170
randomly sample it, you might accidentally pull

761
00:36:05,170 --> 00:36:07,630
a lot of highway driving footage that statistically

762
00:36:07,630 --> 00:36:10,590
resembles Phoenix. Because it is the easiest

763
00:36:10,590 --> 00:36:13,710
to process. Right. Representative sampling fixes

764
00:36:13,710 --> 00:36:15,809
this by mathematically comparing the probability

765
00:36:15,809 --> 00:36:18,670
distribution of the new Seattle data pool against

766
00:36:18,670 --> 00:36:21,230
the old Phoenix training data. It explicitly

767
00:36:21,230 --> 00:36:23,510
calculates and selects the items that have the

768
00:36:23,510 --> 00:36:25,630
highest probability of coming from the new domain

769
00:36:25,630 --> 00:36:28,469
and the lowest probability of coming from the

770
00:36:28,469 --> 00:36:31,789
old domain. You actively force the model to ingest

771
00:36:31,789 --> 00:36:34,150
the environmental difference. And when we talk

772
00:36:34,150 --> 00:36:37,250
about environmental differences in diverse populations,

773
00:36:37,690 --> 00:36:39,929
we have to talk about how these mathematical

774
00:36:39,929 --> 00:36:42,449
sampling techniques intersect with demographic

775
00:36:42,449 --> 00:36:45,449
representation. This is a critical point of failure

776
00:36:45,449 --> 00:36:47,730
in many production systems. Yeah, it is a massive

777
00:36:47,730 --> 00:36:50,710
problem. If you rely purely on random sampling

778
00:36:50,710 --> 00:36:54,449
or even basic uncertainty sampling, your algorithm's

779
00:36:54,449 --> 00:36:56,750
loss function will naturally optimize to perform

780
00:36:56,750 --> 00:36:59,550
best on the majority demographic present in your

781
00:36:59,550 --> 00:37:01,670
data set, simply because that group generates

782
00:37:01,670 --> 00:37:05,130
the most data points. Right. The math will inherently

783
00:37:05,130 --> 00:37:08,409
deprioritize minority demographics because optimizing

784
00:37:08,409 --> 00:37:11,130
for them does not lower the global error rate

785
00:37:11,130 --> 00:37:13,429
as efficiently. Which mathematically leads to

786
00:37:13,429 --> 00:37:16,030
those systemic failures we see, right? Like where

787
00:37:16,030 --> 00:37:19,130
a facial recognition system operates at 99 %

788
00:37:19,130 --> 00:37:21,570
accuracy for one demographic, but completely

789
00:37:21,570 --> 00:37:24,170
fails to calculate the features of another demographic.

790
00:37:24,510 --> 00:37:27,670
Exactly. To solve this mathematically, engineers

791
00:37:27,670 --> 00:37:31,050
must implement stratified sampling. You cannot

792
00:37:31,050 --> 00:37:33,170
evaluate your model by just looking at the top

793
00:37:33,170 --> 00:37:36,210
-line global accuracy metric. A global accuracy

794
00:37:36,210 --> 00:37:40,030
of 95 % is a meaningless metric if the per -demographic

795
00:37:40,030 --> 00:37:42,849
accuracy reveals a severe subpopulation failure.

796
00:37:43,150 --> 00:37:45,789
So what does stratified sampling actually do?

797
00:37:46,190 --> 00:37:48,590
Stratified sampling forces the active learning

798
00:37:48,590 --> 00:37:51,909
loop to explicitly categorize the data into demographic

799
00:37:51,909 --> 00:37:55,230
strata, calculate the error rates for each stratum

800
00:37:55,230 --> 00:37:58,239
independently, and deliberately oversample data

801
00:37:58,239 --> 00:38:00,960
from the failing subpopulations until the mathematical

802
00:38:00,960 --> 00:38:04,539
accuracy reaches parity across all groups. It

803
00:38:04,539 --> 00:38:07,119
is an algorithmic requirement for a robust system.

804
00:38:07,239 --> 00:38:09,940
So we have these two massive engines. We have

805
00:38:09,940 --> 00:38:12,900
uncertainty sampling using softmax and entropy

806
00:38:12,900 --> 00:38:15,940
to sharpen the decision boundaries. And we have

807
00:38:15,940 --> 00:38:18,539
diversity sampling using k -means and hidden

808
00:38:18,539 --> 00:38:20,719
layer activations to explore the unknown map.

809
00:38:20,980 --> 00:38:23,579
Right. But if I am building a system, how do

810
00:38:23,579 --> 00:38:25,780
I actually combine these into a hybrid engine?

811
00:38:25,960 --> 00:38:28,860
How do these mathematical operations chain together

812
00:38:28,860 --> 00:38:31,559
in production? You almost never run these algorithms

813
00:38:31,559 --> 00:38:34,440
in isolation. The standard engineering architecture

814
00:38:34,440 --> 00:38:37,880
is a sequential funnel. Let us trace the data.

815
00:38:38,179 --> 00:38:41,099
You start with a massive raw database of a million

816
00:38:41,099 --> 00:38:44,840
unlabeled images. Step one, you run your uncertainty

817
00:38:44,840 --> 00:38:47,639
sampling, perhaps the least confidence algorithm.

818
00:38:48,349 --> 00:38:51,230
It scans the million images and outputs a filtered

819
00:38:51,230 --> 00:38:53,730
list of the 10 ,000 most mathematically uncertain

820
00:38:53,730 --> 00:38:56,929
items. But we do not just immediately send those

821
00:38:56,929 --> 00:38:59,150
10 ,000 items to our human workforce, right?

822
00:38:59,230 --> 00:39:01,710
Because they might all be identical? Precisely.

823
00:39:02,139 --> 00:39:04,440
If there is a recurring artifact in the data,

824
00:39:04,519 --> 00:39:07,380
say a specific type of camera lens flare, the

825
00:39:07,380 --> 00:39:09,840
model's uncertainty will spike every single time

826
00:39:09,840 --> 00:39:12,510
it sees it. If you just hand the humans the top

827
00:39:12,510 --> 00:39:15,550
10 ,000 most uncertain items, you might be paying

828
00:39:15,550 --> 00:39:17,929
them to label the exact same lens flare 10 ,000

829
00:39:17,929 --> 00:39:20,829
times. You have maximized uncertainty, but you

830
00:39:20,829 --> 00:39:24,010
have zero diversity. So you pipe the output of

831
00:39:24,010 --> 00:39:26,489
the first algorithm into the second. Yes. Step

832
00:39:26,489 --> 00:39:29,210
two. You take that tightly filtered pool of 10

833
00:39:29,210 --> 00:39:31,849
,000 highly uncertain images, and you run your

834
00:39:31,849 --> 00:39:34,329
diversity algorithm, like k -means cluster -based

835
00:39:34,329 --> 00:39:37,929
sampling, strictly on that subpool. You map the

836
00:39:37,929 --> 00:39:40,570
vector geometry of the confusing items. find

837
00:39:40,570 --> 00:39:43,289
the clusters, and extract the centroids and outliers.

838
00:39:43,690 --> 00:39:46,510
And then? Now, the final output that reaches

839
00:39:46,510 --> 00:39:49,150
the human annotator is a small, highly optimized

840
00:39:49,150 --> 00:39:52,070
batch of data points that are both deeply confusing

841
00:39:52,070 --> 00:39:55,190
to the neural network and geometrically distinct

842
00:39:55,190 --> 00:39:57,750
from one another. That is an elegant funnel.

843
00:39:58,090 --> 00:40:00,829
But is there a way to calculate the perfect intersection

844
00:40:00,829 --> 00:40:03,809
of uncertainty and diversity simultaneously in

845
00:40:03,809 --> 00:40:06,329
a single algorithm rather than chaining them

846
00:40:06,329 --> 00:40:10,289
sequentially? There is. And it is a computationally

847
00:40:10,289 --> 00:40:13,469
massive approach called expected error reduction.

848
00:40:14,449 --> 00:40:16,670
Instead of looking for uncertainty and diversity

849
00:40:16,670 --> 00:40:19,090
as separate metrics, this algorithm attempts

850
00:40:19,090 --> 00:40:22,110
to run a global simulation. A global simulation?

851
00:40:22,710 --> 00:40:25,949
Yes. For every single unlabeled item in the database,

852
00:40:26,190 --> 00:40:28,690
the algorithm mathematically simulates what would

853
00:40:28,690 --> 00:40:32,579
happen if a human provided a label for it. It

854
00:40:32,579 --> 00:40:34,840
temporarily adds that simulated label to the

855
00:40:34,840 --> 00:40:37,800
training set. simulates a complete retraining

856
00:40:37,800 --> 00:40:41,000
of the neural network's weights, and then calculates

857
00:40:41,000 --> 00:40:42,860
what the new global error rate of the entire

858
00:40:42,860 --> 00:40:46,239
system would be. Wait, simulating a full neural

859
00:40:46,239 --> 00:40:48,920
network retrain for every single item in a million

860
00:40:48,920 --> 00:40:51,719
-item database? The compute cost for that would

861
00:40:51,719 --> 00:40:55,559
be astronomical. It is incredibly computationally

862
00:40:55,559 --> 00:40:57,980
expensive, which is why it is rarely deployed

863
00:40:57,980 --> 00:41:00,599
in fast -paced, massive -scale production environments.

864
00:41:00,780 --> 00:41:03,900
But theoretically, it represents the absolute

865
00:41:04,349 --> 00:41:08,010
mathematical holy grail of active learning. Here's

866
00:41:08,010 --> 00:41:10,289
where it gets really interesting, though. If

867
00:41:10,289 --> 00:41:11,889
we are talking about cutting -edge architecture,

868
00:41:12,409 --> 00:41:15,989
there is a concept that feels like it crosses

869
00:41:15,989 --> 00:41:18,429
the line from engineering into actual sci -fi.

870
00:41:18,590 --> 00:41:22,769
It is an architecture called ATLAS, active transfer

871
00:41:22,769 --> 00:41:26,449
learning for adaptive sampling. Explain how ATLAS

872
00:41:26,449 --> 00:41:29,630
redefines the entire loop. ATLAS is where we

873
00:41:29,630 --> 00:41:32,500
transition into meta -learning. We build an AI

874
00:41:32,500 --> 00:41:34,699
whose entire purpose is to manage the education

875
00:41:34,699 --> 00:41:38,090
of another AI. Up to this point in our discussion,

876
00:41:38,230 --> 00:41:40,750
we have used hard -coded mathematical algorithms

877
00:41:40,750 --> 00:41:44,429
like softmax exponentials or k -means cosine

878
00:41:44,429 --> 00:41:47,030
similarity to figure out what data to sample.

879
00:41:47,409 --> 00:41:50,969
Atlas asks, why rely on static human -coded math?

880
00:41:51,269 --> 00:41:54,329
Why not instantiate a secondary deep neural network

881
00:41:54,329 --> 00:41:57,090
whose sole objective function is to predict the

882
00:41:57,090 --> 00:42:00,230
failures of the primary neural network? So we

883
00:42:00,230 --> 00:42:02,690
have the primary model. Which is trying to do

884
00:42:02,690 --> 00:42:04,889
the actual job, say, predicting if a medical

885
00:42:04,889 --> 00:42:07,929
scan shows a tumor or not. And we have the secondary

886
00:42:07,929 --> 00:42:11,150
model, the manager AI, watching the primary model

887
00:42:11,150 --> 00:42:13,929
work. Exactly. As the system runs, you feed the

888
00:42:13,929 --> 00:42:16,889
manager AI the raw data. And you feed it the

889
00:42:16,889 --> 00:42:20,309
primary model's resulting accuracy. Over millions

890
00:42:20,309 --> 00:42:23,070
of iterations, the manager AI's weights optimize

891
00:42:23,070 --> 00:42:25,730
to recognize the precise mathematical patterns

892
00:42:25,730 --> 00:42:29,110
that cause the primary model to fail. Wow. It

893
00:42:29,110 --> 00:42:31,940
learns to predict. Ah, based on the pixel gradients

894
00:42:31,940 --> 00:42:34,539
in this specific corner of the scan, I mathematically

895
00:42:34,539 --> 00:42:36,559
guarantee the primary model is going to output

896
00:42:36,559 --> 00:42:39,940
the wrong classification. The manager AI proactively

897
00:42:39,940 --> 00:42:41,880
intercepts and flags the data that will defeat

898
00:42:41,880 --> 00:42:44,400
the primary model, routing it to a human doctor

899
00:42:44,400 --> 00:42:46,239
for annotation before the primary model even

900
00:42:46,239 --> 00:42:48,590
has a chance to process it. So the sampling strategy

901
00:42:48,590 --> 00:42:51,469
itself becomes a continuously learning adaptive

902
00:42:51,469 --> 00:42:54,630
neural network. That is a mind -bending feedback

903
00:42:54,630 --> 00:42:57,510
loop. Now, so far, we have primarily anchored

904
00:42:57,510 --> 00:43:00,190
our math in simple binary classification tasks.

905
00:43:00,489 --> 00:43:03,090
Cat versus dog, disaster versus not disaster.

906
00:43:03,449 --> 00:43:06,429
But how do we translate these uncertainty algorithms

907
00:43:06,429 --> 00:43:09,510
into much more complex, multidimensional tasks?

908
00:43:10,010 --> 00:43:12,849
Let us look at object detection. If I am training

909
00:43:12,849 --> 00:43:15,369
an AI to draw bounding boxes around cars and

910
00:43:15,369 --> 00:43:17,949
pedestrians on a live video feed, How does uncertainty

911
00:43:17,949 --> 00:43:20,869
work? It is not just a single probability distribution

912
00:43:20,869 --> 00:43:23,909
anymore. Object detection requires a two -dimensional

913
00:43:23,909 --> 00:43:26,550
calculation of uncertainty. The first dimension

914
00:43:26,550 --> 00:43:30,110
is label confidence. Is the model's softmax output

915
00:43:30,110 --> 00:43:32,929
confident that the cluster of pixels inside the

916
00:43:32,929 --> 00:43:36,269
box represents a bicycle? But the second, equally

917
00:43:36,269 --> 00:43:38,690
critical dimension is localization confidence.

918
00:43:39,670 --> 00:43:42,539
Localization meaning the actual position. Yes.

919
00:43:42,840 --> 00:43:45,059
Is the model mathematically confident that the

920
00:43:45,059 --> 00:43:47,300
physical coordinates of the box are drawn tightly

921
00:43:47,300 --> 00:43:49,860
around the true edges of the bicycle and not

922
00:43:49,860 --> 00:43:51,980
accidentally cutting off half the front wheel?

923
00:43:52,539 --> 00:43:54,679
Right, because the neural network could be 100

924
00:43:54,679 --> 00:43:57,579
% confident it is looking at a bicycle, but the

925
00:43:57,579 --> 00:43:59,699
bounding box coordinates are floating three feet

926
00:43:59,699 --> 00:44:02,599
above it in the sky. Precisely. So to calculate

927
00:44:02,599 --> 00:44:04,840
the true uncertainty of an object detection model,

928
00:44:05,119 --> 00:44:07,659
engineers must combine the softmax probability

929
00:44:07,659 --> 00:44:11,239
of the label with a geometric calculation called

930
00:44:11,239 --> 00:44:15,820
intersection over union, or IOU. Unpack intersection

931
00:44:15,820 --> 00:44:18,460
over union. How do you mathematically calculate

932
00:44:18,460 --> 00:44:21,219
how tight a box is if the machine does not actually

933
00:44:21,219 --> 00:44:23,920
know where the real edges are? IOU is used to

934
00:44:23,920 --> 00:44:25,940
measure the overlap between two bounding boxes,

935
00:44:26,139 --> 00:44:29,239
usually the model's predicted box and the human's

936
00:44:29,239 --> 00:44:32,050
ground truth box. You mathematically calculate

937
00:44:32,050 --> 00:44:35,190
the area where the two boxes intersect, and you

938
00:44:35,190 --> 00:44:37,570
divide it by the total area of the union of both

939
00:44:37,570 --> 00:44:42,429
boxes. An IOU of 1 .0 means perfect pixel alignment.

940
00:44:42,710 --> 00:44:46,070
During active learning, models often generate

941
00:44:46,070 --> 00:44:48,409
multiple overlapping box proposals for a single

942
00:44:48,409 --> 00:44:51,719
object. By calculating the variance in the IOU

943
00:44:51,719 --> 00:44:54,480
across those internal proposals, we can mathematically

944
00:44:54,480 --> 00:44:56,460
estimate the model's localization confusion.

945
00:44:57,340 --> 00:44:59,840
We sample the frames where the model is confused

946
00:44:59,840 --> 00:45:02,639
about what the object is or geometrically confused

947
00:45:02,639 --> 00:45:05,079
about where its physical boundaries lie. What

948
00:45:05,079 --> 00:45:07,659
about text generation, sequence labeling, and

949
00:45:07,659 --> 00:45:10,760
machine translation? If I want a large language

950
00:45:10,760 --> 00:45:13,139
model to translate a complex legal paragraph

951
00:45:13,139 --> 00:45:16,320
from French to English or generate a summary

952
00:45:16,320 --> 00:45:19,000
of a document, there is no single correct bounding

953
00:45:19,000 --> 00:45:21,539
box. There are thousands of valid ways to write

954
00:45:21,539 --> 00:45:24,099
a sentence. How do you measure mathematical confusion

955
00:45:24,099 --> 00:45:27,460
when generating a fluid sequence of words? This

956
00:45:27,460 --> 00:45:29,260
is where the dropout architecture we discussed

957
00:45:29,260 --> 00:45:31,500
earlier becomes an incredibly powerful tool.

958
00:45:31,940 --> 00:45:34,500
If you ask a neural network to translate a sentence,

959
00:45:34,659 --> 00:45:37,119
it generates a sequential probability distribution

960
00:45:37,119 --> 00:45:40,159
for every single word. Okay. If you intentionally

961
00:45:40,159 --> 00:45:43,179
leave dropouts on, meaning you repeatedly pass

962
00:45:43,179 --> 00:45:45,179
the data through the network while randomly turning

963
00:45:45,179 --> 00:45:47,679
off different neurons, and you ask it to translate

964
00:45:47,679 --> 00:45:50,400
the exact same French sentence five times, you

965
00:45:50,400 --> 00:45:52,139
can measure the mathematical variance in the

966
00:45:52,139 --> 00:45:54,780
generated text sequences. So if it outputs the

967
00:45:54,780 --> 00:45:58,460
exact string the quick brown fox five times in

968
00:45:58,460 --> 00:46:01,239
a row, despite the internal pathways being constantly

969
00:46:01,239 --> 00:46:04,300
scrambled, the internal mathematical representation

970
00:46:04,300 --> 00:46:07,739
is highly stable and confident. Yes. But if it

971
00:46:07,739 --> 00:46:12,380
outputs the fast brown fox, then the speedy dark

972
00:46:12,380 --> 00:46:15,699
wolf, and then a quick orange dog, the model's

973
00:46:15,699 --> 00:46:18,559
vector space is highly unstable and deeply uncertain

974
00:46:18,559 --> 00:46:21,730
about the semantic translation. Exactly. You

975
00:46:21,730 --> 00:46:24,050
calculate the statistical divergence of the generated

976
00:46:24,050 --> 00:46:26,349
sequences, and the sentences with the highest

977
00:46:26,349 --> 00:46:28,389
divergence are the ones you flag and route to

978
00:46:28,389 --> 00:46:31,150
a human translator to correct. Which brings us

979
00:46:31,150 --> 00:46:34,369
to the human translators, the human annotators,

980
00:46:34,409 --> 00:46:37,449
the actual biological intelligence in the loop.

981
00:46:37,630 --> 00:46:39,949
Up to this point in our architecture, we have

982
00:46:39,949 --> 00:46:42,969
treated the human being like an infallible, perfectly

983
00:46:42,969 --> 00:46:45,449
calibrated oracle. We assume they are perfect.

984
00:46:45,929 --> 00:46:47,630
Right. The machine calculates its uncertainty.

985
00:46:47,809 --> 00:46:49,989
It routes the data to the human. The human provides

986
00:46:49,989 --> 00:46:53,090
the perfect mathematical truth and the machine

987
00:46:53,090 --> 00:46:55,710
updates its weights. But the reality of human

988
00:46:55,710 --> 00:46:58,130
annotation completely shatters that assumption.

989
00:46:58,349 --> 00:47:02,309
Human brains are messy, subjective, easily fatigued

990
00:47:02,309 --> 00:47:05,130
and highly prone to error. The human being is

991
00:47:05,130 --> 00:47:07,750
undeniably the most unpredictable high variance

992
00:47:07,750 --> 00:47:10,250
variable in the entire machine learning pipeline.

993
00:47:11,070 --> 00:47:14,059
Humans experience cognitive load. Their attention

994
00:47:14,059 --> 00:47:16,699
spans degrade over an eight -hour shift. They

995
00:47:16,699 --> 00:47:18,659
misinterpret complex guidelines, and they bring

996
00:47:18,659 --> 00:47:21,579
their own latent biases to subjective tasks.

997
00:47:21,960 --> 00:47:24,280
So how do we engineer around the human variable?

998
00:47:24,659 --> 00:47:27,260
First, we have to define who the humans actually

999
00:47:27,260 --> 00:47:30,059
are. The workforce architecture generally breaks

1000
00:47:30,059 --> 00:47:33,239
down into four categories. First, in -house experts.

1001
00:47:33,400 --> 00:47:35,760
These are highly compensated subject matter experts

1002
00:47:35,760 --> 00:47:38,599
like board -certified radiologists or senior

1003
00:47:38,599 --> 00:47:40,420
legal analysts sitting in your office. Right,

1004
00:47:40,460 --> 00:47:43,559
very expensive. Second, outsourced workers. This

1005
00:47:43,559 --> 00:47:46,139
is usually a dedicated managed team operating

1006
00:47:46,139 --> 00:47:49,119
through a specialized third -party company. Third,

1007
00:47:49,320 --> 00:47:51,719
crowdsourced workers. This is the gig economy.

1008
00:47:51,900 --> 00:47:54,039
Thousands of anonymous individuals distributed

1009
00:47:54,039 --> 00:47:57,119
globally, completing microtasks for pennies on

1010
00:47:57,119 --> 00:48:00,519
platforms like Amazon Mechanical Turk. And fourth,

1011
00:48:00,719 --> 00:48:03,480
end users. This is you and me clicking on a capture

1012
00:48:03,480 --> 00:48:05,739
grid to prove we aren't a script, unwittingly

1013
00:48:05,739 --> 00:48:07,619
providing labeled data for Google's autonomous

1014
00:48:07,619 --> 00:48:10,659
vehicle models. The structural choice of which

1015
00:48:10,659 --> 00:48:13,300
workforce you deploy radically dictates the quality

1016
00:48:13,300 --> 00:48:15,639
of the data and the design of the user interface.

1017
00:48:16,260 --> 00:48:18,900
But there is a phenomenal case study regarding

1018
00:48:18,900 --> 00:48:21,159
outsourced workers that completely challenges

1019
00:48:21,159 --> 00:48:23,420
the industry assumptions about who is capable

1020
00:48:23,420 --> 00:48:26,840
of executing highly complex annotation. It revolves

1021
00:48:26,840 --> 00:48:29,880
around a company called iMerit, founded by Radha

1022
00:48:29,880 --> 00:48:33,059
Basu. Yes. Walk us through the iMerit architecture.

1023
00:48:33,300 --> 00:48:35,599
This was so cool. So iMerit specifically engineered

1024
00:48:35,599 --> 00:48:38,699
their workforce to draw from marginalized underserved

1025
00:48:38,699 --> 00:48:41,119
communities, primarily employing young women

1026
00:48:41,119 --> 00:48:44,219
in rural areas of India. Now, a major medical

1027
00:48:44,219 --> 00:48:46,199
client came to them needing incredibly precise

1028
00:48:46,199 --> 00:48:49,619
pixel level annotation of robotic coronary artery

1029
00:48:49,619 --> 00:48:52,329
bypass graft videos. Basically, the annotators

1030
00:48:52,329 --> 00:48:55,309
had to watch surgical video of highly complex

1031
00:48:55,309 --> 00:48:58,610
robotic heart surgery and meticulously draw polygons

1032
00:48:58,610 --> 00:49:01,309
around the different bleeding anatomical structures

1033
00:49:01,309 --> 00:49:04,590
frame by frame. Exactly. The immediate assumption

1034
00:49:04,590 --> 00:49:07,170
from any engineering manager would be that you

1035
00:49:07,170 --> 00:49:09,309
absolutely must have an in -house expert for

1036
00:49:09,309 --> 00:49:12,650
that. A literal cardiovascular surgeon. Right.

1037
00:49:13,130 --> 00:49:15,530
But highly specialized surgeons are far too expensive

1038
00:49:15,530 --> 00:49:18,670
and far too busy saving actual lives to sit at

1039
00:49:18,670 --> 00:49:20,929
a monitor and draw polygons on thousands of hours

1040
00:49:20,929 --> 00:49:24,010
of video. Yeah, that is not happening. So iMerit

1041
00:49:24,010 --> 00:49:26,690
took their outsourced workforce, who had already

1042
00:49:26,690 --> 00:49:29,449
developed incredibly high -resolution 3D spatial

1043
00:49:29,449 --> 00:49:32,230
reasoning and mouse precision from previous computer

1044
00:49:32,230 --> 00:49:35,030
vision tasks, and they brought in a trained clinical

1045
00:49:35,030 --> 00:49:38,590
surgeon, not to annotate, but to teach. Oh, wow.

1046
00:49:39,190 --> 00:49:41,469
The surgeon taught these young workers advanced

1047
00:49:41,469 --> 00:49:44,809
clinical anatomy. These individuals from profoundly

1048
00:49:44,809 --> 00:49:47,690
under -resourced backgrounds became world -class,

1049
00:49:47,730 --> 00:49:50,550
hyper -accurate experts in identifying coronary

1050
00:49:50,550 --> 00:49:53,349
anatomy for AI training. That mathematically

1051
00:49:53,349 --> 00:49:55,570
proves that specialized domain knowledge can

1052
00:49:55,570 --> 00:49:58,010
be transferred and that robust investment in

1053
00:49:58,010 --> 00:50:00,449
workforce training unlocks incredible human potential

1054
00:50:00,449 --> 00:50:03,250
that the market usually ignores. It is a brilliant

1055
00:50:03,250 --> 00:50:06,489
structural solution. But even with highly trained,

1056
00:50:06,550 --> 00:50:08,969
dedicated workers, human brains still drop packets.

1057
00:50:09,230 --> 00:50:11,630
People make mistakes. So from an engineering

1058
00:50:11,630 --> 00:50:13,650
perspective, how do we mathematically verify

1059
00:50:13,650 --> 00:50:16,489
if our humans are actually right? How do we run

1060
00:50:16,489 --> 00:50:18,989
quality control on the teachers? The foundational

1061
00:50:18,989 --> 00:50:21,369
mechanism for quality control is the injection

1062
00:50:21,369 --> 00:50:24,599
of ground truth data. Before you ever unleash

1063
00:50:24,599 --> 00:50:26,679
your annotator workforce on the massive pool

1064
00:50:26,679 --> 00:50:29,960
of unknown data, your project leads, the domain

1065
00:50:29,960 --> 00:50:32,800
experts who design the architecture, meticulously

1066
00:50:32,800 --> 00:50:36,059
label a small, mathematically significant subset

1067
00:50:36,059 --> 00:50:39,400
of the data. These are the pristine gold standard

1068
00:50:39,400 --> 00:50:42,920
items. Like a secret answer key. Precisely. As

1069
00:50:42,920 --> 00:50:44,320
the annotators are working through their queue

1070
00:50:44,320 --> 00:50:47,000
of unknown data, the system's software periodically

1071
00:50:47,000 --> 00:50:50,000
randomly injects these ground truth items into

1072
00:50:50,000 --> 00:50:52,869
their feed. The annotator has no idea which item

1073
00:50:52,869 --> 00:50:55,769
is unknown and which is a test. Sneaky! Very.

1074
00:50:56,559 --> 00:50:58,800
Because the system's backend already has the

1075
00:50:58,800 --> 00:51:00,500
correct mathematical answer for the ground truth

1076
00:51:00,500 --> 00:51:03,360
item, it can instantly calculate the accuracy

1077
00:51:03,360 --> 00:51:05,860
metric of every single annotator in real time.

1078
00:51:06,139 --> 00:51:08,440
If an annotator's accuracy score in the ground

1079
00:51:08,440 --> 00:51:10,780
truth checks drops below a specific statistical

1080
00:51:10,780 --> 00:51:13,920
threshold, the system automatically flags them.

1081
00:51:14,219 --> 00:51:16,440
Because if they are failing the known tests,

1082
00:51:16,659 --> 00:51:18,719
the mathematical probability that their work

1083
00:51:18,719 --> 00:51:21,449
on the unknown data is flawed is unacceptably

1084
00:51:21,449 --> 00:51:24,809
high okay ground truth injection is a solid objective

1085
00:51:24,809 --> 00:51:27,429
metric but what if there is no clear objective

1086
00:51:27,429 --> 00:51:30,110
answer key what if the task is highly subjective

1087
00:51:30,110 --> 00:51:32,429
or the experts themselves look at an image and

1088
00:51:32,429 --> 00:51:35,860
disagree this introduces the concept of inter

1089
00:51:35,860 --> 00:51:38,480
-annotator agreement and i want to push back

1090
00:51:38,480 --> 00:51:41,059
on the complexity here for a second okay go for

1091
00:51:41,059 --> 00:51:43,519
it if i give a photograph to two different human

1092
00:51:43,519 --> 00:51:45,480
workers and they both click the button that says

1093
00:51:45,480 --> 00:51:49,739
dog aren't they just objectively right Why do

1094
00:51:49,739 --> 00:51:53,139
we need the intense, complex statistical formulas

1095
00:51:53,139 --> 00:51:55,559
to measure agreement? That assumption is one

1096
00:51:55,559 --> 00:51:58,300
of the most dangerous, common traps in data science.

1097
00:51:58,519 --> 00:52:01,300
You cannot simply look at the raw percentage

1098
00:52:01,300 --> 00:52:04,099
of agreement between two humans and assume validity.

1099
00:52:04,320 --> 00:52:07,300
You have to mathematically penalize the score

1100
00:52:07,300 --> 00:52:10,139
to adjust for random chance. Adjust for chance.

1101
00:52:10,219 --> 00:52:12,519
Walk me through the math of why that is necessary.

1102
00:52:12,840 --> 00:52:15,099
Imagine you have a strict binary classification

1103
00:52:15,099 --> 00:52:18,210
task. Is this a picture of a hot dog or is it

1104
00:52:18,210 --> 00:52:21,099
not a hot dog? You hire two monkeys to be your

1105
00:52:21,099 --> 00:52:23,219
annotators and you give them each a coin to flip.

1106
00:52:23,800 --> 00:52:26,239
Heads means hot dog, tails means not a hot dog.

1107
00:52:26,940 --> 00:52:29,719
Because it is a completely random binary choice,

1108
00:52:30,000 --> 00:52:32,360
the laws of probability dictate that the two

1109
00:52:32,360 --> 00:52:34,860
monkeys will randomly flip the same result, they

1110
00:52:34,860 --> 00:52:37,440
will quote unquote agree with each other, exactly

1111
00:52:37,440 --> 00:52:40,659
50 % of the time. Right. They both flip heads

1112
00:52:40,659 --> 00:52:44,300
25 % of the time and they both flip tails 25

1113
00:52:44,300 --> 00:52:47,309
% of the time. Total random agreement is 50%.

1114
00:52:47,309 --> 00:52:50,519
Exactly. So if you have two highly paid human

1115
00:52:50,519 --> 00:52:53,460
annotators working on a complex binary task and

1116
00:52:53,460 --> 00:52:55,860
your raw dashboard shows they agree with each

1117
00:52:55,860 --> 00:52:58,480
other 60 % of the time, they are statistically

1118
00:52:58,480 --> 00:53:00,679
barely doing better than the coin flipping monkeys.

1119
00:53:01,159 --> 00:53:04,780
A 60 % raw agreement on a binary task is actually

1120
00:53:04,780 --> 00:53:07,699
an atrocious signal. Oh man, I did not think

1121
00:53:07,699 --> 00:53:09,679
about it like that. You have to mathematically

1122
00:53:09,679 --> 00:53:12,440
calculate the probability of random chance for

1123
00:53:12,440 --> 00:53:15,380
your specific task and subtract it from your

1124
00:53:15,380 --> 00:53:17,639
raw agreement to find the true reliability of

1125
00:53:17,639 --> 00:53:20,099
your data. So what is the actual mathematical

1126
00:53:20,099 --> 00:53:23,420
equation for doing that? There are several statistical

1127
00:53:23,420 --> 00:53:26,639
metrics like Cohen's kappa or Fleiss kappa, but

1128
00:53:26,639 --> 00:53:29,599
the most mathematically robust tool is called

1129
00:53:29,599 --> 00:53:33,679
Krippendorff's alpha. Krippendorff's alpha. Right.

1130
00:53:34,769 --> 00:53:36,829
Krippendorf's alpha is a highly sophisticated

1131
00:53:36,829 --> 00:53:40,150
reliability coefficient. It calculates the observed

1132
00:53:40,150 --> 00:53:43,349
variance in the human annotations and subtracts

1133
00:53:43,349 --> 00:53:45,150
the expected variance that would occur by pure

1134
00:53:45,150 --> 00:53:47,829
chance. What makes alpha so powerful is that

1135
00:53:47,829 --> 00:53:50,670
its math scales perfectly, whether you have two

1136
00:53:50,670 --> 00:53:53,510
annotators or 200, and whether you are dealing

1137
00:53:53,510 --> 00:53:56,889
with binary, categorical labels, or continuous

1138
00:53:56,889 --> 00:53:59,329
numerical variables. Because the probability

1139
00:53:59,329 --> 00:54:01,710
of random chance completely changes depending

1140
00:54:01,710 --> 00:54:04,449
on the architecture of the task, right? If you

1141
00:54:04,449 --> 00:54:06,429
have a task with a thousand possible labels,

1142
00:54:06,650 --> 00:54:09,469
chance agreement plummets to near zero. T -monkeys

1143
00:54:09,469 --> 00:54:11,510
are never going to randomly guess the exact same

1144
00:54:11,510 --> 00:54:13,510
specific breed of dog out of a thousand options.

1145
00:54:13,909 --> 00:54:17,530
Precisely. Krippendorf's alpha dynamically adjusts

1146
00:54:17,530 --> 00:54:19,949
its internal expectations based on the number

1147
00:54:19,949 --> 00:54:22,550
of categories, the number of annotators, and

1148
00:54:22,550 --> 00:54:24,309
the statistical distribution of the specific

1149
00:54:24,309 --> 00:54:27,349
data set. It gives the engineering team a single,

1150
00:54:27,469 --> 00:54:29,929
highly reliable coefficient that represents the

1151
00:54:29,929 --> 00:54:33,030
true structural integrity of the data set. And

1152
00:54:33,030 --> 00:54:35,800
if that score is... Bad. If your alpha score

1153
00:54:35,800 --> 00:54:38,719
is low, it mathematically proves your humans

1154
00:54:38,719 --> 00:54:41,260
are confused or your instructions are vague.

1155
00:54:41,460 --> 00:54:43,880
And if your human teachers are confused, the

1156
00:54:43,880 --> 00:54:46,019
machine learning model trying to learn from them

1157
00:54:46,019 --> 00:54:49,150
does not stand a chance. Which means you have

1158
00:54:49,150 --> 00:54:51,130
to halt the pipeline and rewrite your annotation

1159
00:54:51,130 --> 00:54:53,710
guidelines. But let us say the guidelines are

1160
00:54:53,710 --> 00:54:56,730
clear and humans still disagree. If we have five

1161
00:54:56,730 --> 00:54:58,869
annotators look at a document and three vote

1162
00:54:58,869 --> 00:55:01,429
that it represents positive sentiment and two

1163
00:55:01,429 --> 00:55:03,989
vote that it represents negative sentiment, do

1164
00:55:03,989 --> 00:55:05,750
we just write a script to take a majority vote

1165
00:55:05,750 --> 00:55:08,389
and feed positive to the machine? Taking a simple

1166
00:55:08,389 --> 00:55:11,809
majority vote is a common default, but it structurally

1167
00:55:11,809 --> 00:55:14,929
degrades your model. What if the algorithm reveals

1168
00:55:14,929 --> 00:55:17,250
that the two people who voted negative are your

1169
00:55:17,250 --> 00:55:19,630
historically most accurate, highest performing

1170
00:55:19,630 --> 00:55:22,610
annotators based on ground truth checks, and

1171
00:55:22,610 --> 00:55:24,349
the three who voted positive are historically

1172
00:55:24,349 --> 00:55:29,110
sloppy and fast? Oh, right. A robust system mathematically

1173
00:55:29,110 --> 00:55:31,690
weights the votes based on the annotator's historical

1174
00:55:31,690 --> 00:55:35,170
reliability. You algorithmically trust the proven

1175
00:55:35,170 --> 00:55:38,110
experts more than the crowd. Furthermore, the

1176
00:55:38,110 --> 00:55:40,710
disagreement itself is a massive, valuable data

1177
00:55:40,710 --> 00:55:44,039
signal. How so? If a specific document generates

1178
00:55:44,039 --> 00:55:47,099
a three to two split among experts, that document

1179
00:55:47,099 --> 00:55:50,619
is inherently linguistically ambiguous. You should

1180
00:55:50,619 --> 00:55:53,179
not just flatten that rich ambiguity into a binary

1181
00:55:53,179 --> 00:55:56,239
positive label. You should design your loss function

1182
00:55:56,239 --> 00:55:59,079
to absorb that uncertainty, teaching the neural

1183
00:55:59,079 --> 00:56:01,960
network that this specific phrasing is borderline

1184
00:56:01,960 --> 00:56:04,860
and context dependent. We are explicitly teaching

1185
00:56:04,860 --> 00:56:07,519
the machine the statistical nuances of human

1186
00:56:07,519 --> 00:56:10,929
disagreement. Okay. This leads us perfectly into

1187
00:56:10,929 --> 00:56:13,829
the final dimension of our architecture, advanced

1188
00:56:13,829 --> 00:56:16,449
annotation and interface mechanics. Because if

1189
00:56:16,449 --> 00:56:18,309
humans disagree, sometimes it is not because

1190
00:56:18,309 --> 00:56:20,570
they are wrong or because the text is ambiguous.

1191
00:56:21,070 --> 00:56:23,489
Sometimes it is because the psychological environment

1192
00:56:23,489 --> 00:56:26,530
of the software interface itself is actively

1193
00:56:26,530 --> 00:56:29,090
working against their biology. Let us start with

1194
00:56:29,090 --> 00:56:31,849
highly subjective tasks. How do you engineer

1195
00:56:31,849 --> 00:56:33,889
a mathematical system to handle data where there

1196
00:56:33,889 --> 00:56:36,889
is literally no objective reality? Think about

1197
00:56:36,889 --> 00:56:39,789
a content moderation AI trying to identify toxicity

1198
00:56:39,789 --> 00:56:43,289
in a social media comment. What is perceived

1199
00:56:43,289 --> 00:56:45,469
as a highly offensive attack to one demographic

1200
00:56:45,469 --> 00:56:48,590
might be perceived as a harmless, sarcastic joke

1201
00:56:48,590 --> 00:56:51,550
to another. It is deeply culturally subjective.

1202
00:56:51,969 --> 00:56:55,349
Yeah, that is a mess. In these cases, you cannot

1203
00:56:55,349 --> 00:56:57,349
mathematically define a single ground truth.

1204
00:56:57,530 --> 00:57:00,949
You have to map the cultural variance. A fascinating

1205
00:57:00,949 --> 00:57:04,250
algorithmic approach to this is called the Bayesian

1206
00:57:04,250 --> 00:57:08,079
truth serum. The Bayesian truth serum. How does

1207
00:57:08,079 --> 00:57:12,019
an algorithm map subjective cultural truth? It

1208
00:57:12,019 --> 00:57:15,099
uses game theory and probability to restructure

1209
00:57:15,099 --> 00:57:17,920
the annotator's incentive model. Normally, you

1210
00:57:17,920 --> 00:57:20,599
just ask the human, is this comment toxic? But

1211
00:57:20,599 --> 00:57:22,840
with the Bayesian truth serum, you force the

1212
00:57:22,840 --> 00:57:25,559
annotator to answer two distinct questions. First,

1213
00:57:25,619 --> 00:57:28,320
you ask their subjective opinion, is this toxic?

1214
00:57:28,840 --> 00:57:31,480
Second, you ask them to predict the group's behavior.

1215
00:57:31,760 --> 00:57:34,019
What percentage of the other annotators do you

1216
00:57:34,019 --> 00:57:37,090
predict will say? This is toxic. That is fascinating.

1217
00:57:37,570 --> 00:57:40,230
It forces the human to step outside their own

1218
00:57:40,230 --> 00:57:43,710
subjective neurological bubble and intellectually

1219
00:57:43,710 --> 00:57:46,630
evaluate the broader cultural consensus. But

1220
00:57:46,630 --> 00:57:49,809
how does the math actually reward them? The algorithm

1221
00:57:49,809 --> 00:57:51,849
mathematically compares their prediction of the

1222
00:57:51,849 --> 00:57:54,170
group against the actual group distribution.

1223
00:57:54,610 --> 00:57:57,110
The highest statistical rewards, the highest

1224
00:57:57,110 --> 00:57:59,929
scores, are given to annotators who hold a minority

1225
00:57:59,929 --> 00:58:03,110
subjective opinion, but accurately predict that

1226
00:58:03,110 --> 00:58:05,829
their opinion is the minority. So, for example,

1227
00:58:05,909 --> 00:58:08,230
if someone personally thinks a comment is not

1228
00:58:08,230 --> 00:58:11,309
toxic, but they accurately predict that 80 %

1229
00:58:11,309 --> 00:58:14,030
of the workforce will find it toxic. The math

1230
00:58:14,030 --> 00:58:16,090
reveals that they possess a highly sophisticated,

1231
00:58:16,389 --> 00:58:18,650
objective understanding of the cultural landscape,

1232
00:58:18,869 --> 00:58:21,429
even if they disagree with it personally. It

1233
00:58:21,429 --> 00:58:24,710
is an algorithmic mechanism for mapping the true

1234
00:58:24,710 --> 00:58:28,170
subjective contours of a society. That is game

1235
00:58:28,170 --> 00:58:31,130
theory at its finest. And speaking of algorithms

1236
00:58:31,130 --> 00:58:34,230
monitoring the humans. We can actually use machine

1237
00:58:34,230 --> 00:58:36,809
learning for quality assurance. We can build

1238
00:58:36,809 --> 00:58:39,969
AI models whose only job is to check the humans

1239
00:58:39,969 --> 00:58:43,409
who are teaching the primary AI. Yes, it becomes

1240
00:58:43,409 --> 00:58:46,469
a deeply recursive loop. You can train a secondary

1241
00:58:46,469 --> 00:58:48,730
machine learning model to analyze the behavioral

1242
00:58:48,730 --> 00:58:51,750
metadata of the annotator to predict if they

1243
00:58:51,750 --> 00:58:53,889
are actually a bot. Like looking at how they

1244
00:58:53,889 --> 00:58:57,010
click? Exactly. If the telemetry data shows an

1245
00:58:57,010 --> 00:59:00,480
annotator is executing a highly complex... bounding

1246
00:59:00,480 --> 00:59:04,179
box task at exactly 1 .2 seconds per image. With

1247
00:59:04,179 --> 00:59:06,420
absolutely zero pixel variance in their mouse

1248
00:59:06,420 --> 00:59:09,139
trajectory over hundreds of clicks, the ML model

1249
00:59:09,139 --> 00:59:11,659
mathematically flags the statistical impossibility

1250
00:59:11,659 --> 00:59:14,719
of that human behavior, identifying it as a malicious

1251
00:59:14,719 --> 00:59:18,260
script. Wow. You can also train ML models to

1252
00:59:18,260 --> 00:59:20,179
look at the image, look at the human's applied

1253
00:59:20,179 --> 00:59:22,900
label, and calculate the probability that the

1254
00:59:22,900 --> 00:59:25,519
human made a mistake, automatically routing suspicious

1255
00:59:25,519 --> 00:59:28,360
annotations to a senior human manager. But the

1256
00:59:28,360 --> 00:59:30,400
physical, biological connection between the human

1257
00:59:30,400 --> 00:59:32,860
and the machine happens at the screen, the user

1258
00:59:32,860 --> 00:59:35,179
interface. And this brings up the incredible

1259
00:59:35,179 --> 00:59:37,739
psychological vulnerabilities of human -computer

1260
00:59:37,739 --> 00:59:40,659
interaction, specifically the neurological concept

1261
00:59:40,659 --> 00:59:43,840
of priming. We have to discuss the stuffed kiwi

1262
00:59:43,840 --> 00:59:45,860
in the kangaroo study. It is one of the most

1263
00:59:45,860 --> 00:59:47,679
illuminating psychological studies regarding

1264
00:59:47,679 --> 00:59:50,909
human perception and data integrity. So researchers

1265
00:59:50,909 --> 00:59:52,969
were conducting a linguistic study, bringing

1266
00:59:52,969 --> 00:59:54,889
human participants into a room, playing them

1267
00:59:54,889 --> 00:59:57,250
ambiguous audio clips and asking them to classify

1268
00:59:57,250 --> 00:59:59,949
whether the speaker had an Australian accent

1269
00:59:59,949 --> 01:00:02,889
or a New Zealand accent. But the researchers

1270
01:00:02,889 --> 01:00:05,690
introduced a completely silent visual variable.

1271
01:00:05,889 --> 01:00:08,130
In the room where the participants were working,

1272
01:00:08,289 --> 01:00:10,449
the researchers simply placed a stuffed toy on

1273
01:00:10,449 --> 01:00:12,809
a shelf in their peripheral vision. No one mentioned

1274
01:00:12,809 --> 01:00:15,269
the toy. Sometimes the toy was a stuffed kiwi

1275
01:00:15,269 --> 01:00:17,329
bird, which is the national cultural symbol of

1276
01:00:17,329 --> 01:00:19,610
New Zealand. Sometimes the toy was a stuffed

1277
01:00:19,610 --> 01:00:22,650
kangaroo, the iconic symbol of Australia. And

1278
01:00:22,650 --> 01:00:24,389
the statistical results of the classifications

1279
01:00:24,389 --> 01:00:27,250
were absolutely staggering. What happened? The

1280
01:00:27,250 --> 01:00:30,030
mere passive peripheral presence of the stuffed

1281
01:00:30,030 --> 01:00:32,769
kangaroo literally altered the neurological processing

1282
01:00:32,769 --> 01:00:35,670
of the audio. It caused listeners to systematically

1283
01:00:35,670 --> 01:00:38,889
interpret the ambiguous accents as more Australian.

1284
01:00:39,050 --> 01:00:41,730
When the kiwi bird was on the shelf, the exact

1285
01:00:41,730 --> 01:00:44,519
same audio. was classified as more New Zealand.

1286
01:00:44,699 --> 01:00:48,199
That is wild. Their literal biological sensory

1287
01:00:48,199 --> 01:00:50,820
perception of an acoustic signal was mathematically

1288
01:00:50,820 --> 01:00:54,340
skewed by a completely irrelevant visual object

1289
01:00:54,340 --> 01:00:56,539
in their environment. This phenomenon is called

1290
01:00:56,539 --> 01:00:58,840
priming. So translate the physics of that psychology

1291
01:00:58,840 --> 01:01:01,860
into data annotation. If a stuffed animal on

1292
01:01:01,860 --> 01:01:03,739
a shelf can rewrite human sensory perception,

1293
01:01:04,019 --> 01:01:06,420
what is happening to the neural pathways of our

1294
01:01:06,420 --> 01:01:08,239
workers who are staring at a glowing software

1295
01:01:08,239 --> 01:01:10,929
interface for eight hours a day? The most statistically

1296
01:01:10,929 --> 01:01:12,869
dangerous manifestation of this in our systems

1297
01:01:12,869 --> 01:01:16,449
is repetition priming. Imagine a human annotator

1298
01:01:16,449 --> 01:01:19,230
is classifying social media posts for toxic sentiment.

1299
01:01:19,449 --> 01:01:21,710
Due to random chance in the data queue, they

1300
01:01:21,710 --> 01:01:24,829
receive a batch that contains 99 highly negative,

1301
01:01:24,969 --> 01:01:29,190
intensely toxic posts in a row. As they click

1302
01:01:29,190 --> 01:01:31,309
through, their brain gets into a physical neurological

1303
01:01:31,309 --> 01:01:35,110
rhythm. Negative, click, negative, click, negative,

1304
01:01:35,349 --> 01:01:38,210
click. Their neural pathways are actively...

1305
01:01:38,460 --> 01:01:40,760
physically primed to expect the pattern of toxicity

1306
01:01:40,760 --> 01:01:43,820
to continue. And then the 100th post in the queue

1307
01:01:43,820 --> 01:01:46,239
is actually a positive, uplifting, joyful message.

1308
01:01:46,480 --> 01:01:49,679
Exactly. Because of repetition priming, the annotator's

1309
01:01:49,679 --> 01:01:52,139
brain is moving faster than their conscious comprehension.

1310
01:01:52,780 --> 01:01:55,500
They are highly statistically likely to misread

1311
01:01:55,500 --> 01:01:58,679
the positive text. or their motor cortex will

1312
01:01:58,679 --> 01:02:00,800
instinctively click the negative button before

1313
01:02:00,800 --> 01:02:03,219
their conscious brain can hit the brakes. They

1314
01:02:03,219 --> 01:02:05,320
will inject a mathematically false label into

1315
01:02:05,320 --> 01:02:07,739
the training data simply because of the sequential

1316
01:02:07,739 --> 01:02:10,420
order of the cue. So how do we engineer a solution

1317
01:02:10,420 --> 01:02:13,389
to a biological vulnerability like that? We fix

1318
01:02:13,389 --> 01:02:15,469
it through rigorous algorithmic shuffling and

1319
01:02:15,469 --> 01:02:18,510
interface design. The system backend must mathematically

1320
01:02:18,510 --> 01:02:21,630
randomize the data queue so annotators are never

1321
01:02:21,630 --> 01:02:24,010
subjected to long, unbroken streaks of the exact

1322
01:02:24,010 --> 01:02:26,710
same label. You have to algorithmically break

1323
01:02:26,710 --> 01:02:29,010
the repetition priming to reset their cognitive

1324
01:02:29,010 --> 01:02:32,269
baseline. Just shuffle it up. Right. But interface

1325
01:02:32,269 --> 01:02:35,170
design goes far beyond just shuffling data. When

1326
01:02:35,170 --> 01:02:37,150
building these tools, the paramount ergonomic

1327
01:02:37,150 --> 01:02:39,809
rule of human -computer interaction is keyboard

1328
01:02:39,809 --> 01:02:43,000
is king. meaning you absolutely do not want the

1329
01:02:43,000 --> 01:02:44,900
human taking their hand off the keyboard to use

1330
01:02:44,900 --> 01:02:47,639
the mouse. Fitz's law in ergonomics dictates

1331
01:02:47,639 --> 01:02:51,019
the time required to move to a target. Taking

1332
01:02:51,019 --> 01:02:53,119
your hands off the tactile home row of a keyboard

1333
01:02:53,119 --> 01:02:55,860
to visually locate a cursor, grab a physical

1334
01:02:55,860 --> 01:02:58,159
mouse, drag it across the screen, click a small

1335
01:02:58,159 --> 01:03:00,099
drop -down menu, and return your hand to the

1336
01:03:00,099 --> 01:03:03,559
keyboard destroys cognitive flow and massively

1337
01:03:03,559 --> 01:03:05,940
inflates the temporal cost per annotation. It

1338
01:03:05,940 --> 01:03:08,659
slows everything down. If an engineer can mathematically

1339
01:03:08,659 --> 01:03:11,980
cast their annotation problem as a binary or

1340
01:03:11,980 --> 01:03:15,099
categorical choice, press 1 for yes, 0 for no,

1341
01:03:15,199 --> 01:03:18,639
2 for unsure, the human's motor cortex locks

1342
01:03:18,639 --> 01:03:21,199
in and they can fly through the data with incredible

1343
01:03:21,199 --> 01:03:23,599
speed and accuracy. But not every machine learning

1344
01:03:23,599 --> 01:03:25,659
problem can be reduced to pressing a single number

1345
01:03:25,659 --> 01:03:28,340
key. Let us go back to sequence labeling. Let

1346
01:03:28,340 --> 01:03:31,019
us go back to translation. That inherently requires

1347
01:03:31,019 --> 01:03:33,820
the human to type out complex variable length

1348
01:03:33,820 --> 01:03:36,219
sentences. How do you optimize the UI for that?

1349
01:03:36,760 --> 01:03:39,940
If we look at the historical evolution of translation

1350
01:03:39,940 --> 01:03:42,800
interfaces, it provides the ultimate architectural

1351
01:03:42,800 --> 01:03:45,719
case study for human -computer interaction. For

1352
01:03:45,719 --> 01:03:48,340
decades, computer scientists logically assumed

1353
01:03:48,340 --> 01:03:50,380
that they could make human translators faster

1354
01:03:50,380 --> 01:03:53,500
by using an adjudication interface. Adjudication.

1355
01:03:53,639 --> 01:03:56,340
They would take the output of an early, slightly

1356
01:03:56,340 --> 01:03:59,079
clunky machine translation model, present that

1357
01:03:59,079 --> 01:04:01,739
finished draft to the human, and ask the human

1358
01:04:01,739 --> 01:04:04,440
to simply adjudicate it to edit it and fix the

1359
01:04:04,440 --> 01:04:06,840
errors. That sounds entirely logically sound.

1360
01:04:07,019 --> 01:04:09,760
Starting with an 80 % correct rough draft should

1361
01:04:09,760 --> 01:04:12,059
mathematically require less energy than starting

1362
01:04:12,059 --> 01:04:14,619
from a completely blank page. It sounds mathematically

1363
01:04:14,619 --> 01:04:17,159
sound, but in psychological practice, it failed

1364
01:04:17,159 --> 01:04:20,570
catastrophically. The cognitive load required

1365
01:04:20,570 --> 01:04:23,010
for a human to read a flawed machine translation,

1366
01:04:23,349 --> 01:04:27,050
identify the specific semantic error, highlight

1367
01:04:27,050 --> 01:04:30,210
it with a mouse, delete it, and type the correction

1368
01:04:30,210 --> 01:04:33,550
actually takes longer. and induces more mental

1369
01:04:33,550 --> 01:04:35,489
fatigue than simply allowing the human to translate

1370
01:04:35,489 --> 01:04:37,230
the sentence from scratch, from a blank page.

1371
01:04:37,449 --> 01:04:40,269
Oh, wow. The humans hated the adjudication interfaces.

1372
01:04:40,650 --> 01:04:43,829
It actively slowed down the pipeline. So what

1373
01:04:43,829 --> 01:04:46,469
was the architectural breakthrough? How did they

1374
01:04:46,469 --> 01:04:50,150
finally merge human and machine speed? The breakthrough

1375
01:04:50,150 --> 01:04:52,989
was not a more powerful neural network. The breakthrough

1376
01:04:52,989 --> 01:04:55,590
was a fundamentally redesigned user interface.

1377
01:04:56,110 --> 01:04:58,599
They abandoned the adjudication model. where

1378
01:04:58,599 --> 01:05:00,460
the human reviews the machine's finished work

1379
01:05:00,460 --> 01:05:04,119
and invented an assisted interface. They invented

1380
01:05:04,119 --> 01:05:08,139
predictive autocomplete text. Ah, so the psychological

1381
01:05:08,139 --> 01:05:11,440
dynamic completely flips. The human starts typing

1382
01:05:11,440 --> 01:05:13,460
their translation on a blank page, retaining

1383
01:05:13,460 --> 01:05:16,239
complete cognitive control, and the machine learning

1384
01:05:16,239 --> 01:05:18,860
model silently runs in the background, analyzing

1385
01:05:18,860 --> 01:05:21,260
the keystrokes and suggesting the rest of the

1386
01:05:21,260 --> 01:05:23,420
sentence in light gray text ahead of the cursor.

1387
01:05:23,659 --> 01:05:26,110
Yes. If the human agrees with the math, they

1388
01:05:26,110 --> 01:05:28,829
just hit tab to accept it. If they disagree,

1389
01:05:29,030 --> 01:05:30,929
they do not have to stop and delete anything.

1390
01:05:31,050 --> 01:05:32,789
They just keep typing right through the gray

1391
01:05:32,789 --> 01:05:35,429
text and the machine instantly recalculates its

1392
01:05:35,429 --> 01:05:38,550
probabilities. Exactly. The human retains complete

1393
01:05:38,550 --> 01:05:41,690
cognitive agency and flow. They never have to

1394
01:05:41,690 --> 01:05:43,750
break their physical rhythm to correct the machine's

1395
01:05:43,750 --> 01:05:46,590
mistakes. They simply ignore them. That single

1396
01:05:46,590 --> 01:05:49,550
HCI design shift, transitioning from adjudication

1397
01:05:49,550 --> 01:05:52,900
to assisted autocomplete. was what finally allowed

1398
01:05:52,900 --> 01:05:55,179
biological intelligence and artificial intelligence

1399
01:05:55,179 --> 01:05:58,659
to merge efficiently. It maximized the mathematical

1400
01:05:58,659 --> 01:06:00,920
throughput of the system without sacrificing

1401
01:06:00,920 --> 01:06:03,820
data quality or psychologically frustrating the

1402
01:06:03,820 --> 01:06:06,809
worker. Which perfectly synthesizes everything

1403
01:06:06,809 --> 01:06:09,489
we have explored today. It proves that state

1404
01:06:09,489 --> 01:06:11,869
-of -the -art AI is not just about the massive

1405
01:06:11,869 --> 01:06:14,110
neural network humming in the sterile server

1406
01:06:14,110 --> 01:06:17,690
room. It is entirely about the intricate, meticulously

1407
01:06:17,690 --> 01:06:21,130
evaluated, mathematically rigorous loop between

1408
01:06:21,130 --> 01:06:23,269
the human biology and the machine architecture.

1409
01:06:23,630 --> 01:06:26,469
It really is. We have traveled from the microscopic

1410
01:06:26,469 --> 01:06:29,550
isolation of softmax probabilities and k -means

1411
01:06:29,550 --> 01:06:31,789
vector clustering, plunging deep through the

1412
01:06:31,789 --> 01:06:34,610
hidden activation layers of PyTorch, all the

1413
01:06:34,610 --> 01:06:37,090
way out to the psychological realities of stuffed

1414
01:06:37,090 --> 01:06:40,469
kangaroos, coin -flipping monkeys, and the ergonomic

1415
01:06:40,469 --> 01:06:43,269
physics of keyboard shortcuts. It fundamentally

1416
01:06:43,269 --> 01:06:45,909
reshapes your entire paradigm of artificial intelligence.

1417
01:06:46,269 --> 01:06:49,369
It is not a standalone, pristine brain. It is

1418
01:06:49,369 --> 01:06:51,429
a highly sensitive, continuously maintained,

1419
01:06:51,690 --> 01:06:55,190
beautifully complex ecosystem of human, and machine

1420
01:06:55,190 --> 01:06:57,670
so what does this all mean for you our listener

1421
01:06:57,670 --> 01:07:00,789
we have seen how psychological phenomena like

1422
01:07:00,789 --> 01:07:03,369
repetition priming and the physical design of

1423
01:07:03,369 --> 01:07:06,269
an interface can literally alter an annotator's

1424
01:07:06,269 --> 01:07:09,429
biological perception of reality the decisions

1425
01:07:09,429 --> 01:07:11,710
they make dictate the mathematical weights of

1426
01:07:11,710 --> 01:07:15,920
the model if our future ai models the vast networks

1427
01:07:15,920 --> 01:07:18,079
that will drive our cars analyze our medical

1428
01:07:18,079 --> 01:07:21,840
scans and filter our global information are entirely

1429
01:07:21,840 --> 01:07:24,659
trained by human workers who are physically constrained

1430
01:07:24,659 --> 01:07:28,019
by highly specific machine -generated software

1431
01:07:28,019 --> 01:07:31,460
interfaces. That is the big question. Are we

1432
01:07:31,460 --> 01:07:33,360
actually building systems that understand the

1433
01:07:33,360 --> 01:07:37,550
messy, nuanced, beautiful human world? Or are

1434
01:07:37,550 --> 01:07:40,329
we slowly, iteratively forcing humans to perceive

1435
01:07:40,329 --> 01:07:43,190
the world like machines, clicking binary buttons

1436
01:07:43,190 --> 01:07:45,750
and following algorithmic paths of least resistance

1437
01:07:45,750 --> 01:07:49,789
just so the mathematics finally works out? Something

1438
01:07:49,789 --> 01:07:52,070
to deeply think about. the next time your software

1439
01:07:52,070 --> 01:07:54,090
silently tries to autocomplete your sentence.