1
00:00:00,000 --> 00:00:04,679
Picture a fuzzy yellow banana plushie. OK. I

2
00:00:04,679 --> 00:00:06,799
have it in my head. It's got these little stick

3
00:00:06,799 --> 00:00:10,539
arms and like angry embroidered eyes. Oh, wow.

4
00:00:10,740 --> 00:00:14,240
Yeah. And a really wide screaming mouth. It is

5
00:00:14,240 --> 00:00:17,000
completely chaotic. That sounds terrifying, honestly.

6
00:00:17,160 --> 00:00:19,660
Right. Now, if you take a photo of that exact

7
00:00:19,660 --> 00:00:22,420
toy and you hand it to a billion dollar state

8
00:00:22,420 --> 00:00:25,039
of the art computer vision AI from, say, the

9
00:00:25,039 --> 00:00:28,059
year 2020. It would probably just have a total

10
00:00:28,059 --> 00:00:31,519
meltdown. Exactly. it will likely tell you it's

11
00:00:31,519 --> 00:00:34,200
a throw pillow. Or maybe it'll say it's a dog

12
00:00:34,200 --> 00:00:36,820
toy. Or it might just spit out a complete error

13
00:00:36,820 --> 00:00:40,039
code. Yeah, exactly. But today, we're going to

14
00:00:40,039 --> 00:00:42,039
look at the landmark paper that finally taught

15
00:00:42,039 --> 00:00:44,560
a machine to look at that screaming fuzzy mess

16
00:00:44,560 --> 00:00:47,579
and correctly say, you know, that is a banana.

17
00:00:48,119 --> 00:00:50,759
Which is honestly the ultimate stress test for

18
00:00:50,759 --> 00:00:54,579
an AI. Welcome to today's deep dive. Glad to

19
00:00:54,579 --> 00:00:58,750
be here. That plushie represents, like, everything

20
00:00:58,750 --> 00:01:00,890
that used to absolutely break machine learning

21
00:01:00,890 --> 00:01:04,250
models. It really does. Because whether you are

22
00:01:04,409 --> 00:01:07,310
catching up on AI research for your career or

23
00:01:07,310 --> 00:01:09,730
you're just intensely curious about how machines

24
00:01:09,730 --> 00:01:12,090
actually learn to understand the visual world,

25
00:01:12,989 --> 00:01:15,750
you are getting a front row seat to a massive

26
00:01:15,750 --> 00:01:18,650
paradigm shift today. A huge one. We are going

27
00:01:18,650 --> 00:01:21,829
to save you hours of parsing through dense academic

28
00:01:21,829 --> 00:01:25,469
jargon. We're giving you the absolute core. of

29
00:01:25,469 --> 00:01:28,250
why this shift matters. Yeah, we're doing a meticulous

30
00:01:28,250 --> 00:01:30,870
chapter -by -chapter unpacking of a truly landmark

31
00:01:30,870 --> 00:01:34,170
paper. Right. It's called Learning Transferable

32
00:01:34,170 --> 00:01:37,209
Visual Models from Natural Language Supervision.

33
00:01:37,489 --> 00:01:40,790
And this is by Alec Radford, Jongwook Kim, Chris

34
00:01:40,790 --> 00:01:42,849
Hallisey, and a whole team of researchers at

35
00:01:42,849 --> 00:01:45,989
OpenAI. In this paper, it introduces a model

36
00:01:45,989 --> 00:01:50,150
called CLIP. Right, CLIP. Which stands for Contrastive

37
00:01:50,150 --> 00:01:52,469
Language Image Pre -Training. Which is a lot

38
00:01:52,469 --> 00:01:54,670
of words, but we'll break it down. It's basically

39
00:01:54,670 --> 00:01:57,209
a fundamental rethinking of how we teach computers

40
00:01:57,209 --> 00:01:59,069
to see. And to really ground this, we've got

41
00:01:59,069 --> 00:02:02,549
a highly specific set of 37 visual sources provided

42
00:02:02,549 --> 00:02:04,989
for this deep dive. The banana collection. Yes,

43
00:02:05,069 --> 00:02:06,590
the banana collection. But it's not just a regular

44
00:02:06,590 --> 00:02:08,330
bunch of fruit you'd find at the grocery store.

45
00:02:08,590 --> 00:02:11,110
No, it's way more diverse than that. We've got

46
00:02:11,110 --> 00:02:13,710
real yellow bananas, sure. But we also have rotting

47
00:02:13,710 --> 00:02:16,030
brown bananas lying on concrete. Right. We've

48
00:02:16,030 --> 00:02:19,030
got 3D pixelated bananas with an N64 logo on

49
00:02:19,030 --> 00:02:21,610
them. We have rough sketches of bananas on scrap

50
00:02:21,610 --> 00:02:23,870
paper. And, of course, the screaming plushie.

51
00:02:24,150 --> 00:02:26,310
My personal favorite. We're going to use all

52
00:02:26,310 --> 00:02:28,669
of these to figure out how Clip actually bridges

53
00:02:28,669 --> 00:02:31,830
this massive gap between text and images. It's

54
00:02:31,830 --> 00:02:34,349
a great set of examples. But to really appreciate

55
00:02:34,349 --> 00:02:36,750
the leap that Clip made, I mean, we have to talk

56
00:02:36,750 --> 00:02:38,629
about the landscape right before this paper dropped.

57
00:02:38,810 --> 00:02:40,750
We have to set the stage. Yeah, because there

58
00:02:40,750 --> 00:02:42,969
was this great divide in AI, right? A massive

59
00:02:42,969 --> 00:02:45,590
divide, yes. Natural language processing or NLP

60
00:02:45,590 --> 00:02:47,870
was doing one thing, and computer vision was

61
00:02:47,870 --> 00:02:50,060
doing something completely different. The divergence

62
00:02:50,060 --> 00:02:52,219
was stark. I mean, right around the time this

63
00:02:52,219 --> 00:02:54,020
paper was being developed, we were seeing this

64
00:02:54,020 --> 00:02:57,259
huge revolution at NLP. Right. Models like GPT

65
00:02:57,259 --> 00:03:00,460
-3 had just burst onto the scene. Exactly. And

66
00:03:00,460 --> 00:03:02,979
they were achieving unprecedented successes,

67
00:03:03,180 --> 00:03:05,500
like truly mind blowing stuff. And what was the

68
00:03:05,500 --> 00:03:07,759
secret to their success? It was their training

69
00:03:07,759 --> 00:03:10,819
diet. They were pre -trained directly on raw,

70
00:03:11,159 --> 00:03:13,120
completely unstructured text scraped from the

71
00:03:13,120 --> 00:03:16,500
wild internet. So they weren't being like explicitly

72
00:03:16,500 --> 00:03:18,819
tutored by humans with flashcards. No, not at

73
00:03:18,819 --> 00:03:20,789
all. They were just reading everything. They

74
00:03:20,789 --> 00:03:24,069
were absorbing petabytes of human language. Just

75
00:03:24,069 --> 00:03:26,409
passively reading. Basically, yeah. They were

76
00:03:26,409 --> 00:03:28,469
essentially just predicting the next word over

77
00:03:28,469 --> 00:03:31,150
and over again on billions of websites. And that

78
00:03:31,150 --> 00:03:34,069
made them task agnostic, right? Precisely. Because

79
00:03:34,069 --> 00:03:36,409
they absorbed the underlying structure and the

80
00:03:36,409 --> 00:03:38,729
actual logic of language, you could apply them

81
00:03:38,729 --> 00:03:42,250
to entirely new problems. Like translating languages

82
00:03:42,250 --> 00:03:44,710
or summarizing articles. Or even writing code.

83
00:03:45,050 --> 00:03:47,669
And you could do all that without needing a custom

84
00:03:47,669 --> 00:03:49,930
hand -labeled data set for every single specific

85
00:03:49,930 --> 00:03:52,810
task. I like to think of the NLP approach like

86
00:03:52,810 --> 00:03:55,289
a student who has spent years just sitting in

87
00:03:55,289 --> 00:03:58,389
a massive public library, literally reading everything.

88
00:03:58,669 --> 00:04:01,270
That's a great way to look at it. Like they read

89
00:04:01,270 --> 00:04:04,050
history, science fiction, stereo instruction

90
00:04:04,050 --> 00:04:06,990
manuals, everything. Just gathering general knowledge.

91
00:04:07,349 --> 00:04:10,210
Yeah. So when test day comes, even if you ask

92
00:04:10,210 --> 00:04:12,569
them a completely unexpected question, they can

93
00:04:12,569 --> 00:04:14,830
reason through it. Because they have this massive

94
00:04:14,830 --> 00:04:17,629
interconnected web of context to draw from. Exactly.

95
00:04:18,189 --> 00:04:21,029
But then you look over a computer vision at the

96
00:04:21,029 --> 00:04:24,029
exact same time in history. And it is a completely

97
00:04:24,029 --> 00:04:25,930
different story. It feels like they were stuck

98
00:04:25,930 --> 00:04:28,930
in the flashcard era. They absolutely were. Computer

99
00:04:28,930 --> 00:04:31,670
vision was still relying heavily on meticulously

100
00:04:31,670 --> 00:04:35,209
curated crowd -labeled data sets. And the undisputed

101
00:04:35,209 --> 00:04:37,250
king of those data sets was ImageNet, right?

102
00:04:37,569 --> 00:04:39,910
Oh, without a doubt. ImageNet was the gold standard.

103
00:04:40,389 --> 00:04:42,329
Okay, let's establish the baseline here for everyone.

104
00:04:42,860 --> 00:04:46,459
What is ImageNet, and why was it eventually holding

105
00:04:46,459 --> 00:04:49,459
the field back? Well, ImageNet is a phenomenal

106
00:04:49,459 --> 00:04:51,120
achievement in its own right, let's be clear.

107
00:04:51,199 --> 00:04:54,060
Oh, for sure. It consists of about 1 .28 million

108
00:04:54,060 --> 00:04:56,620
training images. Which sounds like a lot. It

109
00:04:56,620 --> 00:05:00,240
does, but those images are spread across exactly

110
00:05:00,240 --> 00:05:05,000
1 ,000 fixed object categories. Okay, just 1

111
00:05:05,000 --> 00:05:07,980
,000 things. Right. And to build it, you had

112
00:05:07,980 --> 00:05:10,879
thousands of human workers manually looking at

113
00:05:10,879 --> 00:05:13,029
photos and tagging them. tagging them with one

114
00:05:13,029 --> 00:05:16,050
specific label. Yes, a single definitive label

115
00:05:16,050 --> 00:05:19,310
from that fixed list of 1 ,000, like, um, Golden

116
00:05:19,310 --> 00:05:22,709
Retriever. Or Fireboat. Or Granny Smith Apple.

117
00:05:23,350 --> 00:05:25,610
So if the NLP student was the one reading the

118
00:05:25,610 --> 00:05:28,189
whole library, the computer vision student was

119
00:05:28,189 --> 00:05:32,189
handed a massive stack of exactly 1 ,000 specific

120
00:05:32,189 --> 00:05:35,430
flashcards. Yes, and they memorize those flashcards

121
00:05:35,430 --> 00:05:37,970
flawlessly. But if you hand that computer vision

122
00:05:37,970 --> 00:05:40,930
student a test that asks a question about something,

123
00:05:41,269 --> 00:05:44,100
well, Something's not on those cards. Complete

124
00:05:44,100 --> 00:05:46,899
and utter failure. Really? Just crashes? Pretty

125
00:05:46,899 --> 00:05:49,839
much. If you trained an ImageNet model to recognize

126
00:05:49,839 --> 00:05:52,000
a thousand things, and you suddenly needed it

127
00:05:52,000 --> 00:05:54,620
to recognize a thousand and first thing... Like

128
00:05:54,620 --> 00:05:57,319
our banana plushie. Exactly. You couldn't just

129
00:05:57,319 --> 00:05:59,560
ask it. It physically lacked the architecture

130
00:05:59,560 --> 00:06:01,959
to output a new guess. It just didn't have the

131
00:06:01,959 --> 00:06:04,579
word in its vocabulary. It didn't. So you had

132
00:06:04,579 --> 00:06:07,399
to go out, gather thousands of images of that

133
00:06:07,399 --> 00:06:10,720
new concept, manually label them, append them

134
00:06:10,720 --> 00:06:13,180
to the data set and retrain the entire model.

135
00:06:13,420 --> 00:06:16,379
That sounds incredibly rigid. And expensive.

136
00:06:17,000 --> 00:06:19,899
And expensive. It was incredibly rigid. It took

137
00:06:19,899 --> 00:06:22,480
so much time and human effort. This is the part

138
00:06:22,480 --> 00:06:25,550
that always bugged me about this era of AI. What's

139
00:06:25,550 --> 00:06:28,129
that? Well, if raw text free training, you know,

140
00:06:28,189 --> 00:06:30,110
just letting the AI loose on the messy internet,

141
00:06:30,649 --> 00:06:33,550
was such a massive, undeniable slam dunk for

142
00:06:33,550 --> 00:06:35,829
natural language processing. Right, with GPT

143
00:06:35,829 --> 00:06:38,709
and all that. Yeah. Why was computer vision still

144
00:06:38,709 --> 00:06:41,370
obsessing over these curated flashcards? Why

145
00:06:41,370 --> 00:06:43,470
hadn't they just scraped the internet for pictures

146
00:06:43,470 --> 00:06:45,529
and their captions and done the exact same thing?

147
00:06:45,750 --> 00:06:47,889
It's a great question. Yeah. And it wasn't for

148
00:06:47,889 --> 00:06:49,930
lack of imagination, I promise you. Oh, so they

149
00:06:49,930 --> 00:06:52,670
thought about it? Of course. But it was a massive

150
00:06:52,670 --> 00:06:55,629
computational and mathematical barrier. A barrier.

151
00:06:56,230 --> 00:06:59,750
How so? Getting a model to map the chaotic, infinite

152
00:06:59,750 --> 00:07:02,529
variations of natural language to complex arrays

153
00:07:02,529 --> 00:07:04,810
of visual pixels is just fundamentally harder.

154
00:07:05,189 --> 00:07:07,480
Harder than what? Harder than mapping an image

155
00:07:07,480 --> 00:07:10,459
to a simple standardized gold label category

156
00:07:10,459 --> 00:07:13,600
like the ImageNet flashcards. I need you to break

157
00:07:13,600 --> 00:07:16,720
that down for me because to me pixels are just

158
00:07:16,720 --> 00:07:20,550
data, right? And text is just data. They are.

159
00:07:20,930 --> 00:07:23,029
But think about the math of a fixed category.

160
00:07:23,670 --> 00:07:26,689
Think about a dog tag on a flashcard. Okay. It's

161
00:07:26,689 --> 00:07:29,850
what we call a one of N majority vote. It's very

162
00:07:29,850 --> 00:07:32,430
clean. Meaning there's one right answer out of

163
00:07:32,430 --> 00:07:36,279
N options. Exactly. The model processes the image

164
00:07:36,279 --> 00:07:38,959
and the output layer just has to assign a high

165
00:07:38,959 --> 00:07:42,560
probability score to the dog bucket. And low

166
00:07:42,560 --> 00:07:46,259
scores to the other 999 buckets. Right. It's

167
00:07:46,259 --> 00:07:50,500
neat. It's tidy. But natural language isn't a

168
00:07:50,500 --> 00:07:52,879
bucket. No, it's definitely not. A picture of

169
00:07:52,879 --> 00:07:55,579
a dog on the internet isn't neatly labeled dog.

170
00:07:56,420 --> 00:07:58,540
It's usually something weird. Right. It might

171
00:07:58,540 --> 00:08:02,060
be captioned, my good boy Max at the beach or

172
00:08:02,060 --> 00:08:04,790
running in the sand. Or it might just be three

173
00:08:04,790 --> 00:08:08,209
fire emojis and a heart. Exactly. So the AI has

174
00:08:08,209 --> 00:08:10,930
to somehow figure out the phrase, good boy Max,

175
00:08:11,430 --> 00:08:13,750
corresponds to the pixels representing the fur.

176
00:08:13,910 --> 00:08:16,430
Oh, I see. And that the word beach corresponds

177
00:08:16,430 --> 00:08:19,069
to the pixels representing the sand in the background.

178
00:08:19,230 --> 00:08:21,230
And has to do that without anyone explicitly

179
00:08:21,230 --> 00:08:23,550
drawing a line between those specific words and

180
00:08:23,550 --> 00:08:26,449
those specific pixels. Yes. The mapping space

181
00:08:26,449 --> 00:08:29,790
between the visual world and the infinite, poetic,

182
00:08:29,949 --> 00:08:33,509
sarcastic, messy ways humans describe it is just

183
00:08:33,509 --> 00:08:36,889
vast. I guess standardizing the labels into those

184
00:08:36,889 --> 00:08:39,970
1 ,000 neat little boxes was kind of a necessary

185
00:08:39,970 --> 00:08:43,769
crutch then. It was a crucial crutch for a decade.

186
00:08:44,049 --> 00:08:45,929
It was the only way to get computer vision to

187
00:08:45,929 --> 00:08:48,129
function at all. We had to simplify the problem

188
00:08:48,129 --> 00:08:50,830
for the AI. Exactly, because the compute power

189
00:08:50,830 --> 00:08:53,710
and the algorithmic architecture to handle the

190
00:08:53,710 --> 00:08:57,230
raw messy truth just weren't there yet. OK, but

191
00:08:57,230 --> 00:08:58,870
researchers must have been chipping away at this,

192
00:08:58,990 --> 00:09:01,009
right? I mean, someone must have realized that

193
00:09:01,009 --> 00:09:03,289
marrying text and images was the holy grail long

194
00:09:03,289 --> 00:09:05,809
before the OpenAI team published the clip paper.

195
00:09:06,169 --> 00:09:08,590
Oh, absolutely. The clip paper spends a significant

196
00:09:08,590 --> 00:09:10,629
amount of time acknowledging what it calls the

197
00:09:10,629 --> 00:09:13,590
pragmatic middle ground. The middle ground between

198
00:09:13,590 --> 00:09:16,289
flashcards and reading the whole internet. Right.

199
00:09:16,850 --> 00:09:18,789
The history of trying to combine text and images

200
00:09:18,789 --> 00:09:20,909
actually goes back over two decades. Wow, really?

201
00:09:21,049 --> 00:09:23,389
Yeah. And if we want to truly understand why

202
00:09:23,389 --> 00:09:25,330
CLIP is such a breakthrough, we have to look

203
00:09:25,330 --> 00:09:27,330
at why these earlier attempts fell short. Let's

204
00:09:27,330 --> 00:09:29,850
do it. What was the first real swing at this?

205
00:09:30,269 --> 00:09:32,409
One of the earliest notable attempts mentioned

206
00:09:32,409 --> 00:09:35,830
was by Maury and colleagues back in 1999. 1999?

207
00:09:36,409 --> 00:09:39,389
That's the era of dial -up internet. Right. Dial

208
00:09:39,389 --> 00:09:42,190
-up, Netscape, Navigator. I was just wondering

209
00:09:42,190 --> 00:09:44,289
if the Y2K bug was going to shut down the power

210
00:09:44,289 --> 00:09:47,750
grid back then. We all were. But even back then...

211
00:09:47,769 --> 00:09:50,210
These researchers were exploring how to improve

212
00:09:50,210 --> 00:09:53,909
image retrieval. By using text? Yeah. They were

213
00:09:53,909 --> 00:09:56,450
training a model to predict nouns and adjectives

214
00:09:56,450 --> 00:09:58,950
in text documents that were paired with images.

215
00:09:59,370 --> 00:10:01,990
That's pretty forward thinking for 1999. It was.

216
00:10:02,169 --> 00:10:05,049
It was primitive, mostly looking at simple co

217
00:10:05,049 --> 00:10:07,690
-occurrences of words and image regions. But

218
00:10:07,690 --> 00:10:10,690
the seed of the idea was definitely there. But

219
00:10:10,690 --> 00:10:12,789
I'm guessing they didn't have 400 million images

220
00:10:12,789 --> 00:10:15,870
to work with. on a dial -up connection. Not quite.

221
00:10:16,129 --> 00:10:19,149
Yeah, the scale just wasn't possible. So fast

222
00:10:19,149 --> 00:10:21,950
forward to the deep learning era around 2016.

223
00:10:22,149 --> 00:10:24,850
OK, modern times. A researcher named Julin and

224
00:10:24,850 --> 00:10:29,629
his team took a data set called YFCC100M. OK,

225
00:10:29,789 --> 00:10:32,190
catch me up on that acronym. It was a massive

226
00:10:32,190 --> 00:10:35,690
data set of 100 million photos from Flickr. 100

227
00:10:35,690 --> 00:10:37,690
million. 100 million. Now we're talking. Yeah.

228
00:10:38,240 --> 00:10:40,940
They used the metadata from those photos, the

229
00:10:40,940 --> 00:10:43,580
titles, descriptions, and hashtags, to train

230
00:10:43,580 --> 00:10:46,840
convolutional neural networks, or CNNs, to predict

231
00:10:46,840 --> 00:10:49,740
words. Okay, 100 million photos sounds like the

232
00:10:49,740 --> 00:10:51,980
massive scale we're looking for. Were they predicting

233
00:10:51,980 --> 00:10:55,299
full natural language sentences? No, and that's

234
00:10:55,299 --> 00:10:58,179
the rub. They were predicting words based on

235
00:10:58,179 --> 00:11:01,259
metadata, which is incredibly noisy. Noisy meaning

236
00:11:01,259 --> 00:11:04,600
inaccurate? Meaning chaotic. We'll get into just

237
00:11:04,600 --> 00:11:07,320
how bad that specific metadata was in a moment,

238
00:11:07,720 --> 00:11:09,799
but it wasn't true natural language understanding.

239
00:11:10,159 --> 00:11:12,620
It was a step though. It was. It ushered in the

240
00:11:12,620 --> 00:11:15,139
era of what we call weakly supervised learning.

241
00:11:15,860 --> 00:11:17,799
Researchers realized they couldn't hand label

242
00:11:17,799 --> 00:11:20,000
everything anymore, so they started looking for

243
00:11:20,000 --> 00:11:23,190
existing messy labels. Like Instagram hashtags.

244
00:11:23,830 --> 00:11:27,970
Exactly like Instagram hashtags. In 2018, Mahajan

245
00:11:27,970 --> 00:11:32,929
and a massive team used up to 3 .5 billion Instagram

246
00:11:32,929 --> 00:11:36,450
photos to train a model. Wait, I'm sorry. 3 .5

247
00:11:36,450 --> 00:11:40,070
billion? 3 .5 billion. That is a staggering amount

248
00:11:40,070 --> 00:11:43,350
of data. If they had 3 .5 billion images, how

249
00:11:43,350 --> 00:11:45,289
is this still just the middle ground? That sounds

250
00:11:45,289 --> 00:11:47,149
like they should have solved computer vision

251
00:11:47,149 --> 00:11:49,110
right there on the spot. You would think so,

252
00:11:49,169 --> 00:11:51,210
right? Yeah. But it's because of how they built

253
00:11:51,210 --> 00:11:54,649
the model. What do you mean? They had 3 .5 billion

254
00:11:54,649 --> 00:11:57,110
images, but they were training the model to predict

255
00:11:57,110 --> 00:12:00,610
hashtags. And not just any hashtags. What were

256
00:12:00,610 --> 00:12:02,929
they predicting? They carefully designed and

257
00:12:02,929 --> 00:12:04,750
limited their supervision to a static number

258
00:12:04,750 --> 00:12:07,330
of classes. Wait, they limited it? Yeah. The

259
00:12:07,330 --> 00:12:09,250
Instagram hashtag model was strictly limited

260
00:12:09,250 --> 00:12:12,129
to predicting 1 ,000 specific ImageNet related

261
00:12:12,129 --> 00:12:16,330
hashtags. Ah. Oh, I see what they did. They wanted

262
00:12:16,330 --> 00:12:18,580
the massive scale of the internet. But they were

263
00:12:18,580 --> 00:12:20,860
still absolutely terrified of that messy mapping

264
00:12:20,860 --> 00:12:23,899
space you mentioned earlier. Exactly. They were

265
00:12:23,899 --> 00:12:25,879
dipping a toe in the ocean, but keeping their

266
00:12:25,879 --> 00:12:27,840
life jacket on. And there was another massive

267
00:12:27,840 --> 00:12:30,759
study in 2019 by Kolesnikov. What did they do?

268
00:12:31,399 --> 00:12:34,340
They used a data set called JFT -300M, which

269
00:12:34,340 --> 00:12:38,120
had 300 million noisily labeled images. OK. But

270
00:12:38,120 --> 00:12:40,720
again, they explicitly limited the model to predicting

271
00:12:40,720 --> 00:12:45,700
exactly 18 ,291 classes. So they basically took

272
00:12:45,700 --> 00:12:48,240
the CV student out of the curated classroom,

273
00:12:48,740 --> 00:12:50,179
threw them into the giant library, the internet,

274
00:12:50,259 --> 00:12:52,700
but said, hey, you can look at billions of books,

275
00:12:53,120 --> 00:12:54,899
but you are only allowed to learn these very

276
00:12:54,899 --> 00:12:58,159
specific flashcard terms. That is a perfect analogy.

277
00:12:59,039 --> 00:13:00,879
Technically speaking, in the architecture, they

278
00:13:00,879 --> 00:13:03,620
were still using what are called static softmax

279
00:13:03,620 --> 00:13:07,179
classifiers. OK, I'm going to need you to unpack

280
00:13:07,340 --> 00:13:10,379
Static softmax classifier. Sure. Because that

281
00:13:10,379 --> 00:13:12,159
sounds like a highly technical piece of equipment

282
00:13:12,159 --> 00:13:14,279
on a nuclear submarine. What is that actually

283
00:13:14,279 --> 00:13:17,879
doing inside the AI? Fair enough. A softmax classifier

284
00:13:17,879 --> 00:13:20,299
is basically a mathematical function that sits

285
00:13:20,299 --> 00:13:22,779
at the very end of the neural network. The very

286
00:13:22,779 --> 00:13:24,440
end. Like right before it gives you the answer.

287
00:13:24,940 --> 00:13:27,940
Exactly. After the AI has processed all the complex

288
00:13:27,940 --> 00:13:30,360
features of an image, like the edges, the textures,

289
00:13:30,480 --> 00:13:33,379
the shapes, it feeds all that analysis into this

290
00:13:33,379 --> 00:13:36,440
softmax layer. And what's its job? Its job is

291
00:13:36,440 --> 00:13:38,720
to output a list of probabilities that all add

292
00:13:38,720 --> 00:13:41,879
up to 100%. Oh. So it says something like, I'm

293
00:13:41,879 --> 00:13:44,980
80 % sure this is a dog, 15 % sure it's a cat,

294
00:13:45,240 --> 00:13:47,700
5 % sure it's a car. Right. It forces the AI

295
00:13:47,700 --> 00:13:49,700
to make a definitive guess across its available

296
00:13:49,700 --> 00:13:53,720
options. But you called it a static softmax classifier.

297
00:13:54,139 --> 00:13:58,159
Yes. The key word is static. A static softmax

298
00:13:58,159 --> 00:14:00,759
classifier means that the list of options, the

299
00:14:00,759 --> 00:14:03,659
literal buckets for dog, cat, car, and so on.

300
00:14:04,009 --> 00:14:06,090
is hard -coded into the architecture of the model

301
00:14:06,090 --> 00:14:08,710
before the training even begins. So the number

302
00:14:08,710 --> 00:14:11,470
of outputs is locked in? Yes. The number of nodes

303
00:14:11,470 --> 00:14:14,210
at the output layer is physically fixed. Which

304
00:14:14,210 --> 00:14:16,470
means it completely lacks a mechanism for dynamic

305
00:14:16,470 --> 00:14:19,409
outputs. Completely. So if you hand that model

306
00:14:19,409 --> 00:14:21,669
a picture of an airplane, and airplane isn't

307
00:14:21,669 --> 00:14:23,649
one of the hard -coded nodes in its static layer,

308
00:14:23,909 --> 00:14:26,389
it physically cannot guess airplane. It cannot.

309
00:14:26,929 --> 00:14:29,409
The concept of an airplane just doesn't exist

310
00:14:29,409 --> 00:14:31,960
in its output universe. That's wild. It will

311
00:14:31,960 --> 00:14:33,879
just guess whatever is mathematically closest

312
00:14:33,879 --> 00:14:36,379
in its predetermined list. Precisely. It might

313
00:14:36,379 --> 00:14:38,519
guess bird because it has wings, but it will

314
00:14:38,519 --> 00:14:40,879
never guess airplane. This completely crippled

315
00:14:40,879 --> 00:14:43,620
their zero -shot capabilities. Zero -shot meaning

316
00:14:43,620 --> 00:14:45,620
their ability to recognize things they weren't

317
00:14:45,620 --> 00:14:48,440
explicitly trained to recognize. Right. They

318
00:14:48,440 --> 00:14:50,519
had the unbelievable scale of the internet, but

319
00:14:50,519 --> 00:14:52,320
they were still trapped in the rigid paradigm

320
00:14:52,320 --> 00:14:55,100
of those fixed categories. I had to push back

321
00:14:55,100 --> 00:14:57,399
a little on the Instagram hashtag thing too.

322
00:14:57,580 --> 00:15:00,639
Go for it. Because predicting an Instagram hashtag

323
00:15:00,830 --> 00:15:03,149
isn't really understanding natural language,

324
00:15:03,309 --> 00:15:06,129
is it? Not at all. I mean, hashtags are basically

325
00:15:06,129 --> 00:15:09,470
just trendier flashcards. Someone posts a picture

326
00:15:09,470 --> 00:15:12,049
of a beautiful sunset on a beach and tags it

327
00:15:12,049 --> 00:15:14,950
hashtag blessed or hashtag no filter. Right,

328
00:15:14,990 --> 00:15:17,110
which tells you nothing about the visual content.

329
00:15:17,289 --> 00:15:20,600
Exactly. That doesn't teach an AI the complex

330
00:15:20,600 --> 00:15:22,899
syntax and relational logic of human language.

331
00:15:23,399 --> 00:15:25,980
It just teaches it internet slang. That's a vital

332
00:15:25,980 --> 00:15:28,500
point, and it's exactly why this era is called

333
00:15:28,500 --> 00:15:31,000
weak supervision. The signal is just too noisy.

334
00:15:31,379 --> 00:15:33,919
It's culturally specific, too. Very much so.

335
00:15:34,360 --> 00:15:37,340
And the vocabulary is incredibly narrow. It lacks

336
00:15:37,340 --> 00:15:40,580
verbs, complex adjectives, prepositional relationships.

337
00:15:41,340 --> 00:15:43,379
Now, to be totally fair to the field, there were

338
00:15:43,379 --> 00:15:45,600
earlier models that tried to learn directly from

339
00:15:45,600 --> 00:15:48,840
actual, rich natural language descriptions. Really?

340
00:15:49,019 --> 00:15:51,179
Who was doing that? There were models with names

341
00:15:51,179 --> 00:15:56,409
like Vertex, ICMLM, and Convert. Okay, so Vertex

342
00:15:56,409 --> 00:15:58,269
and those others, they were trying to do what

343
00:15:58,269 --> 00:16:01,210
Clip eventually did. Read real sentences paired

344
00:16:01,210 --> 00:16:04,070
with images. Yes. And they were exciting proofs

345
00:16:04,070 --> 00:16:06,370
of concept. They showed that you actually could

346
00:16:06,370 --> 00:16:09,210
use natural language to train a visual representation.

347
00:16:09,649 --> 00:16:12,529
But I'm sensing a butt? Did they actually work?

348
00:16:12,970 --> 00:16:15,549
They ran into a brutal bottleneck, which was

349
00:16:15,549 --> 00:16:20,100
scale. Scale again. Always scale. While the Instagram

350
00:16:20,100 --> 00:16:23,100
hashtag model was trained on 3 .5 billion images,

351
00:16:23,600 --> 00:16:25,340
these natural language models were only trained

352
00:16:25,340 --> 00:16:28,019
on maybe one or two hundred thousand images.

353
00:16:28,340 --> 00:16:31,940
Wait, why so few? If the internet is just overflowing

354
00:16:31,940 --> 00:16:34,779
with images and text, why only grab a couple

355
00:16:34,779 --> 00:16:37,580
hundred thousand? Because of the massive computational

356
00:16:37,580 --> 00:16:39,480
cost of the architectures they were using at

357
00:16:39,480 --> 00:16:41,600
the time. Ah, they were too slow. Unbelievably

358
00:16:41,600 --> 00:16:43,840
slow. We'll get into the specific algorithms

359
00:16:43,840 --> 00:16:45,980
in a bit, but essentially their methods were

360
00:16:45,980 --> 00:16:47,940
so inefficient that running them on billions

361
00:16:47,940 --> 00:16:50,500
of images would have taken lifetimes. Oh wow.

362
00:16:50,740 --> 00:16:53,440
Yeah. They ran on servers for a few days, whereas

363
00:16:53,440 --> 00:16:56,000
modern state -of -the -art models require thousands

364
00:16:56,000 --> 00:16:58,259
of GPU years of compute. So because they had

365
00:16:58,259 --> 00:17:00,159
to keep the data set small, they just weren't

366
00:17:00,159 --> 00:17:02,980
smart enough. Right. As a result of this small

367
00:17:02,980 --> 00:17:05,759
scale, their performance was deeply underwhelming.

368
00:17:05,799 --> 00:17:07,839
When one of these earlier natural language models

369
00:17:07,839 --> 00:17:10,480
was tested on ImageNet in a zero -shot setting,

370
00:17:10,799 --> 00:17:15,799
it only achieved an 11 .5 % accuracy. Ouch. 11

371
00:17:15,799 --> 00:17:18,940
.5%. Yeah, not great. That's like barely better

372
00:17:18,940 --> 00:17:20,880
than guessing randomly on some of these datasets.

373
00:17:21,779 --> 00:17:24,680
So we have a clear problem here. To break free

374
00:17:24,680 --> 00:17:27,579
from the static flashcards, you need rich natural

375
00:17:27,579 --> 00:17:30,279
language. Yes. But to make natural language work

376
00:17:30,279 --> 00:17:33,380
without failing, you need massive scale. Also,

377
00:17:33,500 --> 00:17:35,539
yes. So where do you find a data set that gives

378
00:17:35,539 --> 00:17:38,400
you both? Well, you can't just buy one. You have

379
00:17:38,400 --> 00:17:40,599
to build it yourself from the raw internet. And

380
00:17:40,599 --> 00:17:42,660
that brings us to the creation of the data set

381
00:17:42,660 --> 00:17:45,680
that finally made CLIP possible. Right. If you're

382
00:17:45,680 --> 00:17:48,119
going to train a new kind of AI student, you

383
00:17:48,119 --> 00:17:50,519
need a completely new kind of textbook. They

384
00:17:50,519 --> 00:17:53,660
called this new data set WIT. WIT. But before

385
00:17:53,660 --> 00:17:55,859
we get deep into WIT, I want to go back to that

386
00:17:55,859 --> 00:17:59,160
Flickr data set you mentioned earlier. The YFCC100M.

387
00:17:59,359 --> 00:18:02,349
Yes, that one. You said it had 100 million photos.

388
00:18:02,430 --> 00:18:04,890
Why couldn't the OpenAI researchers just use

389
00:18:04,890 --> 00:18:07,170
that? I mean, 100 million seems like a pretty

390
00:18:07,170 --> 00:18:10,150
fantastic place to start. It does seem like a

391
00:18:10,150 --> 00:18:12,549
massive amount of data on the surface. But remember,

392
00:18:12,789 --> 00:18:15,410
our primary requirement here, we need images

393
00:18:15,410 --> 00:18:18,130
paired with natural language. Right. Real sentences.

394
00:18:18,569 --> 00:18:21,230
Other high quality academic data sets like MSCoco

395
00:18:21,230 --> 00:18:24,809
or Visual Genome, they had beautiful human -written

396
00:18:24,809 --> 00:18:27,109
natural language captions, but they only contain

397
00:18:27,109 --> 00:18:30,009
about 100 ,000 photos. Far too small. So they

398
00:18:30,009 --> 00:18:32,630
needed the 100 million from Flickr. They wanted

399
00:18:32,630 --> 00:18:37,529
it. YFCC100M had the sheer volume, but the metadata

400
00:18:37,529 --> 00:18:40,569
was an absolute disaster. How much of a disaster

401
00:18:40,569 --> 00:18:42,289
are we talking about here? Well, think about

402
00:18:42,289 --> 00:18:44,450
how photos are often saved on a digital camera

403
00:18:44,450 --> 00:18:46,650
or on your phone before you rename them. OK,

404
00:18:46,769 --> 00:18:49,809
yeah. A lot of the titles in that data set were

405
00:18:49,809 --> 00:18:52,650
just auto -generated file names. So the AI would

406
00:18:52,650 --> 00:18:55,029
be looking at this stunning, high -resolution

407
00:18:55,029 --> 00:18:57,490
picture of a mountain range at sunset, and the

408
00:18:57,490 --> 00:19:03,519
text paired with it would just be 20160716113957

409
00:19:03,519 --> 00:19:07,039
.jpg. Oh, that's completely useless. The model

410
00:19:07,039 --> 00:19:09,259
would just be churning through compute power

411
00:19:09,259 --> 00:19:11,859
trying to learn the visual concept of the number

412
00:19:11,859 --> 00:19:16,420
2016 -07 -16. Exactly. It learns nothing about

413
00:19:16,420 --> 00:19:19,599
mountains. Or the descriptions were entirely

414
00:19:19,599 --> 00:19:22,819
technical, just EXIF data detailing the camera

415
00:19:22,819 --> 00:19:25,599
aperture, the ISO, the shutter speed. Which again,

416
00:19:25,809 --> 00:19:27,650
None of that teaches the model what a mountain

417
00:19:27,650 --> 00:19:31,349
actually looks like. Right. So the OpenAI researchers

418
00:19:31,349 --> 00:19:34,970
realized they had to filter YFCC 100M aggressively

419
00:19:34,970 --> 00:19:37,450
if they wanted to use it. They decided to keep

420
00:19:37,450 --> 00:19:40,329
only the images that had actual natural language

421
00:19:40,329 --> 00:19:43,529
titles or descriptions in English. But wait,

422
00:19:43,769 --> 00:19:45,650
if they go through and filter out all those auto

423
00:19:45,650 --> 00:19:48,329
-generated file names and the camera settings

424
00:19:48,329 --> 00:19:51,250
and the non -English text, that hundred million

425
00:19:51,250 --> 00:19:54,309
number has to plummet. It did. They must have

426
00:19:54,309 --> 00:19:55,690
been left with almost nothing compared to what

427
00:19:55,690 --> 00:19:57,730
they started with. You're right on the money.

428
00:19:58,049 --> 00:20:00,809
The data set shrank by a factor of six. It went

429
00:20:00,809 --> 00:20:03,890
from 100 million down to a mere 15 million useful

430
00:20:03,890 --> 00:20:08,250
photos. 15 million, which ironically is basically

431
00:20:08,250 --> 00:20:11,509
the same size as ImageNet. Exactly. So after

432
00:20:11,509 --> 00:20:14,230
all that filtering work, they're right back to

433
00:20:14,230 --> 00:20:16,869
square one in terms of scale. Which is exactly

434
00:20:16,869 --> 00:20:19,569
why they abandoned the idea of using existing

435
00:20:19,569 --> 00:20:22,089
datasets altogether and decided to construct

436
00:20:22,089 --> 00:20:25,410
a brand new one completely from scratch. Enter

437
00:20:25,410 --> 00:20:29,650
WIT. Enter WIT, which stands for Web Image Text.

438
00:20:30,170 --> 00:20:33,490
And this dataset is staggering. It contains 400

439
00:20:33,490 --> 00:20:36,589
million image text pairs collected from a massive

440
00:20:36,589 --> 00:20:38,789
variety of publicly available sources on the

441
00:20:38,789 --> 00:20:43,450
internet. 400 million pairs. That is a massive

442
00:20:43,450 --> 00:20:46,089
leap from 15 million. It is. But how do you even

443
00:20:46,089 --> 00:20:48,390
go about building a data set that huge without

444
00:20:48,390 --> 00:20:51,009
it just turning into a disorganized dump of internet

445
00:20:51,009 --> 00:20:53,150
garbage? Because if you just scrape randomly,

446
00:20:53,269 --> 00:20:55,170
you're going to end up with, like, 300 million

447
00:20:55,170 --> 00:20:57,549
pictures of celebrities, weird memes, and cat

448
00:20:57,549 --> 00:21:00,109
photos. Exactly. And that wouldn't teach the

449
00:21:00,109 --> 00:21:02,890
AI about the real world. They had to be incredibly

450
00:21:02,890 --> 00:21:05,789
systematic to ensure the data set covered a broad,

451
00:21:06,190 --> 00:21:08,630
useful set of visual concepts. They didn't just

452
00:21:08,630 --> 00:21:10,529
turn on a vacuum cleaner and suck up the web.

453
00:21:11,029 --> 00:21:13,400
So what was the strategy? They started with a

454
00:21:13,400 --> 00:21:16,940
base query list of 500 ,000 queries. Half a million

455
00:21:16,940 --> 00:21:19,119
search terms. What kind of words made it onto

456
00:21:19,119 --> 00:21:21,579
that list? They included all words that occurred

457
00:21:21,579 --> 00:21:24,059
at least 100 times in the English version of

458
00:21:24,059 --> 00:21:26,880
Wikipedia. Okay, so that covers a huge chunk

459
00:21:26,880 --> 00:21:29,559
of human knowledge. Right. They also added common

460
00:21:29,559 --> 00:21:32,140
bigrams, which are two -word phrases that frequently

461
00:21:32,140 --> 00:21:34,740
appear together. They added the names of Wikipedia

462
00:21:34,740 --> 00:21:37,519
articles that had high search volume. And crucially,

463
00:21:37,799 --> 00:21:41,579
they added wordnet sinsets. Sinsets. I'm going

464
00:21:41,579 --> 00:21:44,359
to need a quick definition there. Sure. A synset

465
00:21:44,359 --> 00:21:46,900
is essentially a group of synonyms that represent

466
00:21:46,900 --> 00:21:49,559
a single distinct concept. Give me an example.

467
00:21:50,220 --> 00:21:53,680
For example, the words car, auto, automobile,

468
00:21:53,920 --> 00:21:57,539
and motor car all belong to the exact same synset.

469
00:21:57,680 --> 00:22:00,319
By including these, they ensure the search queries

470
00:22:00,319 --> 00:22:02,960
were semantically rich and wouldn't miss images

471
00:22:02,960 --> 00:22:05,220
just because someone used a slightly different

472
00:22:05,220 --> 00:22:08,460
word. OK. So they have this massive encyclopedic

473
00:22:08,460 --> 00:22:10,890
list of half a million concepts. And they just

474
00:22:10,890 --> 00:22:12,630
searched the internet for images paired with

475
00:22:12,630 --> 00:22:15,410
text that include those concepts? Yes. But they

476
00:22:15,410 --> 00:22:16,829
had to solve the exact problem you mentioned

477
00:22:16,829 --> 00:22:19,049
earlier, the dataset being totally overwhelmed

478
00:22:19,049 --> 00:22:21,289
by pictures of a single popular celebrity or

479
00:22:21,289 --> 00:22:24,029
a viral meme. Right, because there are probably

480
00:22:24,029 --> 00:22:26,490
50 million pictures of Taylor Swift out there.

481
00:22:26,730 --> 00:22:29,730
Exactly. So to fix that, they imposed a strict

482
00:22:29,730 --> 00:22:32,509
cap. They balanced the dataset by taking a maximum

483
00:22:32,509 --> 00:22:36,309
of 20 ,000 image text pairs per query. Oh, that's

484
00:22:36,309 --> 00:22:39,130
brilliant. It forces diversity. Yes. It guarantees

485
00:22:39,130 --> 00:22:42,309
that the model sees 20 ,000 pictures of an abacus

486
00:22:42,309 --> 00:22:45,230
and 20 ,000 pictures of Taylor Swift, giving

487
00:22:45,230 --> 00:22:47,869
them equal footing in the training data. Precisely.

488
00:22:48,109 --> 00:22:51,329
It creates a vast, balanced representation of

489
00:22:51,329 --> 00:22:53,769
human knowledge as it exists visually on the

490
00:22:53,769 --> 00:22:56,490
web without letting any one topic drown out the

491
00:22:56,490 --> 00:22:59,309
others. So let's pull back for a second and look

492
00:22:59,309 --> 00:23:02,269
at the big picture here. Why is this specific

493
00:23:02,269 --> 00:23:06,170
move, forging the WIT data set and relying on

494
00:23:06,170 --> 00:23:08,950
unconstrained natural language supervision, why

495
00:23:08,950 --> 00:23:11,349
is this the critical shift for the whole paper?

496
00:23:11,769 --> 00:23:14,049
Because it completely shatters the flashcard

497
00:23:14,049 --> 00:23:16,910
paradigm. By moving to natural language supervision

498
00:23:16,910 --> 00:23:19,990
on this massive scale, they completely bypass

499
00:23:19,990 --> 00:23:23,390
the need for annotations to be in a classic sterile,

500
00:23:23,650 --> 00:23:26,309
machine learning compatible format. There are

501
00:23:26,309 --> 00:23:29,049
no human labelers agonizing in a cubicle over

502
00:23:29,049 --> 00:23:31,210
whether a picture should be categorized as a

503
00:23:31,210 --> 00:23:34,900
mug or a cup or a stein. Which is how the real

504
00:23:34,900 --> 00:23:37,119
world actually works. I mean, if you've ever

505
00:23:37,119 --> 00:23:39,740
tried to search your iPhone photos by just typing

506
00:23:39,740 --> 00:23:43,619
receipts or dog at the beach, and you just marvel

507
00:23:43,619 --> 00:23:46,059
at how it instantly finds those exact pictures

508
00:23:46,059 --> 00:23:48,839
without you ever having tagged them, you are

509
00:23:48,839 --> 00:23:50,799
interacting with the descendants of the exact

510
00:23:50,799 --> 00:23:53,279
philosophy we are talking about right now. Exactly.

511
00:23:53,799 --> 00:23:56,500
The model learns passively from the wild unstructured

512
00:23:56,500 --> 00:23:59,000
supervision contained in the vastness of the

513
00:23:59,000 --> 00:24:01,420
internet text. It learns that the visual features

514
00:24:01,420 --> 00:24:03,680
of a mug, You know, the curve of the cylinder,

515
00:24:03,980 --> 00:24:05,920
the shadow of the interior, the C shape of the

516
00:24:05,920 --> 00:24:09,099
handle. It learns those visual features frequently

517
00:24:09,099 --> 00:24:12,059
co -occur with words like coffee, morning, handle,

518
00:24:12,319 --> 00:24:16,420
ceramic. It learns the rich context of the visual

519
00:24:16,420 --> 00:24:18,799
world. Yes, much like a human child does when

520
00:24:18,799 --> 00:24:20,500
they're just listening to their parents describe

521
00:24:20,500 --> 00:24:22,279
the world around them as they walk through a

522
00:24:22,279 --> 00:24:25,299
grocery store. It's an organic, holistic way

523
00:24:25,299 --> 00:24:28,940
of learning. But I see a massive brick wall approaching.

524
00:24:29,259 --> 00:24:32,099
The computational wall. Yeah. because you now

525
00:24:32,099 --> 00:24:35,119
have 400 million image text pairs. That is a

526
00:24:35,119 --> 00:24:37,519
literal ocean of data. You mentioned earlier

527
00:24:37,519 --> 00:24:39,640
that models like vertex failed because they were

528
00:24:39,640 --> 00:24:41,839
computationally inefficient, even on tiny data

529
00:24:41,839 --> 00:24:44,400
sets of 100 ,000. They did. So how on earth do

530
00:24:44,400 --> 00:24:46,720
you physically compute 400 million pairs without

531
00:24:46,720 --> 00:24:49,200
melting every server farm on the planet? That

532
00:24:49,200 --> 00:24:51,640
was the exact wall the open AI researchers hit.

533
00:24:52,039 --> 00:24:54,759
Having the data is entirely useless if you don't

534
00:24:54,759 --> 00:24:57,000
have an algorithm efficient enough to learn from

535
00:24:57,000 --> 00:24:59,640
it in a human lifetime. So they needed a breakthrough.

536
00:25:00,259 --> 00:25:02,779
Overcoming this computational nightmare is what

537
00:25:02,779 --> 00:25:04,900
actually makes CLIP a breakthrough. Let's talk

538
00:25:04,900 --> 00:25:07,099
about that efficiency breakthrough then. How

539
00:25:07,099 --> 00:25:09,759
did they initially try to train it and why did

540
00:25:09,759 --> 00:25:12,700
it fail so badly? Well, initially they took an

541
00:25:12,700 --> 00:25:14,900
approach very similar to that vertex model we

542
00:25:14,900 --> 00:25:17,740
discussed. They jointly trained an image CNN,

543
00:25:18,279 --> 00:25:21,259
a convolutional neural network, and a text transformer

544
00:25:21,259 --> 00:25:23,980
from scratch. Okay. And the objective they gave

545
00:25:23,980 --> 00:25:26,599
the model was to predict the exact caption of

546
00:25:26,599 --> 00:25:28,900
an image. Okay, let me make sure I have this

547
00:25:28,900 --> 00:25:32,589
right. The AI looks at the pixels of the image

548
00:25:32,680 --> 00:25:36,299
and it has to output the exact word -for -word

549
00:25:36,299 --> 00:25:38,619
sequence of the text that was paired with it

550
00:25:38,619 --> 00:25:41,420
on the internet. Yes, autoregressively predicting

551
00:25:41,420 --> 00:25:44,779
the exact text word by word by word. And how

552
00:25:44,779 --> 00:25:49,049
did that go? It was fundamentally paralyzingly

553
00:25:49,049 --> 00:25:52,190
inefficient. In their tests, a 63 million parameter

554
00:25:52,190 --> 00:25:55,089
transformer model was learning three times slower

555
00:25:55,089 --> 00:25:57,710
than a ridiculously basic baseline model that

556
00:25:57,710 --> 00:26:00,220
just predicted a bag of words. Wait, a bag of

557
00:26:00,220 --> 00:26:04,160
words? What is that? A bag of words is a model

558
00:26:04,160 --> 00:26:06,579
that just guesses the words in the caption in

559
00:26:06,579 --> 00:26:08,819
no particular order, ignoring grammar completely,

560
00:26:08,920 --> 00:26:11,519
just throws words out there. And the state of

561
00:26:11,519 --> 00:26:14,079
the art transformer was three times slower than

562
00:26:14,079 --> 00:26:17,140
that? It was. But a transformer is the engine

563
00:26:17,140 --> 00:26:19,980
behind GPT -3. Why would it be getting beaten

564
00:26:19,980 --> 00:26:23,460
by a dumb bag of words model? Because predicting

565
00:26:23,460 --> 00:26:25,940
the exact words of a caption on the internet

566
00:26:25,940 --> 00:26:29,660
is an impossibly ambiguous task. Why? Think about

567
00:26:29,660 --> 00:26:32,900
a picture of a cat sitting on a windowsill. One

568
00:26:32,900 --> 00:26:36,039
person on Flickr might caption it, my feline

569
00:26:36,039 --> 00:26:38,799
friend enjoying the sun. OK. Another person on

570
00:26:38,799 --> 00:26:41,400
Reddit might post the exact same photo and caption

571
00:26:41,400 --> 00:26:44,859
it, look at those whiskers. A third person on

572
00:26:44,859 --> 00:26:47,799
a blog might just write, grumpy today. I see.

573
00:26:48,380 --> 00:26:50,640
The visual information in the image doesn't dictate

574
00:26:50,640 --> 00:26:54,240
one single deterministic sentence. There's no

575
00:26:54,240 --> 00:26:57,119
mathematical right answer for what is the caption.

576
00:26:57,759 --> 00:27:00,279
Exactly. So the model spends massive amounts

577
00:27:00,279 --> 00:27:02,859
of compute power trying to guess the exact phrasing,

578
00:27:03,279 --> 00:27:06,000
the grammar, the specific stylistic choices of

579
00:27:06,000 --> 00:27:08,700
the person who uploaded the photo, rather than

580
00:27:08,700 --> 00:27:11,420
focusing on learning the visual concepts. It's

581
00:27:11,420 --> 00:27:13,660
trying to be a mind reader instead of just being

582
00:27:13,660 --> 00:27:16,420
an image recognizer. Exactly. It's a waste of

583
00:27:16,420 --> 00:27:18,559
energy. Okay, I think I've got an analogy for

584
00:27:18,559 --> 00:27:21,329
this. Tell me if this works. Go ahead. Imagine

585
00:27:21,329 --> 00:27:24,049
you are playing a massive high -stakes game of

586
00:27:24,049 --> 00:27:26,849
matchmaking. The predictive method, the one that

587
00:27:26,849 --> 00:27:29,829
was failing and taking forever, is like asking

588
00:27:29,829 --> 00:27:32,509
the AI to paint a photorealistic portrait of

589
00:27:32,509 --> 00:27:36,309
my blind date from scratch, just based on a tiny

590
00:27:36,309 --> 00:27:39,789
vague clue I give it. Okay. I tell the AI she

591
00:27:39,789 --> 00:27:42,549
likes dogs and coffee. Now paint her perfectly.

592
00:27:43,069 --> 00:27:45,420
It's an impossible task. The AI would have to

593
00:27:45,420 --> 00:27:47,819
generate millions of iterations, tweaking hair

594
00:27:47,819 --> 00:27:49,839
color, eye shape, height, all based on a clue

595
00:27:49,839 --> 00:27:52,640
that doesn't actually contain that visual information.

596
00:27:53,180 --> 00:27:55,940
It's exhausting. It's going to be wrong 99 %

597
00:27:55,940 --> 00:27:58,099
of the time, and it takes forever. Precisely.

598
00:27:58,200 --> 00:28:00,759
It's trying to generate instead of recognize.

599
00:28:01,380 --> 00:28:03,319
So they needed to change the rules of the game.

600
00:28:03,480 --> 00:28:05,900
What's the alternative they came up with? The

601
00:28:05,900 --> 00:28:08,480
contrastive method. Contrastive language image

602
00:28:08,480 --> 00:28:11,430
pre -training. That's the C in CLIP. Okay, so

603
00:28:11,430 --> 00:28:13,430
if the predictive method is painting a portrait

604
00:28:13,430 --> 00:28:17,589
from scratch The contrast of method is what putting

605
00:28:17,589 --> 00:28:20,329
a hundred people in a room? Yes, exactly You

606
00:28:20,329 --> 00:28:22,650
give the AI that same vague description likes

607
00:28:22,650 --> 00:28:25,250
dogs and coffee and you just ask it to pick out

608
00:28:25,250 --> 00:28:26,990
Which one of the hundred people in the room is

609
00:28:26,990 --> 00:28:29,309
my date? Yes, it doesn't have to generate anything

610
00:28:29,309 --> 00:28:31,509
new. It doesn't have to guess her eye color It

611
00:28:31,509 --> 00:28:33,710
just has to look at the people in the room. Look

612
00:28:33,710 --> 00:28:36,390
at the clue and recognize the most likely pair

613
00:28:36,390 --> 00:28:40,720
that has to be astronomically faster Your analogy

614
00:28:40,720 --> 00:28:43,480
maps perfectly to the architecture. But to really

615
00:28:43,480 --> 00:28:45,660
understand why it's so incredibly efficient,

616
00:28:45,680 --> 00:28:47,859
we have to look at the math of that room full

617
00:28:47,859 --> 00:28:50,359
of 100 people. Let's do it. We have to look at

618
00:28:50,359 --> 00:28:53,259
the end -by -end grid. OK. Let's get into the

619
00:28:53,259 --> 00:28:56,480
weeds here. How does the math of this contrastive

620
00:28:56,480 --> 00:28:59,640
matchmaking game actually work in the code? Let's

621
00:28:59,640 --> 00:29:02,779
visualize a single training batch. In deep learning,

622
00:29:03,059 --> 00:29:05,180
you don't feed the model one image at a time.

623
00:29:05,200 --> 00:29:07,380
You feed it a batch of them. Right. Just feed

624
00:29:07,380 --> 00:29:11,299
things up. Imagine a batch of N image text pairs.

625
00:29:11,619 --> 00:29:14,440
In Clip's case, N is a very large number. They

626
00:29:14,440 --> 00:29:18,619
used a batch size of 32 ,768 pairs processed

627
00:29:18,619 --> 00:29:23,059
simultaneously. Wow. OK, so 32 ,768 images and

628
00:29:23,059 --> 00:29:27,880
32 ,768 captions. Exactly. Now, Clip is trained

629
00:29:27,880 --> 00:29:31,079
to predict the correct pairings out of all possible

630
00:29:31,079 --> 00:29:34,140
pairings in that massive batch. Out of all possible

631
00:29:34,140 --> 00:29:38,759
pairings? Yes. Visualize a giant grid. an n by

632
00:29:38,759 --> 00:29:45,680
n grid. So a grid with 32 ,768 rows and 32 ,768

633
00:29:45,680 --> 00:29:48,940
columns. Yes. Now, the diagonal line running

634
00:29:48,940 --> 00:29:51,420
down the center of that grid represents the real

635
00:29:51,420 --> 00:29:54,740
correct image text pairs. Image 1 with text 1,

636
00:29:54,880 --> 00:29:57,940
image 2 with text 2. The actual matches. Right.

637
00:29:58,359 --> 00:30:01,660
But the rest of the grid, the n squared minus

638
00:30:01,660 --> 00:30:04,559
n combinations are the incorrect pairings. Like

639
00:30:04,559 --> 00:30:08,190
image 1 being paired with text 2. or image five

640
00:30:08,190 --> 00:30:12,049
paired with text 10 ,000. Exactly. And here is

641
00:30:12,049 --> 00:30:14,269
where I think my blind date analogy falls a little

642
00:30:14,269 --> 00:30:16,990
short, right? Because in my analogy, the AI is

643
00:30:16,990 --> 00:30:20,269
just looking for the one right person. But in

644
00:30:20,269 --> 00:30:22,710
this grid, the AI is doing something with all

645
00:30:22,710 --> 00:30:25,710
the wrong pairings, too. That is the absolute

646
00:30:25,710 --> 00:30:28,329
genius of the contrastive loss function. The

647
00:30:28,329 --> 00:30:30,289
model's job isn't just to maximize the score

648
00:30:30,289 --> 00:30:33,210
on that correct diagonal. It must simultaneously

649
00:30:33,210 --> 00:30:35,630
minimize the score everywhere else on the grid.

650
00:30:35,940 --> 00:30:38,940
How does it actually score them, though? It jointly

651
00:30:38,940 --> 00:30:41,420
trains an image encoder and a text encoder. Both

652
00:30:41,420 --> 00:30:43,900
of these encoders convert their inputs, the pixels

653
00:30:43,900 --> 00:30:46,039
of the image and the words of the text, into

654
00:30:46,039 --> 00:30:49,480
a mathematical vector. A string of numbers. Right.

655
00:30:49,640 --> 00:30:52,160
Called an embedding. So both the image and the

656
00:30:52,160 --> 00:30:54,839
text are translated into the exact same mathematical

657
00:30:54,839 --> 00:30:57,480
language? Yes. They are placed into a shared

658
00:30:57,480 --> 00:31:01,339
multi -modal embedding space? Correct. Now, how

659
00:31:01,339 --> 00:31:03,359
do you measure if an image embedding and a text

660
00:31:03,359 --> 00:31:05,849
embedding are a good match in that space. You

661
00:31:05,849 --> 00:31:09,029
calculate their cosine similarity. You do. Basically,

662
00:31:09,049 --> 00:31:10,950
you're measuring the angle between these two

663
00:31:10,950 --> 00:31:13,210
vectors in this crazy multi -dimensional space.

664
00:31:13,809 --> 00:31:15,869
If they point in the exact same direction, they

665
00:31:15,869 --> 00:31:18,309
have a high similarity. If they point in opposite

666
00:31:18,309 --> 00:31:20,849
directions, they are totally unrelated. Precisely.

667
00:31:21,369 --> 00:31:23,470
So for every single batch, the model calculates

668
00:31:23,470 --> 00:31:26,650
the cosine similarity for all n squared pairings

669
00:31:26,650 --> 00:31:29,150
in the grid. All of them. That's over a billion

670
00:31:29,150 --> 00:31:32,940
pairings per batch. It is. Then it uses a symmetric

671
00:31:32,940 --> 00:31:35,319
cross -entropy loss function to optimize those

672
00:31:35,319 --> 00:31:38,039
scores. It tweaks its internal parameters to

673
00:31:38,039 --> 00:31:40,619
push the vectors of the correct pairs closer

674
00:31:40,619 --> 00:31:43,000
together, maximizing the diagonal. And at the

675
00:31:43,000 --> 00:31:45,460
same time? It violently shoves the vectors of

676
00:31:45,460 --> 00:31:47,940
all the incorrect pairs further apart. That is

677
00:31:47,940 --> 00:31:50,599
such a powerful learning signal. Every single

678
00:31:50,599 --> 00:31:52,559
time it looks at a batch, it's not just learning

679
00:31:52,559 --> 00:31:56,579
one right answer. It's explicitly learning 32

680
00:31:56,579 --> 00:31:59,599
,000 wrong answers at the same time. Exactly.

681
00:31:59,799 --> 00:32:02,920
It's learning what a dog is by simultaneously

682
00:32:02,920 --> 00:32:05,619
learning that the dog is not a car, not a mountain,

683
00:32:05,799 --> 00:32:08,440
not a bowl of soup. Exactly. And this single

684
00:32:08,440 --> 00:32:11,660
switch from trying to predict exact words autoregressively

685
00:32:11,660 --> 00:32:14,000
to just trying to match the correct text to the

686
00:32:14,000 --> 00:32:16,880
correct image in an n -by -n grid, it brought

687
00:32:16,880 --> 00:32:20,579
a massive 4x efficiency improvement. 4x faster.

688
00:32:20,799 --> 00:32:22,839
Just by changing the rules of the game. And when

689
00:32:22,839 --> 00:32:25,339
you combine that with abandoning the transformer's

690
00:32:25,339 --> 00:32:27,880
attempt to generate text entirely compared to

691
00:32:27,880 --> 00:32:30,559
the old baseline, it was a 12x efficiency gain

692
00:32:30,559 --> 00:32:34,480
in total. 12x. That 12x multiplier is the only

693
00:32:34,480 --> 00:32:37,740
reason training on 400 million pairs became physically,

694
00:32:38,119 --> 00:32:40,339
temporally possible. It's incredible what happens

695
00:32:40,339 --> 00:32:42,940
when you just ask the AI the right question.

696
00:32:43,420 --> 00:32:46,180
OK, so we have the contrastive matchmaking game

697
00:32:46,180 --> 00:32:47,980
established. That's the software, the logic.

698
00:32:48,640 --> 00:32:50,920
Let's look at the hardware. Let's do it. How

699
00:32:50,920 --> 00:32:53,619
is this model physically built to play this massive

700
00:32:53,619 --> 00:32:56,259
end -by -end grid game? Well, as we established,

701
00:32:56,519 --> 00:32:59,619
Clip needs two brains working in tandem, an image

702
00:32:59,619 --> 00:33:02,480
encoder to process the visual pixels and a text

703
00:33:02,480 --> 00:33:04,480
encoder to read the language. Let's start with

704
00:33:04,480 --> 00:33:06,460
the visual side. What are they using for the

705
00:33:06,460 --> 00:33:09,319
image encoder? Did they invent some totally new

706
00:33:09,319 --> 00:33:11,779
architecture for this? No, they actually tested

707
00:33:11,779 --> 00:33:14,059
two different, highly established architectures

708
00:33:14,059 --> 00:33:16,440
for the image encoder just to see how performance

709
00:33:16,440 --> 00:33:18,920
scaled with compute power. The first was based

710
00:33:18,920 --> 00:33:22,880
on Resonant 50. ResNet -50. That's a classic

711
00:33:22,880 --> 00:33:25,359
convolutional neural network. It's been the workhorse

712
00:33:25,359 --> 00:33:28,220
of computer vision for years. It has. But I'm

713
00:33:28,220 --> 00:33:29,819
guessing they didn't just use it straight out

714
00:33:29,819 --> 00:33:32,460
of the box for something this advanced. No, they

715
00:33:32,460 --> 00:33:34,759
made several significant tweaks. They incorporated

716
00:33:34,759 --> 00:33:38,119
what are known as ResNet -D improvements, which

717
00:33:38,119 --> 00:33:40,240
essentially optimized the downsampling layers.

718
00:33:40,799 --> 00:33:43,259
But the more interesting tweaks were to the pooling

719
00:33:43,259 --> 00:33:45,819
layers. What'd they do? They added something

720
00:33:45,819 --> 00:33:49,559
called anti -aliased blur pooling. OK, time out.

721
00:33:49,910 --> 00:33:53,049
Anti -aliased blur pooling sounds like a magic

722
00:33:53,049 --> 00:33:56,029
spell from a sci -fi movie. It does sound complicated.

723
00:33:56,589 --> 00:33:59,089
I know that in CNN, pooling is how the network

724
00:33:59,089 --> 00:34:01,430
shrinks the resolution of the image as it processes

725
00:34:01,430 --> 00:34:04,069
deeper layers, kind of summarizing the features.

726
00:34:04,650 --> 00:34:07,750
But what is the anti -aliased blur part actually

727
00:34:07,750 --> 00:34:11,190
doing? It solves a classic fragility in CNNs.

728
00:34:11,269 --> 00:34:13,670
In a standard convolutional network, when you

729
00:34:13,670 --> 00:34:16,380
reduce the resolution of an image, when you downsample

730
00:34:16,380 --> 00:34:18,940
it, the network can lose what's called spatial

731
00:34:18,940 --> 00:34:22,099
translation invariance. Which means what in plain

732
00:34:22,099 --> 00:34:24,780
English? It means if you take a picture of a

733
00:34:24,780 --> 00:34:28,139
cat and you shift the entire image just one single

734
00:34:28,139 --> 00:34:31,480
pixel to the left, a standard CNN might suddenly

735
00:34:31,480 --> 00:34:34,099
get completely confused and its output changes

736
00:34:34,099 --> 00:34:37,539
drastically. Because the grid changed. Exactly.

737
00:34:37,820 --> 00:34:40,159
The features don't align with its rigid mathematical

738
00:34:40,159 --> 00:34:43,539
grid anymore. Anti -aliased blur pooling applies

739
00:34:43,539 --> 00:34:45,960
a small subtle blur filter to the feature map

740
00:34:45,960 --> 00:34:49,480
before downsampling it. Oh, I see. This smooths

741
00:34:49,480 --> 00:34:51,559
out the high frequency noise and makes the network

742
00:34:51,559 --> 00:34:53,900
much more robust to tiny little shifts in the

743
00:34:53,900 --> 00:34:56,840
image. That's incredibly clever. It stops the

744
00:34:56,840 --> 00:34:59,739
network from obsessing over the exact rigid placement

745
00:34:59,739 --> 00:35:02,719
of a sharp pixel. What else did they change in

746
00:35:02,719 --> 00:35:05,239
the ResNet? The biggest architectural change

747
00:35:05,239 --> 00:35:07,500
was replacing the global average pooling layer

748
00:35:07,500 --> 00:35:10,559
with an attention pooling mechanism. Attention

749
00:35:10,559 --> 00:35:12,679
pooling. How does that differ from the average?

750
00:35:13,019 --> 00:35:15,420
In a standard ResNet, at the very end of the

751
00:35:15,420 --> 00:35:17,739
visual processing, the network takes a simple

752
00:35:17,739 --> 00:35:20,139
mathematical average of all the features it found

753
00:35:20,139 --> 00:35:23,239
across the entire image to create its final summary

754
00:35:23,239 --> 00:35:25,460
vector. So it treats every part of the image

755
00:35:25,460 --> 00:35:28,199
equally. Right. It treats the background sky

756
00:35:28,199 --> 00:35:30,239
with the exact same weight as the dog sitting

757
00:35:30,239 --> 00:35:33,539
in the center. But attention pooling uses a transformer

758
00:35:33,539 --> 00:35:36,210
-style multi -head attention layer. So it's not

759
00:35:36,210 --> 00:35:38,550
just passively averaging everything together.

760
00:35:39,110 --> 00:35:41,329
It's dynamically deciding which parts of the

761
00:35:41,329 --> 00:35:44,070
image are actually important to focus on when

762
00:35:44,070 --> 00:35:47,010
creating its final embedding. Yes. It can literally

763
00:35:47,010 --> 00:35:49,429
learn to ignore the sky and focus heavily on

764
00:35:49,429 --> 00:35:52,610
the dog's face. Exactly. It's a much more active,

765
00:35:52,929 --> 00:35:56,150
intelligent form of visual processing. So that

766
00:35:56,150 --> 00:35:58,590
was their souped up ResNet architecture. But

767
00:35:58,590 --> 00:36:01,230
they also tested a second architecture, Vision

768
00:36:01,230 --> 00:36:05,650
Transformers, or VIT. Ah. Vision transformers?

769
00:36:05,909 --> 00:36:07,869
These are newer. Let's spend a second on these,

770
00:36:07,969 --> 00:36:10,730
because they represent a massive shift in how

771
00:36:10,730 --> 00:36:13,929
AI looks at an image. How does a vision transformer

772
00:36:13,929 --> 00:36:16,610
differ from a classic convolutional network like

773
00:36:16,610 --> 00:36:19,670
ResNet? A CNN sweeps across an image with a small

774
00:36:19,670 --> 00:36:22,289
filter, right, looking for local features, edges,

775
00:36:22,590 --> 00:36:24,570
corners, and gradually builds them up into complex

776
00:36:24,570 --> 00:36:27,309
shapes. A vision transformer, on the other hand,

777
00:36:27,630 --> 00:36:30,610
chops the image up into a grid of distinct non

778
00:36:30,610 --> 00:36:33,409
-overlapping squares called patches. Like cutting

779
00:36:33,409 --> 00:36:35,989
a photograph into a puzzle. Exactly like that.

780
00:36:36,309 --> 00:36:38,389
And then it treats each of those puzzle patches

781
00:36:38,389 --> 00:36:41,250
exactly like a word in a sentence. It flattens

782
00:36:41,250 --> 00:36:44,210
the pixels in that patch into a vector and feeds

783
00:36:44,210 --> 00:36:46,309
the sequence of all the patches into a standard

784
00:36:46,309 --> 00:36:49,250
transformer architecture. The exact same architecture

785
00:36:49,250 --> 00:36:52,809
used to process paragraphs of text in NLP. The

786
00:36:52,809 --> 00:36:55,769
exact same one. So it's literally reading the

787
00:36:55,769 --> 00:36:58,170
image. It's looking at the relationships between

788
00:36:58,170 --> 00:37:01,630
patch 1 in the top left corner and patch 50 in

789
00:37:01,630 --> 00:37:04,409
the bottom right corner, mapping the global relationships

790
00:37:04,409 --> 00:37:06,909
of the whole image simultaneously. Yes, instead

791
00:37:06,909 --> 00:37:09,349
of just sweeping locally. And the researchers

792
00:37:09,349 --> 00:37:12,250
found that for CLIP, vision transformers were

793
00:37:12,250 --> 00:37:14,750
highly computationally efficient. The largest

794
00:37:14,750 --> 00:37:17,050
model they trained, the one that achieved the

795
00:37:17,050 --> 00:37:19,750
absolute best results, was a massive vision transformer

796
00:37:19,750 --> 00:37:23,090
that processed the image in 14 by 14 pixel patches.

797
00:37:23,710 --> 00:37:26,530
OK, so that's the visual brain. What about the

798
00:37:26,530 --> 00:37:29,429
text encoder? How does it read the messy internet

799
00:37:29,429 --> 00:37:31,969
captions? The text encoder is a relatively standard

800
00:37:31,969 --> 00:37:34,130
transformer model. It's actually quite small

801
00:37:34,130 --> 00:37:37,389
at only 63 million parameters. But how it ingests

802
00:37:37,389 --> 00:37:40,130
the text is important. How does it ingest it?

803
00:37:40,429 --> 00:37:43,269
It uses a lower -cased byte pair encoding, or

804
00:37:43,269 --> 00:37:47,929
BPE, with a vocabulary size of 49 ,152. Let's

805
00:37:47,929 --> 00:37:51,210
break down byte pair encoding. Why not just feed

806
00:37:51,210 --> 00:37:53,769
it whole words? Because the internet is full

807
00:37:53,769 --> 00:37:57,849
of typos, slang, and weird, rare words. If the

808
00:37:57,849 --> 00:38:00,230
AI encounters a word it doesn't have in its vocabulary

809
00:38:00,230 --> 00:38:03,489
list, it crashes or just throws an unknown token.

810
00:38:04,409 --> 00:38:06,409
Right. Byte pair encoding breaks words down into

811
00:38:06,409 --> 00:38:08,989
common subword pieces. So instead of needing

812
00:38:08,989 --> 00:38:11,269
to know the entire word unbelievable, it might

813
00:38:11,269 --> 00:38:13,849
just learn the pieces un, believe, and able.

814
00:38:14,230 --> 00:38:16,489
Oh, that makes sense. This allows it to gracefully

815
00:38:16,489 --> 00:38:18,610
handle misspellings or rare terms by combining

816
00:38:18,610 --> 00:38:21,750
pieces it already knows. Smart. Okay, so we have

817
00:38:21,750 --> 00:38:24,630
these two powerful brains a souped -up resnet

818
00:38:24,630 --> 00:38:28,510
or a vision transformer for the images and a

819
00:38:28,510 --> 00:38:32,190
transformer for the text Now earlier you mentioned

820
00:38:32,190 --> 00:38:34,530
that standard CV models were rigidly trained

821
00:38:34,530 --> 00:38:38,219
on image net Did the researchers take a pre trained

822
00:38:38,219 --> 00:38:40,719
image net model and use that as the starting

823
00:38:40,719 --> 00:38:42,800
point for clip? You know just to give it a head

824
00:38:42,800 --> 00:38:45,929
start on recognizing basic shapes Surprisingly,

825
00:38:46,230 --> 00:38:48,550
no. And this is a vital detail about the purity

826
00:38:48,550 --> 00:38:51,489
of this experiment. Because they had 400 million

827
00:38:51,489 --> 00:38:54,130
image text pairs, overfitting to a small data

828
00:38:54,130 --> 00:38:56,570
set wasn't a major concern anymore. So they didn't

829
00:38:56,570 --> 00:38:59,090
need the crutch? No. So they simplified the training

830
00:38:59,090 --> 00:39:02,030
drastically. They trained Clip entirely from

831
00:39:02,030 --> 00:39:04,750
scratch. Wow. Truly a blank slate. Complete blank

832
00:39:04,750 --> 00:39:07,889
slate. Complete. No image not wait initialization

833
00:39:07,889 --> 00:39:10,670
for the image encoder. No pre -trained word embeddings

834
00:39:10,670 --> 00:39:13,329
for the text encoder. It had to figure out everything.

835
00:39:13,469 --> 00:39:15,989
Everything. It learned everything from what a

836
00:39:15,989 --> 00:39:18,969
basic visual edge is to what the word dog means

837
00:39:18,969 --> 00:39:21,090
purely from the contrast of matchmaking game

838
00:39:21,090 --> 00:39:23,949
on the WIT data set. That is wild. They also

839
00:39:23,949 --> 00:39:26,309
removed the complex non -linear projections that

840
00:39:26,309 --> 00:39:28,409
earlier models used between the encoders and

841
00:39:28,409 --> 00:39:30,489
the embedding space. They just used a simple

842
00:39:30,489 --> 00:39:32,869
linear projection. Keeping it simple. And for

843
00:39:32,869 --> 00:39:35,130
data augmentation, which is how you mathematically

844
00:39:35,130 --> 00:39:37,210
alter images during training to make the model

845
00:39:37,210 --> 00:39:39,489
robust, they used almost none of the standard

846
00:39:39,489 --> 00:39:42,409
tricks. Right. Usually in machine learning, you

847
00:39:42,409 --> 00:39:45,010
take an image and you flip it, rotate it, mess

848
00:39:45,010 --> 00:39:47,030
with the colors, just to force the model to learn

849
00:39:47,030 --> 00:39:49,309
that a dog is still a dog, even if it's upside

850
00:39:49,309 --> 00:39:52,170
down or tinted green. Exactly. But with Clip,

851
00:39:52,289 --> 00:39:55,369
they used only a single random square crop. No

852
00:39:55,369 --> 00:39:57,929
complex color jittering and no weird rotations.

853
00:39:58,409 --> 00:40:00,869
Because when you have 400 million images, the

854
00:40:00,869 --> 00:40:03,010
data set itself contains enough natural real

855
00:40:03,010 --> 00:40:05,730
world variety. You don't need to artificially

856
00:40:05,730 --> 00:40:08,030
warp the images. The internet has already done

857
00:40:08,030 --> 00:40:11,309
that for you. Precisely. The massive scale replaces

858
00:40:11,309 --> 00:40:14,170
the need for heavy engineering tricks. And there's

859
00:40:14,170 --> 00:40:17,010
one final incredibly elegant mathematical detail

860
00:40:17,010 --> 00:40:19,590
about the training. The temperature parameter

861
00:40:19,590 --> 00:40:22,190
in the softmax function. OK, I have to stop you

862
00:40:22,190 --> 00:40:26,289
again. Log parameterized multiplicative scalar.

863
00:40:26,710 --> 00:40:28,630
That's what it says in the paper. I read that

864
00:40:28,630 --> 00:40:31,150
phrase, and my eyes just completely glazed over.

865
00:40:31,769 --> 00:40:33,889
What does this temperature parameter actually

866
00:40:33,889 --> 00:40:37,650
do in plain English? OK. Think back to our massive

867
00:40:37,650 --> 00:40:40,829
end -by -end grid of 32 ,000 pairings. The matchmaking

868
00:40:40,829 --> 00:40:43,750
room. Right. The model is calculating similarity

869
00:40:43,750 --> 00:40:46,929
scores. The temperature essentially controls

870
00:40:46,929 --> 00:40:50,170
the sharpness of those probabilities. How strict

871
00:40:50,170 --> 00:40:53,170
it is. Exactly. A high temperature makes the

872
00:40:53,170 --> 00:40:55,769
model very lenient, seeing similarities everywhere.

873
00:40:56,369 --> 00:40:58,849
A low temperature makes it incredibly strict,

874
00:40:59,150 --> 00:41:01,150
demanding absolute certainty before it declares

875
00:41:01,150 --> 00:41:04,130
a match. Just like a thermostat for the AI's

876
00:41:04,130 --> 00:41:06,409
confidence. Yes. In almost all previous models,

877
00:41:06,710 --> 00:41:08,889
human researchers had to manually tune this as

878
00:41:08,889 --> 00:41:10,730
a hyperparameter. Meaning they had to guess.

879
00:41:11,369 --> 00:41:13,590
They would run the model, guess the temperature,

880
00:41:13,809 --> 00:41:16,110
see if it worked, stop the model, tweak it, run

881
00:41:16,110 --> 00:41:18,369
it again. It was exhausting trial and error.

882
00:41:18,489 --> 00:41:21,550
But in CLIP? In CLIP, they optimized it directly

883
00:41:21,550 --> 00:41:24,869
during training as a log -parameterized multiplicative

884
00:41:24,869 --> 00:41:28,329
scalar. Meaning? Meaning the model acts as its

885
00:41:28,329 --> 00:41:31,429
own thermostat. It automatically learns and adjusts

886
00:41:31,429 --> 00:41:33,449
the optimal temperature itself as it trains,

887
00:41:34,269 --> 00:41:36,190
dynamically tightening its strictness as it gets

888
00:41:36,190 --> 00:41:38,869
smarter. That is brilliant. It just removes human

889
00:41:38,869 --> 00:41:41,849
guesswork entirely. OK, so we've built the engine.

890
00:41:42,309 --> 00:41:44,809
We've fueled it with 400 million unstructured

891
00:41:44,809 --> 00:41:47,789
images from the wild internet. We've optimized

892
00:41:47,789 --> 00:41:50,750
the training with this contrast of matchmaking

893
00:41:50,750 --> 00:41:54,510
game. And we've let the model set its own parameters.

894
00:41:55,230 --> 00:41:58,590
We are ready. Now for the fun part. We are taking

895
00:41:58,590 --> 00:42:01,130
this engine for a test drive to see if it actually

896
00:42:01,130 --> 00:42:03,730
understands concepts. And this brings us back

897
00:42:03,730 --> 00:42:06,510
to our 37 bananas. The ultimate banana test.

898
00:42:07,690 --> 00:42:09,809
But first we need to redefine a crucial term.

899
00:42:09,869 --> 00:42:12,949
You mentioned zero -shot earlier. In traditional

900
00:42:12,949 --> 00:42:14,909
computer vision, what did that actually mean

901
00:42:14,909 --> 00:42:17,730
and how does CLIP change the definition? Well,

902
00:42:17,769 --> 00:42:20,150
in traditional computer vision, zero -shot learning

903
00:42:20,150 --> 00:42:23,690
usually meant generalizing to unseen object categories

904
00:42:23,690 --> 00:42:26,769
within a specific controlled data set. Okay,

905
00:42:26,769 --> 00:42:29,809
give me an example. For example, if an ImageNet

906
00:42:29,809 --> 00:42:32,869
model was trained to recognize horses, and it

907
00:42:32,869 --> 00:42:35,969
was also trained to recognize stripes, could

908
00:42:35,969 --> 00:42:39,409
it recognize a zebra even if it had never explicitly

909
00:42:39,409 --> 00:42:42,429
seen a label for zebra? Right. But it's still

910
00:42:42,429 --> 00:42:45,809
operating within the safe curated visual style

911
00:42:45,809 --> 00:42:49,510
of that specific data set. Exactly. The OpenAI

912
00:42:49,510 --> 00:42:52,010
researchers argue that this is a flawed metric.

913
00:42:52,329 --> 00:42:55,070
They propose using zero -shot in a much broader,

914
00:42:55,190 --> 00:42:58,889
much more difficult sense, generalizing to entirely

915
00:42:58,889 --> 00:43:02,710
unseen data sets. So can a model trained on the

916
00:43:02,710 --> 00:43:04,929
wild, messy internet suddenly be dropped into

917
00:43:04,929 --> 00:43:08,829
a highly specific curated medical data set or

918
00:43:08,829 --> 00:43:11,329
a satellite imagery data set it has never seen

919
00:43:11,329 --> 00:43:13,769
and still perform the classification test flawlessly?

920
00:43:14,050 --> 00:43:16,940
Yes. They argued this is a much better proxy

921
00:43:16,940 --> 00:43:20,139
for measuring true general task learning capabilities.

922
00:43:20,699 --> 00:43:23,719
Okay, but here's the mechanical problem. If Clip

923
00:43:23,719 --> 00:43:26,239
doesn't have a static list of a thousand categories

924
00:43:26,239 --> 00:43:28,900
hard -coded into its output like the old models,

925
00:43:29,460 --> 00:43:31,780
how do you actually ask it a question? Right,

926
00:43:31,800 --> 00:43:33,900
how do you interface with it? Yeah. How do you

927
00:43:33,900 --> 00:43:36,079
hand it an image and say, hey, is this a dog,

928
00:43:36,239 --> 00:43:39,500
a cat, or a car? You use the text encoder. You

929
00:43:39,500 --> 00:43:41,920
essentially use prompt engineering. At test time,

930
00:43:42,039 --> 00:43:45,039
you synthesize a custom zero -shot linear classifier

931
00:43:45,039 --> 00:43:47,400
on the fly. On the fly. Walk me through that

932
00:43:47,400 --> 00:43:50,139
step by step. OK. Let's say you want to evaluate

933
00:43:50,139 --> 00:43:54,300
CLIP on the CIFAR 10 data set. This is a classic

934
00:43:54,300 --> 00:43:57,539
data set with 10 classes. Airplane, automobile,

935
00:43:58,019 --> 00:44:02,880
bird, cat, deer, dog, frog, horse, ship, and

936
00:44:02,880 --> 00:44:05,909
truck. OK. 10 things. You take those ten class

937
00:44:05,909 --> 00:44:08,670
names and you embed them into a natural language

938
00:44:08,670 --> 00:44:11,710
prompt template, usually something like a photo

939
00:44:11,710 --> 00:44:16,300
of a blank. Ah, so you don't just feed it the

940
00:44:16,300 --> 00:44:18,900
single word dog. You generate full sentences.

941
00:44:19,519 --> 00:44:21,539
A photo of an airplane, a photo of an automobile,

942
00:44:21,880 --> 00:44:25,079
a photo of a dog, and so on. Yes. Why use the

943
00:44:25,079 --> 00:44:27,679
full sentence? Why not just the word? Because

944
00:44:27,679 --> 00:44:29,760
of polysemy, words having multiple meanings.

945
00:44:30,119 --> 00:44:33,489
Oh. If you just feed it the single word Crane,

946
00:44:33,670 --> 00:44:35,670
the model doesn't know if you mean the construction

947
00:44:35,670 --> 00:44:39,329
equipment or the bird. Oh, right. But if you

948
00:44:39,329 --> 00:44:42,690
feed it a photo of a crane, a type of bird, it

949
00:44:42,690 --> 00:44:45,590
provides vital context. Got it. OK, so you have

950
00:44:45,590 --> 00:44:48,389
these 10 sentences, then what? You feed those

951
00:44:48,389 --> 00:44:51,710
10 sentences into Clip's text encoder. The encoder

952
00:44:51,710 --> 00:44:54,650
processes them and spits out 10 distinct text

953
00:44:54,650 --> 00:44:56,969
embeddings, 10 vectors in our multi -dimensional

954
00:44:56,969 --> 00:44:59,570
space. OK. Then you take the image you want to

955
00:44:59,570 --> 00:45:02,570
classify, say a picture of a golden retriever.

956
00:45:02,949 --> 00:45:05,070
You feed it through the image encoder. It spits

957
00:45:05,070 --> 00:45:07,650
out one image embedding. And then you just play

958
00:45:07,650 --> 00:45:09,809
the matchmaking game again. Exactly. You calculate

959
00:45:09,809 --> 00:45:12,190
the cosine similarity between the one image embedding

960
00:45:12,190 --> 00:45:14,909
and the 10 text embeddings. Whichever text vector

961
00:45:14,909 --> 00:45:16,929
points in the closest direction to the image

962
00:45:16,929 --> 00:45:19,449
vector wins. Exactly. The one with the highest

963
00:45:19,449 --> 00:45:22,369
similarity score is your prediction. No retraining,

964
00:45:22,710 --> 00:45:25,489
no static layers, just asking the model to compare

965
00:45:25,489 --> 00:45:28,210
embeddings. That. It's unbelievably elegant.

966
00:45:28,710 --> 00:45:30,469
You literally just talk to it. You describe the

967
00:45:30,469 --> 00:45:32,070
categories you want it to look for, and it finds

968
00:45:32,070 --> 00:45:34,989
them. So what were the results? Did this actually

969
00:45:34,989 --> 00:45:37,510
work across different domains? It worked incredibly

970
00:45:37,510 --> 00:45:39,789
well. They tested this zero -shot capability

971
00:45:39,789 --> 00:45:42,769
on over 30 different highly specialized computer

972
00:45:42,769 --> 00:45:45,809
vision datasets. Like what? Everything from recognizing

973
00:45:45,809 --> 00:45:49,750
specific types of meals in Food 101, to recognizing

974
00:45:49,750 --> 00:45:53,369
fine -grained aircraft variants in FGVC aircraft,

975
00:45:53,869 --> 00:45:56,369
to identifying actions in kinetic video datasets.

976
00:45:56,760 --> 00:45:59,380
And what's the headline result? How did it compare

977
00:45:59,380 --> 00:46:02,119
to the old flashcard method? The headline is

978
00:46:02,119 --> 00:46:05,420
this. A zero -shot clip model matches the accuracy

979
00:46:05,420 --> 00:46:09,719
of the original fully supervised ResNet -50 on

980
00:46:09,719 --> 00:46:12,239
the ImageNet dataset. Wait, it matches the gold

981
00:46:12,239 --> 00:46:15,880
standard? It matches it. And it achieved this

982
00:46:15,880 --> 00:46:19,480
without ever seeing any of the 1 .28 million

983
00:46:19,480 --> 00:46:22,179
specific training examples in ImageNet and without

984
00:46:22,179 --> 00:46:24,840
any dataset -specific fine -tuning whatsoever.

985
00:46:25,019 --> 00:46:28,500
That is the paradigm shift right there. It matches

986
00:46:28,500 --> 00:46:31,380
the student who spent years memorizing the specific

987
00:46:31,380 --> 00:46:34,039
flashcards without ever once looking at those

988
00:46:34,039 --> 00:46:37,199
flashcards. Yes. It just reasoned through the

989
00:46:37,199 --> 00:46:39,300
test using its general knowledge of the world.

990
00:46:39,699 --> 00:46:43,139
Yes. And it transfers non -trivially to almost

991
00:46:43,139 --> 00:46:45,639
every task they threw at it. OK. I think it's

992
00:46:45,639 --> 00:46:47,559
finally time to bring in our visual sources.

993
00:46:48,159 --> 00:46:50,760
Let's look at this collection of 37 banana images.

994
00:46:51,280 --> 00:46:54,119
Because I think diving into these images perfectly

995
00:46:54,119 --> 00:46:56,239
illustrates why this language -driven approach

996
00:46:56,239 --> 00:46:59,559
is so revolutionary compared to a standard convolutional

997
00:46:59,559 --> 00:47:02,500
network. Let's do it. It's a fantastic case study

998
00:47:02,500 --> 00:47:05,519
in semantic abstraction. So looking at the folder,

999
00:47:05,920 --> 00:47:09,099
we have standard bananas. Image 2, image 15,

1000
00:47:09,199 --> 00:47:11,780
image 16. These are typical yellow or slightly

1001
00:47:11,780 --> 00:47:14,199
green bananas. Some are in bunches. Some have

1002
00:47:14,199 --> 00:47:17,340
little blue Chiquita stickers on them. If we

1003
00:47:17,340 --> 00:47:19,969
train the traditional supervised ImageNet model

1004
00:47:19,969 --> 00:47:22,130
on standard photos of yellow bananas, it didn't

1005
00:47:22,130 --> 00:47:25,309
nail these, right? Absolutely. A standard CNN

1006
00:47:25,309 --> 00:47:27,769
would excel here. It would have learned the specific

1007
00:47:27,769 --> 00:47:30,849
yellow pixel patterns, the curved edge detection,

1008
00:47:31,349 --> 00:47:33,429
the typical ambient lighting of a kitchen counter.

1009
00:47:34,070 --> 00:47:36,610
But now let's look at the weird stuff. Look at

1010
00:47:36,610 --> 00:47:40,789
image 27. The plushie. The screaming plushie.

1011
00:47:40,909 --> 00:47:43,590
It's a toy banana with angry embroidered eyes,

1012
00:47:44,070 --> 00:47:46,409
a jagged red mouth, little black stick arms,

1013
00:47:46,929 --> 00:47:49,369
and it's sitting in a basket next to fuzzy red

1014
00:47:49,369 --> 00:47:52,530
plush cherries that also have angry faces. Right.

1015
00:47:53,130 --> 00:47:57,190
If I hand Image 27 to that standard CNN that

1016
00:47:57,190 --> 00:48:00,250
perfectly recognized the real banana, what happens?

1017
00:48:01,110 --> 00:48:04,710
A traditional model fails spectacularly. Why?

1018
00:48:04,889 --> 00:48:06,869
I mean, it's shaped like a banana. It's yellow

1019
00:48:06,869 --> 00:48:10,230
like a banana. Because a traditional CNN trained

1020
00:48:10,230 --> 00:48:13,829
on real bananas has learned a very brittle literal

1021
00:48:13,829 --> 00:48:17,289
representation. It is expecting the smooth waxy

1022
00:48:17,289 --> 00:48:20,510
texture of a real peel. It expects the specific

1023
00:48:20,510 --> 00:48:23,570
way light reflects off that smooth surface. It's

1024
00:48:23,570 --> 00:48:26,449
looking for a dark brown woody stem at the top.

1025
00:48:27,150 --> 00:48:29,550
When it looks at the plushie, its low -level

1026
00:48:29,550 --> 00:48:32,389
edge detectors see the fuzzy matte texture of

1027
00:48:32,389 --> 00:48:34,949
the fabric. They see the high contrast black

1028
00:48:34,949 --> 00:48:37,050
and white of the stitched mouth and the stick

1029
00:48:37,050 --> 00:48:39,710
arms. So the pixel patterns completely disrupt

1030
00:48:39,710 --> 00:48:42,809
the rigid correlations the model memorized. Exactly.

1031
00:48:43,349 --> 00:48:46,289
It might confidently predict stuffed animal or

1032
00:48:46,289 --> 00:48:49,389
pillow or craft project, but it wouldn't understand

1033
00:48:49,389 --> 00:48:53,309
the semantic essence of banana -ness being portrayed.

1034
00:48:53,849 --> 00:48:57,650
But Clip succeeds here. Why? Clip succeeds because

1035
00:48:57,650 --> 00:49:00,050
its visual understanding is tethered to language.

1036
00:49:00,610 --> 00:49:03,610
In its 400 million training pairs, it hasn't

1037
00:49:03,610 --> 00:49:05,889
just seen pristine photos of fruit on a white

1038
00:49:05,889 --> 00:49:08,869
background. It has seen pictures of toys on Etsy

1039
00:49:08,869 --> 00:49:11,610
captioned plushie banana. Oh, that makes sense.

1040
00:49:12,030 --> 00:49:15,369
It has seen babies in Halloween costumes captioned

1041
00:49:15,369 --> 00:49:18,250
dressed as a banana. It has learned to abstract

1042
00:49:18,250 --> 00:49:21,170
the semantic concept of a banana, the curve,

1043
00:49:21,309 --> 00:49:24,150
the yellowness, the cultural idea of it rather

1044
00:49:24,150 --> 00:49:26,909
than overfitting to the specific pixel textures

1045
00:49:26,909 --> 00:49:29,349
of organic fruit. That explains so much about

1046
00:49:29,349 --> 00:49:32,369
these other images. Let's look at image 30. Okay.

1047
00:49:32,989 --> 00:49:37,519
It's a 3D low polygon pixelated banana. Looks

1048
00:49:37,519 --> 00:49:39,840
like it's from a retro video game and it has

1049
00:49:39,840 --> 00:49:42,119
a Nintendo 64 logo slapped right on the side

1050
00:49:42,119 --> 00:49:44,579
of it. A traditional CNN would probably just

1051
00:49:44,579 --> 00:49:48,099
see the harsh jagged polygon lines and get completely

1052
00:49:48,099 --> 00:49:50,760
confused, right? It's these sharp corners instead

1053
00:49:50,760 --> 00:49:54,400
of smooth curves. Precisely. A standard model

1054
00:49:54,400 --> 00:49:57,199
sees the sharp geometric edges and might guess

1055
00:49:57,199 --> 00:50:01,420
origami or yellow box. But Clip has likely seen

1056
00:50:01,420 --> 00:50:04,780
images paired with text like low poly 3D banana

1057
00:50:04,780 --> 00:50:09,360
model or retro N64 graphics banana. So it links

1058
00:50:09,360 --> 00:50:12,800
the low fidelity visual representation to the

1059
00:50:12,800 --> 00:50:15,039
high level concept. Exactly. It's incredible.

1060
00:50:15,219 --> 00:50:18,119
What about image 21? This one is fascinating.

1061
00:50:18,280 --> 00:50:21,159
Which one is that? It's a wooden table. covered

1062
00:50:21,159 --> 00:50:23,880
in random scrap paper. And on one of the papers,

1063
00:50:24,079 --> 00:50:26,320
someone has roughly sketched a banana with a

1064
00:50:26,320 --> 00:50:28,679
blue ballpoint pen. And right next to the papers

1065
00:50:28,679 --> 00:50:32,320
are two actual real bananas. Ah, yes. A traditional

1066
00:50:32,320 --> 00:50:34,739
model might spot the real ones, assuming they

1067
00:50:34,739 --> 00:50:37,519
aren't obscured, but it would struggle immensely

1068
00:50:37,519 --> 00:50:40,980
to categorize the blue ink sketch as the same

1069
00:50:40,980 --> 00:50:43,420
object. The textures, the colors, the lighting,

1070
00:50:43,559 --> 00:50:45,719
everything is different. But because natural

1071
00:50:45,719 --> 00:50:48,599
language has words like sketch, drawing, doodle,

1072
00:50:48,780 --> 00:50:51,420
CLIP has a topological map that connects the

1073
00:50:51,420 --> 00:50:54,119
visual features of a line drawing to the concept

1074
00:50:54,119 --> 00:50:57,019
of the object being drawn. It understands that

1075
00:50:57,019 --> 00:51:00,380
a 2D blue line can represent a 3D yellow fruit.

1076
00:51:00,880 --> 00:51:03,880
Exactly. It's learned the concept of representation

1077
00:51:03,880 --> 00:51:06,360
itself. Now let's talk about the state of the

1078
00:51:06,360 --> 00:51:09,320
object, too, because bananas change over time.

1079
00:51:09,820 --> 00:51:13,599
Look at Image 9 and Image 18. Image 9 is a single

1080
00:51:13,599 --> 00:51:16,539
banana lying on cracked concrete, and it's heavily

1081
00:51:16,539 --> 00:51:19,900
bruised. massive black and brown patches. Image

1082
00:51:19,900 --> 00:51:23,739
18 is a banana on a mosaic table that is almost

1083
00:51:23,739 --> 00:51:27,500
entirely dark brown, wrinkled and rotting. Again,

1084
00:51:27,619 --> 00:51:30,079
language provides the robust supervision here.

1085
00:51:30,619 --> 00:51:33,739
Clip has seen phrases like rotten banana, overripe,

1086
00:51:33,920 --> 00:51:36,320
time to make banana bread. I've definitely searched

1087
00:51:36,320 --> 00:51:39,389
for that exact phrase. Right. So it understands

1088
00:51:39,389 --> 00:51:41,570
that the visual color shift from bright yellow

1089
00:51:41,570 --> 00:51:44,929
to dark brown is a valid natural state change

1090
00:51:44,929 --> 00:51:47,869
for this specific object. A strictly supervised

1091
00:51:47,869 --> 00:51:50,530
model, on the other hand, might be heavily biased

1092
00:51:50,530 --> 00:51:53,510
towards yellow, depending on how narrowly its

1093
00:51:53,510 --> 00:51:56,849
human curators defined the banana category during

1094
00:51:56,849 --> 00:51:59,550
data collection. There are also images here where

1095
00:51:59,550 --> 00:52:02,650
the banana isn't the star of the show. It's part

1096
00:52:02,650 --> 00:52:05,610
of a complex cluttered scene. Image seven shows

1097
00:52:05,610 --> 00:52:08,690
a slice of frittata or quiche on a plate with

1098
00:52:08,690 --> 00:52:11,590
tomatoes, a glass of smoothie, and way in the

1099
00:52:11,590 --> 00:52:14,309
background out of focus is a single blurred banana.

1100
00:52:14,769 --> 00:52:19,119
Clutter is tough. Yeah, and image 25 is a bizarre

1101
00:52:19,119 --> 00:52:22,280
still life. A mesa jar of brown liquid, a green

1102
00:52:22,280 --> 00:52:25,099
apple, cranberries, a head of cabbage, and a

1103
00:52:25,099 --> 00:52:27,659
banana framing the bottom edge. This tests the

1104
00:52:27,659 --> 00:52:30,099
model's ability to isolate concepts in a cluttered

1105
00:52:30,099 --> 00:52:32,500
environment. Because standard data sets often

1106
00:52:32,500 --> 00:52:35,139
center the object on a clean background, standard

1107
00:52:35,139 --> 00:52:38,599
models struggle with clutter. But CLIP was trained

1108
00:52:38,599 --> 00:52:41,440
on wild internet images. Which are inherently

1109
00:52:41,440 --> 00:52:44,739
cluttered. Very much so. It is deeply accustomed

1110
00:52:44,739 --> 00:52:47,699
to objects appearing tangentially alongside other,

1111
00:52:47,780 --> 00:52:50,840
entirely unrelated objects. It's not just recognizing

1112
00:52:50,840 --> 00:52:53,699
a banana in a vacuum. It's recognizing a banana

1113
00:52:53,699 --> 00:52:56,280
in the messy real world. Which brings us perfectly

1114
00:52:56,280 --> 00:52:58,579
to the next major point in the paper. Because

1115
00:52:58,579 --> 00:53:00,880
matching ResNet -50 on the standard ImageNet

1116
00:53:00,880 --> 00:53:03,539
test is an impressive party trick, but as we

1117
00:53:03,539 --> 00:53:05,980
just saw with the plushies and the N64 graphics,

1118
00:53:06,420 --> 00:53:08,800
the real world is not a standardized multiple

1119
00:53:08,800 --> 00:53:12,610
-choice test. Right. How does CLIP handle real

1120
00:53:12,610 --> 00:53:15,670
-world messiness at scale? Beyond just our 37

1121
00:53:15,670 --> 00:53:18,090
bananas, did the researchers actually quantify

1122
00:53:18,090 --> 00:53:20,789
this robustness? They did, and the results are

1123
00:53:20,789 --> 00:53:22,670
arguably the most important part of the entire

1124
00:53:22,670 --> 00:53:25,429
paper. They evaluated CLIP on what are called

1125
00:53:25,429 --> 00:53:28,110
natural distribution shift datasets. What are

1126
00:53:28,110 --> 00:53:30,630
those? These are datasets specifically designed

1127
00:53:30,630 --> 00:53:33,070
by the AI community to be incredibly tricky.

1128
00:53:33,610 --> 00:53:36,630
They include datasets like ImageNet V2, ImageNet

1129
00:53:36,630 --> 00:53:40,849
A, ImageNet R, ObjectNet, and ImageNet Sketch.

1130
00:53:40,989 --> 00:53:43,210
Okay, let's break those down. What makes them

1131
00:53:43,210 --> 00:53:47,289
so tricky? Take ImageNet A. It's a dataset consisting

1132
00:53:47,289 --> 00:53:49,849
of images that cause standard ResNet models to

1133
00:53:49,849 --> 00:53:52,590
make catastrophic, highly confident mistakes.

1134
00:53:52,650 --> 00:53:56,130
Like what? They are adversarial, but naturally

1135
00:53:56,130 --> 00:53:58,730
occurring. For example, an image of a bullfrog

1136
00:53:58,730 --> 00:54:00,869
that a standard model insists is a pool table

1137
00:54:00,869 --> 00:54:03,329
because of some weird background texture. Oh

1138
00:54:03,329 --> 00:54:07,030
wow. And ImageNet R contains renditions, so art,

1139
00:54:07,070 --> 00:54:10,429
cartoons, sculptures, origami, of ImageNet classes.

1140
00:54:10,670 --> 00:54:13,110
Oh, so literally testing the plushie and sketch

1141
00:54:13,110 --> 00:54:15,510
banana theories, but on a massive scale across

1142
00:54:15,510 --> 00:54:18,510
thousands of categories. And ImageNet Sketch

1143
00:54:18,510 --> 00:54:21,250
is just black and white line drawings? Exactly.

1144
00:54:22,070 --> 00:54:24,349
So they took a standard, fully supervised model,

1145
00:54:24,570 --> 00:54:27,710
one that gets a very respectable 76 .2 % accuracy

1146
00:54:27,710 --> 00:54:30,650
on regular ImageNet, and they tested it on these

1147
00:54:30,650 --> 00:54:34,710
tricky datasets. How did it do? It failed spectacularly.

1148
00:54:35,010 --> 00:54:38,590
When you test that standard 76 .2 % model on

1149
00:54:38,590 --> 00:54:41,349
ImageNet R, the renditions and cartoons, its

1150
00:54:41,349 --> 00:54:44,809
accuracy drops to 32 .6%. Cut in half? More than

1151
00:54:44,809 --> 00:54:48,050
half. More than half. It gets worse. When you

1152
00:54:48,050 --> 00:54:50,750
test it on ImageNet A, the naturally adversarial

1153
00:54:50,750 --> 00:54:53,849
images, its accuracy plummets to a staggering

1154
00:54:53,849 --> 00:54:58,900
2 .7%. 2 .7%. It basically forgot everything

1155
00:54:58,900 --> 00:55:01,059
it knew. It's practically guessing randomly at

1156
00:55:01,059 --> 00:55:03,659
that point. It shows how incredibly fragile standard

1157
00:55:03,659 --> 00:55:06,059
supervised learning is. The model hasn't learned

1158
00:55:06,059 --> 00:55:09,539
concepts. It has just memorized spurious correlations.

1159
00:55:10,000 --> 00:55:12,599
Spurious correlations. Give me a classic example

1160
00:55:12,599 --> 00:55:15,320
of that. The classic example is the cow on the

1161
00:55:15,320 --> 00:55:17,940
beach. If every single picture of a cow in your

1162
00:55:17,940 --> 00:55:20,099
training data set features the cow standing in

1163
00:55:20,099 --> 00:55:22,760
a grassy green pasture, the neural network might

1164
00:55:22,760 --> 00:55:25,460
take a shortcut. A shortcut? Yeah, it might learn

1165
00:55:25,460 --> 00:55:28,530
that... green pixels at the bottom of the image

1166
00:55:28,530 --> 00:55:32,000
equals the cow. If you show that model a perfectly

1167
00:55:32,000 --> 00:55:34,300
clear picture of a cow standing on a sandy beach,

1168
00:55:34,659 --> 00:55:37,719
it fails. The concept of the animal wasn't learned,

1169
00:55:37,800 --> 00:55:40,579
only the background context. That makes total

1170
00:55:40,579 --> 00:55:43,679
sense. So the standard model fails because it's

1171
00:55:43,679 --> 00:55:46,599
looking for green grass, not the cow. Now let's

1172
00:55:46,599 --> 00:55:49,900
look at clip. How did the zero -shot clip model,

1173
00:55:50,219 --> 00:55:53,989
the one that also gets 76 .2%, on standard ImageNet

1174
00:55:53,989 --> 00:55:57,389
perform on these tricky datasets? On ImageNet

1175
00:55:57,389 --> 00:56:01,469
R, the renditions, Clip scored 77 .1%. Whoa.

1176
00:56:02,050 --> 00:56:04,909
Compared to the standard models, 32 .6%. And

1177
00:56:04,909 --> 00:56:07,590
on ImageNet A, the adversarial one where the

1178
00:56:07,590 --> 00:56:12,690
standard model got 2 .7%, Clip scored 72 .3%.

1179
00:56:12,690 --> 00:56:15,250
That is a night and day difference. It didn't

1180
00:56:15,250 --> 00:56:17,650
drop at all. There is a chart in the paper showing

1181
00:56:17,650 --> 00:56:21,510
this, right? Yes. Figure 13. It is a defining

1182
00:56:21,510 --> 00:56:24,289
visualization for the field. It plots the performance

1183
00:56:24,289 --> 00:56:27,570
on standard ImageNet on the x -axis and the average

1184
00:56:27,570 --> 00:56:29,809
performance under distribution shift on the y

1185
00:56:29,809 --> 00:56:33,070
-axis. There's a dotted diagonal line representing

1186
00:56:33,070 --> 00:56:37,269
the ideal robust model where y equals x. Meaning

1187
00:56:37,269 --> 00:56:40,789
the ideal model performs exactly as well on weird

1188
00:56:40,789 --> 00:56:44,110
tricky data as it does on standard boring data.

1189
00:56:44,409 --> 00:56:47,829
Right. Now the standard supervised models form

1190
00:56:47,829 --> 00:56:51,940
a flat line way below that ideal diagonal. As

1191
00:56:51,940 --> 00:56:54,260
you pump more compute into them and they get

1192
00:56:54,260 --> 00:56:56,980
better at standard ImageNet, they barely improve

1193
00:56:56,980 --> 00:56:59,360
at all on the distribution shifts. They just

1194
00:56:59,360 --> 00:57:01,840
get better at memorizing the standard test. Exactly.

1195
00:57:02,500 --> 00:57:05,320
But the zero -shot clip models form a completely

1196
00:57:05,320 --> 00:57:08,260
different trajectory. They sharply close the

1197
00:57:08,260 --> 00:57:11,179
gap, moving much closer to that ideal y equals

1198
00:57:11,179 --> 00:57:14,199
x line. So I have to ask a big philosophical

1199
00:57:14,199 --> 00:57:17,440
question here. Based on this chart, Does this

1200
00:57:17,440 --> 00:57:20,340
mean that standard supervised learning on closed

1201
00:57:20,340 --> 00:57:22,440
datasets, the flashcard method we've been using

1202
00:57:22,440 --> 00:57:24,920
for a decade, has actually been teaching models

1203
00:57:24,920 --> 00:57:28,039
to be fundamentally fragile? Have we inadvertently

1204
00:57:28,039 --> 00:57:31,219
been training them to be oversensitive to irrelevant

1205
00:57:31,219 --> 00:57:34,420
details? Yes. That is exactly the implication.

1206
00:57:34,780 --> 00:57:36,659
And it's a hard pill for the field to swallow.

1207
00:57:37,360 --> 00:57:40,000
The researchers conclude that zero -shot evaluation

1208
00:57:40,000 --> 00:57:43,000
of task agnostic models is actually a much more

1209
00:57:43,000 --> 00:57:45,480
representative measure of a model's true capability.

1210
00:57:46,039 --> 00:57:48,179
Testing a model on the exact same distribution

1211
00:57:48,179 --> 00:57:50,800
it was trained on is basically cheating. It's

1212
00:57:50,800 --> 00:57:52,599
the difference between a student who memorizes

1213
00:57:52,599 --> 00:57:55,800
the practice exam to get an A versus a student

1214
00:57:55,800 --> 00:57:58,159
who actually understands the core material and

1215
00:57:58,159 --> 00:58:00,440
can answer essay questions on the real exam that

1216
00:58:00,440 --> 00:58:03,300
looked nothing like the practice test. Precisely.

1217
00:58:03,539 --> 00:58:06,639
Clip understands the material. OK. This all sounds

1218
00:58:06,639 --> 00:58:09,039
like absolute magic. It sounds like Clip has

1219
00:58:09,039 --> 00:58:10,980
completely solved computer vision and we can

1220
00:58:10,980 --> 00:58:13,920
all just pack up and go home. But we know there's

1221
00:58:13,920 --> 00:58:16,340
always a catch in AI. What is the hidden cost

1222
00:58:16,340 --> 00:58:18,659
of this breakthrough? And where does the technology

1223
00:58:18,659 --> 00:58:20,800
go from here? You're absolutely right to look

1224
00:58:20,800 --> 00:58:23,000
for the catch. And to their credit, the researchers

1225
00:58:23,000 --> 00:58:25,639
are very transparent about the elephant in the

1226
00:58:25,639 --> 00:58:29,230
room. Poor data efficiency. Data efficiency,

1227
00:58:29,349 --> 00:58:32,409
meaning how much data it needs to ingest to learn

1228
00:58:32,409 --> 00:58:35,449
a single concept. Right. CLIP compensates for

1229
00:58:35,449 --> 00:58:38,170
the inherent fundamental inefficiency of deep

1230
00:58:38,170 --> 00:58:40,829
learning by brute forcing it with a massive scale

1231
00:58:40,829 --> 00:58:43,809
of supervision. It learns incredibly well, but

1232
00:58:43,809 --> 00:58:47,110
it is an unbelievably slow learner. How slow

1233
00:58:47,110 --> 00:58:49,449
are we talking? Give me a number. During training,

1234
00:58:49,710 --> 00:58:53,190
the CLIP model runs for 32 epochs over its 400

1235
00:58:53,190 --> 00:58:57,010
million image text pairs. 32 times 400 million.

1236
00:58:57,159 --> 00:59:00,260
That means the model physically processes 12

1237
00:59:00,260 --> 00:59:04,360
.8 billion images during its training run. To

1238
00:59:04,360 --> 00:59:06,300
put that computational scale in perspective,

1239
00:59:06,800 --> 00:59:09,480
the paper notes that if every single image seen

1240
00:59:09,480 --> 00:59:12,039
during training was presented at a rate of one

1241
00:59:12,039 --> 00:59:16,739
per second, it would take 405 years to complete

1242
00:59:16,739 --> 00:59:20,619
the training. 405 years. One image a second,

1243
00:59:21,019 --> 00:59:25,449
nonstop, for four centuries. That is... horrifyingly

1244
00:59:25,449 --> 00:59:27,489
inefficient when you compare it to biology. It

1245
00:59:27,489 --> 00:59:30,550
really is. Think about a human toddler. A toddler

1246
00:59:30,550 --> 00:59:32,510
can see a picture of a banana in a board book,

1247
00:59:33,070 --> 00:59:34,769
maybe see a real one on the kitchen counter,

1248
00:59:34,969 --> 00:59:37,449
and they can instantly recognize a fuzzy, screaming,

1249
00:59:37,730 --> 00:59:40,349
plushy banana at the toy store. They don't need

1250
00:59:40,349 --> 00:59:43,309
to look at 12 .8 billion flashcards over 400

1251
00:59:43,309 --> 00:59:46,690
years to figure it out. Exactly. Clip achieves

1252
00:59:46,690 --> 00:59:50,269
robustness by brute -forcing scale, not by achieving

1253
00:59:50,269 --> 00:59:53,860
human -like learning efficiency. It is computationally

1254
00:59:53,860 --> 00:59:57,219
incredibly expensive, and that is the major limitation

1255
00:59:57,219 --> 01:00:00,039
of this approach. So how do we fix that? Because

1256
01:00:00,039 --> 01:00:02,320
we can't just keep scaling up to models that

1257
01:00:02,320 --> 01:00:04,880
take 10 ,000 years to train. What's the future

1258
01:00:04,880 --> 01:00:07,320
path here? The paper notes that combining the

1259
01:00:07,320 --> 01:00:10,039
core idea of CLIP, the contrast of matchmaking,

1260
01:00:10,579 --> 01:00:12,619
with self -supervision and self -training methods

1261
01:00:12,619 --> 01:00:15,940
is a highly promising direction. Self -supervision.

1262
01:00:16,260 --> 01:00:18,400
Like what? Techniques where a model learns the

1263
01:00:18,400 --> 01:00:20,619
underlying structure of an image by masking out

1264
01:00:20,619 --> 01:00:23,340
parts of it. You hide 50 % of the pixels and

1265
01:00:23,340 --> 01:00:25,500
force the model to predict what's missing based

1266
01:00:25,500 --> 01:00:27,780
on the surrounding context. Like filling in a

1267
01:00:27,780 --> 01:00:30,219
blank. Right. If you use self -supervision to

1268
01:00:30,219 --> 01:00:32,579
teach the image encoder the basic geometry of

1269
01:00:32,579 --> 01:00:35,019
the visual world before you start the contrast

1270
01:00:35,019 --> 01:00:37,920
of game, it might not need to see 12 .8 billion

1271
01:00:37,920 --> 01:00:40,559
images to understand that edges form shapes.

1272
01:00:40,840 --> 01:00:43,500
Basically, giving the model a solid visual foundation

1273
01:00:43,500 --> 01:00:46,019
before you introduce the messy language, so it's

1274
01:00:46,019 --> 01:00:48,019
not trying to learn what a pixel is and what

1275
01:00:48,019 --> 01:00:50,880
the word dog means at the exact same time. Exactly.

1276
01:00:51,679 --> 01:00:54,019
Improving data efficiency is the next major frontier,

1277
01:00:54,340 --> 01:00:57,260
but even with its current inefficiency, we have

1278
01:00:57,260 --> 01:00:59,360
to acknowledge that the type of supervision Clip

1279
01:00:59,360 --> 01:01:01,980
enables opens doors that were previously nailed

1280
01:01:01,980 --> 01:01:04,840
shut. Broadening supervision. Tell me about that.

1281
01:01:04,860 --> 01:01:07,639
What can Clip do that ImageNet couldn't, aside

1282
01:01:07,639 --> 01:01:09,929
from recognizing plushies? Think back to the

1283
01:01:09,929 --> 01:01:12,769
ImageNet categories. A thousand classes. Almost

1284
01:01:12,769 --> 01:01:15,489
all of them are common nouns. Dogs, cars, apples.

1285
01:01:15,889 --> 01:01:18,010
But because CLIP uses natural language, it can

1286
01:01:18,010 --> 01:01:20,150
learn so much more than just common nouns. It

1287
01:01:20,150 --> 01:01:22,849
can learn proper nouns, for one. Right. It can

1288
01:01:22,849 --> 01:01:25,070
recognize the Eiffel Tower, not just the generic

1289
01:01:25,070 --> 01:01:28,150
concept of a tower. Or Barack Obama, not just

1290
01:01:28,150 --> 01:01:31,289
person. Yes. It can learn complex adjectives,

1291
01:01:31,289 --> 01:01:33,650
as we saw with the rotting or pixelated bananas.

1292
01:01:34,150 --> 01:01:37,730
And crucially, it can learn verbs. Verbs. Action.

1293
01:01:38,070 --> 01:01:40,659
Yes. The paper specifically highlights that this

1294
01:01:40,659 --> 01:01:43,039
opens the door for complex tasks like action

1295
01:01:43,039 --> 01:01:46,019
recognition in videos. A static image data set

1296
01:01:46,019 --> 01:01:48,719
struggles immensely with the concept of running

1297
01:01:48,719 --> 01:01:51,360
versus jumping because those are temporal actions.

1298
01:01:51,500 --> 01:01:53,360
It's hard to capture in a still frame. Right.

1299
01:01:53,940 --> 01:01:56,280
But on the wild internet, images and video frames

1300
01:01:56,280 --> 01:01:58,880
are often explicitly captioned with verbs. Look

1301
01:01:58,880 --> 01:02:01,039
at him jumping over the fence. She is running

1302
01:02:01,039 --> 01:02:04,539
a marathon. So Clip can begin to associate visual

1303
01:02:04,539 --> 01:02:07,320
motion blur or specific human poses with action

1304
01:02:07,320 --> 01:02:10,730
verbs. Yes. We are shifting AI from just identifying

1305
01:02:10,730 --> 01:02:13,409
what is in a static image, this is a banana,

1306
01:02:13,769 --> 01:02:16,690
to understanding what is happening and how things

1307
01:02:16,690 --> 01:02:18,730
relate to each other in a scene. Effectively,

1308
01:02:18,730 --> 01:02:21,170
it's moving from an AI that can merely categorize

1309
01:02:21,170 --> 01:02:24,389
pixels into buckets to an AI that is visually

1310
01:02:24,389 --> 01:02:27,849
literate. It can actually read the scene. Visually

1311
01:02:27,849 --> 01:02:30,570
literate. That is the perfect phrase. It really

1312
01:02:30,570 --> 01:02:32,869
captures the leap from the flash card to the

1313
01:02:32,869 --> 01:02:36,170
library. Okay, let's wrap this up. We've been

1314
01:02:36,170 --> 01:02:38,670
on quite a journey today. We started by looking

1315
01:02:38,670 --> 01:02:41,809
at the Great Divide, how natural language processing

1316
01:02:41,809 --> 01:02:45,210
was racing ahead by reading the raw chaotic internet

1317
01:02:45,210 --> 01:02:48,190
while computer vision was stuck memorizing curated

1318
01:02:48,190 --> 01:02:50,849
one -of -n flashcards on rigid datasets like

1319
01:02:50,849 --> 01:02:53,710
ImageNet. We traced the pragmatic middle ground,

1320
01:02:53,789 --> 01:02:56,090
seeing how early attempts to use Flickr metadata

1321
01:02:56,090 --> 01:02:59,590
or Instagram hashtags just didn't have the scale,

1322
01:02:59,949 --> 01:03:02,469
the computational efficiency, or the dynamic

1323
01:03:02,469 --> 01:03:05,929
output required for true generalized zero -shot

1324
01:03:05,929 --> 01:03:08,980
learning. And then we saw how the OpenAI researchers

1325
01:03:08,980 --> 01:03:13,139
broke the mold by forging the WIT dataset, 400

1326
01:03:13,139 --> 01:03:16,059
million pairs of messy unstructured internet

1327
01:03:16,059 --> 01:03:18,780
data. They threw away the gold labels and embraced

1328
01:03:18,780 --> 01:03:22,039
the wild. And they made it computationally possible

1329
01:03:22,039 --> 01:03:25,639
with the CLIP algorithm, pivoting from a predictive

1330
01:03:25,639 --> 01:03:28,260
bottleneck to a highly efficient contrastive

1331
01:03:28,260 --> 01:03:31,219
matchmaking game on a massive end -by -end grid.

1332
01:03:31,920 --> 01:03:34,760
Which yielded a 12x efficiency gain. allowing

1333
01:03:34,760 --> 01:03:37,320
them to train a multimodal model from scratch

1334
01:03:37,320 --> 01:03:39,840
that could finally pass the ultimate banana test.

1335
01:03:40,500 --> 01:03:42,579
It could recognize a fuzzy screaming plushie

1336
01:03:42,579 --> 01:03:45,519
banana, or a rotting banana on concrete, or a

1337
01:03:45,519 --> 01:03:49,460
retro N64 pixel banana. Not because it memorized

1338
01:03:49,460 --> 01:03:52,199
the exact pixel patterns, but because it understood

1339
01:03:52,199 --> 01:03:54,539
the semantic concept linked through the rich

1340
01:03:54,539 --> 01:03:57,869
context of natural language. It proved that this

1341
01:03:57,869 --> 01:04:01,050
task -agnostic, language -driven approach creates

1342
01:04:01,050 --> 01:04:03,530
models that are fundamentally more robust to

1343
01:04:03,530 --> 01:04:06,210
the messy adversarial distribution shifts of

1344
01:04:06,210 --> 01:04:08,769
the real world, sharply closing the gap toward

1345
01:04:08,769 --> 01:04:12,329
an ideal, robust model. Even if it takes 405

1346
01:04:12,329 --> 01:04:15,289
years of one image a second compute time to brute

1347
01:04:15,289 --> 01:04:18,130
force its way there. The inefficiency is very

1348
01:04:18,130 --> 01:04:20,849
real, but the paradigm shift is undeniable. It

1349
01:04:20,849 --> 01:04:23,050
changed computer vision forever. It really did.

1350
01:04:23,670 --> 01:04:26,300
So, to end today's deep dive. I want to leave

1351
01:04:26,300 --> 01:04:29,139
you, the listener, with a final provocative thought

1352
01:04:29,139 --> 01:04:31,760
to mull over. We've seen today how natural language

1353
01:04:31,760 --> 01:04:34,500
supervision was the key that unlocked visual

1354
01:04:34,500 --> 01:04:36,980
understanding without explicit rigid labeling.

1355
01:04:37,719 --> 01:04:39,820
Language was the bridge between the chaos of

1356
01:04:39,820 --> 01:04:42,019
the visual world and the logic of a machine.

1357
01:04:42,849 --> 01:04:45,789
So, what happens when we apply this exact same

1358
01:04:45,789 --> 01:04:49,150
philosophy to other sensory domains? That's a

1359
01:04:49,150 --> 01:04:51,949
huge question. Could we eventually train an AI?

1360
01:04:52,130 --> 01:04:55,010
to interpret complex, chaotic audio environments

1361
01:04:55,010 --> 01:04:58,269
like a crowded city street or a rainforest canopy?

1362
01:04:58,769 --> 01:05:00,570
Could we train it to understand the sensation

1363
01:05:00,570 --> 01:05:03,469
of physical touch or even complex human emotional

1364
01:05:03,469 --> 01:05:06,309
states in real time? And could we do all of this

1365
01:05:06,309 --> 01:05:08,969
simply by pairing vast, unstructured amounts

1366
01:05:08,969 --> 01:05:11,949
of raw, multi -sensory data with the natural

1367
01:05:11,949 --> 01:05:14,469
language we use to describe our human experience?

1368
01:05:15,050 --> 01:05:17,690
If natural language is the universal API for

1369
01:05:17,690 --> 01:05:20,710
human thought, Perhaps it is destined to be the

1370
01:05:20,710 --> 01:05:23,429
universal API for all machine perception as well.

1371
01:05:23,909 --> 01:05:26,369
If that's the case, the messy reality of the

1372
01:05:26,369 --> 01:05:29,170
world might not be so confusing to machines for

1373
01:05:29,170 --> 01:05:32,000
much longer. Thanks for diving deep with us today.