1
00:00:00,000 --> 00:00:03,600
So in 2010, the global stock market just evaporated

2
00:00:03,600 --> 00:00:07,059
by, I mean, a trillion dollars in just 36 minutes.

3
00:00:07,200 --> 00:00:09,820
Right. The flash crash. Yeah, exactly. And there

4
00:00:09,820 --> 00:00:11,720
was no bomb. You know, there was no declaration

5
00:00:11,720 --> 00:00:14,380
of war. There wasn't even like a massive geopolitical

6
00:00:14,380 --> 00:00:17,460
news event that day. It was entirely algorithmic.

7
00:00:17,500 --> 00:00:21,120
It was just these silent strings of code making

8
00:00:21,120 --> 00:00:24,100
millions of decisions a second, reacting to each

9
00:00:24,100 --> 00:00:27,140
other in this insane cascading feedback loop.

10
00:00:27,239 --> 00:00:29,719
Yeah. And then. Just as quickly as that trillion

11
00:00:29,719 --> 00:00:32,399
dollars vanished, the algorithms recalibrated

12
00:00:32,399 --> 00:00:34,829
and the market just bounced back. Like nothing

13
00:00:34,829 --> 00:00:37,630
happened. Right. So today we're opening up that

14
00:00:37,630 --> 00:00:40,929
black box. We want to see exactly how those supercomputers

15
00:00:40,929 --> 00:00:44,750
learn to print money and occasionally how they

16
00:00:44,750 --> 00:00:46,969
almost destroy the entire system. Because the

17
00:00:46,969 --> 00:00:49,350
financial industry really relies on this perception

18
00:00:49,350 --> 00:00:53,109
that quantitative trading is sort of this dark

19
00:00:53,109 --> 00:00:56,450
art. Oh, absolutely. Wall Street benefits immensely

20
00:00:56,450 --> 00:00:59,429
for making these concepts seem like they're strictly

21
00:00:59,429 --> 00:01:03,009
the domain of rocket scientists or quantum physicists.

22
00:01:03,350 --> 00:01:06,909
But at its core, algorithmic trading is really

23
00:01:06,909 --> 00:01:10,010
just the mechanization of human behavior and

24
00:01:10,099 --> 00:01:13,319
statistical probabilities. And that mechanization

25
00:01:13,319 --> 00:01:15,319
is our mission for you today in this deep dive.

26
00:01:15,519 --> 00:01:19,000
We are pulling back the curtain on this totally

27
00:01:19,000 --> 00:01:22,400
inaccessible mystery box. And we have a great

28
00:01:22,400 --> 00:01:25,319
guide for this. We really do. We're using this

29
00:01:25,319 --> 00:01:29,459
incredibly expansive, just meticulous text, Deep

30
00:01:29,459 --> 00:01:33,060
Learning for Finance. It's a 2024 book by Sofian

31
00:01:33,060 --> 00:01:36,040
Kabar. It's extremely comprehensive. Yeah. And

32
00:01:36,040 --> 00:01:37,959
we aren't just doing a high level overview here.

33
00:01:38,019 --> 00:01:40,340
We're going concept by concept, translating the

34
00:01:40,340 --> 00:01:42,939
raw math into the actual mechanics of modern

35
00:01:42,939 --> 00:01:45,239
trading. We're going to start from the absolute

36
00:01:45,239 --> 00:01:47,000
fundamental building blocks, you know, the raw,

37
00:01:47,079 --> 00:01:49,560
messy financial data. Right. And walk you all

38
00:01:49,560 --> 00:01:51,879
the way to the bleeding edge of autonomous artificial

39
00:01:51,879 --> 00:01:54,819
intelligence. Because the goal here is to interrelate

40
00:01:54,819 --> 00:01:57,280
these highly complex concepts so that they actually

41
00:01:57,280 --> 00:01:59,239
make mechanical sense. Make them accessible.

42
00:01:59,540 --> 00:02:02,560
Exactly. We're going to explore how an algorithm

43
00:02:02,560 --> 00:02:06,079
actually thinks, how it processes the noise of

44
00:02:06,079 --> 00:02:08,300
the market, and how it arrives at a decision

45
00:02:08,300 --> 00:02:10,620
to buy or sell. But we're going to do it without

46
00:02:10,620 --> 00:02:13,159
sacrificing the mathematical rigor that makes

47
00:02:13,159 --> 00:02:15,520
those decisions possible. Right. Because before

48
00:02:15,520 --> 00:02:18,139
a machine can learn anything, it needs a universe

49
00:02:18,139 --> 00:02:21,000
to learn from. It needs raw material. The data.

50
00:02:21,240 --> 00:02:24,020
The data. And the source material focuses really

51
00:02:24,020 --> 00:02:26,879
heavily on demystifying data science and Python

52
00:02:26,879 --> 00:02:29,139
right out of the gate. We have to begin with

53
00:02:29,139 --> 00:02:32,400
how data is gathered and crucially how it's fundamentally

54
00:02:32,400 --> 00:02:35,400
altered before an algorithm ever even like glances

55
00:02:35,400 --> 00:02:38,340
at a price tag. There's this foundational rule

56
00:02:38,340 --> 00:02:40,740
in computer science, right? Garbage in, garbage

57
00:02:40,740 --> 00:02:43,340
out. Oh, yeah. Classic. If the universe you present

58
00:02:43,340 --> 00:02:46,219
to your algorithm is flawed, every subsequent

59
00:02:46,219 --> 00:02:48,240
prediction it makes, no matter how sophisticated

60
00:02:48,240 --> 00:02:51,680
your multilayer neural network is. It's going

61
00:02:51,680 --> 00:02:54,419
to be equally flawed. It's built on sand. Precisely.

62
00:02:54,419 --> 00:02:57,139
I mean, in 2012, a firm called Knight Capital

63
00:02:57,139 --> 00:03:01,919
lost $440 million in 45 minutes. Wow. Yeah. And

64
00:03:01,919 --> 00:03:04,460
it was just because of a flaw in how their software

65
00:03:04,460 --> 00:03:06,639
interacted with the market data. It bankrupted

66
00:03:06,639 --> 00:03:09,599
them. So understanding the raw material, it isn't

67
00:03:09,599 --> 00:03:11,919
some administrative task. It's literally the

68
00:03:11,919 --> 00:03:14,379
foundation of survival. Okay. So looking at the

69
00:03:14,379 --> 00:03:18,240
text, it keeps bringing up OHLC data as the primary

70
00:03:18,240 --> 00:03:20,979
raw material. Right. Open, high, low, close.

71
00:03:21,419 --> 00:03:23,020
Yeah. But is that just the raw ticker price,

72
00:03:23,219 --> 00:03:25,300
like the numbers we see scrolling at the bottom

73
00:03:25,300 --> 00:03:28,120
of the screen on a financial news network? Not

74
00:03:28,120 --> 00:03:31,039
quite. I mean, that scrolling ticker is usually

75
00:03:31,039 --> 00:03:33,360
just the last traded price, just a single data

76
00:03:33,360 --> 00:03:36,800
point. OK. OHLC gives you the open, high, low

77
00:03:36,800 --> 00:03:39,580
and close prices for a specific given time period,

78
00:03:39,699 --> 00:03:42,879
let's say a day or even just a five minute window.

79
00:03:43,099 --> 00:03:45,560
So it's more of a snapshot of the action. Exactly.

80
00:03:46,169 --> 00:03:49,110
It gives the algorithm a multidimensional view

81
00:03:49,110 --> 00:03:51,189
of the price action rather than just a single

82
00:03:51,189 --> 00:03:53,830
point. It tells the machine where the battle

83
00:03:53,830 --> 00:03:56,169
between the buyers and sellers started, right,

84
00:03:56,229 --> 00:03:58,490
the open. Right. How far the buyers pushed it

85
00:03:58,490 --> 00:04:01,009
up, the high. How far the sellers pushed it down,

86
00:04:01,090 --> 00:04:03,389
the low. And who ultimately won at the end of

87
00:04:03,389 --> 00:04:05,909
the period, which is the close. But the data

88
00:04:05,909 --> 00:04:07,469
goes way beyond just numbers on the spreadsheet

89
00:04:07,469 --> 00:04:10,030
now, right? The book breaks down several types

90
00:04:10,030 --> 00:04:12,990
of data used in modern quant finance. We've got

91
00:04:12,990 --> 00:04:16,199
the numerical data, like OHLC. But then there's

92
00:04:16,199 --> 00:04:19,060
categorical data. Right, like labeling specific

93
00:04:19,060 --> 00:04:22,300
market regimes. Is it a bull market, a bear market,

94
00:04:22,459 --> 00:04:25,259
high volatility? Yeah. And then there's text

95
00:04:25,259 --> 00:04:28,000
data, which is just massive right now with natural

96
00:04:28,000 --> 00:04:30,740
language processing. NLP is everywhere. I mean,

97
00:04:30,740 --> 00:04:32,899
algorithms are literally reading news headlines

98
00:04:32,899 --> 00:04:35,939
in milliseconds. Or they're parsing the specific

99
00:04:35,939 --> 00:04:38,279
vocabulary of a central bank speech to gauge

100
00:04:38,279 --> 00:04:40,720
sentiment, which is crazy to me. It gets even

101
00:04:40,720 --> 00:04:43,079
wilder when you bring in visual data. Right,

102
00:04:43,100 --> 00:04:47,500
visual. Like... Charts? Literally images of charts.

103
00:04:47,819 --> 00:04:50,180
Neural networks can actually be trained to recognize

104
00:04:50,180 --> 00:04:53,040
chart patterns by feeding them image files. No

105
00:04:53,040 --> 00:04:55,560
way. Yeah. So instead of teaching a computer

106
00:04:55,560 --> 00:04:58,079
to identify a stop sign for like a self -driving

107
00:04:58,079 --> 00:05:00,639
car, you're training it to visually identify

108
00:05:00,639 --> 00:05:02,939
a descending channel on a stock chart. That's

109
00:05:02,939 --> 00:05:05,100
incredible. Yeah. And the book also mentions

110
00:05:05,100 --> 00:05:08,560
audio data becoming prevalent. It is. Algorithms

111
00:05:08,560 --> 00:05:11,060
are starting to parse the tone of voice or, you

112
00:05:11,060 --> 00:05:13,439
know, the micro hesitations, even pitch changes

113
00:05:13,439 --> 00:05:15,980
of a CEO during a live quarterly earnings call.

114
00:05:16,120 --> 00:05:19,240
Just to detect if they're lying? Basically, yeah.

115
00:05:19,259 --> 00:05:22,680
Or anxious. Basically, yeah. To detect concealed

116
00:05:22,680 --> 00:05:25,079
anxiety or on the flip side, overconfidence.

117
00:05:25,279 --> 00:05:28,399
It's extracting data from human emotion. Which

118
00:05:28,399 --> 00:05:31,839
is completely mind blowing. But OK, to process

119
00:05:31,839 --> 00:05:35,620
any of this, Kabbar outlines six critical steps

120
00:05:35,620 --> 00:05:39,199
of data science. gathering, pre -processing,

121
00:05:39,199 --> 00:05:43,160
exploration, visualization, analysis, and interpretation.

122
00:05:43,519 --> 00:05:47,129
The core workflow. Right. And to anchor this

123
00:05:47,129 --> 00:05:49,490
really abstract process, the book gives a very

124
00:05:49,490 --> 00:05:53,129
specific, practical example using the VIX. Right,

125
00:05:53,189 --> 00:05:55,990
the VIX. So the VIX is widely known as the fear

126
00:05:55,990 --> 00:05:59,220
index. I hear that term a lot on the news. Yeah,

127
00:05:59,259 --> 00:06:02,079
it tracks the implied volatility of the S &P

128
00:06:02,079 --> 00:06:04,920
500 using options pricing. So it doesn't measure

129
00:06:04,920 --> 00:06:08,300
price directly. It measures how much traders

130
00:06:08,300 --> 00:06:10,519
are willing to pay for insurance against massive

131
00:06:10,519 --> 00:06:12,860
price swings. So it's basically a gauge of panic.

132
00:06:13,139 --> 00:06:16,019
Exactly. When the market is panicking, the VIX

133
00:06:16,019 --> 00:06:19,300
spikes violently. But when the market is complacent,

134
00:06:19,319 --> 00:06:21,120
you know, just grinding higher day after day,

135
00:06:21,220 --> 00:06:24,970
the VIX slowly bleeds downward. Okay, so the

136
00:06:24,970 --> 00:06:27,269
source walks through using a Python library called...

137
00:06:27,629 --> 00:06:30,550
pandas data reader to gather this VIX data. Right.

138
00:06:30,629 --> 00:06:32,470
It's an API call. Yeah. You just write a few

139
00:06:32,470 --> 00:06:34,990
lines of code to connect to a data source and

140
00:06:34,990 --> 00:06:38,449
you request the VIX data from like January 1st,

141
00:06:38,449 --> 00:06:42,250
1990, all the way to early 2023. And in seconds,

142
00:06:42,290 --> 00:06:46,009
you have over 30 years of digitized market fear

143
00:06:46,009 --> 00:06:48,589
just sitting on your screen. It's incredibly

144
00:06:48,589 --> 00:06:51,250
powerful. But when you look closely at that downloaded

145
00:06:51,250 --> 00:06:54,470
data, it isn't perfect. The text highlights this

146
00:06:54,470 --> 00:06:57,250
massive roadblock, which is handling invalid

147
00:06:57,250 --> 00:07:00,790
data. Specifically, in that 30 -year data set,

148
00:07:00,930 --> 00:07:06,910
the author found 292 NAN values. N -A -N. Not

149
00:07:06,910 --> 00:07:09,509
a number. Yeah, finding those blank spots in

150
00:07:09,509 --> 00:07:12,350
a historical timeline, that's the rule. It's

151
00:07:12,350 --> 00:07:14,529
not the exception, yeah. Really? Yeah. I think

152
00:07:14,529 --> 00:07:17,149
historical data, it's pretty solid. You'd hope

153
00:07:17,149 --> 00:07:18,990
so, but data gets corrupted during transmission

154
00:07:18,990 --> 00:07:21,500
all the time. Exchanges suffer micro outages

155
00:07:21,500 --> 00:07:23,860
or maybe there are unexpected market holidays

156
00:07:23,860 --> 00:07:27,040
that mess up the formatting. Oh, OK. But here's

157
00:07:27,040 --> 00:07:29,500
the issue. You absolutely cannot feed a machine

158
00:07:29,500 --> 00:07:31,720
learning algorithm a blank space. Because it's

159
00:07:31,720 --> 00:07:34,560
math. Right. If a neural network tries to multiply

160
00:07:34,560 --> 00:07:38,000
a weight by not a number, the math just breaks.

161
00:07:38,120 --> 00:07:40,319
The code throws an error and the entire system

162
00:07:40,319 --> 00:07:44,079
halts. So it has to be pre -processed. So the

163
00:07:44,079 --> 00:07:46,519
text gives three specific mathematical ways to

164
00:07:46,519 --> 00:07:50,199
handle missing data, like those 292 NANDs in

165
00:07:50,199 --> 00:07:54,019
the VIX, or, say, missing quarterly GDP numbers

166
00:07:54,019 --> 00:07:56,819
from some economic database. What are the three

167
00:07:56,819 --> 00:07:59,459
options it outlines? Well, first, you can just

168
00:07:59,459 --> 00:08:02,040
delete the row entirely. Second, you can assume

169
00:08:02,040 --> 00:08:05,339
the missing cell equals the exact value of the

170
00:08:05,339 --> 00:08:08,199
previous cell, so just carrying it forward. Or

171
00:08:08,199 --> 00:08:10,519
third, you can calculate a mean, an average of

172
00:08:10,519 --> 00:08:12,720
the surrounding cells to kind of smooth it over.

173
00:08:13,079 --> 00:08:14,860
And I really want to emphasize the philosophical

174
00:08:14,860 --> 00:08:16,779
weight of that choice because it seems like a

175
00:08:16,779 --> 00:08:19,079
mundane administrative task, right? Just cleaning

176
00:08:19,079 --> 00:08:20,860
up a spreadsheet. Yeah, it sounds like housekeeping.

177
00:08:21,199 --> 00:08:24,540
But it's not. Pre -processing is literally altering

178
00:08:24,540 --> 00:08:26,839
the fabric of reality that the algorithm is going

179
00:08:26,839 --> 00:08:29,420
to learn from. Wait, how do you mean? Well, let's

180
00:08:29,420 --> 00:08:31,600
say you choose to fill a missing GDP number with

181
00:08:31,600 --> 00:08:33,679
the mathematical average of the previous quarter

182
00:08:33,679 --> 00:08:36,399
and the next quarter. Okay. You are commanding

183
00:08:36,399 --> 00:08:38,419
the algorithm to learn from an economic trajectory

184
00:08:38,419 --> 00:08:41,590
that never actually happened. You've artificially

185
00:08:41,590 --> 00:08:43,870
erased a potential shock and smoothed out the

186
00:08:43,870 --> 00:08:47,500
volatility of the past. Okay, hold on. I'm thinking

187
00:08:47,500 --> 00:08:49,240
about that first option now, deleting the row.

188
00:08:49,419 --> 00:08:52,100
Right. If it's Tuesday and Wednesday's data is

189
00:08:52,100 --> 00:08:55,279
corrupted, so we just delete Wednesday. Thursday

190
00:08:55,279 --> 00:08:57,480
is now sitting right next to Tuesday in the spreadsheet.

191
00:08:57,759 --> 00:09:00,039
It is. The algorithm is going to think Thursday

192
00:09:00,039 --> 00:09:03,090
happened right after Tuesday. Doesn't that completely

193
00:09:03,090 --> 00:09:05,669
destroy the timeline of the market? Like in a

194
00:09:05,669 --> 00:09:07,590
system where momentum and the distance between

195
00:09:07,590 --> 00:09:10,629
events really matter. Aren't we blinding the

196
00:09:10,629 --> 00:09:12,990
machine to the actual passage of time? It's a

197
00:09:12,990 --> 00:09:15,610
really valid concern. And it highlights a crucial

198
00:09:15,610 --> 00:09:18,230
vulnerability in proof processing. Now, when

199
00:09:18,230 --> 00:09:20,730
you use a Python library like Pandas and you

200
00:09:20,730 --> 00:09:23,230
invoke a function like Dropna to delete those

201
00:09:23,230 --> 00:09:26,129
NANs. Right, Dropna. The remaining data isn't

202
00:09:26,129 --> 00:09:29,009
just mindlessly mashed together like a text document.

203
00:09:29,210 --> 00:09:31,909
The timestamps attached to those prices, which

204
00:09:31,909 --> 00:09:35,490
is called the index, they remain intact. Oh,

205
00:09:35,490 --> 00:09:38,169
OK. So Thursday still mathematically knows it's

206
00:09:38,169 --> 00:09:41,110
Thursday. Exactly. Advanced models can be engineered

207
00:09:41,110 --> 00:09:43,669
to look at the timestamp gap rather than just

208
00:09:43,669 --> 00:09:47,169
sequential steps. Whew, okay. However, a lot

209
00:09:47,169 --> 00:09:49,529
of standard models do process data sequentially.

210
00:09:49,629 --> 00:09:52,129
And for those models, deleting a row absolutely

211
00:09:52,129 --> 00:09:55,330
warps the temporal distance between events. You

212
00:09:55,330 --> 00:09:57,769
are, in effect, artificially accelerating time.

213
00:09:58,090 --> 00:10:00,669
See, basically every method of dealing with missing

214
00:10:00,669 --> 00:10:03,870
data is a compromise. Always. You're either losing

215
00:10:03,870 --> 00:10:06,350
time by deleting it, you're repeating the past

216
00:10:06,350 --> 00:10:08,350
artificially by carrying the last number forward,

217
00:10:08,529 --> 00:10:11,730
or you're hallucinating this smooth, fake reality

218
00:10:11,730 --> 00:10:14,950
by taking an average. And you have to understand

219
00:10:14,950 --> 00:10:16,970
the specific biases you're introducing when you

220
00:10:16,970 --> 00:10:19,370
make that choice. Because the algorithm, it's

221
00:10:19,370 --> 00:10:21,909
going to treat whatever you hand it as the absolute,

222
00:10:22,129 --> 00:10:24,389
unquestionable truth. It doesn't know you manipulated

223
00:10:24,389 --> 00:10:27,450
it. Right. If you fill missing data with averages,

224
00:10:27,769 --> 00:10:29,990
your model will eventually assume the market

225
00:10:29,990 --> 00:10:32,370
is much safer and smoother than it really is.

226
00:10:32,549 --> 00:10:35,049
And that leads to taking on excessive risk when

227
00:10:35,049 --> 00:10:39,029
things get chaotic. Man. OK, so we've gathered

228
00:10:39,029 --> 00:10:41,769
our raw material. We've made our philosophical

229
00:10:41,769 --> 00:10:44,129
choices about the missing pieces. And we now

230
00:10:44,129 --> 00:10:47,549
have a pristine pre -processed data set. Ready

231
00:10:47,549 --> 00:10:50,850
to go. But raw prices are just a historical ledger.

232
00:10:51,320 --> 00:10:53,659
We have to translate that data into a language

233
00:10:53,659 --> 00:10:56,200
the machine can actually mathematically parse.

234
00:10:56,379 --> 00:10:59,220
Yes. We're moving from the hard computer science

235
00:10:59,220 --> 00:11:02,299
of data gathering into the subjective psychological

236
00:11:02,299 --> 00:11:05,059
history of market analysis. We have to figure

237
00:11:05,059 --> 00:11:07,720
out how to translate human panic and greed into

238
00:11:07,720 --> 00:11:10,679
stationary statistics. And the absolute gateway

239
00:11:10,679 --> 00:11:12,980
to that translation and arguably the most important

240
00:11:12,980 --> 00:11:15,460
statistical concept in all of quantitative finance

241
00:11:15,460 --> 00:11:19,000
is stationarity. Stationarity. Yes. Machine learning

242
00:11:19,000 --> 00:11:22,159
models require data with stable statistical properties.

243
00:11:22,279 --> 00:11:24,600
If the environment is constantly shifting beneath

244
00:11:24,600 --> 00:11:26,840
their feet, they literally cannot learn meaningful

245
00:11:26,840 --> 00:11:30,039
rules. Let's break that down for everyone. When

246
00:11:30,039 --> 00:11:31,960
we say stable statistical properties, we are

247
00:11:31,960 --> 00:11:34,840
talking about things like a stable mean, right?

248
00:11:34,899 --> 00:11:38,080
Like a stable average over time. Correct. Consider

249
00:11:38,080 --> 00:11:41,379
the raw price of the S &P 500. Over a 30 -year

250
00:11:41,379 --> 00:11:44,220
horizon, it trends massively upward, right? From

251
00:11:44,220 --> 00:11:47,320
around 300 in the early 1990s to well over 4

252
00:11:47,320 --> 00:11:50,399
,000 today. Right. If you feed an algorithm those

253
00:11:50,399 --> 00:11:52,620
raw prices, it really struggles to find recurring

254
00:11:52,620 --> 00:11:55,159
patterns because the baseline just keeps drifting

255
00:11:55,159 --> 00:11:57,480
away. A pattern that occurred when the price

256
00:11:57,480 --> 00:12:00,039
was at 500 might look mathematically completely

257
00:12:00,039 --> 00:12:02,080
different to the machine than the exact same

258
00:12:02,080 --> 00:12:04,600
pattern when the price is at 4 ,000. Because

259
00:12:04,600 --> 00:12:07,620
the scales totally warped. Exactly. Stationary

260
00:12:07,620 --> 00:12:09,980
data is data where the mean and the variance,

261
00:12:10,080 --> 00:12:12,820
so the average and the spread, they don't care

262
00:12:12,820 --> 00:12:15,139
about the time at which they're measured. They

263
00:12:15,139 --> 00:12:18,259
remain relatively flat. Okay, so the text mentions

264
00:12:18,259 --> 00:12:21,120
using the augmented Dickey -Fuller test, the

265
00:12:21,120 --> 00:12:24,320
ADF test. Yes. And this is to mathematically

266
00:12:24,320 --> 00:12:27,659
prove if a data set is stationary. It basically

267
00:12:27,659 --> 00:12:30,320
spits out a p -value, and if that value is low

268
00:12:30,320 --> 00:12:33,120
enough, you can trust your data is stable. That's

269
00:12:33,120 --> 00:12:35,940
the threshold, yeah. And to force the raw VIX

270
00:12:35,940 --> 00:12:38,500
data into stationarity, the author uses this

271
00:12:38,500 --> 00:12:41,000
process called differencing. Differencing is

272
00:12:41,000 --> 00:12:44,320
crucial. It transforms the data from absolute

273
00:12:44,320 --> 00:12:48,019
values into relative changes. How so? Instead

274
00:12:48,019 --> 00:12:49,879
of telling the algorithm that the price today

275
00:12:49,879 --> 00:12:53,539
is $1 .50 and yesterday it was $1 .48, you just

276
00:12:53,539 --> 00:12:55,980
tell the algorithm the price changed by positive

277
00:12:55,980 --> 00:12:59,580
two. Oh, okay. Right? And then the next day,

278
00:12:59,620 --> 00:13:02,059
it might change by negative one. By looking only

279
00:13:02,059 --> 00:13:05,220
at the daily changes, the data loses that massive

280
00:13:05,220 --> 00:13:07,500
upward trend. It begins to hover tightly around

281
00:13:07,500 --> 00:13:10,919
a mean of zero. And boom, the data becomes stationary.

282
00:13:11,179 --> 00:13:13,740
Exactly. And once we have that stable mathematical

283
00:13:13,740 --> 00:13:16,580
foundation, we can start applying technical analysis

284
00:13:16,580 --> 00:13:18,639
to create the actual features we'll feed our

285
00:13:18,639 --> 00:13:21,600
model. Right, the inputs. And technical analysis

286
00:13:21,600 --> 00:13:23,559
is fascinating to me because it's built on three

287
00:13:23,559 --> 00:13:27,299
core, almost philosophical beliefs. Let's hear

288
00:13:27,299 --> 00:13:32,029
them. One, history repeats itself. Two, the market

289
00:13:32,029 --> 00:13:34,649
discounts everything, meaning the current price

290
00:13:34,649 --> 00:13:36,830
already reflects all public information, news,

291
00:13:36,929 --> 00:13:39,289
earnings, geopolitical tension. It's all baked

292
00:13:39,289 --> 00:13:41,309
into the price already. The efficient market

293
00:13:41,309 --> 00:13:44,570
hypothesis, essentially. Yeah. And three, the

294
00:13:44,570 --> 00:13:47,269
market moves in waves, not straight lines. And

295
00:13:47,269 --> 00:13:49,250
Khabar divides the application of those beliefs

296
00:13:49,250 --> 00:13:52,429
into two distinct camps. You have charting and

297
00:13:52,429 --> 00:13:55,669
you have indicators. Charting is what immediately

298
00:13:55,669 --> 00:13:57,690
comes to mind when you picture like a day trader,

299
00:13:57,830 --> 00:14:00,769
right? Drawing angled lines on a screen, connecting

300
00:14:00,769 --> 00:14:03,429
the bottoms of a price chart to find support

301
00:14:03,429 --> 00:14:06,389
or connecting the tops to find resistance. Yep.

302
00:14:06,809 --> 00:14:09,950
Classical technical analysis. The text talks

303
00:14:09,950 --> 00:14:12,629
about ascending and descending channels where

304
00:14:12,629 --> 00:14:14,710
the price bounces back and forth between two

305
00:14:14,710 --> 00:14:17,309
parallel lines. And there's a really fascinating

306
00:14:17,309 --> 00:14:19,590
psychological bias the author mentions here called

307
00:14:19,590 --> 00:14:22,490
the invisible hand. The invisible hand bias is

308
00:14:22,490 --> 00:14:24,830
so interesting because it illustrates how charting

309
00:14:24,830 --> 00:14:26,909
can become a self -fulfilling prophecy. Right.

310
00:14:26,990 --> 00:14:29,309
When human traders look at an ascending channel,

311
00:14:29,470 --> 00:14:32,789
so a stock bouncing upward between two upward

312
00:14:32,789 --> 00:14:35,750
sloping lines, they're heavily influenced by

313
00:14:35,750 --> 00:14:38,669
market axioms like Martin Zweig's famous quote,

314
00:14:38,870 --> 00:14:41,809
the trend is your friend until the end when it

315
00:14:41,809 --> 00:14:45,139
bends. Classic trader speak. Right. Because thousands

316
00:14:45,139 --> 00:14:47,620
of human traders see the exact same line on their

317
00:14:47,620 --> 00:14:49,799
screens and they hold the same ingrained belief,

318
00:14:50,080 --> 00:14:52,720
whenever the price drops to touch that bottom

319
00:14:52,720 --> 00:14:55,799
support line, an invisible hand of bullish pressure

320
00:14:55,799 --> 00:14:58,340
enters the market. Because they all buy at the

321
00:14:58,340 --> 00:15:00,899
exact same time. Exactly. They automatically

322
00:15:00,899 --> 00:15:03,200
hit the buy button because they collectively

323
00:15:03,200 --> 00:15:05,779
believe the trend will hold. Okay, but this is

324
00:15:05,779 --> 00:15:07,679
where I struggle with the premise of this part

325
00:15:07,679 --> 00:15:10,019
of the book. I'm looking at my notes and traditional

326
00:15:10,019 --> 00:15:12,649
chart reading. Literally drawing lines on a screen

327
00:15:12,649 --> 00:15:15,169
based on where your human eyes think a trend

328
00:15:15,169 --> 00:15:18,049
should be. It feels a bit like financial astrology.

329
00:15:18,470 --> 00:15:21,009
It gets that criticism a lot. Right. Because

330
00:15:21,009 --> 00:15:22,450
two different analysts could look at the exact

331
00:15:22,450 --> 00:15:25,389
same S &P chart and draw totally conflicting

332
00:15:25,389 --> 00:15:28,250
support lines. Like one guy includes the wick

333
00:15:28,250 --> 00:15:30,370
of a candlestick and the other just uses the

334
00:15:30,370 --> 00:15:32,970
body. True. So why in the world does a meticulous

335
00:15:32,970 --> 00:15:35,889
book about math -driven deep learning dedicate

336
00:15:35,889 --> 00:15:39,460
space to subjective human charting? It's a fair

337
00:15:39,460 --> 00:15:42,149
question. Drawing lines by hand is inherently

338
00:15:42,149 --> 00:15:44,490
subjective. And you're completely correct that

339
00:15:44,490 --> 00:15:46,889
a machine learning model cannot ingest a trader's

340
00:15:46,889 --> 00:15:49,669
visual interpretation of a chart. A neural network

341
00:15:49,669 --> 00:15:52,450
doesn't have a gut feeling about a support zone.

342
00:15:52,629 --> 00:15:55,450
It doesn't. But the text includes charting to

343
00:15:55,450 --> 00:15:57,730
establish the underlying psychology that drives

344
00:15:57,730 --> 00:16:00,210
those market waves. To actually capture that

345
00:16:00,210 --> 00:16:02,669
psychology for the machine, though, we rely on

346
00:16:02,669 --> 00:16:05,830
the second camp of technical analysis. Indicators.

347
00:16:06,330 --> 00:16:09,610
Okay, like the moving average. Or the RSI. Yes.

348
00:16:10,870 --> 00:16:13,769
Indicators digitize that subjective human psychology

349
00:16:13,769 --> 00:16:17,590
into objective mathematical formulas. Ah, I see.

350
00:16:17,789 --> 00:16:21,169
Take the RSI, the Relative Strength Index. It's

351
00:16:21,169 --> 00:16:23,509
just a formula that measures the speed and change

352
00:16:23,509 --> 00:16:25,889
of price movements on a scale from 0 to 100.

353
00:16:26,210 --> 00:16:29,049
Okay. The industry standard looks at the last

354
00:16:29,049 --> 00:16:33,389
14 periods. When the RSI gets above 70, human

355
00:16:33,389 --> 00:16:35,629
traders collectively look at the chart. They

356
00:16:35,629 --> 00:16:37,789
feel that the asset has gone up way too far,

357
00:16:37,850 --> 00:16:40,509
way too fast. They declare it overbought, and

358
00:16:40,509 --> 00:16:42,610
they start selling. Right. And then the opposite

359
00:16:42,610 --> 00:16:45,629
is true. Yep. When it falls below 30, they declare

360
00:16:45,629 --> 00:16:47,529
it oversold, and they expect a bounce, so they

361
00:16:47,529 --> 00:16:49,970
buy. Awesome. Okay, so the algorithm isn't learning

362
00:16:49,970 --> 00:16:53,210
to draw subjective lines on a chart. It's ingesting

363
00:16:53,210 --> 00:16:57,330
the RSI data as a raw number. Precisely. The

364
00:16:57,330 --> 00:16:59,590
deep learning models, they do not care about

365
00:16:59,590 --> 00:17:02,090
the visual elegance of a chart pattern, but they

366
00:17:02,090 --> 00:17:04,829
can easily ingest that 14 period RSI value as

367
00:17:04,829 --> 00:17:07,410
a stationary statistical feature. Because it's

368
00:17:07,410 --> 00:17:09,549
just a number between zero and 100. Exactly.

369
00:17:09,710 --> 00:17:12,569
The model can process 10 million historical data

370
00:17:12,569 --> 00:17:15,150
points and mathematically learn exactly when

371
00:17:15,150 --> 00:17:17,589
human traders are statistically likely to panic

372
00:17:17,589 --> 00:17:21,690
because the RSI hit 15 or get irrationally exuberant

373
00:17:21,690 --> 00:17:24,609
because it hit 85. Wow. We are turning messy

374
00:17:24,609 --> 00:17:26,970
human emotional cycles into crap. cold, hard

375
00:17:26,970 --> 00:17:29,730
numbers that a neural network can process and

376
00:17:29,730 --> 00:17:32,730
optimize. Okay, that bridges the gap perfectly

377
00:17:32,730 --> 00:17:36,589
for me. So we have digitized human trading psychology.

378
00:17:37,069 --> 00:17:39,849
We've forced those emotional waves into stationary

379
00:17:39,849 --> 00:17:42,869
statistics using differencing. Our raw materials

380
00:17:42,869 --> 00:17:45,029
are completely prepped. They're mathematically

381
00:17:45,029 --> 00:17:48,569
sound. We're ready for the algorithm. Yes. Now

382
00:17:48,569 --> 00:17:51,390
we enter the machine. We step into the realm

383
00:17:51,390 --> 00:17:54,690
of introductory machine learning. And crucially...

384
00:17:55,279 --> 00:17:58,079
how we evaluate if the machine is actually doing

385
00:17:58,079 --> 00:18:01,079
anything useful. Right. To build a machine learning

386
00:18:01,079 --> 00:18:03,559
model, the author sets up a very strict framework.

387
00:18:04,039 --> 00:18:07,980
We need features, our x values. Those are the

388
00:18:07,980 --> 00:18:10,900
inputs we feed into the algorithm. Like the RSI.

389
00:18:11,119 --> 00:18:14,059
Like the RSI, exactly. And we need a target,

390
00:18:14,200 --> 00:18:16,759
our y value. That's the specific outcome we're

391
00:18:16,759 --> 00:18:20,119
trying to predict. So the text uses a specific

392
00:18:20,119 --> 00:18:24,099
4x example here. It uses lagged daily price differences

393
00:18:24,099 --> 00:18:27,819
on the UER -USD currency pair. The euro against

394
00:18:27,819 --> 00:18:30,839
the U .S. dollar? Right. It uses those daily

395
00:18:30,839 --> 00:18:33,759
differences as the features, the X values. And

396
00:18:33,759 --> 00:18:36,200
it uses those to try and predict the future returns,

397
00:18:36,299 --> 00:18:38,980
which is the Y value. So we're basically looking

398
00:18:38,980 --> 00:18:41,920
at how much the euro to dollar exchange rate

399
00:18:41,920 --> 00:18:44,740
changed over the last few days to guess what

400
00:18:44,740 --> 00:18:46,849
it's going to do tomorrow. But before deploying

401
00:18:46,849 --> 00:18:49,250
some massively complex algorithm to solve this,

402
00:18:49,470 --> 00:18:52,589
Khabar introduces a really vital concept. It

403
00:18:52,589 --> 00:18:55,150
prevents quantitative researchers from fooling

404
00:18:55,150 --> 00:18:58,369
themselves. Oh, the dummy regression model. Yes,

405
00:18:58,490 --> 00:19:01,049
the dummy model. It sounds insulting, honestly,

406
00:19:01,230 --> 00:19:03,329
but it's actually this really elegant reality

407
00:19:03,329 --> 00:19:06,130
check. It really is. A dummy model is just a

408
00:19:06,130 --> 00:19:08,890
basic benchmark algorithm that uses incredibly

409
00:19:08,890 --> 00:19:12,190
simple, naive rules. In predicting a time series,

410
00:19:12,509 --> 00:19:14,910
a common dummy model simply predicts that Whatever

411
00:19:14,910 --> 00:19:17,710
happened today will happen again tomorrow. Wait,

412
00:19:17,769 --> 00:19:20,589
really? That's it. That's it. If the euro was

413
00:19:20,589 --> 00:19:23,009
up today, the dummy model blindly predicts the

414
00:19:23,009 --> 00:19:26,009
euro will be up tomorrow. The rule is simple.

415
00:19:26,109 --> 00:19:29,049
If your computationally expensive, highly complex

416
00:19:29,049 --> 00:19:31,329
artificial intelligence cannot beat the performance

417
00:19:31,329 --> 00:19:34,250
of this completely naive dummy model. Your AI

418
00:19:34,250 --> 00:19:37,150
adds absolutely zero value. Exactly. You scrap

419
00:19:37,150 --> 00:19:39,049
it. It's worthless. It's the ultimate gatekeeper.

420
00:19:39,170 --> 00:19:42,339
I love that. So the text evaluates these models

421
00:19:42,339 --> 00:19:45,019
using a few key performance metrics. We have

422
00:19:45,019 --> 00:19:47,380
accuracy, which, you know, is pretty straightforward.

423
00:19:47,680 --> 00:19:50,519
And then we have root mean squared error, or

424
00:19:50,519 --> 00:19:54,299
RMSE. RMSE is really critical because of how

425
00:19:54,299 --> 00:19:56,220
it treats errors mathematically. How does it

426
00:19:56,220 --> 00:19:58,619
treat them? By squaring the difference between

427
00:19:58,619 --> 00:20:01,200
the predicted price and the actual price, the

428
00:20:01,200 --> 00:20:03,680
formula heavily penalizes large mistakes compared

429
00:20:03,680 --> 00:20:05,720
to small ones. Okay, let me make sure I get this.

430
00:20:05,859 --> 00:20:08,769
Think of it this way. If your model misses a

431
00:20:08,769 --> 00:20:11,369
price prediction by one penny, the squared error

432
00:20:11,369 --> 00:20:14,329
is tiny. One squared is just one. But if it misses

433
00:20:14,329 --> 00:20:18,950
by $10, 10 squared is 100. Squaring that error

434
00:20:18,950 --> 00:20:22,210
explodes the penalty. I see. So it trains the

435
00:20:22,210 --> 00:20:25,349
model to avoid catastrophic misses at all costs.

436
00:20:25,549 --> 00:20:28,869
Exactly. It forces the model to be robust against

437
00:20:28,869 --> 00:20:32,089
massive outliers. Okay. And here is the specific

438
00:20:32,089 --> 00:20:34,150
example from the book that literally stopped

439
00:20:34,150 --> 00:20:36,049
me in my tracks when I read it. Oh, the baseline

440
00:20:36,049 --> 00:20:39,289
model. Yes. Kabar runs a baseline machine learning

441
00:20:39,289 --> 00:20:43,730
model on the US dollar USD testing data using

442
00:20:43,730 --> 00:20:45,970
all these clean features we just talked about.

443
00:20:46,089 --> 00:20:52,150
And the final result is 49 .33 % accuracy. 49

444
00:20:52,150 --> 00:20:56,170
.33%. I mean, this isn't just an academic exercise,

445
00:20:56,369 --> 00:21:00,049
right? This 49 % accuracy model might be the

446
00:21:00,049 --> 00:21:03,470
exact foundational code managing someone's 401k

447
00:21:03,470 --> 00:21:06,369
right now. It very well could be. Why on earth

448
00:21:06,369 --> 00:21:08,650
are we spending hours learning Python, writing

449
00:21:08,650 --> 00:21:12,390
complex code, processing data, and building sophisticated

450
00:21:12,390 --> 00:21:14,710
machine learning models if they are statistically

451
00:21:14,710 --> 00:21:18,150
worse than just flipping a quarter? We built

452
00:21:18,150 --> 00:21:20,849
a robot that loses to a coin flip? The frustration

453
00:21:20,849 --> 00:21:24,329
is entirely warranted. I get it. But the context

454
00:21:24,329 --> 00:21:27,809
of that 49 .33 % reveals the core challenge of

455
00:21:27,809 --> 00:21:31,089
quantitative finance. First, we really have to

456
00:21:31,089 --> 00:21:33,670
clarify what that metric actually measures. Okay.

457
00:21:33,769 --> 00:21:36,230
It's directional accuracy on a daily horizon,

458
00:21:36,430 --> 00:21:38,789
meaning did it go up or did it go down tomorrow?

459
00:21:38,890 --> 00:21:41,230
It does not account for the magnitude of the

460
00:21:41,230 --> 00:21:44,349
move. Okay. So it just cares about green or red.

461
00:21:44,529 --> 00:21:47,369
Right. If it was wrong on 10 days where the market

462
00:21:47,369 --> 00:21:49,809
barely moved at all, just chopped sideways, but

463
00:21:49,809 --> 00:21:52,130
it correctly predicted one day where the market

464
00:21:52,130 --> 00:21:55,210
rallied massively, the model might actually be

465
00:21:55,210 --> 00:21:57,890
wildly profitable despite a terrible accuracy

466
00:21:57,890 --> 00:22:01,559
rate. Oh. Okay, that makes a lot of sense. Furthermore,

467
00:22:01,740 --> 00:22:04,559
this is a raw model output. There are no transaction

468
00:22:04,559 --> 00:22:07,099
costs applied, no risk management frameworks,

469
00:22:07,259 --> 00:22:09,940
and no stop losses. So it's literally just raw,

470
00:22:10,079 --> 00:22:13,119
unfiltered directional guessing without any actual

471
00:22:13,119 --> 00:22:16,740
trading logic applied. Exactly. But more importantly,

472
00:22:16,940 --> 00:22:19,200
the author deliberately includes this mediocre

473
00:22:19,200 --> 00:22:21,900
result to teach the most dangerous concept in

474
00:22:21,900 --> 00:22:24,839
all of machine learning. Overfitting. Overfitting,

475
00:22:24,940 --> 00:22:27,779
yes. Overfitting is like a student who memorizes

476
00:22:27,779 --> 00:22:29,660
the answers to the practice test, right? But

477
00:22:29,660 --> 00:22:31,539
they don't actually learn the subject. That's

478
00:22:31,539 --> 00:22:34,759
a perfect analogy. A data scientist could easily

479
00:22:34,759 --> 00:22:36,640
build a machine learning model that achieves

480
00:22:36,640 --> 00:22:39,880
99 % accuracy on past data. You achieve that

481
00:22:39,880 --> 00:22:42,019
by making the model incredibly complex, giving

482
00:22:42,019 --> 00:22:44,779
it tons of parameters, and just allowing it to

483
00:22:44,779 --> 00:22:47,039
memorize the historical data set completely.

484
00:22:47,440 --> 00:22:49,640
It just remembers what happened in 2008. Right.

485
00:22:49,869 --> 00:22:52,950
It will capture every tiny random fluctuation,

486
00:22:52,970 --> 00:22:55,210
and it'll look like a flawless money printing

487
00:22:55,210 --> 00:22:58,069
machine on paper. But the moment you deploy it

488
00:22:58,069 --> 00:23:01,069
in the real world on unseen data, it will utterly

489
00:23:01,069 --> 00:23:03,390
fail. Because it learned the random noise of

490
00:23:03,390 --> 00:23:05,829
the past, not the underlying signal. Exactly.

491
00:23:05,950 --> 00:23:08,670
The mathematical bias -variance tradeoff explains

492
00:23:08,670 --> 00:23:11,230
this perfectly. A model with high complexity

493
00:23:11,230 --> 00:23:13,589
has low bias. It fits the training data perfectly,

494
00:23:13,789 --> 00:23:16,210
but massive variance when exposed to new unseen

495
00:23:16,210 --> 00:23:21,299
data. So a 49 .33 % accuracy on testing data,

496
00:23:21,420 --> 00:23:23,880
meaning data the model hasn't seen during its

497
00:23:23,880 --> 00:23:26,579
training phase. That's actually a feature, not

498
00:23:26,579 --> 00:23:28,640
a bug for this introductory part of the book.

499
00:23:28,819 --> 00:23:33,880
Yes. A 49 % baseline on unseen test data without

500
00:23:33,880 --> 00:23:36,900
overfitting is the honest starting point for

501
00:23:36,900 --> 00:23:39,599
optimization. It proves the model is attempting

502
00:23:39,599 --> 00:23:42,259
to generalize the real rules of the market, even

503
00:23:42,259 --> 00:23:44,400
if it's currently failing to beat a coin flip.

504
00:23:44,579 --> 00:23:47,539
It's the raw clay. It is. It proves you haven't

505
00:23:47,539 --> 00:23:50,599
cheated by just memorizing the past. Okay. Which

506
00:23:50,599 --> 00:23:52,819
brings us to the revolution. Because if we want

507
00:23:52,819 --> 00:23:55,000
to beat that coin flip, if we want to extract

508
00:23:55,000 --> 00:23:58,339
real tradable signal from the chaotic noise of

509
00:23:58,339 --> 00:24:00,880
the market, shallow machine learning just isn't

510
00:24:00,880 --> 00:24:03,039
enough. No, it hits a wall. We have to change

511
00:24:03,039 --> 00:24:05,700
the very architecture of the algorithm to discover

512
00:24:05,700 --> 00:24:08,680
relationships that human analysts literally cannot

513
00:24:08,680 --> 00:24:10,700
physically see. We're talking about deep learning.

514
00:24:11,180 --> 00:24:13,220
The transition from machine learning to deep

515
00:24:13,220 --> 00:24:15,880
learning is fundamentally about depth and nonlinear

516
00:24:15,880 --> 00:24:18,980
complexity. In the text, Kabar focuses on artificial

517
00:24:18,980 --> 00:24:23,160
neural networks, or ANNs. Specifically, he utilizes

518
00:24:23,160 --> 00:24:27,359
multilayer perceptrons, MLPs, and long short

519
00:24:27,359 --> 00:24:29,579
-term memory models, which are called LSTMs.

520
00:24:29,720 --> 00:24:32,420
And he uses these to forecast S &P 500 returns.

521
00:24:32,720 --> 00:24:35,380
Let's talk about the architecture of these networks,

522
00:24:35,599 --> 00:24:38,279
because this is where the magic really happens.

523
00:24:38,970 --> 00:24:41,089
The text explains that these deep models contain

524
00:24:41,089 --> 00:24:44,809
what are called hidden layers. Yes. So you have

525
00:24:44,809 --> 00:24:47,630
an input layer where the stationary data like

526
00:24:47,630 --> 00:24:50,730
our RSI indicators goes in. You have an output

527
00:24:50,730 --> 00:24:53,549
layer where the prediction comes out. And in

528
00:24:53,549 --> 00:24:57,430
between is this vast interconnected web of hidden

529
00:24:57,430 --> 00:25:00,759
layers. Right. A shallow machine learning model

530
00:25:00,759 --> 00:25:03,380
might just look at the direct linear relationship

531
00:25:03,380 --> 00:25:05,400
between yesterday's price and today's price.

532
00:25:06,069 --> 00:25:08,609
But a deep model with all its hidden layers applies

533
00:25:08,609 --> 00:25:11,930
activation functions that allow it to catch nonlinear

534
00:25:11,930 --> 00:25:14,650
relationships. Let's visualize how those layers

535
00:25:14,650 --> 00:25:17,230
actually stack up. Like layer one might look

536
00:25:17,230 --> 00:25:19,789
at basic price momentum, right? Sure. Then layer

537
00:25:19,789 --> 00:25:22,329
two might take the output of layer one and combine

538
00:25:22,329 --> 00:25:25,289
it with volatility data. Then layer three might

539
00:25:25,289 --> 00:25:27,869
take the output of layer two and analyze how

540
00:25:27,869 --> 00:25:30,210
momentum and volatility interact with the RSI

541
00:25:30,210 --> 00:25:33,910
on Tuesdays during high market volume. Exactly.

542
00:25:34,380 --> 00:25:36,059
By the time the data cascades through all the

543
00:25:36,059 --> 00:25:39,400
hidden layers and reaches the output, the network

544
00:25:39,400 --> 00:25:42,859
has synthesized patterns so complex, so multidimensional,

545
00:25:42,920 --> 00:25:45,200
that a human mathematician couldn't possibly

546
00:25:45,200 --> 00:25:47,940
graph them on a screen. It's beyond human comprehension

547
00:25:47,940 --> 00:25:51,730
at that level. And the long short -term memory

548
00:25:51,730 --> 00:25:54,410
model, the LSTM, takes this a step further. How

549
00:25:54,410 --> 00:25:57,190
so? Traditional neural networks process data

550
00:25:57,190 --> 00:26:00,049
in isolated snapshots. But financial markets

551
00:26:00,049 --> 00:26:02,799
are sequential. The order of events matters deeply.

552
00:26:03,579 --> 00:26:06,319
LSTMs contain specialized memory cells with gates.

553
00:26:06,539 --> 00:26:10,180
Gates? Yeah. An input gate, an output gate, and

554
00:26:10,180 --> 00:26:13,220
crucially, a forget gate. This allows the model

555
00:26:13,220 --> 00:26:15,900
to process a sequence of time, explicitly deciding

556
00:26:15,900 --> 00:26:18,619
which information from 10 days ago is important

557
00:26:18,619 --> 00:26:21,099
enough to keep in its memory, and which information

558
00:26:21,099 --> 00:26:22,940
is just noise that should be forgotten entirely.

559
00:26:23,420 --> 00:26:26,640
Wow. So it's actively pruning its own memory.

560
00:26:27,589 --> 00:26:29,970
But to make these massive networks actually function

561
00:26:29,970 --> 00:26:33,049
profitably, the text introduces optimization

562
00:26:33,049 --> 00:26:36,950
satellites. Yes. These aren't the neural network

563
00:26:36,950 --> 00:26:39,609
itself, but they're techniques and formulas that

564
00:26:39,609 --> 00:26:42,289
kind of hover around the network to refine the

565
00:26:42,289 --> 00:26:45,230
data and the training process. Things like fractional

566
00:26:45,230 --> 00:26:48,380
differentiation. Fractional differentiation is

567
00:26:48,380 --> 00:26:51,220
an incredibly elegant mathematical solution to

568
00:26:51,220 --> 00:26:53,220
the stationarity problem we discussed earlier.

569
00:26:53,460 --> 00:26:55,599
Right, because standard differencing... where

570
00:26:55,599 --> 00:26:57,740
we just subtract yesterday's price from today's

571
00:26:57,740 --> 00:27:00,500
price is kind of like having severe amnesia every

572
00:27:00,500 --> 00:27:02,779
morning. That's a good way to put it. You only

573
00:27:02,779 --> 00:27:05,680
know how today changed from yesterday, but you

574
00:27:05,680 --> 00:27:07,240
completely lose the memory of what the price

575
00:27:07,240 --> 00:27:10,039
was five years ago. You lose the overarching

576
00:27:10,039 --> 00:27:12,940
trend of the whole market. So how does fractional

577
00:27:12,940 --> 00:27:15,299
differencing fix that? Well, instead of integer

578
00:27:15,299 --> 00:27:17,259
differencing, which applies a weight of one to

579
00:27:17,259 --> 00:27:19,339
today and minus one to yesterday and completely

580
00:27:19,339 --> 00:27:22,460
ignores everything before that. Fractional differencing,

581
00:27:22,539 --> 00:27:26,160
say an order of 0 .5, applies a slowly decaying

582
00:27:26,160 --> 00:27:29,599
mathematical weight to all past prices. It considers

583
00:27:29,599 --> 00:27:32,740
yesterday heavily, the day before slightly less,

584
00:27:32,940 --> 00:27:35,440
and the price from five years ago only a tiny

585
00:27:35,440 --> 00:27:38,380
fraction. It mathematically flattens the curve

586
00:27:38,380 --> 00:27:40,920
just enough to pass the augmented Dickey -Fuller

587
00:27:40,920 --> 00:27:43,440
test for stationarity, but it retains the long

588
00:27:43,440 --> 00:27:45,680
-term memory of the overarching market trend.

589
00:27:46,180 --> 00:27:48,319
So it gives the deep learning model the best

590
00:27:48,319 --> 00:27:50,700
of both worlds. Exactly. The short -term stability

591
00:27:50,700 --> 00:27:53,980
and the long -term context. The book also covers

592
00:27:53,980 --> 00:27:56,539
forecasting thresholds and continuous retraining

593
00:27:56,539 --> 00:28:00,279
as vital optimization satellites. And continuous

594
00:28:00,279 --> 00:28:02,779
retraining is pretty intuitive, right? The market

595
00:28:02,779 --> 00:28:06,000
is a living, breathing ecosystem. You can't train

596
00:28:06,000 --> 00:28:08,420
a model on the low inflation environment of 2018

597
00:28:08,420 --> 00:28:11,680
and expect it to profitably trade the high interest

598
00:28:11,680 --> 00:28:14,799
rate market of 2024 without continuously updating

599
00:28:14,799 --> 00:28:17,380
its internal weights. It would get crushed. But

600
00:28:17,380 --> 00:28:19,799
this brings up the mystery box dilemma again.

601
00:28:20,099 --> 00:28:23,359
If a multilayer... LSTM neural network finds

602
00:28:23,359 --> 00:28:26,539
a highly complex nonlinear hidden pattern, but

603
00:28:26,539 --> 00:28:28,940
we have literally no idea what that pattern actually

604
00:28:28,940 --> 00:28:31,859
is because it's buried in the math of a thousand

605
00:28:31,859 --> 00:28:35,559
hidden nodes. How can a hedge fund possibly trust

606
00:28:35,559 --> 00:28:37,720
it with hundreds of millions of dollars? That's

607
00:28:37,720 --> 00:28:40,299
terrifying. Right. How do you give real capital

608
00:28:40,299 --> 00:28:43,519
to a black box? Institutional funds grapple with

609
00:28:43,519 --> 00:28:46,400
this exact dilemma every single day. The regulatory

610
00:28:46,400 --> 00:28:49,039
and financial risk of deploying an opaque model

611
00:28:49,039 --> 00:28:53,460
is massive. The answer is you do not trust the

612
00:28:53,460 --> 00:28:56,160
black box blindly. You mitigate the risk of the

613
00:28:56,160 --> 00:28:58,700
unknown using rigorous validation frameworks.

614
00:28:59,079 --> 00:29:02,339
So you cross -validate it. The book mentions

615
00:29:02,339 --> 00:29:05,900
time series cross -validation. Yes. Cross -validating

616
00:29:05,900 --> 00:29:09,279
time series data is like training a weather forecaster.

617
00:29:09,559 --> 00:29:12,630
Okay. How so? With normal data, you might train

618
00:29:12,630 --> 00:29:14,690
a model on random data points and test it on

619
00:29:14,690 --> 00:29:16,769
other random points. But time series data has

620
00:29:16,769 --> 00:29:19,390
a strict chronology. You cannot train your weather

621
00:29:19,390 --> 00:29:22,089
forecaster on random days from the summer and

622
00:29:22,089 --> 00:29:24,089
test their predictive accuracy on random days

623
00:29:24,089 --> 00:29:25,809
in the winter. Right, because time only moves

624
00:29:25,809 --> 00:29:28,710
forward. Exactly. You have to train them on the

625
00:29:28,710 --> 00:29:32,390
data from, say, 1990 to 2010, and then test their

626
00:29:32,390 --> 00:29:34,569
predictive power on the unseen year of 2011.

627
00:29:35,109 --> 00:29:37,960
Then you roll the window forward. So then you'd

628
00:29:37,960 --> 00:29:42,880
train on 1991 to 2011 and test on 2012. Precisely.

629
00:29:42,880 --> 00:29:45,400
And by rolling that window chronologically forward

630
00:29:45,400 --> 00:29:48,039
through decades of market history, you basically

631
00:29:48,039 --> 00:29:50,200
proved that the model can consistently adapt

632
00:29:50,200 --> 00:29:52,920
to unseen data without peering into the future

633
00:29:52,920 --> 00:29:57,420
to cheat. Exactly. The text is unambiguous about

634
00:29:57,420 --> 00:29:59,559
this. Deep learning in finance is fundamentally

635
00:29:59,559 --> 00:30:02,440
an exercise in massive, structured, and validated

636
00:30:02,440 --> 00:30:05,230
trial and error. You are structurally weeding

637
00:30:05,230 --> 00:30:08,069
out market noise from true alpha. You might not

638
00:30:08,069 --> 00:30:10,670
know exactly what mathematical correlations the

639
00:30:10,670 --> 00:30:13,309
hidden layers are prioritizing, but your rigorous

640
00:30:13,309 --> 00:30:15,990
chronological cross -validation framework mathematically

641
00:30:15,990 --> 00:30:18,309
proves that the model's logic works reliably

642
00:30:18,309 --> 00:30:21,250
over time. All right, let's turn the dial up

643
00:30:21,250 --> 00:30:23,710
even further. Everything we've discussed so far,

644
00:30:23,869 --> 00:30:27,670
shallow machine learning, deep learning, LSTMs,

645
00:30:27,750 --> 00:30:30,230
it all culminates in predicting a price or at

646
00:30:30,230 --> 00:30:33,279
least a direction, right? Yes, forecasting. But

647
00:30:33,279 --> 00:30:35,500
what if the algorithm didn't just predict the

648
00:30:35,500 --> 00:30:37,700
price? What if it could actually learn how to

649
00:30:37,700 --> 00:30:40,640
trade that price on its own, managing its own

650
00:30:40,640 --> 00:30:44,359
portfolio? We are moving into the absolute bleeding

651
00:30:44,359 --> 00:30:47,740
edge of the book here. Deep reinforcement learning.

652
00:30:48,019 --> 00:30:51,759
This represents a massive paradigm shift in artificial

653
00:30:51,759 --> 00:30:54,019
intelligence. With supervised deep learning,

654
00:30:54,200 --> 00:30:56,619
you provide the model with the answers, the historical

655
00:30:56,619 --> 00:30:59,650
target prices. And you say. Adjust your internal

656
00:30:59,650 --> 00:31:01,809
weights until you learn how to predict this answer.

657
00:31:01,990 --> 00:31:04,410
Right. You give it the test key. With reinforcement

658
00:31:04,410 --> 00:31:07,329
learning, there are no answers provided. You

659
00:31:07,329 --> 00:31:09,450
create a framework where an autonomous agent

660
00:31:09,450 --> 00:31:12,009
interacts with an environment, learning an optimal

661
00:31:12,009 --> 00:31:14,789
policy of behavior entirely through trial and

662
00:31:14,789 --> 00:31:18,309
error. The text lays out the framework for this

663
00:31:18,309 --> 00:31:20,789
autonomous agent really clearly. You have states,

664
00:31:21,029 --> 00:31:24,730
actions, and rewards. Let's unpack the states

665
00:31:24,730 --> 00:31:28,160
first. The state is basically the current environment

666
00:31:28,160 --> 00:31:30,900
the agent finds itself in. And the book mentions

667
00:31:30,900 --> 00:31:34,220
using cue tables to represent this. I need a

668
00:31:34,220 --> 00:31:36,460
bit of help translating cue tables into plain

669
00:31:36,460 --> 00:31:39,740
English. Sure. Think of a cue table as a massive,

670
00:31:39,779 --> 00:31:42,859
multidimensional cheat sheet that the agent constantly

671
00:31:42,859 --> 00:31:45,240
updates. Okay. A cheat sheet. It's a matrix.

672
00:31:45,519 --> 00:31:48,220
The rows of the matrix represent every possible

673
00:31:48,220 --> 00:31:51,140
state the market could be in. For example, a

674
00:31:51,140 --> 00:31:54,519
state where the RSI is at 70, the MACD is crossing

675
00:31:54,519 --> 00:31:57,019
over, and the price is touching an upper Bollinger

676
00:31:57,019 --> 00:31:59,680
Band. Got it. And the columns of the matrix represent

677
00:31:59,680 --> 00:32:02,720
the actions the agent can take, buy, sell, or

678
00:32:02,720 --> 00:32:05,119
hold. And what goes inside the intersecting cells

679
00:32:05,119 --> 00:32:08,750
of that matrix? The expected future reward. Those

680
00:32:08,750 --> 00:32:11,869
are the Q values. When the agent is in a specific

681
00:32:11,869 --> 00:32:14,309
state and takes a specific action, it receives

682
00:32:14,309 --> 00:32:17,410
a reward or a penalty from the environment. Based

683
00:32:17,410 --> 00:32:19,730
on whether the trade made money or lost money.

684
00:32:19,930 --> 00:32:22,950
Exactly. It then uses a mathematical formula,

685
00:32:23,150 --> 00:32:25,730
it's called the Bellman equation, to update the

686
00:32:25,730 --> 00:32:28,769
Q value in that specific cell. Over time, the

687
00:32:28,769 --> 00:32:31,269
agent looks at its cheat sheet, finds the state

688
00:32:31,269 --> 00:32:33,589
it's currently in, and simply chooses the action

689
00:32:33,589 --> 00:32:36,720
with the highest Q value. Okay. And the rewards

690
00:32:36,720 --> 00:32:39,740
are the ultimate goals you program into the system.

691
00:32:39,819 --> 00:32:42,640
You can tell it to just maximize absolute profit

692
00:32:42,640 --> 00:32:46,460
per trade or, more intelligently, you can tell

693
00:32:46,460 --> 00:32:49,299
it to maximize the Sharpe ratio, which rewards

694
00:32:49,299 --> 00:32:52,200
risk -adjusted returns, forcing the agent to

695
00:32:52,200 --> 00:32:55,339
prioritize smooth equity curves rather than wild,

696
00:32:55,420 --> 00:32:59,000
volatile gains. Yes. You can even severely penalize

697
00:32:59,000 --> 00:33:01,539
the agent for maximum drawdowns, teaching it

698
00:33:01,539 --> 00:33:04,900
to be terrified of losing capital. And initially,

699
00:33:05,099 --> 00:33:07,339
the agent begins by taking completely random

700
00:33:07,339 --> 00:33:10,420
actions. It buys at the wrong time, loses money,

701
00:33:10,559 --> 00:33:13,480
and receives a negative reward, a penalty. The

702
00:33:13,480 --> 00:33:16,180
Q value for that action drops. Makes sense. But

703
00:33:16,180 --> 00:33:17,799
then it sells at the right time, makes money,

704
00:33:17,960 --> 00:33:20,259
gets a positive reward, and the Q value increases.

705
00:33:21,500 --> 00:33:24,480
Over millions of simulated iterations, it populates

706
00:33:24,480 --> 00:33:27,480
that entire queue table and learns a comprehensive

707
00:33:27,480 --> 00:33:31,259
policy of behavior that maximizes its total cumulative

708
00:33:31,259 --> 00:33:34,539
reward. But the text outlines a massive hurdle

709
00:33:34,539 --> 00:33:36,880
for these agents, the exploration versus exploitation

710
00:33:36,880 --> 00:33:40,759
dilemma. Ah, yes. The classic RL problem. The

711
00:33:40,759 --> 00:33:43,519
algorithm constantly has to make a choice. Does

712
00:33:43,519 --> 00:33:45,740
it stick with the trading strategy that has generated

713
00:33:45,740 --> 00:33:48,210
a steady profit so far? which is exploitation?

714
00:33:48,710 --> 00:33:51,690
Or does it randomly try a completely new, untested

715
00:33:51,690 --> 00:33:55,309
strategy, just in case it yields a massive undiscovered

716
00:33:55,309 --> 00:33:57,990
payout, which is exploration? This is managed

717
00:33:57,990 --> 00:34:00,230
mathematically using what is called an epsilon

718
00:34:00,230 --> 00:34:03,789
greedy strategy. Epsilon greedy. Yeah. Early

719
00:34:03,789 --> 00:34:06,190
in the training, the epsilon value is high, meaning

720
00:34:06,190 --> 00:34:08,329
the agent ignores its cue table and explores

721
00:34:08,329 --> 00:34:10,610
randomly most of the time. It tries everything.

722
00:34:11,510 --> 00:34:14,309
As training progresses, the epsilon value decays

723
00:34:14,309 --> 00:34:16,530
and the agent begins to exploit the knowledge

724
00:34:16,530 --> 00:34:19,050
it's gathered. So it settles down. Right. But

725
00:34:19,050 --> 00:34:22,230
if the epsilon decays too fast, the agent exploits

726
00:34:22,230 --> 00:34:24,909
too early, locking into a mediocre strategy and

727
00:34:24,909 --> 00:34:27,090
missing a vastly more profitable one hidden in

728
00:34:27,090 --> 00:34:30,630
the data. If it explores too much, it never establishes

729
00:34:30,630 --> 00:34:32,909
a consistent strategy. You know, it sounds incredibly

730
00:34:32,909 --> 00:34:35,769
similar to training a puppy with treats. If you

731
00:34:35,769 --> 00:34:37,989
tell a puppy to sit and it sits, you give it

732
00:34:37,989 --> 00:34:40,889
a treat. Positive reinforcement. It updates its

733
00:34:40,889 --> 00:34:44,130
internal cue table and repeats the action. It

734
00:34:44,130 --> 00:34:46,809
is a bit like training a puppy. But imagine if

735
00:34:46,809 --> 00:34:48,789
the living room floor was constantly shifting,

736
00:34:48,989 --> 00:34:51,389
the rules of gravity randomly reversed every

737
00:34:51,389 --> 00:34:53,909
few weeks, and the puppy occasionally got a massive

738
00:34:53,909 --> 00:34:55,949
pile of treats for chewing up the couch due to

739
00:34:55,949 --> 00:34:58,389
sheer luck. Oh, my God. That is the reality of

740
00:34:58,389 --> 00:35:01,469
financial markets. They are chaotic, non -stationary

741
00:35:01,469 --> 00:35:03,590
environments heavily driven by random noise.

742
00:35:03,829 --> 00:35:06,650
And that's exactly where my concern lies. What

743
00:35:06,650 --> 00:35:08,690
if the reinforcement learning agent makes a completely

744
00:35:08,690 --> 00:35:11,750
random, statistically terrible trade like buying

745
00:35:11,750 --> 00:35:13,670
at the absolute peak of a speculative bubble,

746
00:35:13,769 --> 00:35:16,730
but because of sheer dumb luck, the market glitches

747
00:35:16,730 --> 00:35:19,349
higher for a single millisecond? The agent closes

748
00:35:19,349 --> 00:35:21,570
the trade, makes a massive profit, and receives

749
00:35:21,570 --> 00:35:24,409
a huge reward. It happens. Doesn't the algorithm

750
00:35:24,409 --> 00:35:27,070
learn the exact wrong lesson? Doesn't the noise

751
00:35:27,070 --> 00:35:29,769
of the market just completely destroy the reinforcement

752
00:35:29,769 --> 00:35:32,849
learning process? The author explicitly outlines

753
00:35:32,849 --> 00:35:35,860
these exact challenges, acknowledging that Reinforcement

754
00:35:35,860 --> 00:35:38,300
learning is exceedingly fragile in financial

755
00:35:38,300 --> 00:35:41,639
contexts. Exceedingly fragile? Yes. RL agents

756
00:35:41,639 --> 00:35:44,400
are notorious for overfitting to specific historical

757
00:35:44,400 --> 00:35:47,719
noise rather than generalized patterns. Furthermore,

758
00:35:47,920 --> 00:35:50,119
they struggle immensely with reward function

759
00:35:50,119 --> 00:35:53,800
design. How so? Well, if you tell an RL agent

760
00:35:53,800 --> 00:35:56,059
to simply maximize profit by the end of the year,

761
00:35:56,139 --> 00:35:59,539
it might take catastrophic leveraged risks to

762
00:35:59,539 --> 00:36:02,260
get there. It might blow up the account 99 times

763
00:36:02,260 --> 00:36:04,969
out of 100 during training just to... hit a massive

764
00:36:04,969 --> 00:36:08,329
lucky home run once. Yikes. And going back to

765
00:36:08,329 --> 00:36:10,869
the black box problem, the reasoning behind an

766
00:36:10,869 --> 00:36:13,670
RL agent's decisions is even more opaque than

767
00:36:13,670 --> 00:36:16,090
a standard deep learning model. Oh, absolutely.

768
00:36:16,449 --> 00:36:19,030
If an autonomous agent suddenly decides to aggressively

769
00:36:19,030 --> 00:36:22,610
short the entire S &P 500, you have zero idea

770
00:36:22,610 --> 00:36:25,809
if it found a genius hidden correlation in the

771
00:36:25,809 --> 00:36:28,550
macro environment or if it just learned a flawed

772
00:36:28,550 --> 00:36:31,309
lucky lesson from a random noise spike three

773
00:36:31,309 --> 00:36:33,460
years ago. This is exactly why reinforcement

774
00:36:33,460 --> 00:36:36,400
learning is not yet dominating quantitative trading

775
00:36:36,400 --> 00:36:38,719
execution the way standard supervised machine

776
00:36:38,719 --> 00:36:41,619
learning is. It is the bleeding edge, but it

777
00:36:41,619 --> 00:36:45,690
bleeds easily. So if the deep neural networks

778
00:36:45,690 --> 00:36:48,130
and the autonomous reinforcement agents all have

779
00:36:48,130 --> 00:36:50,429
these profound structural limitations when fed

780
00:36:50,429 --> 00:36:53,230
standard price data, if they all eventually struggle

781
00:36:53,230 --> 00:36:55,949
with the chaotic noise of the market, then maybe

782
00:36:55,949 --> 00:36:57,670
the ultimate edge doesn't come from building

783
00:36:57,670 --> 00:36:59,630
a more complicated algorithm. Where does it come

784
00:36:59,630 --> 00:37:01,989
from? Maybe the edge comes from feeding a slightly

785
00:37:01,989 --> 00:37:05,570
simpler algorithm entirely unique data, which

786
00:37:05,570 --> 00:37:08,369
brings us to advanced strategies and unconventional

787
00:37:08,369 --> 00:37:11,760
inputs in the book. When every hedge fund in

788
00:37:11,760 --> 00:37:14,940
the world has access to the same LSTM architectures

789
00:37:14,940 --> 00:37:18,179
and the same OHLC price data, the algorithm itself

790
00:37:18,179 --> 00:37:20,780
is no longer the competitive advantage. The data

791
00:37:20,780 --> 00:37:24,340
is. The data is the moat. Exactly. This is where

792
00:37:24,340 --> 00:37:26,360
a quantitative researcher truly earns their keep.

793
00:37:26,599 --> 00:37:29,460
The text introduces the Commitments of Traders

794
00:37:29,460 --> 00:37:33,000
Report, commonly known as the COT Report. Released

795
00:37:33,000 --> 00:37:35,719
by the CFTC, the Commodity Futures Trading Commission

796
00:37:35,719 --> 00:37:37,780
in the U .S. It's a weekly government publication

797
00:37:37,780 --> 00:37:40,559
that breaks down the aggregate open positions

798
00:37:40,559 --> 00:37:42,699
of different types of institutional traders in

799
00:37:42,699 --> 00:37:45,280
the futures market. It provides a literal map

800
00:37:45,280 --> 00:37:47,860
of where the big money is positioned. Follow

801
00:37:47,860 --> 00:37:51,179
the money. Exactly. And the text highlights two

802
00:37:51,179 --> 00:37:53,380
main groups detailed in the report. commercial

803
00:37:53,380 --> 00:37:56,320
traders and non -commercial traders. Let's define

804
00:37:56,320 --> 00:37:59,320
those groups because analyzing them kind of digitizes

805
00:37:59,320 --> 00:38:02,860
human economic behavior on a massive macro scale.

806
00:38:03,739 --> 00:38:07,500
Commercial traders are the hedgers. And the text

807
00:38:07,500 --> 00:38:10,000
notes, their open positions in the futures market

808
00:38:10,000 --> 00:38:12,599
are usually negatively correlated to the actual

809
00:38:12,599 --> 00:38:15,320
price of the asset. Think of a massive corporate

810
00:38:15,320 --> 00:38:17,940
airline. Right. Their entire business model is

811
00:38:17,940 --> 00:38:20,199
heavily exposed to the fluctuating price of jet

812
00:38:20,199 --> 00:38:23,239
fuel. Right. They use the energy futures market

813
00:38:23,239 --> 00:38:25,460
to lock in prices and protect their profit margins.

814
00:38:25,760 --> 00:38:27,960
They are not speculating to make a quick trade.

815
00:38:28,159 --> 00:38:30,800
They are hedging their operational risk. So when

816
00:38:30,800 --> 00:38:33,699
the price of oil skyrockets, what do they do?

817
00:38:33,840 --> 00:38:36,139
The airline aggressively sells futures contracts

818
00:38:36,139 --> 00:38:38,199
to lock in current rates before it goes even

819
00:38:38,199 --> 00:38:40,780
higher. They're selling increases as the price

820
00:38:40,780 --> 00:38:44,449
increases, which is a negative correlation. And

821
00:38:44,449 --> 00:38:46,070
on the other side, the non -commercial traders

822
00:38:46,070 --> 00:38:49,050
are the hedge funds, the massive speculators,

823
00:38:49,130 --> 00:38:51,789
the trend followers. Their positions are generally

824
00:38:51,789 --> 00:38:54,210
positively correlated. Right. They are the ones

825
00:38:54,210 --> 00:38:56,090
trying to ride the wave. If oil is going up,

826
00:38:56,150 --> 00:38:58,190
they are buying oil futures to profit from the

827
00:38:58,190 --> 00:39:00,730
momentum. So the book outlines three specific

828
00:39:00,730 --> 00:39:03,530
algorithmic architectures to process this weekly

829
00:39:03,530 --> 00:39:06,889
COT data and extract actionable forecasts. There

830
00:39:06,889 --> 00:39:10,210
is the indirect one -step, the MPF direct, and

831
00:39:10,210 --> 00:39:13,420
the MPF recursive models. Yes. How do those models

832
00:39:13,420 --> 00:39:15,659
actually function mechanically? Like, what's

833
00:39:15,659 --> 00:39:17,860
the difference? Well, the indirect one -step

834
00:39:17,860 --> 00:39:20,420
model involves predicting the future COT data

835
00:39:20,420 --> 00:39:23,659
first, so guessing what the hedgers and speculators

836
00:39:23,659 --> 00:39:26,500
will do next week, and then using that internal

837
00:39:26,500 --> 00:39:29,079
prediction to forecast the actual market price.

838
00:39:29,500 --> 00:39:32,239
Okay, kind of a two -step dance. Exactly. Now,

839
00:39:32,300 --> 00:39:34,900
the MPF Direct, which stands for multi -period

840
00:39:34,900 --> 00:39:38,139
forecasting, bypasses that middle step. It attempts

841
00:39:38,139 --> 00:39:40,539
to predict the market price several steps into

842
00:39:40,539 --> 00:39:43,119
the future directly from the current week's COT

843
00:39:43,119 --> 00:39:46,079
data. Straight to the point. Right. And the recursive

844
00:39:46,079 --> 00:39:48,800
one? The MPF recursive model predicts the price

845
00:39:48,800 --> 00:39:51,420
one step ahead and then feeds its own prediction

846
00:39:51,420 --> 00:39:54,380
back into itself as a new input to predict the

847
00:39:54,380 --> 00:39:57,260
second step, just cascading forward. That's super

848
00:39:57,260 --> 00:40:00,500
clever. The author also covers predicting Bitcoin

849
00:40:00,500 --> 00:40:03,820
volatility using deep learning, utilizing a rolling

850
00:40:03,820 --> 00:40:06,659
10 -day standard deviation as a proxy target

851
00:40:06,659 --> 00:40:09,699
for volatility. Crypto is perfect for these models.

852
00:40:10,059 --> 00:40:12,880
Yeah. And crucially, the book provides the Python

853
00:40:12,880 --> 00:40:15,920
code to generate real -time visual outputs of

854
00:40:15,920 --> 00:40:18,980
the LSTM model's internal architecture to monitor

855
00:40:18,980 --> 00:40:21,940
the training process. Generating visual maps

856
00:40:21,940 --> 00:40:24,119
of the architecture is a vital practice for avoiding

857
00:40:24,119 --> 00:40:27,719
that mystery box anxiety we talked about. You

858
00:40:27,719 --> 00:40:29,800
want to see what it's doing. It allows the researcher

859
00:40:29,800 --> 00:40:32,699
to actually see the layers, the node connections,

860
00:40:32,800 --> 00:40:34,960
and the shape of the data tensors as they flow

861
00:40:34,960 --> 00:40:37,300
through the network during training, ensuring

862
00:40:37,300 --> 00:40:39,579
the model is structured correctly before it ever

863
00:40:39,579 --> 00:40:42,739
touches real capital. But the COT data is the

864
00:40:42,739 --> 00:40:44,960
real strategic pivot here. And this is where

865
00:40:44,960 --> 00:40:47,800
the mechanics confuse me a little bit. The COT

866
00:40:47,800 --> 00:40:50,820
report is a slow, bureaucratic release. It comes

867
00:40:50,820 --> 00:40:53,900
out once a week on Fridays. And the data is already

868
00:40:53,900 --> 00:40:55,980
a few days old by the time it's even published.

869
00:40:56,099 --> 00:40:59,159
It's lagging data. Yeah. But modern markets are

870
00:40:59,159 --> 00:41:02,199
trading millions of times a second. How on earth

871
00:41:02,199 --> 00:41:05,519
does an advanced, hyperfast, deep learning algorithm

872
00:41:05,519 --> 00:41:10,239
use slow, weakly lagging government data to trade

873
00:41:10,239 --> 00:41:12,699
in a high frequency environment? Isn't there

874
00:41:12,699 --> 00:41:15,809
a massive temporal mismatch there? What's brilliant

875
00:41:15,809 --> 00:41:18,469
about this approach is how you combine entire

876
00:41:18,469 --> 00:41:20,670
different time frames to create a structural

877
00:41:20,670 --> 00:41:23,489
edge. You are absolutely correct. You cannot

878
00:41:23,489 --> 00:41:25,750
use weekly data for high frequency execution.

879
00:41:25,949 --> 00:41:29,829
But the text explains that the COT data analysis

880
00:41:29,829 --> 00:41:34,289
acts as a macro level guide, an overarching fundamental

881
00:41:34,289 --> 00:41:38,360
bias. Oh, so it's like using a climate model

882
00:41:38,360 --> 00:41:41,239
to determine the general season rather than a

883
00:41:41,239 --> 00:41:43,820
radar to predict today's rainstorm. That is an.

884
00:41:44,000 --> 00:41:46,719
Excellent way to frame it. You run your deep

885
00:41:46,719 --> 00:41:49,500
learning models on the slow weekly COT data.

886
00:41:49,960 --> 00:41:52,840
Let's say your LSTM model processes the historical

887
00:41:52,840 --> 00:41:55,159
patterns and predicts that over the next two

888
00:41:55,159 --> 00:41:57,340
weeks, commercial hedgers are going to massively

889
00:41:57,340 --> 00:41:59,360
increase their short positions on the Canadian

890
00:41:59,360 --> 00:42:02,239
dollar. Okay. The algorithm sees a fundamental

891
00:42:02,239 --> 00:42:05,320
institutional shift coming. You don't execute

892
00:42:05,320 --> 00:42:08,340
a high -frequency trade based just on that weekly

893
00:42:08,340 --> 00:42:11,800
signal. Instead, you use that slow data to filter

894
00:42:11,800 --> 00:42:14,780
the execution of your fast algorithms. So the

895
00:42:14,780 --> 00:42:17,840
slow AI gives orders to the fast AI. Precisely.

896
00:42:18,269 --> 00:42:20,869
You program a rule for your fast five -minute

897
00:42:20,869 --> 00:42:23,550
chart algorithm. For the next two weeks, because

898
00:42:23,550 --> 00:42:26,550
the macro COT model anticipates heavy institutional

899
00:42:26,550 --> 00:42:29,769
selling, you are only allowed to execute on short

900
00:42:29,769 --> 00:42:32,849
signals for the Canadian dollar. Ignore all technical

901
00:42:32,849 --> 00:42:35,389
bisignals, no matter how good the RSI looks.

902
00:42:35,590 --> 00:42:38,710
Wow. You are utilizing the deep learning analysis

903
00:42:38,710 --> 00:42:42,210
of slow, unconventional data to provide a fundamental

904
00:42:42,210 --> 00:42:45,829
tailwind for your fast technical execution. It

905
00:42:45,829 --> 00:42:47,550
keeps your high -frequency bots from fighting

906
00:42:47,550 --> 00:42:51,210
the overarching institutional tide. That is incredibly

907
00:42:51,210 --> 00:42:54,730
elegant. You're marrying fundamental macro data

908
00:42:54,730 --> 00:42:59,239
with quantitative micro execution. which actually

909
00:42:59,239 --> 00:43:00,900
sets us up beautifully for the conclusion of

910
00:43:00,900 --> 00:43:03,380
this journey. It all comes together. It really

911
00:43:03,380 --> 00:43:05,139
does, because ultimately, no matter how deep

912
00:43:05,139 --> 00:43:08,860
the machine learning is, no matter how many LSTM

913
00:43:08,860 --> 00:43:11,760
hidden layers you build, and no matter how cleverly

914
00:43:11,760 --> 00:43:14,619
you filter your timeframe biases, these algorithms

915
00:43:14,619 --> 00:43:17,539
operate in a reality governed by human emotion

916
00:43:17,539 --> 00:43:21,639
and real -world economics. We are looking at

917
00:43:21,639 --> 00:43:24,559
the final chapter of the text, Market Drivers

918
00:43:24,559 --> 00:43:28,070
and Risk Management. Cobber makes a vital editorial

919
00:43:28,070 --> 00:43:30,349
choice to end the book by grounding all this

920
00:43:30,349 --> 00:43:33,909
high level mathematics in the messy reality of

921
00:43:33,909 --> 00:43:37,190
the trading floor. A human element. Yes. He outlines

922
00:43:37,190 --> 00:43:39,610
core market drivers that algorithms struggle

923
00:43:39,610 --> 00:43:43,710
to anticipate. Economic intuition and the interpretation

924
00:43:43,710 --> 00:43:47,710
of unprecedented news. When the FOMC, the Federal

925
00:43:47,710 --> 00:43:50,190
Open Market Committee of the U .S. Central Bank,

926
00:43:50,289 --> 00:43:53,130
makes an unexpected, massive shift in interest

927
00:43:53,130 --> 00:43:56,780
rate policy. it instantly reprices every algorithmic

928
00:43:56,780 --> 00:43:59,400
model across stocks, bonds, commodities, and

929
00:43:59,400 --> 00:44:02,440
currencies globally. A human being steps to a

930
00:44:02,440 --> 00:44:05,940
podium in Washington, D .C., says a few cautiously

931
00:44:05,940 --> 00:44:09,039
worded sentences about inflation, and a million

932
00:44:09,039 --> 00:44:11,920
lines of optimized Python code suddenly have

933
00:44:11,920 --> 00:44:14,860
to recalibrate instantly. And they often fail

934
00:44:14,860 --> 00:44:16,860
in the process. Because the underlying reality,

935
00:44:17,139 --> 00:44:19,340
the cost of capital that drives the entire mathematical

936
00:44:19,340 --> 00:44:22,710
system, just fundamentally shifted. Right. The

937
00:44:22,710 --> 00:44:25,510
text also delves into behavioral finance. It

938
00:44:25,510 --> 00:44:27,809
highlights the enduring power of cognitive biases.

939
00:44:28,170 --> 00:44:31,210
It discusses anchoring, where traders irrationally

940
00:44:31,210 --> 00:44:34,070
fixate on a past price. We all do that. We do.

941
00:44:34,170 --> 00:44:36,630
And confirmation bias, where they actively seek

942
00:44:36,630 --> 00:44:38,929
out only the information that validates their

943
00:44:38,929 --> 00:44:42,090
existing failing trades. And overconfidence.

944
00:44:42,289 --> 00:44:45,190
These human biases dictate the market anomalies

945
00:44:45,190 --> 00:44:47,530
that algorithms attempt to exploit, but they

946
00:44:47,530 --> 00:44:50,190
also create the unpredictable panics that destroy

947
00:44:50,190 --> 00:44:53,269
algorithms. And finally, the text hammers home

948
00:44:53,269 --> 00:44:56,789
risk management, the absolute non -negotiable

949
00:44:56,789 --> 00:44:59,530
necessity of setting hard stop losses, defining

950
00:44:59,530 --> 00:45:02,530
profit targets, utilizing trailing stops, and

951
00:45:02,530 --> 00:45:05,349
prioritizing capital preservation above all else.

952
00:45:05,730 --> 00:45:09,559
And honestly, looking at the arc of this entire

953
00:45:09,559 --> 00:45:13,039
deep dive, I see a profound irony here. How so?

954
00:45:13,360 --> 00:45:15,960
Well... We just spent this entire time and the

955
00:45:15,960 --> 00:45:18,780
author spent an entire meticulous book building

956
00:45:18,780 --> 00:45:21,500
mathematically flawless, emotionless, highly

957
00:45:21,500 --> 00:45:24,340
sophisticated, deep learning robots in Python.

958
00:45:24,519 --> 00:45:27,559
Right. We spent hours discussing stationary data,

959
00:45:27,840 --> 00:45:30,099
multilayer perceptrons, fractional differentiation

960
00:45:30,099 --> 00:45:33,880
and autonomous reinforcement agents. We built

961
00:45:33,880 --> 00:45:36,559
the ultimate quantitative machine. The perfect

962
00:45:36,559 --> 00:45:39,139
robot. Yeah. Only to end the book talking about

963
00:45:39,139 --> 00:45:42,210
human panic. central bank press conferences and

964
00:45:42,210 --> 00:45:44,670
the absolute necessity of manually setting a

965
00:45:44,670 --> 00:45:47,949
hard stop loss just in case the ai gets it completely

966
00:45:47,949 --> 00:45:51,250
wrong it feels like we spent a decade building

967
00:45:51,250 --> 00:45:54,230
an indestructible autonomous spaceship but we

968
00:45:54,230 --> 00:45:56,429
are still relying on a guy with a canvas parachute

969
00:45:56,429 --> 00:45:58,969
just in case the engine explodes the irony you're

970
00:45:58,969 --> 00:46:01,110
observing is the exact synthesis of quantitative

971
00:46:01,110 --> 00:46:04,369
finance the algorithms the python libraries the

972
00:46:04,369 --> 00:46:06,690
neural network architectures they are perfectly

973
00:46:06,690 --> 00:46:09,579
objective mathematically sound tools yeah But

974
00:46:09,579 --> 00:46:11,519
what they are attempting to measure, forecast

975
00:46:11,519 --> 00:46:14,639
and navigate is fundamentally subjective. They

976
00:46:14,639 --> 00:46:17,599
are trying to map human behavior, geopolitical

977
00:46:17,599 --> 00:46:20,199
chaos and economic fear. It is math attempting

978
00:46:20,199 --> 00:46:23,340
to predict madness. The markets are made of people

979
00:46:23,340 --> 00:46:25,699
and even the algorithms are written by people

980
00:46:25,699 --> 00:46:28,199
carrying the structural biases of their creators.

981
00:46:28,690 --> 00:46:31,070
Without strict, non -negotiable risk management,

982
00:46:31,309 --> 00:46:32,909
without an understanding of the macro drivers

983
00:46:32,909 --> 00:46:35,710
like the FOMC, and without respect for behavioral

984
00:46:35,710 --> 00:46:38,150
biases, even the most beautifully optimized,

985
00:46:38,489 --> 00:46:40,989
multi -layer LSTM neural network will eventually

986
00:46:40,989 --> 00:46:43,650
encounter a black swan human panic it hasn't

987
00:46:43,650 --> 00:46:45,949
seen in its training data. And it will blow up

988
00:46:45,949 --> 00:46:48,570
the account. It will blow up the account. The

989
00:46:48,570 --> 00:46:50,750
sophisticated math only works long term if it

990
00:46:50,750 --> 00:46:53,590
is protected by simple risk management. That

991
00:46:53,590 --> 00:46:55,909
brings everything full circle. Let's recap this

992
00:46:55,909 --> 00:46:58,610
incredible journey. We started with the absolute

993
00:46:58,610 --> 00:47:01,550
raw materials, downloading historical VIX data,

994
00:47:01,710 --> 00:47:04,329
and philosophically debating the mechanics of

995
00:47:04,329 --> 00:47:06,889
altering time itself by dropping missing data

996
00:47:06,889 --> 00:47:09,730
rows or filling them with averages. The pre -processing

997
00:47:09,730 --> 00:47:12,070
phase. Yeah. And then we explored how to force

998
00:47:12,070 --> 00:47:14,530
drifting market data into stationarity using

999
00:47:14,530 --> 00:47:17,530
differencing and the Dickey -Fuller test. We

1000
00:47:17,530 --> 00:47:20,170
learned how to translate subjective human charting

1001
00:47:20,170 --> 00:47:22,949
psychology into objective statistical indicators

1002
00:47:22,949 --> 00:47:25,329
like the RSI. Setting up the features. Right.

1003
00:47:25,449 --> 00:47:28,349
And then we built basic dummy machine learning

1004
00:47:28,349 --> 00:47:31,469
models, learned why a 49 % coin flip accuracy

1005
00:47:31,469 --> 00:47:33,969
on unseen data is essential to avoid the trap

1006
00:47:33,969 --> 00:47:36,869
of overfitting, and then plunged into the complex

1007
00:47:36,869 --> 00:47:39,090
hidden layers and memory gates of deep learning

1008
00:47:39,090 --> 00:47:42,550
LSTMs. The architecture. We trained a reinforcement

1009
00:47:42,550 --> 00:47:44,889
learning agent that occasionally gets lucky for

1010
00:47:44,889 --> 00:47:48,289
the wrong reasons. We used slow macro COT data

1011
00:47:48,289 --> 00:47:51,030
to filter fast micro markets. And we ended by

1012
00:47:51,030 --> 00:47:53,090
realizing that even the most advanced AI needs

1013
00:47:53,090 --> 00:47:56,489
a stop loss because human panic remains undefeated.

1014
00:47:56,570 --> 00:47:59,070
It really is a remarkable synthesis of computer

1015
00:47:59,070 --> 00:48:01,710
science, statistical probability, and behavioral

1016
00:48:01,710 --> 00:48:04,929
psychology, rigorously documented in code. And

1017
00:48:04,929 --> 00:48:07,030
for you listening... Whether you are an aspiring

1018
00:48:07,030 --> 00:48:10,230
quantitative researcher, a casual investor managing

1019
00:48:10,230 --> 00:48:13,329
your own portfolio, or just somewhat fascinated

1020
00:48:13,329 --> 00:48:16,090
by how artificial intelligence is mapping human

1021
00:48:16,090 --> 00:48:19,030
behavior, understanding how these models ingest

1022
00:48:19,030 --> 00:48:21,769
reality changes how you view the modern financial

1023
00:48:21,769 --> 00:48:24,489
world. It really does. The next time you look

1024
00:48:24,489 --> 00:48:27,289
at a stock chart, you aren't just seeing a squiggly

1025
00:48:27,289 --> 00:48:30,030
line of prices. You are seeing the exact data

1026
00:48:30,030 --> 00:48:32,639
points. that thousands of hidden algorithmic

1027
00:48:32,639 --> 00:48:35,440
layers are currently digesting and fighting over

1028
00:48:35,440 --> 00:48:38,599
in milliseconds. The market becomes infinitely

1029
00:48:38,599 --> 00:48:40,780
more fascinating when you can see the architecture

1030
00:48:40,780 --> 00:48:43,579
operating beneath the surface. But before we

1031
00:48:43,579 --> 00:48:45,800
go, I want to leave you with a final provocative

1032
00:48:45,800 --> 00:48:47,800
thought to mull over, something that builds on

1033
00:48:47,800 --> 00:48:49,760
everything we've discussed today, but pushes

1034
00:48:49,760 --> 00:48:52,019
it one step further. Oh, this should be good.

1035
00:48:52,420 --> 00:48:54,639
We talked extensively about how these deep learning

1036
00:48:54,639 --> 00:48:57,239
models learn from historical data and human biases

1037
00:48:57,239 --> 00:49:00,230
to predict what the market will do next. But

1038
00:49:00,230 --> 00:49:04,210
if every major quantitative hedge fund and institutional

1039
00:49:04,210 --> 00:49:06,809
bank in the world is eventually running very

1040
00:49:06,809 --> 00:49:09,010
similar deep learning and reinforcement models.

1041
00:49:09,230 --> 00:49:11,570
Which they are. And they are all training on

1042
00:49:11,570 --> 00:49:14,929
the exact same historical data sets, ingesting

1043
00:49:14,929 --> 00:49:17,289
the exact same stationary features, and learning

1044
00:49:17,289 --> 00:49:21,190
the exact same human behavioral biases. Are these

1045
00:49:21,190 --> 00:49:24,010
algorithms actually predicting the market anymore?

1046
00:49:24,329 --> 00:49:27,250
That's a great question. Or by acting in unison

1047
00:49:27,250 --> 00:49:30,320
at light speed? Are they mathematically creating

1048
00:49:30,320 --> 00:49:33,019
the market's future movements as one massive

1049
00:49:33,019 --> 00:49:36,050
automated self -fulfilling prophecy? Have we

1050
00:49:36,050 --> 00:49:38,909
used advanced math to clear the murky waters

1051
00:49:38,909 --> 00:49:40,909
of human finance only to realize that the machine

1052
00:49:40,909 --> 00:49:43,829
itself is now deciding where the waves will crash?

1053
00:49:44,530 --> 00:49:46,949
Fascinating to think about. Thanks for joining

1054
00:49:46,949 --> 00:49:49,369
us on this deep dive. Keep questioning the data.