1 00:00:00,000 --> 00:00:02,319 You know, usually when we talk about a diagnostic 2 00:00:02,319 --> 00:00:05,360 reading or getting a sensor measurement in engineering, 3 00:00:05,419 --> 00:00:08,619 there's this really profound expectation of absolute 4 00:00:08,619 --> 00:00:11,519 precision. Oh, yeah. You expect a completely 5 00:00:11,519 --> 00:00:13,640 clear -cut answer. Right. Like, you break your 6 00:00:13,640 --> 00:00:16,260 arm, the x -ray shows that jagged white line, 7 00:00:16,280 --> 00:00:18,559 and the doctor just points at the film and says, 8 00:00:18,600 --> 00:00:20,739 you know, there it is. It's a binary reality, 9 00:00:20,859 --> 00:00:23,579 broken or not broken. And it's comforting. I 10 00:00:23,579 --> 00:00:26,320 mean, we really like things to be perfectly categorized 11 00:00:26,320 --> 00:00:29,179 without any ambiguity at all. Exactly. It's clean. 12 00:00:29,640 --> 00:00:32,200 But the moment you step into the world of robotics, 13 00:00:32,960 --> 00:00:35,939 and specifically computer vision, that pristine 14 00:00:35,939 --> 00:00:38,939 X -ray machine just totally shatters. Suddenly 15 00:00:38,939 --> 00:00:41,359 you're looking at this sensory landscape that 16 00:00:41,359 --> 00:00:45,219 is incredibly murky. It is. It's filled with 17 00:00:45,219 --> 00:00:48,719 optical illusions and shifting lighting and these 18 00:00:48,719 --> 00:00:51,250 really profound geometric assumptions. Yeah, 19 00:00:51,310 --> 00:00:53,130 which brings us to today's deep dive. Welcome 20 00:00:53,130 --> 00:00:55,689 back, everyone. This is the continuation of our 21 00:00:55,689 --> 00:00:58,210 Marginalia series, exploring Peter Corke's textbook, 22 00:00:58,409 --> 00:01:00,649 Robotics, Vision, and Control. And for those 23 00:01:00,649 --> 00:01:02,409 of you who have been traversing this material 24 00:01:02,409 --> 00:01:04,989 with us, you know we've spent parts one through 25 00:01:04,989 --> 00:01:07,230 three building the physical foundations of a 26 00:01:07,230 --> 00:01:09,430 robot. Right, we've mapped out mobile robots, 27 00:01:09,430 --> 00:01:11,569 you know, rolling through physical spaces using 28 00:01:11,569 --> 00:01:14,329 wheel encoders. We've done all that heavy mathematical 29 00:01:14,329 --> 00:01:17,489 lifting for complex manipulator arms, calculating 30 00:01:17,489 --> 00:01:20,469 the joint kinematics and the rotation matrices 31 00:01:20,469 --> 00:01:24,230 required to reach out and weld a car chassis 32 00:01:24,230 --> 00:01:27,650 or grab a coffee cup. We even got into SLAM, 33 00:01:27,769 --> 00:01:29,849 you know, simultaneous localization and mapping, 34 00:01:30,189 --> 00:01:33,230 but there is this massive glaring limitation 35 00:01:33,230 --> 00:01:36,189 to everything we've built so far. Yes. Moving 36 00:01:36,189 --> 00:01:39,379 blindly. Exactly. If you're relying strictly 37 00:01:39,379 --> 00:01:41,540 on internal joint sensors and wheel encoders, 38 00:01:41,659 --> 00:01:44,079 it leaves the robot fundamentally isolated from 39 00:01:44,079 --> 00:01:46,439 the realities of a totally unpredictable environment. 40 00:01:46,680 --> 00:01:48,700 Right, because if a coffee cup is moved, say, 41 00:01:48,900 --> 00:01:52,060 two inches to the left, a blind manipulator arm 42 00:01:52,060 --> 00:01:55,280 will just confidently grasp empty air. It has 43 00:01:55,280 --> 00:01:58,359 no idea the world changed. So for a robot to 44 00:01:58,359 --> 00:02:00,680 dynamically interact with the world, it actually 45 00:02:00,680 --> 00:02:03,409 has to see it. Which brings us to our focus today, 46 00:02:03,689 --> 00:02:07,430 part four of five in Quark's text, focusing entirely 47 00:02:07,430 --> 00:02:10,509 on computer vision. And our mission today is 48 00:02:10,509 --> 00:02:13,469 to follow that entire journey of sight. We are 49 00:02:13,469 --> 00:02:15,849 going to trace the literal physics of a photon 50 00:02:15,849 --> 00:02:18,449 bouncing off a brick, follow it into a digital 51 00:02:18,449 --> 00:02:22,370 sensor, figure out how to extract actual actionable 52 00:02:22,370 --> 00:02:25,090 meaning from a chaotic sea of millions of pixels. 53 00:02:25,590 --> 00:02:28,770 Which is no small feat. And then finally, reconstruct 54 00:02:28,770 --> 00:02:31,629 a full three -dimensional world from those flattened 55 00:02:31,629 --> 00:02:33,789 two -dimensional images. You know, if there's 56 00:02:33,789 --> 00:02:36,069 one overarching thesis to keep in mind as we 57 00:02:36,069 --> 00:02:38,509 navigate this, it's that while computer vision 58 00:02:38,509 --> 00:02:41,550 feels like it should be magic. Or at least an 59 00:02:41,550 --> 00:02:43,849 effortless replica of our own human sight. Right. 60 00:02:43,949 --> 00:02:47,009 It is actually this incredibly rigorous, deeply 61 00:02:47,009 --> 00:02:50,229 interpretive, and sometimes very fragile algorithmic 62 00:02:50,229 --> 00:02:53,889 process. A digital camera is not an eye. And 63 00:02:53,889 --> 00:02:56,969 an image array is definitely not a scene. So 64 00:02:56,969 --> 00:02:59,210 understanding exactly where those mathematical 65 00:02:59,210 --> 00:03:02,210 interpretations break down, like where the algorithms 66 00:03:02,210 --> 00:03:05,110 get tricked by physical reality, that is the 67 00:03:05,110 --> 00:03:08,430 true secret to modern robotics. Absolutely. So 68 00:03:08,430 --> 00:03:11,389 let's start at the very beginning with the physical 69 00:03:11,389 --> 00:03:15,770 medium itself, light. Chapter one, Cork mentions 70 00:03:15,770 --> 00:03:19,150 this fascinating historical footnote that in 71 00:03:19,150 --> 00:03:21,430 ancient times people actually believed in the 72 00:03:21,430 --> 00:03:23,830 extra mission theory of vision. Oh, right, the 73 00:03:23,830 --> 00:03:26,509 visual beams. Yeah. It was this idea that our 74 00:03:26,509 --> 00:03:28,849 eyes were essentially shooting out these visual 75 00:03:28,849 --> 00:03:31,310 beams, almost like invisible lasers that would 76 00:03:31,310 --> 00:03:34,090 actively scan and interact with objects in the 77 00:03:34,090 --> 00:03:36,569 world so we could perceive them. Right. And of 78 00:03:36,569 --> 00:03:38,990 course, this sounds absurd today, knowing what 79 00:03:38,990 --> 00:03:41,550 we know about how light actually works. Definitely. 80 00:03:42,129 --> 00:03:44,030 But it does make a certain kind of intuitive 81 00:03:44,030 --> 00:03:46,689 sense, right? I mean, you open your eyes and 82 00:03:46,689 --> 00:03:50,030 vision feels so instantaneous. so active, like 83 00:03:50,030 --> 00:03:52,330 you're reaching out with your sight to grab the 84 00:03:52,330 --> 00:03:55,050 world, instead of waiting for these kind of passive 85 00:03:55,050 --> 00:03:57,629 streams of photons to come and land under your 86 00:03:57,629 --> 00:04:00,569 eyes. I mean, it sounds totally absurd to a modern 87 00:04:00,569 --> 00:04:03,370 scientific mind. But intuitively, it speaks to 88 00:04:03,370 --> 00:04:05,710 how active and directed the psychological sensation 89 00:04:05,710 --> 00:04:08,650 of vision really feels to us. It does. When you 90 00:04:08,650 --> 00:04:11,110 focus intensely on an object, it genuinely feels 91 00:04:11,110 --> 00:04:13,129 like you're projecting your attention onto it. 92 00:04:13,310 --> 00:04:15,629 Right. But of course, the physics of optics shifted 93 00:04:15,629 --> 00:04:17,610 our understanding to the intramission theory. 94 00:04:18,009 --> 00:04:21,399 Right. The reality that light originates from 95 00:04:21,399 --> 00:04:24,800 an illuminant bounces off a scene. And what we 96 00:04:24,800 --> 00:04:27,060 perceive is merely the tiny fraction of that 97 00:04:27,060 --> 00:04:29,720 scattered light that just happens to pass through 98 00:04:29,720 --> 00:04:32,660 our pupil or a camera lens. And to truly understand 99 00:04:32,660 --> 00:04:35,360 what a robot is looking at, we have to understand 100 00:04:35,360 --> 00:04:37,839 the nature of that bouncing light. If we go back 101 00:04:37,839 --> 00:04:42,779 to 1665, the plague years, Isaac Newton is quarantined 102 00:04:42,779 --> 00:04:45,089 at home. right? Playing with glass prisms. As 103 00:04:45,089 --> 00:04:47,870 one does. Yeah, as one does. And he empirically 104 00:04:47,870 --> 00:04:51,189 demonstrates that white light isn't a pure fundamental 105 00:04:51,189 --> 00:04:53,709 element, but rather a mixture of different colors. 106 00:04:54,189 --> 00:04:56,750 He proved that color is an intrinsic property 107 00:04:56,750 --> 00:04:59,689 of the light itself. Right. Not a property of 108 00:04:59,689 --> 00:05:02,949 the object it hits. And that is a crucial distinction 109 00:05:02,949 --> 00:05:05,069 that honestly plagues machine vision to this 110 00:05:05,069 --> 00:05:08,170 day. How so? Well, what Newton discovered and 111 00:05:08,170 --> 00:05:10,370 what James Clerk Maxwell later formalized through 112 00:05:10,370 --> 00:05:13,639 electromagnetism is that visible light is just 113 00:05:13,639 --> 00:05:16,600 a remarkably narrow band of electromagnetic radiation. 114 00:05:17,019 --> 00:05:20,199 The spectrum. Exactly. We are talking about wavelengths 115 00:05:20,199 --> 00:05:23,100 specifically between 400 and 700 nanometers. 116 00:05:23,740 --> 00:05:26,740 If you trace the spectrum, 400 nanometers sits 117 00:05:26,740 --> 00:05:29,019 down in the high -energy violet and blue range. 118 00:05:29,139 --> 00:05:31,899 Okay. And as the physical wavelength stretches 119 00:05:31,899 --> 00:05:34,560 out, you move through the greens and yellows, 120 00:05:34,740 --> 00:05:37,000 finally hitting 700 nanometers at the red end. 121 00:05:37,459 --> 00:05:40,079 And anything longer than 700 is infrared, right? 122 00:05:40,250 --> 00:05:43,370 which our skin feels as heat, but our eyes can't 123 00:05:43,370 --> 00:05:45,850 actually see. Right. And anything shorter than 124 00:05:45,850 --> 00:05:48,769 400 is ultraviolet. So the primary source of 125 00:05:48,769 --> 00:05:51,310 this light in the natural world and in, you know, 126 00:05:51,490 --> 00:05:54,129 early artificial lighting is incandescence, basically 127 00:05:54,129 --> 00:05:56,269 something getting so physically hot that it begins 128 00:05:56,269 --> 00:05:58,829 to glow. The strict physics term for this is 129 00:05:58,829 --> 00:06:02,449 a black body radiator or a plankian source. And 130 00:06:02,449 --> 00:06:04,629 the mathematical models of black body radiation 131 00:06:04,629 --> 00:06:07,209 actually dictate the entire evolutionary history 132 00:06:07,209 --> 00:06:10,420 of human sight. They absolutely do. Max Planck's 133 00:06:10,420 --> 00:06:12,899 radiation formula is this cornerstone of quantum 134 00:06:12,899 --> 00:06:16,040 mechanics, describing exactly how much electromagnetic 135 00:06:16,040 --> 00:06:19,300 power a hot object emits across different wavelengths. 136 00:06:19,879 --> 00:06:22,199 It's a really dense equation. It integrates the 137 00:06:22,199 --> 00:06:24,680 object's absolute temperature, Planck's constant, 138 00:06:25,220 --> 00:06:26,860 Boltzmann's constant, and the speed of light. 139 00:06:27,279 --> 00:06:29,500 But for our purposes in robotics, we really just 140 00:06:29,500 --> 00:06:31,420 need the conceptual takeaway, which is governed 141 00:06:31,420 --> 00:06:35,259 by two derived laws, right? Yes. the Stefan -Boltzmann 142 00:06:35,259 --> 00:06:37,600 law, which says hotter objects emit exponentially 143 00:06:37,600 --> 00:06:40,779 more total energy, and Wien's displacement law. 144 00:06:41,319 --> 00:06:43,660 And Wien's displacement law is the one that really 145 00:06:43,660 --> 00:06:46,279 dictates color. It states there's a strict inverse 146 00:06:46,279 --> 00:06:48,680 relationship between the temperature of an object 147 00:06:48,680 --> 00:06:51,699 and the peak wavelength of the light it emits. 148 00:06:51,959 --> 00:06:54,639 So the hotter the object gets, the shorter the 149 00:06:54,639 --> 00:06:57,769 wavelength of its peak emission. Exactly. Let's 150 00:06:57,769 --> 00:07:00,290 apply Wien's law to two familiar light sources 151 00:07:00,290 --> 00:07:02,509 to see why this actually matters for robots. 152 00:07:03,129 --> 00:07:05,870 The surface of our Sun is an incredibly hot plasma 153 00:07:05,870 --> 00:07:10,269 burning at roughly 5,778 Kelvin. Right. And 154 00:07:10,269 --> 00:07:13,730 if you plug 5,778 Kelvin into Planck's formula 155 00:07:13,730 --> 00:07:16,709 and calculate the derivative to find the peak, 156 00:07:17,310 --> 00:07:20,610 that emission spectrum lands right around 500 157 00:07:20,610 --> 00:07:23,949 nanometers. Which is completely non -coincidental. 158 00:07:24,170 --> 00:07:27,060 Not at all. 500 nanometers is smack in the middle 159 00:07:27,060 --> 00:07:29,740 of the visible light spectrum, right around the 160 00:07:29,740 --> 00:07:32,360 blue -green transition. Human eyes evolve specifically 161 00:07:32,360 --> 00:07:35,240 to be sensitive to the exact wavelengths of electromagnetic 162 00:07:35,240 --> 00:07:38,120 radiation that our local star happens to pump 163 00:07:38,120 --> 00:07:40,879 out in the highest abundance. It's amazing. But 164 00:07:40,879 --> 00:07:43,680 if you compare the sun to an old -school artificial 165 00:07:43,680 --> 00:07:45,980 light source like a traditional tungsten filament 166 00:07:45,980 --> 00:07:48,620 light bulb, the math changes drastically. It 167 00:07:48,620 --> 00:07:51,180 really does. Because a tungsten filament is hot, 168 00:07:51,339 --> 00:07:53,860 sure, but compared to a star, it's quite cool. 169 00:07:54,379 --> 00:07:57,379 It burns at about 2,600 Kelvin. So because it's 170 00:07:57,379 --> 00:07:59,839 a lower temperature, Wien's law dictates that 171 00:07:59,839 --> 00:08:02,600 its peak emission has to shift to a longer wavelength. 172 00:08:03,040 --> 00:08:06,500 Yes. And indeed, a 2,600 Kelvin tungsten lamp 173 00:08:06,500 --> 00:08:09,800 peaks at around 1,100 nanometers. But wait, 174 00:08:10,040 --> 00:08:13,079 1,100 nanometers is way past the 700 nanometer 175 00:08:13,079 --> 00:08:16,300 cutoff for visible red light. Exactly. It's deep 176 00:08:16,300 --> 00:08:19,459 into the infrared spectrum. Which means the vast 177 00:08:19,459 --> 00:08:21,939 majority of the energy emitted by a traditional 178 00:08:21,939 --> 00:08:24,360 incandescent light bulb isn't visible light at 179 00:08:24,360 --> 00:08:27,949 all. It is literally invisible heat. This is 180 00:08:27,949 --> 00:08:31,209 why those bulbs are so terribly inefficient and 181 00:08:31,209 --> 00:08:33,090 why they've largely been phased out in favor 182 00:08:33,090 --> 00:08:36,330 of LEDs. With incandescent bulbs, you're pouring 183 00:08:36,330 --> 00:08:38,190 massive amounts of electricity into generating 184 00:08:38,190 --> 00:08:41,210 this huge peak of infrared radiation, all just 185 00:08:41,210 --> 00:08:44,049 to harvest the tiny sloping tail of the turf 186 00:08:44,049 --> 00:08:46,889 that just happens to bleed into the 400 to 700 187 00:08:46,889 --> 00:08:50,179 nanometer range. That implies we were basically 188 00:08:50,179 --> 00:08:52,080 lighting our homes with space feeders for over 189 00:08:52,080 --> 00:08:54,399 a century, just scraping off the visible light 190 00:08:54,399 --> 00:08:57,860 as a byproduct. Pretty much. And Cork points 191 00:08:57,860 --> 00:09:00,419 out that even when you have a perfectly balanced 192 00:09:00,419 --> 00:09:03,460 light source, like the sun, the physical environment 193 00:09:03,460 --> 00:09:07,000 acts as a massive, selective filter before that 194 00:09:07,000 --> 00:09:09,840 light ever actually reaches a surface. Right, 195 00:09:09,840 --> 00:09:12,779 which introduces Beer's Law, or the Beer -Lambert 196 00:09:12,779 --> 00:09:15,899 Law. It mathematically models how light is absorbed 197 00:09:15,899 --> 00:09:18,340 as it travels through any medium. OK, how does 198 00:09:18,340 --> 00:09:21,279 that work? It states that transmittance, which 199 00:09:21,279 --> 00:09:23,480 is the fraction of light that successfully passes 200 00:09:23,480 --> 00:09:26,139 through the medium, decays exponentially based 201 00:09:26,139 --> 00:09:29,159 on two factors, the physical distance traveled 202 00:09:29,159 --> 00:09:31,840 and an absorption coefficient that is highly 203 00:09:31,840 --> 00:09:34,440 specific to the wavelength of the light. OK, 204 00:09:34,480 --> 00:09:37,259 so our atmosphere does this, right? Ozone absorbs 205 00:09:37,259 --> 00:09:40,340 specific ultraviolet bands. Carbon dioxide absorbs 206 00:09:40,340 --> 00:09:44,000 specific infrared bands. Exactly. But for a roboticist, 207 00:09:44,250 --> 00:09:47,090 The most extreme example of Beer's Law in action 208 00:09:47,090 --> 00:09:50,429 is underwater exploration. Oh, because water 209 00:09:50,429 --> 00:09:53,389 is such a heavy filter. Underwater vision is 210 00:09:53,389 --> 00:09:56,049 notoriously difficult because water has a highly 211 00:09:56,049 --> 00:09:59,190 aggressive wavelength -dependent absorption spectrum. 212 00:09:59,929 --> 00:10:01,769 It allows shorter wavelengths like blues and 213 00:10:01,769 --> 00:10:04,470 greens to pass relatively easily, but it acts 214 00:10:04,470 --> 00:10:07,710 as a severe exponential filter against longer 215 00:10:07,710 --> 00:10:10,710 red wavelengths. So if you look at the transmission 216 00:10:10,710 --> 00:10:13,659 curve for just like five meters of clear water, 217 00:10:13,860 --> 00:10:16,559 the red light is almost entirely attenuated. 218 00:10:16,820 --> 00:10:19,259 Almost gone completely. So imagine you deploy 219 00:10:19,259 --> 00:10:21,519 an underwater inspection robot to check on a 220 00:10:21,519 --> 00:10:24,759 bright red pipeline. At five meters depth, that 221 00:10:24,759 --> 00:10:26,720 pipe is not going to look red to the robot's 222 00:10:26,720 --> 00:10:29,519 camera. No, because the red wavelengths originating 223 00:10:29,519 --> 00:10:31,639 from the sun at the surface were absorbed by 224 00:10:31,639 --> 00:10:33,700 the water column and never even made it down 225 00:10:33,700 --> 00:10:35,870 to the pipe in the first place. And whatever 226 00:10:35,870 --> 00:10:38,230 tiny fraction of red light does manage to reflect 227 00:10:38,230 --> 00:10:41,169 off the pipe gets exponentially filtered a second 228 00:10:41,169 --> 00:10:42,990 time on its journey back through the water to 229 00:10:42,990 --> 00:10:45,429 the camera lens. Precisely. So the resulting 230 00:10:45,429 --> 00:10:47,929 image looks overwhelmingly washed out in blue 231 00:10:47,929 --> 00:10:50,629 and green. The technical term we use in the field 232 00:10:50,629 --> 00:10:52,529 for the light that finally reaches the camera 233 00:10:52,529 --> 00:10:56,450 sensor is luminance. Luminance. Yes. And it's 234 00:10:56,450 --> 00:10:58,950 important to know that luminance is not the fundamental 235 00:10:58,950 --> 00:11:02,840 color of the object. Luminance is the mathematical 236 00:11:02,840 --> 00:11:05,299 product of the incoming illumination spectrum, 237 00:11:05,960 --> 00:11:08,559 multiplied by the object's surface reflectance 238 00:11:08,559 --> 00:11:11,419 spectrum, heavily modified by the medium it travels 239 00:11:11,419 --> 00:11:13,720 through. The book illustrates this beautifully 240 00:11:13,720 --> 00:11:16,379 with the example of a weathered red house brick. 241 00:11:16,860 --> 00:11:18,779 The physical material properties of the brick 242 00:11:18,779 --> 00:11:21,639 mean it reflects longer wavelengths, the reds, 243 00:11:21,759 --> 00:11:24,519 highly efficiently, while absorbing shorter blues. 244 00:11:24,639 --> 00:11:28,500 Right. But the luminance The actual photon count 245 00:11:28,500 --> 00:11:31,799 hitting a robot's sensor is the solar spectrum 246 00:11:31,799 --> 00:11:35,460 minus atmospheric absorption multiplied by that 247 00:11:35,460 --> 00:11:38,200 brick's reflectance curve. So if you move that 248 00:11:38,200 --> 00:11:41,299 exact same brick underwater or take it inside 249 00:11:41,299 --> 00:11:45,440 under a 2600 Kelvin tungsten lamp, the incoming 250 00:11:45,440 --> 00:11:48,019 illumination spectrum changes, which means the 251 00:11:48,019 --> 00:11:50,940 final luminance changes entirely. The robot sees 252 00:11:50,940 --> 00:11:54,159 a mathematically different object. which forces 253 00:11:54,159 --> 00:11:56,899 us to look at the biological miracle of how human 254 00:11:56,899 --> 00:12:00,500 beings process this ever -changing luminance 255 00:12:00,500 --> 00:12:02,539 without getting completely helplessly confused. 256 00:12:03,039 --> 00:12:04,980 Because to understand how we engineer digital 257 00:12:04,980 --> 00:12:07,279 sensors, we really have to look at the biological 258 00:12:07,279 --> 00:12:09,840 sensors on the human retina, right? The rod cells 259 00:12:09,840 --> 00:12:12,080 and cone cells. The numbers here are just staggering. 260 00:12:12,700 --> 00:12:17,360 We have about 120 million rod cells. 120 million. 261 00:12:17,720 --> 00:12:20,200 Yeah. And they are incredibly sensitive to low 262 00:12:20,200 --> 00:12:23,169 light. capable of triggering off a single photon, 263 00:12:23,509 --> 00:12:26,210 but they only detect overall intensity. They 264 00:12:26,210 --> 00:12:28,409 are completely color -planned. Which is why our 265 00:12:28,409 --> 00:12:30,230 vision drops to grayscale when we walk around 266 00:12:30,230 --> 00:12:32,690 in the dark. Exactly. But then we have about 267 00:12:32,690 --> 00:12:36,330 six million cone cells, which are our high -resolution 268 00:12:36,330 --> 00:12:39,389 daylight color sensors. And humans, like most 269 00:12:39,389 --> 00:12:42,429 higher primates, are trichromats. We possess 270 00:12:42,429 --> 00:12:45,450 three distinct types of cone cells, categorized 271 00:12:45,450 --> 00:12:47,610 by the specific wavelengths they are most sensitive 272 00:12:47,610 --> 00:12:51,139 to. Long, medium, and short wavelengths. Though, 273 00:12:51,279 --> 00:12:53,100 colloquially, they're almost always just referred 274 00:12:53,100 --> 00:12:55,679 to as red, green, and blue cones. Right. But 275 00:12:55,679 --> 00:12:58,080 here is the biological bottleneck that I find 276 00:12:58,080 --> 00:13:01,240 so fascinating. The textbook explains that the 277 00:13:01,240 --> 00:13:03,919 retina has this central pit called the fovea. 278 00:13:03,960 --> 00:13:06,879 Yes, the fovea. And it is microscopic. It's only 279 00:13:06,879 --> 00:13:10,120 about 0 .6 millimeters in diameter. And it only 280 00:13:10,120 --> 00:13:13,379 provides a five -degree field of view. Yet it 281 00:13:13,379 --> 00:13:16,419 contains nearly all of those six million cone 282 00:13:16,419 --> 00:13:19,220 cells tightly packed together. It's an incredibly 283 00:13:19,220 --> 00:13:22,340 dense cluster. So our sharp, full color, high 284 00:13:22,340 --> 00:13:25,399 resolution vision is essentially a tiny, highly 285 00:13:25,399 --> 00:13:27,620 concentrated flashlight beam. That's a great 286 00:13:27,620 --> 00:13:29,720 way to put it. We are just frantically darting 287 00:13:29,720 --> 00:13:32,320 this tiny, five degree flashlight around our 288 00:13:32,320 --> 00:13:35,240 environment, and our visual cortex is buffering 289 00:13:35,240 --> 00:13:37,500 and stitching all those rapid little snapshots 290 00:13:37,500 --> 00:13:40,460 together to generate the seamless, high resolution 291 00:13:40,460 --> 00:13:43,340 illusion of the world we think we are seeing. 292 00:13:43,679 --> 00:13:46,100 It really is an illusion constructed by the brain. 293 00:13:46,340 --> 00:13:49,860 So my question is, When we mount a standard digital 294 00:13:49,860 --> 00:13:52,820 camera on a robot, are we forcing its processor 295 00:13:52,820 --> 00:13:55,919 to evaluate the entire 90 -degree field of view 296 00:13:55,919 --> 00:13:59,360 at full foveal resolution all at once? Yes, and 297 00:13:59,360 --> 00:14:01,820 that is exactly the dichotomy between biological 298 00:14:01,820 --> 00:14:04,460 evolution and early machine vision engineering. 299 00:14:04,659 --> 00:14:07,139 Because if so, no wonder computer vision requires 300 00:14:07,139 --> 00:14:10,159 so much computing power. It's massive. A standard 301 00:14:10,159 --> 00:14:13,440 digital camera captures, say, 12 million pixels 302 00:14:13,440 --> 00:14:17,320 of RGB data simultaneously across a massive field 303 00:14:17,320 --> 00:14:20,340 of view. It generates hundreds of megabytes of 304 00:14:20,340 --> 00:14:23,899 raw data every single second. Just a fire hose 305 00:14:23,899 --> 00:14:26,840 of data. Right. The hardware doesn't have a Fovea. 306 00:14:27,419 --> 00:14:29,980 It captures the completely irrelevant ceiling 307 00:14:29,980 --> 00:14:31,720 tiles in the corner of the room with the exact 308 00:14:31,720 --> 00:14:35,139 same high resolution fidelity as the obstacle 309 00:14:35,139 --> 00:14:37,759 directly in front of the robot's wheels. It is 310 00:14:37,759 --> 00:14:41,580 a relentless tidal wave of useless data drowning 311 00:14:41,580 --> 00:14:44,320 the processor. It is. And it explains why the 312 00:14:44,320 --> 00:14:46,740 algorithms we'll discuss later regarding feature 313 00:14:46,740 --> 00:14:50,200 extraction are so fundamentally vital. In software, 314 00:14:50,620 --> 00:14:53,379 we have to artificially recreate that biological 315 00:14:53,379 --> 00:14:56,740 foveal focus. We have to ruthlessly throw away 316 00:14:56,740 --> 00:14:59,720 the 99 % of the image that doesn't contain useful 317 00:14:59,720 --> 00:15:02,919 geometry. So the robot's controller can actually 318 00:15:02,919 --> 00:15:05,860 calculate a safe path in real time. Exactly. 319 00:15:06,720 --> 00:15:09,159 Before we get into data reduction, we have to 320 00:15:09,159 --> 00:15:11,240 understand how those three color channels, the 321 00:15:11,240 --> 00:15:13,960 red, green, and blue, are mathematically interpreted. 322 00:15:14,600 --> 00:15:16,759 Right, which brings us to the trichromatic theory. 323 00:15:16,960 --> 00:15:19,240 This was proposed by Thomas Young way back in 324 00:15:19,240 --> 00:15:22,519 1802 and later rigorously championed by Hermann 325 00:15:22,519 --> 00:15:25,139 von Helmholtz and James Clerk Maxwell. And they 326 00:15:25,139 --> 00:15:27,460 propose that all the millions of color sensations 327 00:15:27,460 --> 00:15:30,679 we experience are merely psychological combinations 328 00:15:30,679 --> 00:15:33,710 of these three fundamental signals. The trichromatic 329 00:15:33,710 --> 00:15:35,830 theory perfectly describes the physical hardware 330 00:15:35,830 --> 00:15:38,230 of the eye, right? The three physical types of 331 00:15:38,230 --> 00:15:41,250 cone cells. It does. But for decades, it competed 332 00:15:41,250 --> 00:15:44,149 with a rival idea, the opponent color theory, 333 00:15:44,730 --> 00:15:47,409 primarily pushed by Carl Ewald Herring. And Herring 334 00:15:47,409 --> 00:15:50,009 argued what, exactly? He argued that we don't 335 00:15:50,009 --> 00:15:53,070 perceive colors as a mix of red, green, and blue, 336 00:15:53,450 --> 00:15:56,309 but rather along two mutually exclusive opposing 337 00:15:56,309 --> 00:15:59,889 axes, a red -green axis and a blue -yellow axis. 338 00:16:00,169 --> 00:16:02,690 And the intuitive evidence for opponent color 339 00:16:02,690 --> 00:16:04,929 theory is something everyone has experienced. 340 00:16:05,690 --> 00:16:08,110 Think about color after images. Right. Staring 341 00:16:08,110 --> 00:16:10,470 at a screen. Yeah. If you stare intensely at 342 00:16:10,470 --> 00:16:12,889 a bright red square on your screen for 30 seconds 343 00:16:12,889 --> 00:16:16,350 and then look away at a blank white wall, you 344 00:16:16,350 --> 00:16:19,509 don't see a faint red square. You see a glowing 345 00:16:19,509 --> 00:16:23,450 cyan green square. Or look at our language. We 346 00:16:23,450 --> 00:16:25,809 easily conceptualize a color being reddish -blue, 347 00:16:25,990 --> 00:16:28,409 which we call purple, but it is psychologically 348 00:16:28,409 --> 00:16:30,889 impossible to picture a color that is reddish 349 00:16:30,889 --> 00:16:33,409 -green. The brain simply cannot combine those 350 00:16:33,409 --> 00:16:36,210 two concepts. This led to incredibly acrimonious 351 00:16:36,210 --> 00:16:38,809 debates in the 19th century between the Helmholtz 352 00:16:38,809 --> 00:16:40,809 camp and the Herring camp over who was right. 353 00:16:41,190 --> 00:16:43,250 And the beautiful resolution, discovered much 354 00:16:43,250 --> 00:16:45,750 later by neuroscientists, is that biology is 355 00:16:45,750 --> 00:16:48,509 heavily layered and they were both right. Both 356 00:16:48,509 --> 00:16:51,299 of them. We absolutely have three types of cone 357 00:16:51,299 --> 00:16:53,700 cells acting as trichromatic hardware at the 358 00:16:53,700 --> 00:16:56,039 very back of the retina. But immediately in front 359 00:16:56,039 --> 00:16:58,159 of them, in the retinal ganglion cell layer, 360 00:16:58,500 --> 00:17:01,419 the biological wiring performs a matrix multiplication. 361 00:17:02,059 --> 00:17:05,019 It converts those raw RGB trichromatic signals 362 00:17:05,019 --> 00:17:08,240 into opponent color representations, essentially 363 00:17:08,240 --> 00:17:10,500 calculating the differences between the cones 364 00:17:10,500 --> 00:17:13,039 before sending the compressed data down the optic 365 00:17:13,039 --> 00:17:16,009 nerve to the brain. It is exactly like the eye 366 00:17:16,009 --> 00:17:18,769 has its own built -in pre -processing microchip. 367 00:17:18,829 --> 00:17:20,710 It's compressing the data before transmission. 368 00:17:20,950 --> 00:17:23,470 It's incredibly efficient. And speaking of built 369 00:17:23,470 --> 00:17:25,849 -in processing, there's another biological feature 370 00:17:25,849 --> 00:17:29,430 that robots desperately lack. Color constancy. 371 00:17:29,750 --> 00:17:32,769 Oh, yes. Humans have this profound ability to 372 00:17:32,769 --> 00:17:35,430 adaptively tune out the ambient illumination. 373 00:17:35,490 --> 00:17:37,970 Like, if I take a white piece of printer paper 374 00:17:37,970 --> 00:17:41,289 outside under a bright blue sky, my brain perceives 375 00:17:41,289 --> 00:17:43,779 it as white. Right. If I carry that exact same 376 00:17:43,779 --> 00:17:47,380 piece of paper indoors under a dim yellow 2600 377 00:17:47,380 --> 00:17:49,880 Kelvin tungsten lamp, it still looks perfectly 378 00:17:49,880 --> 00:17:52,380 white to me. My visual cortex understands the 379 00:17:52,380 --> 00:17:55,000 context and actively white balances the scene 380 00:17:55,000 --> 00:17:57,880 in real time. But a raw robotic vision system 381 00:17:57,880 --> 00:18:01,700 possesses zero inherent color constancy. To a 382 00:18:01,700 --> 00:18:04,740 digital sensor, the RGB pixel values of that 383 00:18:04,740 --> 00:18:07,740 paper outdoors versus indoors are drastically 384 00:18:07,740 --> 00:18:11,380 mathematically different. And that creates catastrophic 385 00:18:11,380 --> 00:18:14,579 failures in simple industrial robotics, doesn't 386 00:18:14,579 --> 00:18:17,339 it? Absolutely. Imagine an agricultural robot 387 00:18:17,339 --> 00:18:20,000 programmed to harvest ripe tomatoes. You tune 388 00:18:20,000 --> 00:18:21,980 the algorithm to look for a specific threshold 389 00:18:21,980 --> 00:18:24,960 of red pixel values. Okay, simple enough. At 390 00:18:24,960 --> 00:18:27,759 noon, on a clear day, it works flawlessly. But 391 00:18:27,759 --> 00:18:30,200 if a heavy thunderstorm rolls in, drastically 392 00:18:30,200 --> 00:18:32,599 shifting the illumination spectrum toward the 393 00:18:32,599 --> 00:18:35,819 blue -gray, the luminance reflecting off the 394 00:18:35,819 --> 00:18:39,809 tomato changes. Suddenly, The strict mathematical 395 00:18:39,809 --> 00:18:42,970 threshold for red is no longer met. And the robot 396 00:18:42,970 --> 00:18:45,470 simply stops harvesting, just sitting there surrounded 397 00:18:45,470 --> 00:18:47,849 by perfectly ripe tomatoes. Because it mathematically 398 00:18:47,849 --> 00:18:50,309 doesn't think they are red anymore. Unless the 399 00:18:50,309 --> 00:18:52,950 software engineers explicitly design an algorithm 400 00:18:52,950 --> 00:18:55,049 to mathematically correct for that illumination 401 00:18:55,049 --> 00:18:57,750 shift. Which is really hard to do. And Cork provides 402 00:18:57,750 --> 00:19:00,430 a stunning mathematical application of this regarding 403 00:19:00,430 --> 00:19:02,789 shadows. Because shadows are a nightmare for 404 00:19:02,789 --> 00:19:05,730 navigation algorithms. To a naive edge detector. 405 00:19:05,920 --> 00:19:08,380 A dark shadow cast across a sidewalk looks exactly 406 00:19:08,380 --> 00:19:11,380 like a physical dark object sitting on the ground. 407 00:19:12,039 --> 00:19:14,220 A robot might slam on its brakes to avoid driving 408 00:19:14,220 --> 00:19:16,700 over the shadow of a tree branch. But if we leverage 409 00:19:16,700 --> 00:19:20,019 the physics of light, shadows have two very specific 410 00:19:20,019 --> 00:19:22,880 exploitable properties. First, they are obviously 411 00:19:22,880 --> 00:19:25,259 darker because the direct ray of the sun is occluded. 412 00:19:25,420 --> 00:19:28,140 Right. But second, the light that does reach 413 00:19:28,140 --> 00:19:30,460 the shadowed area is illuminated almost entirely 414 00:19:30,460 --> 00:19:33,839 by the ambient sky. And because of atmospheric 415 00:19:33,839 --> 00:19:36,799 Rayleigh scattering, the ambient sky is blue. 416 00:19:38,119 --> 00:19:40,519 Therefore, outdoor shadows are never purely black 417 00:19:40,519 --> 00:19:42,779 or gray. They have a slight, highly predictable 418 00:19:42,779 --> 00:19:46,220 blue spectral tint. Right. Using this knowledge, 419 00:19:46,380 --> 00:19:48,460 Peter Corke demonstrates a technique where he 420 00:19:48,460 --> 00:19:50,779 models the camera sensor's spectral response 421 00:19:50,779 --> 00:19:53,759 and mathematically factors out this specific 422 00:19:53,759 --> 00:19:56,240 blue sky illumination signature. By projecting 423 00:19:56,240 --> 00:20:00,099 the RGB pixel values into a specific chromaticity 424 00:20:00,099 --> 00:20:02,940 space that accounts for the Planckian locus of 425 00:20:02,940 --> 00:20:06,099 daylight, you can mathematically collapse the 426 00:20:06,099 --> 00:20:09,599 illumination differences. The result is a shadow 427 00:20:09,599 --> 00:20:12,500 invariant image. I've seen the visual examples 428 00:20:12,500 --> 00:20:15,539 in the text and it genuinely looks like dark 429 00:20:15,539 --> 00:20:18,519 magic. It really does. You're in this matrix 430 00:20:18,519 --> 00:20:21,220 projection over the image array and the deep, 431 00:20:21,519 --> 00:20:23,819 confusing shadows of a tree cast on the road 432 00:20:23,819 --> 00:20:27,220 literally vanish. The asphalt looks completely 433 00:20:27,220 --> 00:20:29,859 uniform in color, allowing the robot's navigation 434 00:20:29,859 --> 00:20:32,400 stack to extract the actual physical boundaries 435 00:20:32,400 --> 00:20:35,559 of the road without getting terrified by a shadow. 436 00:20:35,920 --> 00:20:39,099 And this transition from raw photons to manipulatable 437 00:20:39,099 --> 00:20:41,359 numbers leads us perfectly into the next phase, 438 00:20:41,940 --> 00:20:45,380 Chapter 11, image processing. How do we take 439 00:20:45,380 --> 00:20:47,759 that continuous wave of light and turn it into 440 00:20:47,759 --> 00:20:50,500 something a processor can digest? We must digitize 441 00:20:50,500 --> 00:20:53,680 it. The physical sensor, typically a CMOS chip, 442 00:20:53,799 --> 00:20:56,579 is a dense two -dimensional array of microscopic 443 00:20:56,579 --> 00:20:59,799 photo sites. Okay. When phatons strike these 444 00:20:59,799 --> 00:21:02,599 silicon sites, they dislodge electrons due to 445 00:21:02,599 --> 00:21:04,900 the photoelectric effect building up a tiny electrical 446 00:21:04,900 --> 00:21:08,039 charge. And then that analog voltage is measured 447 00:21:08,039 --> 00:21:11,140 by an analog -to -digital converter and translated 448 00:21:11,140 --> 00:21:13,579 into a discrete digital number. Usually between 449 00:21:13,579 --> 00:21:17,240 0 and 255 for a standard 8 -bit system. And that 450 00:21:17,240 --> 00:21:20,240 gives us our digital canvas, just a massive two 451 00:21:20,240 --> 00:21:23,220 -dimensional matrix of integers. Exactly. But 452 00:21:23,220 --> 00:21:25,740 as we establish with the fovea discussion, this 453 00:21:25,740 --> 00:21:28,180 massive matrix contains way too much raw data 454 00:21:28,180 --> 00:21:30,200 and not nearly enough actionable information. 455 00:21:30,339 --> 00:21:32,839 We have to filter it. Right. Quark categorizes 456 00:21:32,839 --> 00:21:35,420 these early filtering steps by the type of mathematical 457 00:21:35,420 --> 00:21:38,640 operation used. The first category is dyadic 458 00:21:38,640 --> 00:21:41,799 operations, which involve comparing two entire 459 00:21:41,799 --> 00:21:44,339 equal -sized images against each other pixel 460 00:21:44,339 --> 00:21:46,740 by pixel. The most intuitive application for 461 00:21:46,740 --> 00:21:49,579 this is motion detection, isn't it? Yes. The 462 00:21:49,579 --> 00:21:51,980 logic seems straightforward. If a security robot 463 00:21:51,980 --> 00:21:55,359 is staring down a hallway, it holds frame A from 464 00:21:55,359 --> 00:21:58,000 one second ago in memory and compares it to frame 465 00:21:58,000 --> 00:22:01,319 B captured right now. You just mathematically 466 00:22:01,319 --> 00:22:04,259 subtract the matrices. Frame B minus frame A. 467 00:22:04,480 --> 00:22:07,440 Any pixel that result in a value of zero hasn't 468 00:22:07,440 --> 00:22:10,480 changed. Any pixel with a significant non -zero 469 00:22:10,480 --> 00:22:13,000 value must represent an object that just moved 470 00:22:13,000 --> 00:22:15,380 through the hallway. It is mathematically elegant, 471 00:22:15,380 --> 00:22:18,400 but it relies on a dangerously brittle assumption. 472 00:22:18,880 --> 00:22:21,480 Which is? that the background of the world is 473 00:22:21,480 --> 00:22:24,339 perfectly, statically illuminated. In any real 474 00:22:24,339 --> 00:22:26,240 -world environment, backgrounds are incredibly 475 00:22:26,240 --> 00:22:28,900 dynamic. The wind blows leaves in the background, 476 00:22:29,339 --> 00:22:31,799 fluorescent lights imperceptibly flicker. The 477 00:22:31,799 --> 00:22:34,599 camera's own auto -exposure algorithm might subtly 478 00:22:34,599 --> 00:22:38,500 tweak the gain. Wait. If the camera just subtracts 479 00:22:38,500 --> 00:22:41,079 pixels to find motion, what happens when a cloud 480 00:22:41,079 --> 00:22:44,460 slowly passes over the sun? That's exactly the 481 00:22:44,460 --> 00:22:47,140 issue. Because as the shadow slides across the 482 00:22:47,140 --> 00:22:49,160 grass, the grass goes from bright green to dark 483 00:22:49,160 --> 00:22:51,559 green, if you subtract those two frames, the 484 00:22:51,559 --> 00:22:54,140 math is going to return a massive non -zero value 485 00:22:54,140 --> 00:22:56,660 everywhere. The algorithm will tell the robot 486 00:22:56,660 --> 00:22:59,299 that the entire lawn is physically moving towards 487 00:22:59,299 --> 00:23:01,819 it. That is the exact vulnerability of simple 488 00:23:01,819 --> 00:23:04,599 frame differencing. To combat this false positive 489 00:23:04,599 --> 00:23:07,200 nightmare, robust computer vision systems utilize 490 00:23:07,200 --> 00:23:09,740 a recursive background estimation algorithm. 491 00:23:09,910 --> 00:23:12,950 often implemented as an infinite impulse response 492 00:23:12,950 --> 00:23:16,150 or IIR filter. Yes. How does that filter work 493 00:23:16,150 --> 00:23:18,650 in practice? Well, instead of blindly subtracting 494 00:23:18,650 --> 00:23:21,690 a static frame A, the algorithm maintains an 495 00:23:21,690 --> 00:23:23,750 actively updating mathematical model of what 496 00:23:23,750 --> 00:23:26,230 it thinks the background looks like. Okay. At 497 00:23:26,230 --> 00:23:28,910 every new time step, it updates this estimated 498 00:23:28,910 --> 00:23:31,710 background matrix using a weighted average. It 499 00:23:31,710 --> 00:23:34,339 might take just say 5 % of the pixel values from 500 00:23:34,339 --> 00:23:37,259 the brand new frame and blend them into 95 % 501 00:23:37,259 --> 00:23:39,660 of the historical background image. I see. So 502 00:23:39,660 --> 00:23:41,980 if a person walks quickly through the frame, 503 00:23:42,180 --> 00:23:44,579 they're only occupying those pixels for a fraction 504 00:23:44,579 --> 00:23:47,559 of a second. The 5 % waiting isn't enough for 505 00:23:47,559 --> 00:23:49,859 the algorithm to absorb the person into the background 506 00:23:49,859 --> 00:23:52,920 model. Right. But if a cloud slowly rolls over, 507 00:23:53,259 --> 00:23:55,259 dimming the entire scene over the course of 10 508 00:23:55,259 --> 00:23:58,799 seconds, that continuous 5 % compounding update 509 00:23:58,799 --> 00:24:01,319 allows the background model to smoothly darken 510 00:24:01,319 --> 00:24:03,559 along with the physical world. perfectly preventing 511 00:24:03,559 --> 00:24:06,240 that massive false positive promotion. Exactly. 512 00:24:06,559 --> 00:24:09,559 It acts as a temporal low pass filter, smoothing 513 00:24:09,559 --> 00:24:12,359 out gradual changes while isolating sharp high 514 00:24:12,359 --> 00:24:14,980 frequency kinetic events. But even with recursive 515 00:24:14,980 --> 00:24:17,539 filtering, comparing full images pixel by pixel 516 00:24:17,539 --> 00:24:20,500 is computationally expensive and conceptually 517 00:24:20,500 --> 00:24:23,900 fragile, right? It is. We need to analyze the 518 00:24:23,900 --> 00:24:26,400 internal structure of a single image. This requires 519 00:24:26,400 --> 00:24:29,420 moving from dyadic operations to spatial operations. 520 00:24:29,880 --> 00:24:32,619 Which introduces the concept of a Neighborhood 521 00:24:32,619 --> 00:24:36,039 and the mathematical tool of convolution. Yes, 522 00:24:36,039 --> 00:24:39,119 convolution. Instead of looking at pixel X in 523 00:24:39,119 --> 00:24:41,680 a vacuum, we analyze pixel X based on the values 524 00:24:41,680 --> 00:24:45,599 of the immediate 3x3 or 5x5 grid of pixels surrounding 525 00:24:45,599 --> 00:24:49,220 it. This is achieved using a kernel, right? A 526 00:24:49,220 --> 00:24:52,160 kernel is simply a small secondary matrix of 527 00:24:52,160 --> 00:24:56,619 numbers, say a 3x3 grid. In a convolution operation, 528 00:24:56,980 --> 00:24:59,339 you mathematically slide this kernel over every 529 00:24:59,339 --> 00:25:01,309 single pixel in the main image. It's kind of 530 00:25:01,309 --> 00:25:03,990 like a magnifying glass moving over a map. Exactly. 531 00:25:04,630 --> 00:25:06,789 At each stop, you multiply the overlapping pixel 532 00:25:06,789 --> 00:25:08,890 values by the corresponding numbers in the kernel, 533 00:25:09,509 --> 00:25:12,089 sum all nine resulting products together, and 534 00:25:12,089 --> 00:25:14,230 write that final sum as the new value for the 535 00:25:14,230 --> 00:25:17,150 center pixel in a brand new output image. OK, 536 00:25:17,150 --> 00:25:19,569 so if I want to blur or smooth an image to get 537 00:25:19,569 --> 00:25:23,230 rid of random sensor noise or static, I might 538 00:25:23,230 --> 00:25:26,089 use a uniform smoothing kernel. Right. In a 3 539 00:25:26,089 --> 00:25:28,150 by 3 uniform kernel, every number in the grid 540 00:25:28,150 --> 00:25:31,069 is just the fraction 1 ninth. By multiplying 541 00:25:31,069 --> 00:25:33,089 and summing, I am literally just calculating 542 00:25:33,089 --> 00:25:36,490 the strict mathematical average of that 3x3 neighborhood 543 00:25:36,490 --> 00:25:39,549 and replacing the center pixel with that average. 544 00:25:39,829 --> 00:25:42,670 That perfectly describes a mean filter. But Cork 545 00:25:42,670 --> 00:25:45,329 points out a severe geometric problem with uniform 546 00:25:45,329 --> 00:25:48,430 smoothing. It introduces ringing artifacts into 547 00:25:48,430 --> 00:25:53,009 the image. Ringing artifacts. Huh. I think I 548 00:25:53,009 --> 00:25:56,369 can visualize why. A uniform kernel is essentially 549 00:25:56,369 --> 00:25:59,509 a rigid mathematical box. If I slide that box 550 00:25:59,509 --> 00:26:02,210 over a smooth gradient, it's fine. But if you 551 00:26:02,210 --> 00:26:04,730 slide it over a sharp, high -contrast edge? Like 552 00:26:04,730 --> 00:26:07,750 a black shadow falling on white concrete. The 553 00:26:07,750 --> 00:26:10,309 sudden, abrupt entry of those extreme white pixel 554 00:26:10,309 --> 00:26:12,690 values into the leading edge of the kernel's 555 00:26:12,690 --> 00:26:15,630 window causes the output average to abruptly 556 00:26:15,630 --> 00:26:18,579 spike. Yes. As it crosses the boundary, the output 557 00:26:18,579 --> 00:26:22,140 oscillates, creating a ghostly echo or a ringing 558 00:26:22,140 --> 00:26:25,220 line parallel to the true edge. It blurs the 559 00:26:25,220 --> 00:26:28,220 noise, but it catastrophically distorts the sharp 560 00:26:28,220 --> 00:26:31,160 geometric features the robot might need to navigate. 561 00:26:31,599 --> 00:26:34,420 Which is why roboticists overwhelmingly prefer 562 00:26:34,420 --> 00:26:37,740 a Gaussian smoothing kernel. Okay, how does that 563 00:26:37,740 --> 00:26:40,839 differ? Instead of a uniform box, the values 564 00:26:40,839 --> 00:26:43,839 inside a Gaussian kernel map to a two -dimensional 565 00:26:43,839 --> 00:26:46,970 bell curve. The center pixel is weighted most 566 00:26:46,970 --> 00:26:50,230 heavily and the multiplier weights smoothly and 567 00:26:50,230 --> 00:26:52,890 exponentially taper off toward the outer edges 568 00:26:52,890 --> 00:26:55,329 of the kernel matrix, determined by a specific 569 00:26:55,329 --> 00:26:58,150 standard deviation parameter. So when that bell 570 00:26:58,150 --> 00:27:00,250 curve slides over the sharp black -to -white 571 00:27:00,250 --> 00:27:03,170 edge, the extreme values are introduced gradually 572 00:27:03,170 --> 00:27:05,769 through the tapered tails of the curve. Precisely. 573 00:27:06,309 --> 00:27:08,569 You get beautiful noise reduction without generating 574 00:27:08,569 --> 00:27:11,710 those harsh ringing artifacts. And preserving 575 00:27:11,710 --> 00:27:14,289 the integrity of edges is paramount because edge 576 00:27:14,289 --> 00:27:16,750 detection is arguably the holy grail of early 577 00:27:16,750 --> 00:27:19,349 image processing. Why are we so desperate to 578 00:27:19,349 --> 00:27:22,210 find edges? Because edges represent the physical 579 00:27:22,210 --> 00:27:25,009 boundaries of objects. And crucially, they are 580 00:27:25,009 --> 00:27:27,930 highly robust against all the illumination nightmares 581 00:27:27,930 --> 00:27:30,930 we discussed in Chapter 10. Because as we established, 582 00:27:31,069 --> 00:27:33,490 the absolute grayscale value of a pixel is fragile. 583 00:27:33,890 --> 00:27:35,809 It changes the moment a cloud covers the sun. 584 00:27:36,150 --> 00:27:39,220 Right. A pixel representing a gray asphalt road 585 00:27:39,220 --> 00:27:42,140 might plummet in intensity from a value of 200 586 00:27:42,140 --> 00:27:45,059 down to 100. But the edge of that road, the exact 587 00:27:45,059 --> 00:27:47,160 pixel boundary where the gray asphalt meets the 588 00:27:47,160 --> 00:27:50,079 green grass, that represents a sharp spatial 589 00:27:50,079 --> 00:27:53,140 derivative. It is a rapid rate of change in brightness. 590 00:27:53,700 --> 00:27:56,480 Yes. Even if the sun goes behind a cloud and 591 00:27:56,480 --> 00:27:58,500 the entire scene gets mathematically darker, 592 00:27:58,900 --> 00:28:01,460 the asphalt remains abruptly different from the 593 00:28:01,460 --> 00:28:04,470 grass right at that boundary line. The magnitude 594 00:28:04,470 --> 00:28:06,690 of the brightness drops, but the relative difference 595 00:28:06,690 --> 00:28:09,509 across the boundary is preserved. Exactly. To 596 00:28:09,509 --> 00:28:12,069 find these edges, we use specific convolution 597 00:28:12,069 --> 00:28:14,329 kernels designed to compute spatial derivatives, 598 00:28:14,410 --> 00:28:17,130 like the Sobel operator. The Sobel operator. 599 00:28:17,569 --> 00:28:19,750 It uses two kernels, one that calculates the 600 00:28:19,750 --> 00:28:22,609 gradient of pixel intensity horizontally, and 601 00:28:22,609 --> 00:28:25,670 another that calculates it vertically. By combining 602 00:28:25,670 --> 00:28:27,990 them, we aren't just looking at how bright the 603 00:28:27,990 --> 00:28:30,910 world is. We are calculating a vector field showing 604 00:28:30,910 --> 00:28:33,150 exactly where the brightness changes and in what 605 00:28:33,150 --> 00:28:36,269 direction. You're extracting the underlying geometric 606 00:28:36,269 --> 00:28:39,009 skeleton of the scene while intentionally throwing 607 00:28:39,009 --> 00:28:42,069 away the fragile, absolute illumination data. 608 00:28:42,470 --> 00:28:44,890 David Marr, a legendary pioneer in computational 609 00:28:44,890 --> 00:28:48,069 neuroscience at MIT, posited that the mammalian 610 00:28:48,069 --> 00:28:50,470 brain does exactly this in the early stages of 611 00:28:50,470 --> 00:28:52,710 the visual cortex. You did. He built what he 612 00:28:52,710 --> 00:28:55,690 called a primal sketch of the visual world, constructed 613 00:28:55,690 --> 00:28:58,680 almost entirely from these extracted edges. It 614 00:28:58,680 --> 00:29:01,259 is incredible how often our silicon engineering 615 00:29:01,259 --> 00:29:04,059 just ends up mimicking millions of years of biological 616 00:29:04,059 --> 00:29:07,160 evolution. It really is. But what if the robot 617 00:29:07,160 --> 00:29:10,299 isn't just looking for generic edges to avoid 618 00:29:10,299 --> 00:29:13,119 running into walls? What if it is actively searching 619 00:29:13,119 --> 00:29:16,240 for a specific target, like a charging port or 620 00:29:16,240 --> 00:29:19,539 a specific valve handle? This brings us to template 621 00:29:19,539 --> 00:29:22,339 matching. Cork uses this great example of searching 622 00:29:22,339 --> 00:29:25,599 a high resolution image of the Mona Lisa. Template 623 00:29:25,599 --> 00:29:28,079 matching is the process of searching a massive 624 00:29:28,079 --> 00:29:30,900 image array for a specific much smaller pattern 625 00:29:30,900 --> 00:29:33,720 of pixels, right? Exactly. Cork demonstrates 626 00:29:33,720 --> 00:29:36,359 taking a small crop of just the Mona Lisa's right 627 00:29:36,359 --> 00:29:40,480 eye, perhaps a 20 by 20 pixel template, and mathematically 628 00:29:40,480 --> 00:29:43,920 sliding it pixel by pixel across the entire full 629 00:29:43,920 --> 00:29:45,880 -size portrait to find the location where the 630 00:29:45,880 --> 00:29:48,779 template matches the underlying image best. But 631 00:29:48,779 --> 00:29:51,410 how does the algorithm define best. It needs 632 00:29:51,410 --> 00:29:54,230 a mathematical similarity measure to score every 633 00:29:54,230 --> 00:29:56,849 single location. The simplest one the book mentions 634 00:29:56,849 --> 00:30:00,490 is the sum of absolute differences, or SAD. Right. 635 00:30:00,730 --> 00:30:03,109 At every location, you subtract the template's 636 00:30:03,109 --> 00:30:05,470 pixel values from the current window's pixel 637 00:30:05,470 --> 00:30:08,250 values, take the absolute values so negatives 638 00:30:08,250 --> 00:30:10,809 don't cancel out positives, and sum them all 639 00:30:10,809 --> 00:30:13,470 up. The lower the sum, the better the match. 640 00:30:14,130 --> 00:30:16,269 A sum of zero means you've found an identical 641 00:30:16,269 --> 00:30:19,380 pixel -perfect match. But just like frame differencing, 642 00:30:19,559 --> 00:30:21,960 SAD has a fatal flaw regarding illumination, 643 00:30:22,119 --> 00:30:24,299 doesn't it? Oh, absolutely. If the template of 644 00:30:24,299 --> 00:30:26,460 the eye was captured from a well -lit photograph, 645 00:30:27,059 --> 00:30:28,640 but the robot is currently searching a version 646 00:30:28,640 --> 00:30:31,559 of the portrait that is dimly lit, the absolute 647 00:30:31,559 --> 00:30:34,400 pixel differences will be mathematically massive. 648 00:30:34,940 --> 00:30:36,900 Even when the template is perfectly aligned over 649 00:30:36,900 --> 00:30:39,859 the eye. The pattern is identical, but the baseline 650 00:30:39,859 --> 00:30:42,799 brightness is shifted. Right. Wait, I see how 651 00:30:42,799 --> 00:30:45,119 to fix this. If the problem is just a uniform 652 00:30:45,119 --> 00:30:47,619 shift in baseline brightness, couldn't the algorithm 653 00:30:47,619 --> 00:30:49,779 just calculate the average brightness of the 654 00:30:49,779 --> 00:30:52,359 template and the average brightness of the current 655 00:30:52,359 --> 00:30:55,079 search window and mathematically subtract those 656 00:30:55,079 --> 00:30:57,440 averages from the pixels before running the comparison? 657 00:30:58,000 --> 00:31:00,559 You have just deduced the exact mechanism behind 658 00:31:00,559 --> 00:31:03,500 zero -mean normalized cross -correlation, or 659 00:31:03,500 --> 00:31:07,769 ZNCC. It is a much more robust similarity measure. 660 00:31:08,210 --> 00:31:10,109 By subtracting the local mean brightness from 661 00:31:10,109 --> 00:31:12,910 both the template and the image patch, it effectively 662 00:31:12,910 --> 00:31:15,569 normalizes the data. It centers the pixel values 663 00:31:15,569 --> 00:31:17,930 around zero. So a shadow darkening the whole 664 00:31:17,930 --> 00:31:20,710 area just shifts the mean down. But the relative 665 00:31:20,710 --> 00:31:22,910 variations between the pixels, the actual pattern 666 00:31:22,910 --> 00:31:26,549 of the eye, stay identical. Exactly. The ZNCC 667 00:31:26,549 --> 00:31:29,490 score will boldly spike right over the Mona Lisa's 668 00:31:29,490 --> 00:31:31,410 eye, regardless of whether the painting is sitting 669 00:31:31,410 --> 00:31:34,220 in bright sunlight or deep shadow. It cancels 670 00:31:34,220 --> 00:31:37,480 out the ambient lighting. That's brilliant. It 671 00:31:37,480 --> 00:31:40,339 is computationally much heavier than SAD, requiring 672 00:31:40,339 --> 00:31:42,759 multiple passes to calculate means and variances, 673 00:31:42,940 --> 00:31:46,200 but it is practically mandatory for real -world 674 00:31:46,200 --> 00:31:48,759 robotics. However, everything we have discussed 675 00:31:48,759 --> 00:31:50,700 so far, you know, smoothing, edge detection, 676 00:31:50,880 --> 00:31:53,019 template matching, is just image processing. 677 00:31:53,380 --> 00:31:55,599 Right, it's just preparation. It takes us to 678 00:31:55,599 --> 00:31:58,140 the critical threshold of Chapter 12, extracting 679 00:31:58,140 --> 00:32:01,710 meaning or image features. We defined feature 680 00:32:01,710 --> 00:32:04,490 extraction earlier as the ultimate data reduction 681 00:32:04,490 --> 00:32:08,250 step. Yes, we have heavily filtered our 10 megabyte 682 00:32:08,250 --> 00:32:10,450 image array, but we still have millions of pixels. 683 00:32:11,009 --> 00:32:13,029 We need to distill it down to a tiny handful 684 00:32:13,029 --> 00:32:15,609 of scalars and vectors so the robot's controller 685 00:32:15,609 --> 00:32:18,369 can actually execute a command. We are going 686 00:32:18,369 --> 00:32:21,470 from raw data to actionable information. If an 687 00:32:21,470 --> 00:32:23,789 industrial robot arm is tasked with picking up 688 00:32:23,789 --> 00:32:25,670 a red brick from a conveyor belt, it doesn't 689 00:32:25,670 --> 00:32:27,930 need to process 10 million pixels at 60 frames 690 00:32:27,930 --> 00:32:31,259 a second. No. It just needs the x and y coordinates 691 00:32:31,259 --> 00:32:34,319 of the brick center of mass, its calculated area, 692 00:32:34,599 --> 00:32:37,140 and the orientation of its bounding box. It is 693 00:32:37,140 --> 00:32:39,599 essentially reading the bulleted executive summary 694 00:32:39,599 --> 00:32:42,400 of a 500 page report. You lose all the nuance, 695 00:32:42,599 --> 00:32:44,680 the texture, and the beautiful high resolution 696 00:32:44,680 --> 00:32:48,039 pros of the individual pixels. But the robot 697 00:32:48,039 --> 00:32:50,740 gets exactly the low bandwidth actionable data 698 00:32:50,740 --> 00:32:53,519 it needs to compute joint angles. But as with 699 00:32:53,519 --> 00:32:56,259 all summaries, if the executive summary is based 700 00:32:56,259 --> 00:32:59,710 on a bad assumption, the robot will fail catastrophically. 701 00:33:00,250 --> 00:33:02,250 And this is where the tension of machine vision 702 00:33:02,250 --> 00:33:05,789 really peaks. A system degrades based on how 703 00:33:05,789 --> 00:33:07,769 well the algorithm anticipates the violation 704 00:33:07,769 --> 00:33:10,730 of its basic geometric assumptions. Let's look 705 00:33:10,730 --> 00:33:13,589 at the first method of extraction, region features 706 00:33:13,589 --> 00:33:17,109 and segmentation. This is where we heuristically 707 00:33:17,109 --> 00:33:20,069 group pixels into application -meaningful regions. 708 00:33:20,910 --> 00:33:22,869 The simplest heuristic is color thresholding, 709 00:33:22,890 --> 00:33:25,569 right? The algorithm scans the image and says, 710 00:33:25,930 --> 00:33:28,509 find all the pixels that meet our RGB threshold 711 00:33:28,509 --> 00:33:31,529 for red and group them together as a contiguous 712 00:33:31,529 --> 00:33:34,190 blob representing the brick. Right. And then 713 00:33:34,190 --> 00:33:36,470 the algorithm calculates the geometric moments 714 00:33:36,470 --> 00:33:39,589 of that blob to find its center of mass and passes 715 00:33:39,589 --> 00:33:41,730 those coordinates to the robot. But think about 716 00:33:41,730 --> 00:33:44,410 the cascading chain of physical assumptions required 717 00:33:44,410 --> 00:33:47,589 to actually trust that blob. Let's say the algorithm 718 00:33:47,589 --> 00:33:50,109 successfully segments the image and tells the 719 00:33:50,109 --> 00:33:52,430 robot controller, hey, I found a blob consisting 720 00:33:52,430 --> 00:33:56,700 of 4,000 red pixels. OK. For the robot to use 721 00:33:56,700 --> 00:33:58,859 that pixel count to estimate how far away the 722 00:33:58,859 --> 00:34:01,859 brick is, so it can extend its arm, it has to 723 00:34:01,859 --> 00:34:04,460 implicitly assume four distinct things about 724 00:34:04,460 --> 00:34:07,359 the world. Yes. The first assumption is that 725 00:34:07,359 --> 00:34:09,440 it knows the true physical size of the target 726 00:34:09,440 --> 00:34:11,719 object in the real world. If it doesn't know 727 00:34:11,719 --> 00:34:14,599 how big the brick actually is, 4,000 pixels 728 00:34:14,599 --> 00:34:17,610 means nothing. Right. The second assumption is 729 00:34:17,610 --> 00:34:20,150 that it knows the exact focal length of the camera 730 00:34:20,150 --> 00:34:23,469 lens, which mathematically dictates how physical 731 00:34:23,469 --> 00:34:25,909 real -world sizes scale down to pixel sizes on 732 00:34:25,909 --> 00:34:28,590 the sensor. And the third assumption is terrifying. 733 00:34:28,730 --> 00:34:31,869 Yeah. It must assume there is zero occlusion. 734 00:34:32,130 --> 00:34:34,730 Yes. It assumes nothing is sitting in front of 735 00:34:34,730 --> 00:34:36,969 the brick, blocking half of it from the camera's 736 00:34:36,969 --> 00:34:40,190 view. If a stray leaf or a piece of packing paper 737 00:34:40,190 --> 00:34:42,349 is covering half the brick, the camera only sees 738 00:34:42,349 --> 00:34:44,699 2,000 red pixels. And the algorithm does the 739 00:34:44,699 --> 00:34:47,099 math and concludes, ah, the blob is half the 740 00:34:47,099 --> 00:34:49,619 expected size. Therefore, the brick must be twice 741 00:34:49,619 --> 00:34:52,500 as far away as I thought. And then the robot 742 00:34:52,500 --> 00:34:54,860 extends its arm too far and smashes into the 743 00:34:54,860 --> 00:34:57,320 conveyor belt. Exactly. And the fourth assumption, 744 00:34:57,460 --> 00:34:59,960 looping back to our earlier discussion, is that 745 00:34:59,960 --> 00:35:02,079 the lighting hasn't shifted the object's color 746 00:35:02,079 --> 00:35:04,260 spectrum out of the hard -coded threshold we 747 00:35:04,260 --> 00:35:07,219 defined as red. If any one of those four assumptions 748 00:35:07,219 --> 00:35:10,380 is violated, size, focal length, occlusion, or 749 00:35:10,380 --> 00:35:13,039 illumination, the executive summary is entirely 750 00:35:13,039 --> 00:35:15,559 wrong. which makes simple region segmentation 751 00:35:15,559 --> 00:35:18,559 feel incredibly fragile for something like autonomous 752 00:35:18,559 --> 00:35:21,960 navigation. A mobile robot trying to map a building 753 00:35:21,960 --> 00:35:24,179 frequently needs to answer the question, have 754 00:35:24,179 --> 00:35:26,519 I been here before? Right, it needs to recognize 755 00:35:26,519 --> 00:35:28,940 if it just looped back into the main lobby after 756 00:35:28,940 --> 00:35:31,539 exploring a long hallway. And we know that simple 757 00:35:31,539 --> 00:35:34,000 pixel by pixel template matching fails completely 758 00:35:34,000 --> 00:35:36,519 for this task. Because if the robot enters the 759 00:35:36,519 --> 00:35:39,039 lobby from an angle that is even five degrees 760 00:35:39,039 --> 00:35:41,690 different from its first visit, or if the camera 761 00:35:41,690 --> 00:35:45,210 translates two inches to the left, the raw perspective 762 00:35:45,210 --> 00:35:48,030 geometry of the pixels changes entirely. The 763 00:35:48,030 --> 00:35:50,969 ZNCC score will be terrible. Because template 764 00:35:50,969 --> 00:35:53,690 matching is not rotationally or scale invariant. 765 00:35:54,090 --> 00:35:57,389 So to solve the lobby problem, we need what Cork 766 00:35:57,389 --> 00:36:00,500 calls interest points. We need to design algorithms 767 00:36:00,500 --> 00:36:03,840 that scan an image not for red blobs, but for 768 00:36:03,840 --> 00:36:06,340 highly distinctive localized geometric features 769 00:36:06,340 --> 00:36:09,559 that associate reliably with specific 3D coordinates 770 00:36:09,559 --> 00:36:11,719 in the physical world, regardless of how the 771 00:36:11,719 --> 00:36:14,420 camera's oriented. Corners are the classic example 772 00:36:14,420 --> 00:36:17,460 of this, right? They are. A sharp 90 -degree 773 00:36:17,460 --> 00:36:19,960 corner on a dark wooden door frame intersecting 774 00:36:19,960 --> 00:36:22,900 with a white wall is a massive spatial derivative 775 00:36:22,900 --> 00:36:26,210 in two different directions simultaneously. So 776 00:36:26,210 --> 00:36:28,710 algorithms, like the Harris corner detector, 777 00:36:29,469 --> 00:36:31,570 essentially simulate shifting a small window 778 00:36:31,570 --> 00:36:34,369 in every possible direction. Right. If the pixel 779 00:36:34,369 --> 00:36:36,329 intensities change drastically, no matter which 780 00:36:36,329 --> 00:36:38,989 way you move the window, you know you are sitting 781 00:36:38,989 --> 00:36:41,429 on a corner. And the corner is mathematically 782 00:36:41,429 --> 00:36:44,610 robust. It remains a high gradient corner, whether 783 00:36:44,610 --> 00:36:48,050 you are 10 feet away or 2 feet away, providing 784 00:36:48,050 --> 00:36:50,900 scale invariance. And it remains a corner, whether 785 00:36:50,900 --> 00:36:54,420 the camera is perfectly level or tilted 45 degrees, 786 00:36:54,539 --> 00:36:57,539 providing rotational invariance. By contrast, 787 00:36:57,699 --> 00:37:00,539 a flat uniform white wall has zero interest points. 788 00:37:00,760 --> 00:37:04,059 Exactly. So if a robot enters the lobby, extracts 789 00:37:04,059 --> 00:37:06,519 a constellation of 500 unique Harris corners, 790 00:37:07,139 --> 00:37:09,000 and compares the geometric arrangement of that 791 00:37:09,000 --> 00:37:11,820 constellation to its memory banks, it can confidently 792 00:37:11,820 --> 00:37:15,239 say, I am in the lobby. Even if the ambient lighting 793 00:37:15,239 --> 00:37:17,800 has totally changed and it is entering from an 794 00:37:17,800 --> 00:37:20,639 awful angle. Which brings us to the most mathematically 795 00:37:20,639 --> 00:37:23,320 dense but absolutely foundational part of the 796 00:37:23,320 --> 00:37:27,179 text, Chapter 13. Image formation and geometry. 797 00:37:27,519 --> 00:37:29,880 Yes. Because now that we have extracted these 798 00:37:29,880 --> 00:37:33,039 features, these XY pixel coordinates of corners 799 00:37:33,039 --> 00:37:35,860 and interest points, we must rigorously define 800 00:37:35,860 --> 00:37:38,380 exactly how the three -dimensional physical world 801 00:37:38,380 --> 00:37:41,539 was mathematically flattened onto that two -dimensional 802 00:37:41,539 --> 00:37:44,139 image sensor. Because if we don't deeply understand 803 00:37:44,139 --> 00:37:46,239 the geometry of the camera lens, the features 804 00:37:46,239 --> 00:37:48,449 are just useless numbers. We have to introduce 805 00:37:48,449 --> 00:37:51,329 the perspective camera model based on the classic 806 00:37:51,329 --> 00:37:53,989 pinhole camera. The concept is that light rays 807 00:37:53,989 --> 00:37:56,210 from the 3D world travel in perfectly straight 808 00:37:56,210 --> 00:37:58,849 lines, pass through a single infinitely small 809 00:37:58,849 --> 00:38:01,710 point in space, the pinhole or the optical center, 810 00:38:02,090 --> 00:38:05,570 and project an inverted image onto the 2D image 811 00:38:05,570 --> 00:38:07,909 plane sitting at a specific focal distance behind 812 00:38:07,909 --> 00:38:11,170 it. It is beautiful in its simplicity. Using 813 00:38:11,170 --> 00:38:13,969 similar triangles, you can easily map a 3D coordinate, 814 00:38:14,010 --> 00:38:17,340 x, y, z, to a 2D coordinate on the plane. But 815 00:38:17,340 --> 00:38:19,780 this projection immediately reveals the fundamental 816 00:38:19,780 --> 00:38:22,739 mathematical ambiguity of vision, the complete 817 00:38:22,739 --> 00:38:25,159 loss of depth scale. The mountain versus the 818 00:38:25,159 --> 00:38:28,019 pebble. Yes. If I hold a tiny pebble one inch 819 00:38:28,019 --> 00:38:30,519 away from the camera lens, it blocks out a certain 820 00:38:30,519 --> 00:38:33,559 cone of light and takes up, say, 100 pixels on 821 00:38:33,559 --> 00:38:36,320 the sensor. Right. If a massive mountain is five 822 00:38:36,320 --> 00:38:39,199 miles away, its cone of light is identical by 823 00:38:39,199 --> 00:38:41,260 the time it reaches the pinhole, and it also 824 00:38:41,260 --> 00:38:44,000 takes up exactly 100 pixels. On the 2D image 825 00:38:44,000 --> 00:38:47,070 plane, the pebble and the mountain are mathematically 826 00:38:47,070 --> 00:38:49,610 indistinguishable. Without stereoscopic vision 827 00:38:49,610 --> 00:38:53,170 or outside context, the absolute scale of the 828 00:38:53,170 --> 00:38:56,130 z -axis is permanently lost in the projection. 829 00:38:56,690 --> 00:38:59,449 So to manage this projection mathematically and 830 00:38:59,449 --> 00:39:02,269 allow software to manipulate it, Cork introduces 831 00:39:02,269 --> 00:39:05,250 homogeneous coordinates and breaks the camera's 832 00:39:05,250 --> 00:39:08,590 projection matrix into two distinct, multiplied 833 00:39:08,590 --> 00:39:12,090 components. The extrinsic parameters and the 834 00:39:12,090 --> 00:39:15,289 intrinsic parameters. Now, the extrinsic parameters 835 00:39:15,289 --> 00:39:17,349 should be very familiar to anyone who followed 836 00:39:17,349 --> 00:39:20,210 parts 1 through 3 of our series. The extrinsic 837 00:39:20,210 --> 00:39:23,289 matrix is literally just the rigid body motion 838 00:39:23,289 --> 00:39:26,170 of the camera in space. It is a translation vector 839 00:39:26,170 --> 00:39:29,289 and a rotation matrix that maps the global world 840 00:39:29,289 --> 00:39:31,889 coordinate frame to the camera's local coordinate 841 00:39:31,889 --> 00:39:34,329 frame. It just tells the math where the camera 842 00:39:34,329 --> 00:39:36,530 is physically bolted onto the robot's chassis 843 00:39:36,530 --> 00:39:39,760 and which way it is pointing. Exactly. The new 844 00:39:39,760 --> 00:39:42,159 critical piece is the intrinsic parameter matrix, 845 00:39:42,619 --> 00:39:45,199 universally denoted in the literature as the 846 00:39:45,199 --> 00:39:49,780 K -matrix. The K -matrix. Yes, this 3x3 matrix 847 00:39:49,780 --> 00:39:52,340 models the internal physical characteristics 848 00:39:52,340 --> 00:39:55,239 of the specific camera hardware. Let's break 849 00:39:55,239 --> 00:39:57,739 down what's actually inside the K -matrix. First 850 00:39:57,739 --> 00:40:00,139 and foremost, it contains the focal length, which 851 00:40:00,139 --> 00:40:02,500 determines the optical magnification and field 852 00:40:02,500 --> 00:40:05,239 of view. But it's represented as two values, 853 00:40:05,599 --> 00:40:09,750 fx and fy. Y2. Because it factors in the physical 854 00:40:09,750 --> 00:40:12,849 size of the pixels on the silicon sensor. If 855 00:40:12,849 --> 00:40:15,210 the microscopic CMOS pixels aren't perfectly 856 00:40:15,210 --> 00:40:18,929 square, if they are slightly rectangular, a physical 857 00:40:18,929 --> 00:40:21,389 square in the real world will project as a rectangle 858 00:40:21,389 --> 00:40:24,789 on the sensor. The K matrix scales the X and 859 00:40:24,789 --> 00:40:27,449 Y axes independently to correct for this pixel 860 00:40:27,449 --> 00:40:30,789 geometry. Okay, that makes sense. The K matrix 861 00:40:30,789 --> 00:40:33,389 also contains the principal point offset labeled 862 00:40:33,389 --> 00:40:38,250 as CX and CY, right? Yes. In an ideal mathematical 863 00:40:38,250 --> 00:40:41,289 pinhole, the optical axis perfectly intersects 864 00:40:41,289 --> 00:40:44,230 the dead center of the digital sensor. The coordinate 865 00:40:44,230 --> 00:40:47,869 0, 0 is right in the middle. But in reality, 866 00:40:48,110 --> 00:40:50,909 due to manufacturing tolerances, the lens is 867 00:40:50,909 --> 00:40:53,570 never perfectly centered over the silicon. The 868 00:40:53,570 --> 00:40:56,369 principal point offset shifts the origin of the 869 00:40:56,369 --> 00:40:58,690 coordinate system to align the math with the 870 00:40:58,690 --> 00:41:01,550 physical optical center. By multiplying a 3D 871 00:41:01,550 --> 00:41:04,480 point by the extrinsic matrix, and then multiplying 872 00:41:04,480 --> 00:41:08,000 the result by the intrinsic K matrix, we can 873 00:41:08,000 --> 00:41:10,579 precisely predict which specific pixel index, 874 00:41:10,800 --> 00:41:13,940 which row and column, that 3D point will illuminate 875 00:41:13,940 --> 00:41:17,340 on the sensor. But there is a massive physical 876 00:41:17,340 --> 00:41:19,880 catch. The pinhole model assumes light travels 877 00:41:19,880 --> 00:41:22,500 perfectly straight through an infinitely small 878 00:41:22,500 --> 00:41:25,460 hole. And infinitely small holes don't let in 879 00:41:25,460 --> 00:41:27,579 enough photons to generate a usable image at 880 00:41:27,579 --> 00:41:29,880 60 frames per second. The sensor would just read 881 00:41:29,880 --> 00:41:32,800 black. So we are forced to widen the aperture 882 00:41:32,800 --> 00:41:35,659 and use curved glass lenses to gather and focus 883 00:41:35,659 --> 00:41:38,900 the light. And glass lenses warp reality. They 884 00:41:38,900 --> 00:41:42,320 do. They introduce geometric distortion. The 885 00:41:42,320 --> 00:41:45,380 most prominent is radial distortion, mathematically 886 00:41:45,380 --> 00:41:48,280 modeled by the Brown -Conradty model. Because 887 00:41:48,280 --> 00:41:50,840 a lens is essentially a sphere, it bends light 888 00:41:50,840 --> 00:41:52,920 more severely at the outer edges than it does 889 00:41:52,920 --> 00:41:55,320 in the dead center. This causes straight lines 890 00:41:55,320 --> 00:41:58,139 in the physical world to appear curved or bowed 891 00:41:58,139 --> 00:42:01,059 in the image. You see this heavily in wide angle 892 00:42:01,059 --> 00:42:03,500 security cameras, creating a barrel distortion 893 00:42:03,500 --> 00:42:06,159 effect. There is also tangential distortion, 894 00:42:06,300 --> 00:42:08,360 which happens if the glass lens isn't mounted 895 00:42:08,360 --> 00:42:11,219 perfectly parallel to the image sensor. One side 896 00:42:11,219 --> 00:42:13,179 of the image gets slightly magnified compared 897 00:42:13,179 --> 00:42:15,760 to the other. Right. And if a robot is trying 898 00:42:15,760 --> 00:42:18,559 to navigate a hallway, and it assumes the baseboards 899 00:42:18,559 --> 00:42:21,400 are straight parallel lines, but the raw image 900 00:42:21,400 --> 00:42:23,900 shows them curving violently outward due to radial 901 00:42:23,900 --> 00:42:26,059 distortion, The navigation stack is going to 902 00:42:26,059 --> 00:42:28,000 calculate a completely warped trajectory and 903 00:42:28,000 --> 00:42:30,260 drive the robot into the wall. So how do we fix 904 00:42:30,260 --> 00:42:32,519 the glass? We don't fix the glass. Well, we fix 905 00:42:32,519 --> 00:42:36,199 the math. Through camera calibration. OK. The 906 00:42:36,199 --> 00:42:38,559 industry standard is Zong's method, popularized 907 00:42:38,559 --> 00:42:42,059 by the OpenCV software library. We show the camera 908 00:42:42,059 --> 00:42:45,139 a known, highly precise geometric pattern, typically 909 00:42:45,139 --> 00:42:47,920 a printed checkerboard mounted on a flat, rigid 910 00:42:47,920 --> 00:42:50,820 board. We take dozens of pictures of this checkerboard 911 00:42:50,820 --> 00:42:53,659 from various angles and distances. The algorithm 912 00:42:53,659 --> 00:42:55,639 detects all the corners of the black and white 913 00:42:55,639 --> 00:42:58,360 squares. Because the algorithm knows that physical 914 00:42:58,360 --> 00:43:00,860 squares are perfectly straight and evenly spaced 915 00:43:00,860 --> 00:43:04,340 in reality, it observes exactly how those known 916 00:43:04,340 --> 00:43:06,440 straight lines are bowed and distorted in the 917 00:43:06,440 --> 00:43:09,639 pixel data. Right. Using a complex optimization 918 00:43:09,639 --> 00:43:14,039 algorithm, usually Levenberg -Markhardt, It iteratively 919 00:43:14,039 --> 00:43:17,119 solves for all 11 intrinsic and extrinsic parameters, 920 00:43:17,539 --> 00:43:20,079 plus a set of radial and tangential distortion 921 00:43:20,079 --> 00:43:24,599 coefficients, usually K1, K2, K3, P1, P2. Once 922 00:43:24,599 --> 00:43:26,960 we have those coefficients, the software performs 923 00:43:26,960 --> 00:43:30,400 a pixel remapping. It takes the warped incoming 924 00:43:30,400 --> 00:43:33,199 image array and mathematically pushes every pixel 925 00:43:33,199 --> 00:43:35,840 back to where it would have been if the camera 926 00:43:35,840 --> 00:43:38,539 were a perfect pinhole. It undistorts the image 927 00:43:38,539 --> 00:43:41,519 in real time, before the feature extraction algorithms 928 00:43:41,519 --> 00:43:44,639 ever even see it. What is profound about this 929 00:43:44,639 --> 00:43:47,360 is the hardware paradigm shift it created. For 930 00:43:47,360 --> 00:43:50,679 decades, the goal in optics was to grind flawless 931 00:43:50,679 --> 00:43:53,199 multi -element distortion -free glass lenses, 932 00:43:53,500 --> 00:43:56,360 which cost thousands of dollars. But computation 933 00:43:56,360 --> 00:43:58,760 and matrix math have become vastly cheaper than 934 00:43:58,760 --> 00:44:01,920 precision glass. This means we can slap a $5 935 00:44:01,920 --> 00:44:05,219 highly distorted plastic lens on a robot and 936 00:44:05,219 --> 00:44:07,880 run an open CV calibration script to fix the 937 00:44:07,880 --> 00:44:10,519 geometry purely in software. It is the primary 938 00:44:10,519 --> 00:44:13,260 driving force behind why robot vision is so ubiquitous 939 00:44:13,260 --> 00:44:15,840 and affordable today. It even allows us to use 940 00:44:15,840 --> 00:44:18,219 extreme fisheye lenses, where the field of view 941 00:44:18,219 --> 00:44:20,739 is so wide it literally wraps into a circle. 942 00:44:21,320 --> 00:44:24,559 The math changes. You have to use a unified spherical 943 00:44:24,559 --> 00:44:26,699 imaging model instead of the standard pinhole 944 00:44:26,699 --> 00:44:29,280 projection to unfold the sphere, but the software 945 00:44:29,280 --> 00:44:32,199 handles it beautifully. But as powerful as intrinsic 946 00:44:32,199 --> 00:44:35,500 calibration is, it still does not solve the fundamental 947 00:44:35,500 --> 00:44:38,059 ambiguity we mentioned earlier. The mountain 948 00:44:38,059 --> 00:44:41,489 versus the pebble problem. A single camera, no 949 00:44:41,489 --> 00:44:44,369 matter how perfectly calibrated and undistorted, 950 00:44:44,610 --> 00:44:48,429 is still trapped in a 2D projection. The Z -axis 951 00:44:48,429 --> 00:44:51,659 is lost. Which brings us to our final chapter, 952 00:44:51,780 --> 00:44:55,780 Chapter 14, using multiple images. If a single 953 00:44:55,780 --> 00:44:58,420 camera loses the scale of depth, how do we get 954 00:44:58,420 --> 00:45:02,510 it back? We use two cameras. We introduce stereo 955 00:45:02,510 --> 00:45:05,150 vision. mimicking the biological advantage of 956 00:45:05,150 --> 00:45:08,050 our own two eyes. By observing the exact same 957 00:45:08,050 --> 00:45:11,730 3D scene from two slightly offset physical viewpoints, 958 00:45:11,989 --> 00:45:14,889 we can triangulate the depth of the world. But 959 00:45:14,889 --> 00:45:17,250 implementing stereo vision in software introduces 960 00:45:17,250 --> 00:45:20,590 arguably the most notoriously difficult computational 961 00:45:20,590 --> 00:45:23,389 challenge in computer vision. The correspondence 962 00:45:23,389 --> 00:45:26,550 problem. The correspondence problem. To triangulate 963 00:45:26,550 --> 00:45:28,889 depth, you have to find the exact same point 964 00:45:28,889 --> 00:45:32,050 in both images. If I run a Harris corner detector, 965 00:45:32,380 --> 00:45:35,000 and find a highly distinctive corner on a coffee 966 00:45:35,000 --> 00:45:38,099 cup in the left camera's image, how do I find 967 00:45:38,099 --> 00:45:40,480 that exact same physical corner in the right 968 00:45:40,480 --> 00:45:43,739 camera's image? Well, if the images are 10 megapixels 969 00:45:43,739 --> 00:45:45,659 each, you can't just run a template matching 970 00:45:45,659 --> 00:45:48,239 algorithm across the entire right image. It would 971 00:45:48,239 --> 00:45:50,340 require millions of calculations for a single 972 00:45:50,340 --> 00:45:53,360 point. It would take seconds to process a single 973 00:45:53,360 --> 00:45:55,679 frame. This is where the elegant mathematics 974 00:45:55,679 --> 00:45:58,000 of epipolar geometry come to the rescue, right? 975 00:45:58,219 --> 00:46:01,710 Yes. Epipolar geometry defines the intrinsic, 976 00:46:01,750 --> 00:46:04,429 projective relationship between two cameras viewing 977 00:46:04,429 --> 00:46:08,690 the same 3D point. It dictates a hard geometric 978 00:46:08,690 --> 00:46:12,280 constraint. OK, what is the constraint? If you 979 00:46:12,280 --> 00:46:14,800 find a point in the left image, you do not have 980 00:46:14,800 --> 00:46:16,539 to search the entire right image to find its 981 00:46:16,539 --> 00:46:19,940 match. The geometry of the stereo setup constrains 982 00:46:19,940 --> 00:46:22,360 the possible location of that point in the right 983 00:46:22,360 --> 00:46:25,260 image to a single, one -dimensional straight 984 00:46:25,260 --> 00:46:28,559 line. The epipolar line. Let's visualize how 985 00:46:28,559 --> 00:46:30,940 this works. You have the optical center of the 986 00:46:30,940 --> 00:46:33,380 left camera, the optical center of the right 987 00:46:33,380 --> 00:46:35,639 camera, and the physical 3D point out in the 988 00:46:35,639 --> 00:46:38,300 real world. Those three points in space form 989 00:46:38,300 --> 00:46:41,039 a perfect triangle, known as the epipolar plane. 990 00:46:41,619 --> 00:46:44,860 Exactly. Now imagine projecting that flat triangular 991 00:46:44,860 --> 00:46:47,519 plane through the lenses and onto the 2D image 992 00:46:47,519 --> 00:46:50,619 sensors of both cameras. Where that plane intersects 993 00:46:50,619 --> 00:46:52,719 the 2D image sensor, it forms a straight line. 994 00:46:53,099 --> 00:46:55,920 Because the 3D point must lie somewhere on that 995 00:46:55,920 --> 00:46:58,619 plane, its projection in the right image must 996 00:46:58,619 --> 00:47:01,599 lie somewhere on that intersected line. It is 997 00:47:01,599 --> 00:47:04,960 brilliant. It mathematically reduces a massive, 998 00:47:05,420 --> 00:47:07,619 computationally impossible 2D search problem 999 00:47:07,619 --> 00:47:10,900 into a quick, bounded 1D search along a single 1000 00:47:10,900 --> 00:47:13,820 line of pixels. The algebraic mapping of this 1001 00:47:13,820 --> 00:47:16,900 geometry is captured in a 3x3 matrix called the 1002 00:47:16,900 --> 00:47:19,849 fundamental matrix, denoted as F. The fundamental 1003 00:47:19,849 --> 00:47:22,789 matrix relates corresponding points in uncalibrated 1004 00:47:22,789 --> 00:47:25,349 stereo images. If you take a pixel coordinate 1005 00:47:25,349 --> 00:47:28,550 in the left image, represented as a vector, and 1006 00:47:28,550 --> 00:47:31,010 multiply it by the fundamental matrix, the output 1007 00:47:31,010 --> 00:47:34,150 is not a point. No, the output is the algebraic 1008 00:47:34,150 --> 00:47:36,670 equation of the epipolar line in the right image. 1009 00:47:37,070 --> 00:47:41,289 The constraint is written as x' transpose f x 1010 00:47:41,289 --> 00:47:43,829 equals zero. But the fundamental matrix only 1011 00:47:43,829 --> 00:47:46,780 maps raw pixels. If we have already calibrated 1012 00:47:46,780 --> 00:47:49,679 our cameras, if we know their intrinsic K matrices 1013 00:47:49,679 --> 00:47:52,139 from the checkerboard process, we can upgrade 1014 00:47:52,139 --> 00:47:54,380 the math. We can factor out the focal lengths 1015 00:47:54,380 --> 00:47:56,760 and principal points to create the essential 1016 00:47:56,760 --> 00:48:00,280 matrix, denoted as E. And the essential matrix 1017 00:48:00,280 --> 00:48:02,659 is incredibly powerful because it encapsulates 1018 00:48:02,659 --> 00:48:05,260 only the extrinsic physical relationship between 1019 00:48:05,260 --> 00:48:07,559 the two cameras. It is purely a function of the 1020 00:48:07,559 --> 00:48:11,019 relative 3D translation vector and the 3D rotation 1021 00:48:11,019 --> 00:48:14,079 matrix between the left and right lenses. Through 1022 00:48:14,079 --> 00:48:16,440 a process called Singular Value Decomposition, 1023 00:48:16,679 --> 00:48:19,880 or SVD, we can mathematically decompose the essential 1024 00:48:19,880 --> 00:48:22,960 matrix to extract exactly how the right camera 1025 00:48:22,960 --> 00:48:25,579 is rotated and translated relative to the left 1026 00:48:25,579 --> 00:48:28,880 camera. But Cork explicitly points out a massive, 1027 00:48:29,260 --> 00:48:31,599 inescapable physical limitation here. And what 1028 00:48:31,599 --> 00:48:34,960 is that? Even with two cameras. Even with perfect 1029 00:48:34,960 --> 00:48:37,780 epipolar geometry. A flawless essential matrix 1030 00:48:37,780 --> 00:48:40,559 and brilliant feature matching. without knowing 1031 00:48:40,559 --> 00:48:43,159 the absolute physical baseline distance between 1032 00:48:43,159 --> 00:48:46,340 the two camera lenses. The scale of depth is 1033 00:48:46,340 --> 00:48:49,219 still completely arbitrary. This is a concept 1034 00:48:49,219 --> 00:48:51,760 that often trips up students. Two cameras can 1035 00:48:51,760 --> 00:48:53,699 give you the relative structure of a scene, but 1036 00:48:53,699 --> 00:48:56,719 not the absolute scale. Imagine holding a tiny, 1037 00:48:57,000 --> 00:49:00,039 miniaturized stereo camera rig up to a perfectly 1038 00:49:00,039 --> 00:49:03,139 scaled, 1 -100th architectural model of a city. 1039 00:49:03,699 --> 00:49:06,260 Now imagine holding a giant stereo rig with the 1040 00:49:06,260 --> 00:49:09,000 camera spaced 10 feet apart, looking at the real 1041 00:49:09,090 --> 00:49:11,849 full -size city. If the proportions of the camera 1042 00:49:11,849 --> 00:49:14,030 baseline perfectly match the proportions of the 1043 00:49:14,030 --> 00:49:16,949 scene, the 2D images produced by both systems 1044 00:49:16,949 --> 00:49:20,110 are mathematically identical. A tiny model world 1045 00:49:20,110 --> 00:49:23,429 looks exactly like a giant real world. The essential 1046 00:49:23,429 --> 00:49:25,849 matrix can tell you the right camera is translated 1047 00:49:25,849 --> 00:49:28,969 along the x -axis, but it cannot tell you if 1048 00:49:28,969 --> 00:49:31,829 it's translated by 10 centimeters or 10 miles. 1049 00:49:32,090 --> 00:49:34,190 You must physically measure the baseline distance 1050 00:49:34,190 --> 00:49:36,809 between the cameras with a ruler and hard code 1051 00:49:36,809 --> 00:49:39,570 that scalar value into the algorithm. to anchor 1052 00:49:39,570 --> 00:49:42,269 the mathematics to physical reality. Assuming 1053 00:49:42,269 --> 00:49:44,690 we have a rigid stereo rig with a known baseline, 1054 00:49:45,110 --> 00:49:48,409 say, exactly 12 centimeters, we can compute absolute 1055 00:49:48,409 --> 00:49:51,230 depth. And we don't just do this for a handful 1056 00:49:51,230 --> 00:49:54,269 of Harris corners. No, using the epipolar constraint, 1057 00:49:54,269 --> 00:49:57,510 we can perform dense stereo. This algorithm computes 1058 00:49:57,510 --> 00:50:00,090 a depth estimate for every single pixel in the 1059 00:50:00,090 --> 00:50:03,320 image, generating a disparity map. The process 1060 00:50:03,320 --> 00:50:06,360 is fascinating. For every pixel in the left image, 1061 00:50:06,800 --> 00:50:09,079 the algorithm grabs a small surrounding template, 1062 00:50:09,539 --> 00:50:11,500 calculates the epipolar line in the right image, 1063 00:50:12,059 --> 00:50:14,139 and slides the template along that line using 1064 00:50:14,139 --> 00:50:17,739 a metric like ZNCC to find the best match. The 1065 00:50:17,739 --> 00:50:19,679 physical difference in pixel coordinates between 1066 00:50:19,679 --> 00:50:21,800 the left image location and the matching right 1067 00:50:21,800 --> 00:50:24,280 image location is called the disparity. And because 1068 00:50:24,280 --> 00:50:27,019 of the geometry of triangulation, disparity is 1069 00:50:27,019 --> 00:50:30,039 inversely proportional to depth. A massive disparity. 1070 00:50:30,190 --> 00:50:32,550 meaning the object shifted significantly between 1071 00:50:32,550 --> 00:50:34,889 the left and right views, means the object is 1072 00:50:34,889 --> 00:50:37,909 very close to the lenses. A tiny disparity means 1073 00:50:37,909 --> 00:50:40,409 the object is far away. At optical infinity, 1074 00:50:40,809 --> 00:50:42,909 like looking at the moon, the disparity is zero 1075 00:50:42,909 --> 00:50:45,329 because the rays are parallel. But dense stereo 1076 00:50:45,329 --> 00:50:48,690 introduces some hilarious and catastrophic failure 1077 00:50:48,690 --> 00:50:51,329 modes that Quark specifically highlights. The 1078 00:50:51,329 --> 00:50:54,670 first major one is the picket fence effect. Ah, 1079 00:50:54,690 --> 00:50:57,559 the picket fence. This occurs when the stereo 1080 00:50:57,559 --> 00:50:59,739 algorithm attempts to process a periodically 1081 00:50:59,739 --> 00:51:03,010 repeating texture in the real world, like a white 1082 00:51:03,010 --> 00:51:06,610 picket fence, a brick wall, or a person wearing 1083 00:51:06,610 --> 00:51:09,170 a tightly striped shirt. The algorithm grabs 1084 00:51:09,170 --> 00:51:11,190 a template of one specific fence post in the 1085 00:51:11,190 --> 00:51:13,969 left image and slides it along the epipolar line 1086 00:51:13,969 --> 00:51:16,769 in the right image. And it immediately finds 1087 00:51:16,769 --> 00:51:19,789 ten perfect matches, because all the fence posts 1088 00:51:19,789 --> 00:51:23,039 look physically identical. The ZNCC similarity 1089 00:51:23,039 --> 00:51:25,579 score doesn't just peak once. It peaks multiple 1090 00:51:25,579 --> 00:51:28,800 times at wildly different disparities. The algorithm 1091 00:51:28,800 --> 00:51:31,239 is completely confused about which post is the 1092 00:51:31,239 --> 00:51:34,239 true match. So the resulting depth map becomes 1093 00:51:34,239 --> 00:51:36,920 a jagged erratic mess, jumping back and forth 1094 00:51:36,920 --> 00:51:40,199 between near and far. It is essentially spatial 1095 00:51:40,199 --> 00:51:43,369 aliasing. The signal lacks uniqueness. It is 1096 00:51:43,369 --> 00:51:45,550 exactly like trying to figure out where you are 1097 00:51:45,550 --> 00:51:47,769 in a pop song that just repeats the exact same 1098 00:51:47,769 --> 00:51:49,610 four chords for three minutes straight. There 1099 00:51:49,610 --> 00:51:51,829 is no unique landmark to anchor your position 1100 00:51:51,829 --> 00:51:54,769 to. You could be in the first chorus or the third. 1101 00:51:54,889 --> 00:51:57,610 They sound identical. The correspondence algorithm 1102 00:51:57,610 --> 00:52:00,769 needs uniqueness. Which is why roboticists often 1103 00:52:00,769 --> 00:52:03,630 joke that robots absolutely love messy, chaotic 1104 00:52:03,630 --> 00:52:06,690 environments and struggle immensely in clean, 1105 00:52:06,989 --> 00:52:09,480 sterile ones. A cluttered laboratory bench full 1106 00:52:09,480 --> 00:52:12,300 of distinct textures, oscilloscopes, random wires, 1107 00:52:12,420 --> 00:52:15,019 and coffee mugs is an absolute goldmine for feature 1108 00:52:15,019 --> 00:52:17,500 extraction and stereo correspondence. Almost 1109 00:52:17,500 --> 00:52:20,420 every pixel neighborhood is unique. But if you 1110 00:52:20,420 --> 00:52:23,500 deploy that same robot into a pristine modern 1111 00:52:23,500 --> 00:52:27,340 hospital hallway with uniform blank white walls 1112 00:52:27,340 --> 00:52:30,380 and highly polished floors? It goes completely 1113 00:52:30,380 --> 00:52:33,239 blind. There are no spatial gradients, no corners, 1114 00:52:33,440 --> 00:52:36,869 no unique templates, just a sea. of identical 1115 00:52:36,869 --> 00:52:39,829 white pixels. The disparity map fails entirely. 1116 00:52:40,190 --> 00:52:42,190 But wait, there is a second failure mode for 1117 00:52:42,190 --> 00:52:44,570 dense stereo that the book details, which happens 1118 00:52:44,570 --> 00:52:47,769 even in highly textured environments. The missing 1119 00:52:47,769 --> 00:52:51,210 parts or occlusion problem. This is a strict 1120 00:52:51,210 --> 00:52:54,429 geometric inevitability, right? Because the two 1121 00:52:54,429 --> 00:52:56,489 cameras are separated by a physical baseline 1122 00:52:56,489 --> 00:52:59,110 distance, they have slightly different perspectives 1123 00:52:59,110 --> 00:53:01,829 on the 3D world. This inherently means there 1124 00:53:01,829 --> 00:53:04,090 will always be physical points in the scene that 1125 00:53:04,090 --> 00:53:06,489 are visible to the left camera, but physically 1126 00:53:06,489 --> 00:53:09,329 hidden behind a foreground object from the perspective 1127 00:53:09,329 --> 00:53:12,110 of the right camera. The book uses a great visual 1128 00:53:12,110 --> 00:53:14,849 of a rock pile to demonstrate this. If the left 1129 00:53:14,849 --> 00:53:16,849 camera is looking at the extreme right edge of 1130 00:53:16,849 --> 00:53:19,389 a foreground rock, it sees the texture of the 1131 00:53:19,389 --> 00:53:22,050 rock right up to the boundary. But from the right 1132 00:53:22,050 --> 00:53:24,510 camera's perspective, positioned further to the 1133 00:53:24,510 --> 00:53:27,289 right, that rock's boundary has physically shifted 1134 00:53:27,289 --> 00:53:29,570 and is now blocking the background behind it. 1135 00:53:29,840 --> 00:53:32,380 So when the stereo algorithm takes a template 1136 00:53:32,380 --> 00:53:34,360 of that background texture from the left image 1137 00:53:34,360 --> 00:53:37,300 and searches the epipolar line in the right image, 1138 00:53:37,659 --> 00:53:40,280 the true matching pixels simply do not exist 1139 00:53:40,280 --> 00:53:42,360 in the right image. They are occluded by the 1140 00:53:42,360 --> 00:53:45,880 rock. But the dead stereo algorithm is relentless. 1141 00:53:46,400 --> 00:53:48,679 It is forced to pick the best available match 1142 00:53:48,679 --> 00:53:52,550 along that line, which will inevitably be a completely 1143 00:53:52,550 --> 00:53:54,769 incorrect piece of texture on the rock itself. 1144 00:53:55,369 --> 00:53:57,650 And this causes massive artificial spikes in 1145 00:53:57,650 --> 00:53:59,940 the depth calculation right at the boundaries 1146 00:53:59,940 --> 00:54:03,320 of objects. The resulting disparity map will 1147 00:54:03,320 --> 00:54:05,739 tell the robot there is a bizarre jagged spike 1148 00:54:05,739 --> 00:54:08,380 of material protruding into empty space right 1149 00:54:08,380 --> 00:54:10,920 at the edge of the rock, purely because the mathematics 1150 00:54:10,920 --> 00:54:14,099 forced a false correspondence. This is why raw 1151 00:54:14,099 --> 00:54:17,159 stereo depth maps always look noisy and haloed 1152 00:54:17,159 --> 00:54:20,000 around the edges of foreground objects. Despite 1153 00:54:20,000 --> 00:54:22,659 these localized flaws, tracking these features 1154 00:54:22,659 --> 00:54:25,420 across multiple images is not just used for 3D 1155 00:54:25,420 --> 00:54:28,760 depth reconstruction. It completely revolutionized 1156 00:54:28,760 --> 00:54:31,519 the way mobile robots navigate, leading to the 1157 00:54:31,519 --> 00:54:35,139 development of visual odometry. Right. If we 1158 00:54:35,139 --> 00:54:37,400 loop all the way back to part two of our series, 1159 00:54:37,659 --> 00:54:40,039 our ground -based mobile robots navigated using 1160 00:54:40,039 --> 00:54:43,039 wheel encoders. The wheels turned, the internal 1161 00:54:43,039 --> 00:54:45,500 sensors counted the magnetic clicks, and we used 1162 00:54:45,500 --> 00:54:47,739 standard trigonometry to dead -recon where the 1163 00:54:47,739 --> 00:54:50,900 robot had driven. But wheel encoders suffer terribly 1164 00:54:50,900 --> 00:54:54,639 from physical slippage on dirt or gravel. And 1165 00:54:54,639 --> 00:54:57,239 more importantly, wheel encoders are entirely 1166 00:54:57,239 --> 00:54:59,360 useless if your robot isn't touching the ground. 1167 00:55:00,000 --> 00:55:03,840 Exactly. How does an underwater ROV or an aerial 1168 00:55:03,840 --> 00:55:06,719 quadcopter drone know how far it has translated 1169 00:55:06,719 --> 00:55:09,380 and rotated in space? They use visual odometry. 1170 00:55:09,960 --> 00:55:12,340 By extracting a constellation of robust interest 1171 00:55:12,340 --> 00:55:16,019 points like Harris corners in frame one and tracking 1172 00:55:16,019 --> 00:55:18,179 where those exact same feature points have moved 1173 00:55:18,179 --> 00:55:20,679 to in frame two, a fraction of a second later, 1174 00:55:21,039 --> 00:55:23,440 we can run the epipolar geometry in reverse. 1175 00:55:23,860 --> 00:55:26,900 Instead of using known camera motion to triangulate 1176 00:55:26,900 --> 00:55:30,039 3D points, we used the apparent 2D movement of 1177 00:55:30,039 --> 00:55:32,659 the points on the sensor to mathematically estimate 1178 00:55:32,659 --> 00:55:35,260 the camera's motion. The algorithm watches the 1179 00:55:35,260 --> 00:55:38,380 world slide past its lens, calculates a massive 1180 00:55:38,380 --> 00:55:41,219 optimization matrix to minimize the reprojection 1181 00:55:41,219 --> 00:55:44,579 error of those points, and deduces, okay, based 1182 00:55:44,579 --> 00:55:47,099 on how these 500 features scaled and shifted, 1183 00:55:47,579 --> 00:55:50,420 I must have moved forward 10 centimeters and 1184 00:55:50,420 --> 00:55:53,079 rotated exactly 2 degrees around the yaw axis. 1185 00:55:53,550 --> 00:55:56,690 It replaces a physical mechanical sensor with 1186 00:55:56,690 --> 00:56:00,190 pure algorithmic geometry. It is the absolute 1187 00:56:00,190 --> 00:56:03,030 foundation of modern autonomous navigation, tying 1188 00:56:03,030 --> 00:56:05,769 the physics of light, image processing, feature 1189 00:56:05,769 --> 00:56:08,550 extraction, and projective geometry into a single 1190 00:56:08,550 --> 00:56:12,250 cohesive real -time capability. So we have covered 1191 00:56:12,250 --> 00:56:15,590 an immense amount of ground today. If we distill 1192 00:56:15,590 --> 00:56:18,949 all of this down, what is the ultimate main takeaway 1193 00:56:18,949 --> 00:56:21,400 for you, the listener? The central thesis of 1194 00:56:21,400 --> 00:56:24,539 part four is that computer vision is not a passive 1195 00:56:24,539 --> 00:56:27,380 recording of physical reality. An image array 1196 00:56:27,380 --> 00:56:30,980 is not a scene. Vision is an active, interpretive, 1197 00:56:31,199 --> 00:56:33,960 and highly constrained algorithmic process. A 1198 00:56:33,960 --> 00:56:36,079 robotic system must constantly make educated 1199 00:56:36,079 --> 00:56:38,579 mathematically -bound guesses to reconstruct 1200 00:56:38,579 --> 00:56:41,659 a complex three -dimensional physical world from 1201 00:56:41,659 --> 00:56:44,340 flattened, distorted, and information -lossy 1202 00:56:44,340 --> 00:56:47,380 two -dimensional data arrays. When those mathematical 1203 00:56:47,380 --> 00:56:49,940 assumptions hold true, when the lighting is stable, 1204 00:56:50,139 --> 00:56:52,019 the textures are unique, and the calibration 1205 00:56:52,019 --> 00:56:55,519 is precise, the robot appears brilliantly intelligent. 1206 00:56:56,239 --> 00:56:59,000 When those assumptions fail, when a cloud rolls 1207 00:56:59,000 --> 00:57:02,420 in, or it faces a blank white wall, or occlusion 1208 00:57:02,420 --> 00:57:04,980 hides a feature, the system breaks down entirely. 1209 00:57:05,840 --> 00:57:08,980 It is entirely about managing ambiguity through 1210 00:57:08,980 --> 00:57:11,519 mathematics. And before we wrap up, I want to 1211 00:57:11,519 --> 00:57:13,559 leave you with a provocative thought, building 1212 00:57:13,559 --> 00:57:16,460 on something we touched on early in our biological 1213 00:57:16,460 --> 00:57:20,199 discussion. We noted that the human retina relies 1214 00:57:20,199 --> 00:57:23,199 on three types of color sensing cones, and we 1215 00:57:23,199 --> 00:57:25,440 have engineered almost all of our digital camera 1216 00:57:25,440 --> 00:57:28,420 hardware to closely mimic this trichromatic biology. 1217 00:57:29,039 --> 00:57:32,639 Red, green, and blue pixels on a CMOS chip. But 1218 00:57:32,639 --> 00:57:35,380 biological vision is vastly more diverse than 1219 00:57:35,380 --> 00:57:38,619 ours. The mantis shrimp possesses between 12 1220 00:57:38,619 --> 00:57:41,239 and 16 different types of color photoreceptors. 1221 00:57:41,449 --> 00:57:44,150 Dragonflies have up to 30 different types of 1222 00:57:44,150 --> 00:57:47,230 cone cells. Their compound eyes give them nearly 1223 00:57:47,230 --> 00:57:50,989 360 -degree spherical vision, and they can natively 1224 00:57:50,989 --> 00:57:53,690 perceive polarized light and ultraviolet spectra. 1225 00:57:54,050 --> 00:57:55,809 Their biological fovea and neural processing 1226 00:57:55,809 --> 00:57:58,309 architecture are distributed completely differently 1227 00:57:58,309 --> 00:58:01,889 than our mammalian visual cortex. Exactly. So 1228 00:58:01,889 --> 00:58:03,750 the thought to chew on as we leave this topic 1229 00:58:03,750 --> 00:58:06,519 is this. If we stopped using human trichromatic 1230 00:58:06,519 --> 00:58:09,480 biology as the default blueprint for our engineering, 1231 00:58:10,019 --> 00:58:13,480 what entirely new optical geometries, what radical 1232 00:58:13,480 --> 00:58:16,079 new mathematical matrices, and what profoundly 1233 00:58:16,079 --> 00:58:19,139 alien feature extraction assumptions would we 1234 00:58:19,139 --> 00:58:21,079 need to build a robot that sees the world like 1235 00:58:21,079 --> 00:58:23,960 a dragonfly? How much more efficient, robust, 1236 00:58:24,139 --> 00:58:26,440 or computationally lightweight could robotic 1237 00:58:26,440 --> 00:58:28,699 vision algorithms be if we finally broke free 1238 00:58:28,699 --> 00:58:30,599 from the constraints of the human perspective? 1239 00:58:30,920 --> 00:58:34,300 That is a staggering and fascinating engineering 1240 00:58:34,300 --> 00:58:36,519 challenge for the next generation of roboticists. 1241 00:58:36,719 --> 00:58:38,679 It really is. So to bring it all back to our 1242 00:58:38,679 --> 00:58:41,059 opening thought, the pristine X -ray machine 1243 00:58:41,059 --> 00:58:44,599 is broken. The clean binary data of mechanical 1244 00:58:44,599 --> 00:58:47,179 joint sensors and wheel encoders has been left 1245 00:58:47,179 --> 00:58:50,280 behind. We are swimming in the muddy, ambiguous 1246 00:58:50,280 --> 00:58:53,360 waters of bouncing photons, spectral attenuation, 1247 00:58:53,719 --> 00:58:56,369 and perspective geometry. But as we've seen today, 1248 00:58:56,630 --> 00:58:58,809 if you apply the right convolutional math and 1249 00:58:58,809 --> 00:59:01,650 rigid projective matrices, you can navigate that 1250 00:59:01,650 --> 00:59:04,750 mud with incredible precision. Next time, we 1251 00:59:04,750 --> 00:59:06,570 are going to take all of this extracted visual 1252 00:59:06,570 --> 00:59:09,090 data, the interest points, the stereo depth maps, 1253 00:59:09,369 --> 00:59:11,969 the undistorted geometries, and finally close 1254 00:59:11,969 --> 00:59:15,750 the loop. Join us for part five of five, vision 1255 00:59:15,750 --> 00:59:18,519 -based control. where we will formally introduce 1256 00:59:18,519 --> 00:59:21,820 visual servoing and explore exactly how a robot 1257 00:59:21,820 --> 00:59:24,719 takes these calculated pixels and uses them to 1258 00:59:24,719 --> 00:59:26,900 dynamically steer its wheels and actuate its 1259 00:59:26,900 --> 00:59:29,019 arms in real time. We will see you then.