1 00:00:00,000 --> 00:00:02,359 You know, usually when we talk about medical 2 00:00:02,359 --> 00:00:06,080 diagnosis, there's this expectation of absolute 3 00:00:06,080 --> 00:00:08,580 precision. Right. Like you break your arm, the 4 00:00:08,580 --> 00:00:10,599 x -ray shows that jagged white line, and the 5 00:00:10,599 --> 00:00:12,439 doctor just points and says, you know, there 6 00:00:12,439 --> 00:00:15,140 it is. It's binary. Exactly. Broken or not broken. 7 00:00:15,580 --> 00:00:18,160 Exactly. It's clean. And because it's visible, 8 00:00:18,199 --> 00:00:21,260 it's comforting. But if you step into like a 9 00:00:21,260 --> 00:00:24,480 neurology ward or deal with complex trauma, that 10 00:00:24,480 --> 00:00:27,079 x -ray machine is pretty much useless. Absolutely. 11 00:00:27,460 --> 00:00:30,410 The landscape gets murky. Yeah, the rigid models 12 00:00:30,410 --> 00:00:32,969 we rely on just start to break down entirely. 13 00:00:33,429 --> 00:00:36,350 And that same exact transition, you know, from 14 00:00:36,350 --> 00:00:39,750 rigid, comforting certainty to this messy dynamic 15 00:00:39,750 --> 00:00:42,729 reality is what we are looking at today in the 16 00:00:42,729 --> 00:00:45,509 world of robotics. It's a big shift. It really 17 00:00:45,509 --> 00:00:48,909 is. So welcome to the grand finale, part five 18 00:00:48,909 --> 00:00:52,049 of five, in our marginalia deep dive on Peter 19 00:00:52,049 --> 00:00:54,890 Cork's textbook, Robotics, Vision and Control. 20 00:00:55,079 --> 00:00:57,719 And we've built up quite a foundation for you 21 00:00:57,719 --> 00:01:00,179 over the previous four parts. I mean, we mapped 22 00:01:00,179 --> 00:01:03,880 out the spatial geometry, the differential kinematics 23 00:01:03,880 --> 00:01:06,540 of mobile platforms and robot arms. Right. The 24 00:01:06,540 --> 00:01:09,340 heavy math. Yeah, the heavy lifting. And most 25 00:01:09,340 --> 00:01:11,579 recently, in part four, we tackled computer vision. 26 00:01:12,280 --> 00:01:14,780 We essentially unpacked how a camera lens projects 27 00:01:14,780 --> 00:01:19,519 the 3D world onto a 2D sensor array, translating 28 00:01:19,519 --> 00:01:23,099 photons into matrices of pixel intensities. Basically, 29 00:01:23,099 --> 00:01:25,319 we took the blindfold off the robot. Exactly. 30 00:01:25,500 --> 00:01:28,420 We gave it eyes. But seeing the jagged line on 31 00:01:28,420 --> 00:01:31,040 the x -ray, or seeing a coffee cup on the table, 32 00:01:31,040 --> 00:01:33,730 that's only half the battle. If a machine can 33 00:01:33,730 --> 00:01:36,230 process the pixels that represent the cup, but 34 00:01:36,230 --> 00:01:38,870 it lacks the mathematical framework to coordinate 35 00:01:38,870 --> 00:01:40,609 its servos to reach out and actually grab it, 36 00:01:40,629 --> 00:01:42,629 then the vision system is functionally useless. 37 00:01:43,049 --> 00:01:44,709 Right. It just remains this passive observer. 38 00:01:45,469 --> 00:01:47,750 So today, our mission is to cross that final 39 00:01:47,750 --> 00:01:50,069 bridge for you. We are covering chapters 15 and 40 00:01:50,069 --> 00:01:53,489 16, vision -based control and advanced visual 41 00:01:53,489 --> 00:01:55,700 servoing. and we are going to wire those pixel 42 00:01:55,700 --> 00:01:58,459 measurements directly into the velocity controllers 43 00:01:58,459 --> 00:02:01,079 of the robot. And to appreciate the gravity of 44 00:02:01,079 --> 00:02:03,760 that shift, I think we have to isolate the fundamental 45 00:02:03,760 --> 00:02:06,239 limitation of classic robotic manipulation. The 46 00:02:06,239 --> 00:02:08,979 blind robot problem. Exactly. In a traditional 47 00:02:08,979 --> 00:02:11,199 setup, say an assembly line stamping out car 48 00:02:11,199 --> 00:02:13,979 doors, the entire operation relies on the assumption 49 00:02:13,979 --> 00:02:16,699 of a perfectly known universe. Which doesn't 50 00:02:16,699 --> 00:02:19,319 really exist. No, it doesn't. But when you watch 51 00:02:19,319 --> 00:02:21,360 that industrial arm swing over and pick up a 52 00:02:21,360 --> 00:02:23,599 bolt, human intuition suggests the robot is, 53 00:02:23,599 --> 00:02:26,500 you know, looking for or moving to the object. 54 00:02:27,000 --> 00:02:28,819 But based on the kinematics we covered back in 55 00:02:28,819 --> 00:02:31,759 Part 3, it absolutely is not moving to an object. 56 00:02:32,340 --> 00:02:35,080 It's executing a trajectory to a mathematically 57 00:02:35,080 --> 00:02:38,599 defined pose in Cartesian space. Right. In SE3 58 00:02:38,599 --> 00:02:42,280 matrix? Yeah. Representing a predetermined XYZ 59 00:02:42,280 --> 00:02:45,259 translation in a roll -pitch yaw orientation. 60 00:02:45,550 --> 00:02:48,050 It moves to the coordinate where the factory 61 00:02:48,050 --> 00:02:50,610 engineers promised the bolt would be. And the 62 00:02:50,610 --> 00:02:53,509 financial cost of that promise is just staggering. 63 00:02:54,009 --> 00:02:56,449 To guarantee that the physical world perfectly 64 00:02:56,449 --> 00:02:58,830 matches the internal mathematical map of the 65 00:02:58,830 --> 00:03:01,729 robot controller, manufacturers have to invest 66 00:03:01,729 --> 00:03:05,210 millions. Oh, massive steel jigs, precise conveyors, 67 00:03:05,389 --> 00:03:08,689 rigid fixtures. You are literally paying to physically 68 00:03:08,689 --> 00:03:10,949 eliminate all uncertainty from the environment. 69 00:03:11,490 --> 00:03:13,430 Because if a vibration shifts that bolt a fraction 70 00:03:13,430 --> 00:03:15,530 of a millimeter outside the expected tolerance, 71 00:03:15,770 --> 00:03:17,949 the robot's end effector just blindly closes 72 00:03:17,949 --> 00:03:21,250 on empty air. Or worse, it crashes into the workpiece 73 00:03:21,250 --> 00:03:23,930 and completely shatters it. Exactly. The open 74 00:03:23,930 --> 00:03:25,830 -loop nature of the execution makes the robot 75 00:03:25,830 --> 00:03:28,189 incredibly brittle to environmental variation. 76 00:03:28,810 --> 00:03:32,150 You get immense repeatability, but zero adaptability. 77 00:03:32,349 --> 00:03:34,189 I was thinking about this and it's kind of like 78 00:03:34,189 --> 00:03:36,129 reaching for a glass of water on your nightstand 79 00:03:36,129 --> 00:03:38,590 in pitch -black darkness. Oh, that's a good analogy. 80 00:03:38,830 --> 00:03:41,189 Right? You're doing it based purely on muscle 81 00:03:41,189 --> 00:03:44,330 memory. That's traditional robotics. You know 82 00:03:44,330 --> 00:03:46,990 exactly where the glass should be, but if you 83 00:03:46,990 --> 00:03:49,629 bump the table earlier in the day... You're knocking 84 00:03:49,629 --> 00:03:52,030 water all over the floor. Exactly. So what we're 85 00:03:52,030 --> 00:03:55,409 talking about today, visual servoing, it's like 86 00:03:55,409 --> 00:03:57,449 finally turning on the bedside lamp and letting 87 00:03:57,449 --> 00:03:59,909 your eyes continuously guide your hand to the 88 00:03:59,909 --> 00:04:02,550 glass. Which brings us to the core thesis of 89 00:04:02,550 --> 00:04:05,689 Chapter 15. Instead of treating vision as a distinct, 90 00:04:05,990 --> 00:04:08,530 isolated step where a camera takes a static snapshot, 91 00:04:09,169 --> 00:04:11,689 a computer calculates the target pose, hands 92 00:04:11,689 --> 00:04:14,169 that coordinate to the robot controller, and 93 00:04:14,169 --> 00:04:16,069 then the robot basically closes its eyes and 94 00:04:16,069 --> 00:04:18,970 executes the move. We weave the visual data directly 95 00:04:18,970 --> 00:04:22,410 into the low -level motion control loop. We establish 96 00:04:22,410 --> 00:04:24,670 continuous hand -eye coordination for the machine. 97 00:04:25,009 --> 00:04:27,670 Right. The robot keeps its eyes open for the 98 00:04:27,670 --> 00:04:30,649 entire duration of the trajectory. It continuously 99 00:04:30,649 --> 00:04:32,769 measures the visual error between its current 100 00:04:32,769 --> 00:04:35,389 state and the target state and uses that error 101 00:04:35,389 --> 00:04:37,870 signal to drive the joint velocities until the 102 00:04:37,870 --> 00:04:40,449 discrepancy hits zero. Peter Corke models a few 103 00:04:40,449 --> 00:04:42,829 different physical architectures for this, but 104 00:04:42,829 --> 00:04:45,189 the primary distinction is where you actually 105 00:04:45,189 --> 00:04:48,850 mount the camera, right? Yes. You have the endpoint 106 00:04:48,850 --> 00:04:51,310 open -loop configuration, which is effectively 107 00:04:51,310 --> 00:04:53,569 an external surveillance camera bolted to the 108 00:04:53,569 --> 00:04:56,290 ceiling. It has a global view of both the robot 109 00:04:56,290 --> 00:04:59,670 manipulator and the target object. So it calculates 110 00:04:59,670 --> 00:05:02,170 the relative distance between the two and guides 111 00:05:02,170 --> 00:05:05,230 the arm from afar. That seems like a massive 112 00:05:05,230 --> 00:05:08,089 calibration headache. Oh, it is. You'd have to 113 00:05:08,089 --> 00:05:10,189 perfectly map the coordinate frame of the ceiling 114 00:05:10,189 --> 00:05:12,870 camera to the base frame of the robot. And any 115 00:05:12,870 --> 00:05:15,310 microscopic shift in the camera mount, say a 116 00:05:15,310 --> 00:05:17,189 truck drives by outside and shakes the building, 117 00:05:17,269 --> 00:05:19,970 ruins the entire spatial relationship. It introduces 118 00:05:19,970 --> 00:05:22,649 significant transformation errors. Yeah. Which 119 00:05:22,649 --> 00:05:25,110 is why the bulk of Quark's analysis, and where 120 00:05:25,110 --> 00:05:27,589 we will focus entirely for this deep dive, is 121 00:05:27,589 --> 00:05:30,990 the eye -in -hand configuration. Okay, so the 122 00:05:30,990 --> 00:05:33,529 camera is rigidly mounted directly onto the robot's 123 00:05:33,529 --> 00:05:36,350 end effector. Right, it moves with the arm. The 124 00:05:36,350 --> 00:05:38,769 camera frame and the end effector frame are locked 125 00:05:38,769 --> 00:05:42,310 together by a constant known rigid body transformation. 126 00:05:42,689 --> 00:05:44,470 So the camera is essentially strapped to the 127 00:05:44,470 --> 00:05:47,269 robot's wrist, acting like a flashlight beam. 128 00:05:47,490 --> 00:05:50,490 pointing toward the gripper's workspace. That 129 00:05:50,490 --> 00:05:52,689 instantly changes the mathematical objective. 130 00:05:53,069 --> 00:05:55,790 How so? Well, we aren't trying to calculate where 131 00:05:55,790 --> 00:05:58,649 the robot is in the global room anymore. We only 132 00:05:58,649 --> 00:06:00,529 care about the relative pose between the camera 133 00:06:00,529 --> 00:06:02,730 on the wrist and the target object right in front 134 00:06:02,730 --> 00:06:05,550 of it. Exactly. The control objective simplifies 135 00:06:05,550 --> 00:06:08,009 down to driving the camera to a very specific 136 00:06:08,009 --> 00:06:11,470 desired pose relative to the object. If the camera 137 00:06:11,470 --> 00:06:14,129 reaches that desired pose, then by definition, 138 00:06:14,290 --> 00:06:16,490 the gripper attached to it is in the perfect 139 00:06:16,490 --> 00:06:18,870 position to grasp the object. That makes total 140 00:06:18,870 --> 00:06:21,970 sense. And to achieve this, Cork divides the 141 00:06:21,970 --> 00:06:25,769 entire field of visual servoing into two massive 142 00:06:25,769 --> 00:06:29,029 competing paradigms, right? He does. Position 143 00:06:29,029 --> 00:06:32,209 -based visual servoing, or PBVS, and image -based 144 00:06:32,209 --> 00:06:35,800 visual servoing, IBVS. Let's attack position 145 00:06:35,800 --> 00:06:38,639 -based visual servo wing first. PBVS feels like 146 00:06:38,639 --> 00:06:41,199 the most natural evolution if you are coming 147 00:06:41,199 --> 00:06:43,500 straight out of traditional kinematics. It's 148 00:06:43,500 --> 00:06:47,160 essentially thinking in 3D. Right. We are still 149 00:06:47,160 --> 00:06:49,300 operating in Cartesian space, but we are using 150 00:06:49,300 --> 00:06:51,980 the camera to continuously update our understanding 151 00:06:51,980 --> 00:06:54,379 of that space. So how does the control loop actually 152 00:06:54,379 --> 00:06:58,240 work in PBVS? Well, it relies heavily on 3D pose 153 00:06:58,240 --> 00:07:00,540 estimation. The camera captures an image of the 154 00:07:00,540 --> 00:07:03,110 target. The computer vision algorithms extract 155 00:07:03,110 --> 00:07:05,870 specific features, say, the four corners of a 156 00:07:05,870 --> 00:07:08,689 machined block on a table. OK. Then the system 157 00:07:08,689 --> 00:07:11,689 feeds those 2D pixel coordinates into an estimation 158 00:07:11,689 --> 00:07:14,589 algorithm alongside two non -negotiable pieces 159 00:07:14,589 --> 00:07:17,129 of prior knowledge. Right. You need the camera 160 00:07:17,129 --> 00:07:20,709 intrinsic matrix, the focal length, optical center, 161 00:07:20,829 --> 00:07:22,610 and distortion coefficients we covered in part 162 00:07:22,610 --> 00:07:25,290 four, so you know exactly how the lens spends 163 00:07:25,290 --> 00:07:28,269 light. Exactly. And crucially, you need a highly 164 00:07:28,269 --> 00:07:31,430 accurate 3D CAD model of the target object. You 165 00:07:31,430 --> 00:07:33,290 have to know the exact physical distances between 166 00:07:33,290 --> 00:07:35,930 those four corners in the real world. So using 167 00:07:35,930 --> 00:07:39,009 the known 3D geometry and the calibrated camera 168 00:07:39,009 --> 00:07:43,069 model, algorithms like perspective endpoint or 169 00:07:43,069 --> 00:07:46,509 PNP can mathematically reconstruct the exact 170 00:07:46,509 --> 00:07:48,990 3D spatial relationship between the camera and 171 00:07:48,990 --> 00:07:51,949 the object. Spot on. The robot looks at the skewed 172 00:07:51,949 --> 00:07:54,529 quadrilateral on its 2D image sensor, crunches 173 00:07:54,529 --> 00:07:57,129 the geometry, and determines to produce this 174 00:07:57,129 --> 00:07:59,589 specific projection, the block must be exactly 175 00:07:59,589 --> 00:08:02,629 42 centimeters away, shifted 5 centimeters to 176 00:08:02,629 --> 00:08:05,110 the left, and rotated 15 degrees around the z 177 00:08:05,110 --> 00:08:09,110 -axis. So it reconstructs a localized SE3 transformation 178 00:08:09,110 --> 00:08:12,269 matrix on the fly. It builds a localized 3D map. 179 00:08:12,550 --> 00:08:14,649 And because it does this at the frame rate of 180 00:08:14,649 --> 00:08:17,600 the camera, so... 30 times a second, it generates 181 00:08:17,600 --> 00:08:20,399 a continuous stream of Cartesian error data. 182 00:08:20,959 --> 00:08:23,600 The control law then operates entirely in the 183 00:08:23,600 --> 00:08:26,139 3D task space. It calculates the difference between 184 00:08:26,139 --> 00:08:28,579 its current estimated 3D pose and its desired 185 00:08:28,579 --> 00:08:31,680 3D pose. Right. So to make that concrete, the 186 00:08:31,680 --> 00:08:35,000 desired pose might be hovering exactly 10 centimeters 187 00:08:35,000 --> 00:08:38,679 above the block. Perfectly frontoparallel. But 188 00:08:38,679 --> 00:08:41,639 the current pose is 42 centimeters away and tilted. 189 00:08:41,940 --> 00:08:44,320 So the controller calculates the 3D translation 190 00:08:44,320 --> 00:08:46,799 vector. and the rotation matrix required to bridge 191 00:08:46,799 --> 00:08:50,259 that gap. Exactly. The control algorithm applies 192 00:08:50,259 --> 00:08:52,799 a proportional gain to that Cartesian error. 193 00:08:53,220 --> 00:08:55,620 At each time step, it commands a spatial velocity, 194 00:08:56,139 --> 00:08:59,379 a six degrees of freedom twist, that moves the 195 00:08:59,379 --> 00:09:01,500 end effector a fraction of the way toward the 196 00:09:01,500 --> 00:09:03,899 goal. And as it moves, the camera captures the 197 00:09:03,899 --> 00:09:07,299 new frame. It runs the PNP estimation again, 198 00:09:07,679 --> 00:09:10,980 recalculates the slightly smaller 3D error, and 199 00:09:10,980 --> 00:09:13,419 adjusts the velocity. The beauty of this is that 200 00:09:13,419 --> 00:09:15,460 it effortlessly handles a dynamic environment. 201 00:09:15,720 --> 00:09:18,419 If the conveyor belt unexpectedly lurches forward, 202 00:09:18,820 --> 00:09:21,259 shifting the block, the next frame of video simply 203 00:09:21,259 --> 00:09:23,539 results in a newly updated pose estimate. The 204 00:09:23,539 --> 00:09:25,960 robot recalculates the Cartesian error from this 205 00:09:25,960 --> 00:09:28,620 new position and seamlessly alters its physical 206 00:09:28,620 --> 00:09:31,620 trajectory to intercept it. It acts as an incredibly 207 00:09:31,620 --> 00:09:34,240 robust feedback mechanism against environmental 208 00:09:34,240 --> 00:09:36,539 perturbations. And what's the defining characteristic 209 00:09:36,539 --> 00:09:40,379 of PBVS? Why do engineers love it? It's the trajectory 210 00:09:40,379 --> 00:09:43,039 it produces in the physical workspace. Because 211 00:09:43,039 --> 00:09:46,039 the error is calculated and minimized in SE3 212 00:09:46,039 --> 00:09:48,759 Cartesian space, the resulting physical movement 213 00:09:48,759 --> 00:09:51,120 of the end effector is a perfectly efficient 214 00:09:51,120 --> 00:09:53,059 straight Cartesian line from the starting point 215 00:09:53,059 --> 00:09:56,159 to the goal. It doesn't take weird swooping arcs. 216 00:09:56,440 --> 00:09:59,100 The translation and orientation interpolate smoothly 217 00:09:59,100 --> 00:10:01,779 and directly. The trajectory in the 3D workspace 218 00:10:01,779 --> 00:10:05,440 is optimal. But, and this is a huge but, If we 219 00:10:05,440 --> 00:10:07,720 analyze the system at its operational extremes, 220 00:10:08,419 --> 00:10:11,840 PBVS harbors a severe vulnerability. Here we 221 00:10:11,840 --> 00:10:15,000 go. It optimizes for physical workspace efficiency 222 00:10:15,000 --> 00:10:17,600 with complete disregard for the sensor generating 223 00:10:17,600 --> 00:10:20,879 its data. I can see where this breaks down. If 224 00:10:20,879 --> 00:10:23,259 the math perfectly calculates the 3D path, this 225 00:10:23,259 --> 00:10:25,059 sounds like a solved problem, but what's the 226 00:10:25,059 --> 00:10:28,149 catch? The control law is blindly demanding a 227 00:10:28,149 --> 00:10:30,529 straight -line path for the end effector in 3D 228 00:10:30,529 --> 00:10:33,190 space. But to execute that straight path while 229 00:10:33,190 --> 00:10:36,009 simultaneously fixing a large rotational error, 230 00:10:36,389 --> 00:10:39,250 the robot's wrist might have to perform a massive 231 00:10:39,250 --> 00:10:41,870 sweeping maneuver. Right. And as the wrist swings, 232 00:10:42,070 --> 00:10:44,809 the camera, rigidly attached to it, swings violently 233 00:10:44,809 --> 00:10:47,809 as well. Which means the target object could 234 00:10:47,809 --> 00:10:50,149 easily slide right off the edge of the camera's 235 00:10:50,149 --> 00:10:52,669 image sensor. The robot essentially gives itself 236 00:10:52,669 --> 00:10:55,370 tunnel vision. It is so focused on the perfect 237 00:10:55,370 --> 00:10:57,909 3D path that it accidentally looks away from 238 00:10:57,909 --> 00:11:00,529 the very object it's trying to track. That is 239 00:11:00,529 --> 00:11:03,149 the fatal flaw of position -based visual servoing. 240 00:11:03,549 --> 00:11:05,889 The moment those tracked visual features leave 241 00:11:05,889 --> 00:11:07,950 the limited field of view of the camera, the 242 00:11:07,950 --> 00:11:10,850 system collapses. Because the PNP pose estimation 243 00:11:10,850 --> 00:11:14,090 algorithm requires those specific points. Without 244 00:11:14,090 --> 00:11:17,309 them, the algorithm fails to converge, the SE3 245 00:11:17,309 --> 00:11:19,190 error calculation throws an exception, and the 246 00:11:19,190 --> 00:11:21,809 whole system crashes. The robot stops dead in 247 00:11:21,809 --> 00:11:24,470 its tracks. It's incredibly brittle to field 248 00:11:24,470 --> 00:11:26,570 of view constraints. And even if the features 249 00:11:26,570 --> 00:11:29,649 stay in the frame, if the camera isn't perfectly 250 00:11:29,649 --> 00:11:32,730 calibrated, or if the 3D CAD model of the object 251 00:11:32,730 --> 00:11:35,929 has a slight dimensional error. The pose estimation 252 00:11:35,929 --> 00:11:39,009 will output a warped, inaccurate 3D coordinate. 253 00:11:39,509 --> 00:11:42,730 The robot will confidently drive itself to the 254 00:11:42,730 --> 00:11:45,090 wrong location because the entire system is built 255 00:11:45,090 --> 00:11:47,889 on a house of cards. The assumption of perfect 256 00:11:47,889 --> 00:11:50,919 3D reconstruction. which necessitates the paradigm 257 00:11:50,919 --> 00:11:54,379 shift introduced in Chapter 15. If reconstructing 258 00:11:54,379 --> 00:11:56,960 the 3D world is computationally expensive, highly 259 00:11:56,960 --> 00:11:59,139 sensitive to calibration errors, and prone to 260 00:11:59,139 --> 00:12:01,879 losing features off the sensor edge, what if 261 00:12:01,879 --> 00:12:04,940 we simply stop trying to calculate 3D space altogether? 262 00:12:05,419 --> 00:12:08,259 This is where we cross into image -based visual 263 00:12:08,259 --> 00:12:14,679 surveying, IBVS. If PBVS is thinking in 3D, IBVS 264 00:12:14,679 --> 00:12:17,100 is the radical commitment to thinking entirely 265 00:12:17,100 --> 00:12:21,620 in 2D. We abandon the SE3 pose estimation. We 266 00:12:21,620 --> 00:12:24,220 throw out the CAD model of the object. We stop 267 00:12:24,220 --> 00:12:26,740 calculating distances and meters and angles and 268 00:12:26,740 --> 00:12:29,580 radians. We define the entire robotic control 269 00:12:29,580 --> 00:12:32,179 problem implicitly in the pixel coordinate space 270 00:12:32,179 --> 00:12:34,679 of the image sensor. Let's return to the target 271 00:12:34,679 --> 00:12:37,720 object with the four distinct corners. How does 272 00:12:37,720 --> 00:12:42,220 IBVS handle that? In IBVS, you first manually 273 00:12:42,220 --> 00:12:44,840 maneuver the robot to the perfect grasping position. 274 00:12:45,039 --> 00:12:48,220 You take a snapshot with a wrist camera. In that 275 00:12:48,220 --> 00:12:51,179 snapshot, those four corners land on very specific 276 00:12:51,179 --> 00:12:54,100 pixel coordinates. Let's say, U and V coordinates 277 00:12:54,100 --> 00:12:57,759 on the image plane. Maybe they form a 200 by 278 00:12:57,759 --> 00:13:00,419 200 pixel square, perfectly centered on the sensor. 279 00:13:00,720 --> 00:13:03,220 That set of 2D coordinates becomes your desired 280 00:13:03,220 --> 00:13:06,320 state, your reference image. Now you move the 281 00:13:06,320 --> 00:13:09,299 robot away. The target object appears skewed, 282 00:13:09,720 --> 00:13:11,960 smaller, and off -center in the current camera 283 00:13:11,960 --> 00:13:14,379 view. The four corners have entirely different 284 00:13:14,379 --> 00:13:17,889 UV pixel coordinates. Right. And the error signal 285 00:13:17,889 --> 00:13:20,190 driving the robot is no longer a 3D distance. 286 00:13:20,929 --> 00:13:22,769 The error is strictly the difference between 287 00:13:22,769 --> 00:13:25,129 where the pixels are on the sensor right now 288 00:13:25,129 --> 00:13:28,990 and where you want those pixels to be. It's literally 289 00:13:28,990 --> 00:13:31,809 an array of 2D coordinate subtractions. So the 290 00:13:31,809 --> 00:13:34,610 controller's sole objective is to compute a 3D 291 00:13:34,610 --> 00:13:36,690 camera velocity that will drag those features 292 00:13:36,690 --> 00:13:38,690 across the image plane until they settle into 293 00:13:38,690 --> 00:13:41,429 their desired pixel locations. Conceptually, 294 00:13:41,429 --> 00:13:43,429 it's like playing a first -person shooter video 295 00:13:43,429 --> 00:13:45,990 game where you are just trying to line up a crosshair 296 00:13:45,990 --> 00:13:48,889 on a target, totally ignoring the fact that the 297 00:13:48,889 --> 00:13:51,549 crosshair is physically attached to a one -ton 298 00:13:51,549 --> 00:13:54,149 industrial robotic arm moving through physical 299 00:13:54,149 --> 00:13:57,090 space. You only care about the screen. But mathematically, 300 00:13:57,250 --> 00:13:59,769 bridging that gap, calculating how to move a 301 00:13:59,769 --> 00:14:02,809 six -axis physical machine based purely on 2D 302 00:14:02,809 --> 00:14:05,549 pixel drift, that requires some heavy lifting. 303 00:14:05,950 --> 00:14:08,110 It requires the cornerstone of visual control 304 00:14:08,110 --> 00:14:11,620 theory, the image Jacobian. also widely referred 305 00:14:11,620 --> 00:14:13,879 to in the literature as the interaction matrix, 306 00:14:14,580 --> 00:14:16,940 Cork devotes a huge portion of the text to deriving 307 00:14:16,940 --> 00:14:19,679 and analyzing the specific matrix, usually denoted 308 00:14:19,679 --> 00:14:23,059 as L. Let's dissect this matrix because it is 309 00:14:23,059 --> 00:14:25,200 the translation engine for the whole system. 310 00:14:25,679 --> 00:14:28,519 What exactly is the image Jacobian doing mathematically? 311 00:14:29,139 --> 00:14:31,220 The image Jacobian maps the spatial velocity 312 00:14:31,220 --> 00:14:33,740 of the camera in the 3D world to the resulting 313 00:14:33,740 --> 00:14:36,940 velocity of a pixel on the 2D image plane. Okay, 314 00:14:37,059 --> 00:14:39,690 break that down for me. Sure. The spatial velocity 315 00:14:39,690 --> 00:14:42,529 of the camera, let's call it nu, is a six -dimensional 316 00:14:42,529 --> 00:14:44,950 vector. It consists of the linear velocities 317 00:14:44,950 --> 00:14:49,350 along the x, y, and z axes, vx, vy, vz, and the 318 00:14:49,350 --> 00:14:52,509 angular velocities around those axes, omega x, 319 00:14:52,549 --> 00:14:55,289 omega y, omega z. Right. Six degrees of freedom 320 00:14:55,289 --> 00:14:57,789 for the camera's physical movement. The resulting 321 00:14:57,789 --> 00:15:00,970 image velocity, let's call it p dot, is a two 322 00:15:00,970 --> 00:15:03,190 -dimensional vector for a single tracked point. 323 00:15:03,629 --> 00:15:06,230 It's the rate of change of the pixel's u coordinate, 324 00:15:06,330 --> 00:15:09,250 horizontal motion on the screen, and its V coordinate, 325 00:15:09,330 --> 00:15:11,850 vertical motion, on the screen. So for a single 326 00:15:11,850 --> 00:15:15,070 point, the image Jacobian is a 2 by 6 matrix. 327 00:15:15,789 --> 00:15:20,490 The equation is P dot equals L times nu. Exactly. 328 00:15:20,830 --> 00:15:23,649 If the robot translates left at 1 meter per second, 329 00:15:23,909 --> 00:15:26,230 multiplying that spatial velocity vector by the 330 00:15:26,230 --> 00:15:29,490 Jacobian matrix outputs a pixel velocity, telling 331 00:15:29,490 --> 00:15:31,549 you the feature will slide right across the sensor 332 00:15:31,549 --> 00:15:34,409 at, say, 50 pixels per second. It's formalizing 333 00:15:34,409 --> 00:15:37,580 optical flow. Precisely. And Cork's derivation 334 00:15:37,580 --> 00:15:40,179 of the matrix elements reveals the deep coupling 335 00:15:40,179 --> 00:15:43,039 between physical motion and optical phenomena. 336 00:15:43,539 --> 00:15:45,620 If we examine the columns of the Jacobian, we 337 00:15:45,620 --> 00:15:48,000 can mathematically predict the optical flow fields 338 00:15:48,000 --> 00:15:50,700 generated by different camera motions. The translational 339 00:15:50,700 --> 00:15:54,039 columns are absolutely fascinating. If you command 340 00:15:54,039 --> 00:15:57,259 a purely positive Z velocity, pushing the camera 341 00:15:57,259 --> 00:16:00,220 straight forward along its optical axis, the 342 00:16:00,220 --> 00:16:02,419 math shows that every feature point on the image 343 00:16:02,419 --> 00:16:05,100 sensor will move radially away from the principal 344 00:16:05,100 --> 00:16:08,090 point. the dead center of the image. And the 345 00:16:08,090 --> 00:16:10,950 magnitude of that radial pixel velocity increases 346 00:16:10,950 --> 00:16:13,450 the further the point is from the center. It's 347 00:16:13,450 --> 00:16:16,090 the classic Star Trek warp speed effect. The 348 00:16:16,090 --> 00:16:17,809 stars in the center of the screen barely move 349 00:16:17,809 --> 00:16:19,889 while the stars on the edge of the screen streak 350 00:16:19,889 --> 00:16:23,149 past you rapidly. That visual phenomenon is literally 351 00:16:23,149 --> 00:16:25,210 hard -coded into the third column of the image 352 00:16:25,210 --> 00:16:29,970 Jacobian. It is. Similarly, a pure angular velocity 353 00:16:29,970 --> 00:16:32,409 around the z -axis, rolling the camera like a 354 00:16:32,409 --> 00:16:34,970 steering wheel, creates a rotational flow field. 355 00:16:35,179 --> 00:16:37,600 The Jacobian shows the pixels will travel in 356 00:16:37,600 --> 00:16:39,700 concentric circular arcs around the center of 357 00:16:39,700 --> 00:16:42,200 the image. OK, but as Corky analyzes the columns, 358 00:16:42,659 --> 00:16:44,720 a severe mathematical vulnerability emerges, 359 00:16:44,879 --> 00:16:48,059 right? We call it motion ambiguity. Yes. Let's 360 00:16:48,059 --> 00:16:50,039 drill into that. Look at the first column of 361 00:16:50,039 --> 00:16:52,419 the Jacobian, which maps translation along the 362 00:16:52,419 --> 00:16:54,960 x -axis, basically sliding the camera horizontally. 363 00:16:55,559 --> 00:16:58,120 Now look at the fifth column, which maps rotation 364 00:16:58,120 --> 00:17:00,360 around the y -axis, panning the camera left or 365 00:17:00,360 --> 00:17:03,149 right, like turning your head. OK. For features 366 00:17:03,149 --> 00:17:05,609 located near the center of the image sensor, 367 00:17:05,829 --> 00:17:08,269 the mathematical terms in those two columns become 368 00:17:08,269 --> 00:17:11,549 nearly identical. OK, so the mapping from a 3D 369 00:17:11,549 --> 00:17:13,730 slide and a 3D pan result in almost the exact 370 00:17:13,730 --> 00:17:16,789 same 2D pixel motion? The pixels just slide horizontally 371 00:17:16,789 --> 00:17:20,880 across the screen in both scenarios. If the camera 372 00:17:20,880 --> 00:17:23,839 has a standard or narrow field of view, the visual 373 00:17:23,839 --> 00:17:26,579 flow fields generated by an X translation and 374 00:17:26,579 --> 00:17:29,279 a Y rotation are virtually indistinguishable. 375 00:17:29,500 --> 00:17:32,000 It's the equivalent of looking out a narrow window 376 00:17:32,000 --> 00:17:34,619 on an incredibly smooth high -speed train. If 377 00:17:34,619 --> 00:17:36,680 the scenery is slowly drifting to the right across 378 00:17:36,680 --> 00:17:38,960 your window, your brain really struggles to determine 379 00:17:38,960 --> 00:17:41,980 the cause. Is the entire train sliding laterally 380 00:17:41,980 --> 00:17:44,400 sideways on the tracks, or is the train simply 381 00:17:44,400 --> 00:17:47,039 navigating a very long, slow curve, changing 382 00:17:47,039 --> 00:17:50,039 its orientation? The 2D visual input projected 383 00:17:50,039 --> 00:17:52,579 onto your retina is nearly identical for both 384 00:17:52,579 --> 00:17:55,869 cases. And that ambiguity translates into severe 385 00:17:55,869 --> 00:17:58,769 problems for the matrix conditioning. When columns 386 00:17:58,769 --> 00:18:01,829 of a matrix are nearly linearly dependent, the 387 00:18:01,829 --> 00:18:04,309 matrix is ill conditioned. Meaning if the control 388 00:18:04,309 --> 00:18:07,730 system sees a pixel sliding left, it struggles 389 00:18:07,730 --> 00:18:10,230 to determine whether it should command an X translation 390 00:18:10,230 --> 00:18:13,250 or a Y rotation to correct it. Right. And it 391 00:18:13,250 --> 00:18:15,930 can lead to unstable oscillating control signals 392 00:18:15,930 --> 00:18:18,609 where the robot basically flails between sliding 393 00:18:18,609 --> 00:18:20,690 and panning, trying to find the true source of 394 00:18:20,690 --> 00:18:22,839 the error. And that's just the ambiguity between 395 00:18:22,839 --> 00:18:25,839 specific movements. There's a much larger fundamental 396 00:18:25,839 --> 00:18:28,740 mathematical problem here. The image Jacobian 397 00:18:28,740 --> 00:18:31,279 is mapping a six -dimensional spatial velocity 398 00:18:31,279 --> 00:18:34,539 down to a two -dimensional pixel velocity. You 399 00:18:34,539 --> 00:18:37,220 are compressing a massive amount of spatial information 400 00:18:37,220 --> 00:18:40,619 through a tiny dimensional bottleneck. From 6D 401 00:18:40,619 --> 00:18:43,059 down to 2D, you are mathematically guaranteed 402 00:18:43,059 --> 00:18:45,440 to lose information. This introduces the concept 403 00:18:45,440 --> 00:18:48,279 of the null space. The null space being the set 404 00:18:48,279 --> 00:18:50,680 of all input vectors that result in an output 405 00:18:50,680 --> 00:18:54,259 of zero. Exactly. For a single tracked point, 406 00:18:54,400 --> 00:18:57,359 the Jacobian matrix is two by six. Assuming the 407 00:18:57,359 --> 00:18:59,259 point isn't exactly at the origin, the matrix 408 00:18:59,259 --> 00:19:02,799 has a rank of two. By the rank nullity theorem, 409 00:19:03,180 --> 00:19:05,859 a two by six matrix with rank two must have a 410 00:19:05,859 --> 00:19:08,279 null space of dimension four. Meaning there are 411 00:19:08,279 --> 00:19:11,279 four distinct combinations of 3D spatial velocity 412 00:19:11,279 --> 00:19:14,640 that will cause absolutely zero movement of that 413 00:19:14,640 --> 00:19:17,500 pixel on the screen. Right. If you are tracking 414 00:19:17,500 --> 00:19:19,839 a single dot in the dead center of the camera 415 00:19:19,839 --> 00:19:22,539 frame, and you command the robot to move straight 416 00:19:22,539 --> 00:19:25,480 forward along the z -axis, the dot remains perfectly 417 00:19:25,480 --> 00:19:28,059 centered. The pixel velocity is zero. Right. 418 00:19:28,220 --> 00:19:30,579 It just gets closer. If you command the robot 419 00:19:30,579 --> 00:19:33,140 to roll around the z -axis, spinning the camera, 420 00:19:33,859 --> 00:19:37,029 that center dot simply rotates in place. As a 421 00:19:37,029 --> 00:19:39,670 mathematical point has no dimensions, its pixel 422 00:19:39,670 --> 00:19:42,329 coordinates do not change. So if you try to build 423 00:19:42,329 --> 00:19:44,970 a visual servo controller for a six -axis robotic 424 00:19:44,970 --> 00:19:48,690 arm using only one tracked point, the system 425 00:19:48,690 --> 00:19:51,970 is catastrophically under constraint. The robot 426 00:19:51,970 --> 00:19:54,529 could be physically plunging forward while violently 427 00:19:54,529 --> 00:19:56,970 spinning, completely destroying the workspace, 428 00:19:57,170 --> 00:20:00,329 and the camera would report zero visual error. 429 00:20:00,970 --> 00:20:02,589 And the controller wouldn't even try to stop 430 00:20:02,589 --> 00:20:04,529 it because from the perspective of that single 431 00:20:04,529 --> 00:20:08,170 pixel, Everything is perfect. Wow. So how do 432 00:20:08,170 --> 00:20:11,049 we stabilize the control law? To stabilize it, 433 00:20:11,130 --> 00:20:13,650 we must eliminate the null space. We do this 434 00:20:13,650 --> 00:20:15,809 by tracking multiple features simultaneously. 435 00:20:16,490 --> 00:20:18,809 We vertically stack their individual Jacobian 436 00:20:18,809 --> 00:20:21,750 matrices to create a larger combined interaction 437 00:20:21,750 --> 00:20:24,789 matrix. If we track two points, we get a four 438 00:20:24,789 --> 00:20:27,750 by six matrix. The rank increases, but we still 439 00:20:27,750 --> 00:20:30,420 have a null space of dimension two. The camera 440 00:20:30,420 --> 00:20:32,640 could still physically rotate around the invisible 441 00:20:32,640 --> 00:20:35,220 cylindrical axis, connecting those two points 442 00:20:35,220 --> 00:20:37,799 in 3D space, and the two pixels on the screen 443 00:20:37,799 --> 00:20:41,049 wouldn't shift. To achieve a full rank matrix, 444 00:20:41,170 --> 00:20:43,609 meaning a rank of six, where the null space is 445 00:20:43,609 --> 00:20:46,390 reduced to zero dimension, we need to track a 446 00:20:46,390 --> 00:20:49,250 minimum of three non -colinear points, three 447 00:20:49,250 --> 00:20:51,470 points that do not lie on a single straight line. 448 00:20:51,869 --> 00:20:55,710 Which generates a six by six Jacobian. Only then 449 00:20:55,710 --> 00:20:58,670 does every possible physical 6D movement of the 450 00:20:58,670 --> 00:21:00,849 camera result in a completely unique combination 451 00:21:00,849 --> 00:21:04,250 of 2D pixel movements. The mapping becomes rigid. 452 00:21:05,109 --> 00:21:07,329 Okay, so once we have a full -ranked stacked 453 00:21:07,329 --> 00:21:09,789 Jacobian, we can finally close the control loop. 454 00:21:10,470 --> 00:21:12,990 Remember, the forward Jacobian equation is P 455 00:21:12,990 --> 00:21:16,589 dot equals L times nu. It tells us pixel velocity 456 00:21:16,589 --> 00:21:19,940 given camera velocity. But for control, we need 457 00:21:19,940 --> 00:21:22,640 the inverse. We know the pixel error on the screen 458 00:21:22,640 --> 00:21:25,059 and we want to compute the commanded camera velocity 459 00:21:25,059 --> 00:21:28,049 to fix it. We need to invert the matrix to solve 460 00:21:28,049 --> 00:21:32,890 for nu. So nu equals L inverse times p dot. Yes, 461 00:21:32,890 --> 00:21:35,089 but in practice we usually track far more than 462 00:21:35,089 --> 00:21:37,349 three points to increase robustness against noise. 463 00:21:37,809 --> 00:21:40,650 Maybe we track four, ten, or even fifty points. 464 00:21:40,950 --> 00:21:43,309 This makes the stacked Jacobian matrix tall and 465 00:21:43,309 --> 00:21:46,349 rectangular, say eight by six or a hundred by 466 00:21:46,349 --> 00:21:48,859 six. And you cannot directly invert a non -square 467 00:21:48,859 --> 00:21:52,779 matrix. You cannot. So we use the Mohr -Penrose 468 00:21:52,779 --> 00:21:57,099 pseudo -inverse, often denoted as L+. We multiply 469 00:21:57,099 --> 00:21:59,839 the transpose of the Jacobian by the original 470 00:21:59,839 --> 00:22:03,339 Jacobian to create a square 6x6 matrix, invert 471 00:22:03,339 --> 00:22:06,000 that, and then multiply by the transpose again. 472 00:22:06,640 --> 00:22:09,500 The formula being L +, equals the inverse of 473 00:22:09,500 --> 00:22:12,640 L transpose times L, all multiplied by L transpose. 474 00:22:13,019 --> 00:22:15,460 That mathematical operation provides the least 475 00:22:15,460 --> 00:22:17,619 squares optimal solution for the overdetermined 476 00:22:17,619 --> 00:22:20,450 system. We multiply the pseudo -inverse of the 477 00:22:20,450 --> 00:22:23,329 Jacobian by our vector of pixel errors, apply 478 00:22:23,329 --> 00:22:25,829 a negative proportional gain, usually lambda, 479 00:22:26,349 --> 00:22:29,029 and the output is the optimal 6 degree of freedom 480 00:22:29,029 --> 00:22:31,769 spatial velocity command for the robot. So nu 481 00:22:31,769 --> 00:22:36,170 sub c equals negative lambda times L plus times 482 00:22:36,170 --> 00:22:40,990 E. It's an incredibly elegant control law. We 483 00:22:40,990 --> 00:22:43,309 figure out how we want the pixels to slide across 484 00:22:43,309 --> 00:22:45,369 the screen to reach their target coordinates, 485 00:22:46,069 --> 00:22:48,950 we run that desired 2D flow backward through 486 00:22:48,950 --> 00:22:52,369 the pseudo -inverse matrix, and out pops the 487 00:22:52,369 --> 00:22:54,670 3D motor speeds required to make it happen. It's 488 00:22:54,670 --> 00:22:59,490 beautiful. But, hold on. If we actually look 489 00:22:59,490 --> 00:23:02,230 at the specific algebraic terms inside Cork's 490 00:23:02,230 --> 00:23:05,410 interaction matrix, there's a massive glaring 491 00:23:05,410 --> 00:23:08,750 contradiction to the entire philosophy of IBVS. 492 00:23:08,880 --> 00:23:13,039 Ah, you found the depth paradox. Yes. We established 493 00:23:13,039 --> 00:23:15,480 that the whole point of IBVS is the completely 494 00:23:15,480 --> 00:23:18,680 abandoned 3D space. We don't do pose estimation. 495 00:23:19,160 --> 00:23:21,619 We don't care how far away the object is in meters. 496 00:23:22,140 --> 00:23:25,319 We only care about pixel coordinates u and v. 497 00:23:25,619 --> 00:23:29,599 Right. But, if you look at the first three columns 498 00:23:29,599 --> 00:23:32,500 of the Jacobian, the columns handling translation, 499 00:23:33,480 --> 00:23:37,140 every single term has the variable z in the denominator. 500 00:23:37,559 --> 00:23:41,220 And in Z represents the depth. The physical distance 501 00:23:41,220 --> 00:23:43,799 from the camera focal point to the target feature 502 00:23:43,799 --> 00:23:47,180 in the 3D world. So the math for mapping spatial 503 00:23:47,180 --> 00:23:50,180 velocity to optical flow explicitly requires 504 00:23:50,180 --> 00:23:53,240 knowing the depth of the features. I mean, it 505 00:23:53,240 --> 00:23:55,799 makes physical sense. If the camera translates 506 00:23:55,799 --> 00:23:58,640 sideways at one meter per second, a feature that 507 00:23:58,640 --> 00:24:01,240 is one meter away will streak rapidly across 508 00:24:01,240 --> 00:24:04,099 the sensor, generating a massive pixel velocity. 509 00:24:04,519 --> 00:24:07,400 But a feature that is a mountain 10 kilometers 510 00:24:07,400 --> 00:24:10,160 away will barely shift a single pixel. Exactly. 511 00:24:10,339 --> 00:24:12,859 The Jacobian absolutely must have Z to scale 512 00:24:12,859 --> 00:24:15,440 the translational flow correctly. It's an inescapable 513 00:24:15,440 --> 00:24:18,420 geometric reality. But doesn't that entirely 514 00:24:18,420 --> 00:24:21,400 defeat the purpose of IBVS? If we have to know 515 00:24:21,400 --> 00:24:24,200 Z to populate the interaction matrix and calculate 516 00:24:24,200 --> 00:24:27,420 the pseudo -inverse, we are back to doing 3D 517 00:24:27,420 --> 00:24:29,960 pose estimation. We need a 3D map again to get 518 00:24:29,960 --> 00:24:32,630 the depth of every pixel. And this is where the 519 00:24:32,630 --> 00:24:35,650 sheer robustness of closed -loop visual control 520 00:24:35,650 --> 00:24:38,190 reveals its power. Cork demonstrates that while 521 00:24:38,190 --> 00:24:41,369 the matrix mathematically requires z, the control 522 00:24:41,369 --> 00:24:44,430 loop as a whole is shockingly resilient to terrible 523 00:24:44,430 --> 00:24:46,549 estimates of z. How terrible are we talking? 524 00:24:47,029 --> 00:24:50,190 You don't need to run complex 3D pose estimation 525 00:24:50,190 --> 00:24:53,269 algorithms to find the true z. You can simply 526 00:24:53,269 --> 00:24:56,150 hard -code a constant, wildly inaccurate guess 527 00:24:56,150 --> 00:24:58,630 for the depth into the matrix, and the system 528 00:24:58,630 --> 00:25:01,470 will still successfully complete the task. That 529 00:25:01,470 --> 00:25:03,730 sounds impossible. If the matrix is populated 530 00:25:03,730 --> 00:25:06,089 with the wrong depth, the pseudo -inverse is 531 00:25:06,089 --> 00:25:08,490 wrong, which means the spatial velocity vector 532 00:25:08,490 --> 00:25:10,809 commanded to the robot is mathematically incorrect. 533 00:25:11,369 --> 00:25:14,670 The magnitude of the commanded velocity will 534 00:25:14,670 --> 00:25:17,970 be incorrect, yes, but the direction of the gradient 535 00:25:17,970 --> 00:25:20,349 descent, the general vector pushing the pixels 536 00:25:20,349 --> 00:25:23,849 toward their goal, remains overwhelmingly correct. 537 00:25:24,289 --> 00:25:26,410 Cork runs a simulation to prove this. Okay, walk 538 00:25:26,410 --> 00:25:29,309 me through the simulation. The robot's end effector 539 00:25:29,309 --> 00:25:31,410 is looking at a target that is physically three 540 00:25:31,410 --> 00:25:34,049 meters away. He programs the IBVS controller 541 00:25:34,049 --> 00:25:37,250 with a hard -coded assumed depth of z equals 542 00:25:37,250 --> 00:25:39,970 one meter. So the controller operates into the 543 00:25:39,970 --> 00:25:42,990 delusion that the object is three times closer 544 00:25:42,990 --> 00:25:45,390 than it actually is. The robot computes the Jacobian 545 00:25:45,390 --> 00:25:48,829 using z equals one, inverses it, and commands 546 00:25:48,829 --> 00:25:51,410 a velocity. Because the controller thinks the 547 00:25:51,410 --> 00:25:54,190 target is very close, it assumes a small physical 548 00:25:54,190 --> 00:25:57,170 translation will result in a massive pixel shift. 549 00:25:57,839 --> 00:26:00,579 OK. Therefore, to avoid overshooting the pixel 550 00:26:00,579 --> 00:26:03,279 target, it commands a very, very small physical 551 00:26:03,279 --> 00:26:06,200 velocity. But because the object is actually 552 00:26:06,200 --> 00:26:08,900 three meters away, that tiny physical movement 553 00:26:08,900 --> 00:26:10,859 barely shifts the pixels on the screen at all. 554 00:26:11,339 --> 00:26:14,759 Exactly. OK. The error barely decreases. But, 555 00:26:14,880 --> 00:26:16,799 and this is the crucial part, the error does 556 00:26:16,799 --> 00:26:19,579 decrease. The pixels moved slightly in the correct 557 00:26:19,579 --> 00:26:21,920 direction. Oh, I see. The system captures the 558 00:26:21,920 --> 00:26:24,799 next frame, sees the error is still large, runs 559 00:26:24,799 --> 00:26:26,960 the flawed math again, and commands another tiny 560 00:26:26,960 --> 00:26:29,880 velocity. The robot effectively crawls toward 561 00:26:29,880 --> 00:26:32,900 the target, exponentially slowly, but it does 562 00:26:32,900 --> 00:26:35,380 ultimately converge. And what if we guess in 563 00:26:35,380 --> 00:26:37,359 the opposite direction? What if we hard code 564 00:26:37,359 --> 00:26:41,059 the matrix with an assumed depth of z equals 565 00:26:41,059 --> 00:26:43,279 10 meters, when the object is only three meters 566 00:26:43,279 --> 00:26:45,240 away? Now the controller thinks it's looking 567 00:26:45,240 --> 00:26:47,369 at something distant. It assumes it requires 568 00:26:47,369 --> 00:26:49,789 a massive physical translation to cause even 569 00:26:49,789 --> 00:26:53,170 a tiny pixel shift. So it commands an aggressive 570 00:26:53,170 --> 00:26:55,789 high -speed velocity vector. Which causes the 571 00:26:55,789 --> 00:26:57,730 end effector to surge forward rapidly. Right. 572 00:26:57,750 --> 00:27:00,309 But because the object is actually close, that 573 00:27:00,309 --> 00:27:02,890 massive physical jump causes the pixels to absolutely 574 00:27:02,890 --> 00:27:05,710 fly across the sensor, likely overshooting the 575 00:27:05,710 --> 00:27:08,210 target coordinates entirely. It does overshoot. 576 00:27:08,690 --> 00:27:10,670 But the closed loop system instantly captures 577 00:27:10,670 --> 00:27:13,369 a new frame. registers the massive negative error, 578 00:27:13,769 --> 00:27:16,170 reverses the velocity command, and violently 579 00:27:16,170 --> 00:27:19,509 jerks the arm back. The trajectory becomes highly 580 00:27:19,509 --> 00:27:23,150 oscillatory. The robot takes aggressive, jagged, 581 00:27:23,210 --> 00:27:26,230 zigzagging steps. But the continuous visual feedback 582 00:27:26,230 --> 00:27:29,710 reigns it in. Even with a violently incorrect 583 00:27:29,710 --> 00:27:32,390 interaction matrix, the persistent error correction 584 00:27:32,390 --> 00:27:34,430 forces the robot to eventually settle on the 585 00:27:34,430 --> 00:27:38,089 target. The wrong depth estimate corrupts the 586 00:27:38,089 --> 00:27:40,869 efficiency of the 3D trajectory, making it a 587 00:27:40,869 --> 00:27:43,150 slow crawl or an aggressive oscillation, but 588 00:27:43,150 --> 00:27:46,269 it does not prevent convergence. It proves that 589 00:27:46,269 --> 00:27:49,750 prioritizing 2D visual feedback over rigid 3D 590 00:27:49,750 --> 00:27:51,970 modeling creates an incredibly fault -tolerant 591 00:27:51,970 --> 00:27:54,970 system. Now, if you want to optimize the trajectory 592 00:27:54,970 --> 00:27:58,750 and avoid the crawl or the oscillation, you don't 593 00:27:58,750 --> 00:28:01,210 have to rely on a bad guess. You can dynamically 594 00:28:01,210 --> 00:28:04,430 estimate depth on the fly using a technique called 595 00:28:04,430 --> 00:28:07,809 flow derotation. Wait, how do you extract depth 596 00:28:07,809 --> 00:28:10,930 from a 2D image without a CAD model? By analyzing 597 00:28:10,930 --> 00:28:13,289 the different properties of the Jacobian columns. 598 00:28:13,910 --> 00:28:16,829 We noted earlier that translational optical flow 599 00:28:16,829 --> 00:28:19,529 is highly dependent on depth. The Z variable 600 00:28:19,529 --> 00:28:22,190 is in the denominator. But if you look at the 601 00:28:22,190 --> 00:28:25,109 columns for rotational optical flow, spinning 602 00:28:25,109 --> 00:28:27,589 or panning the camera, the Z variable doesn't 603 00:28:27,589 --> 00:28:30,029 exist in those terms. Because rotating a camera 604 00:28:30,029 --> 00:28:32,309 purely around its focal point doesn't change 605 00:28:32,309 --> 00:28:35,289 the distance to the object. the pixels shift 606 00:28:35,289 --> 00:28:38,250 on the sensor, but that shift is completely independent 607 00:28:38,250 --> 00:28:41,630 of how far away the object is. Precisely. So 608 00:28:41,630 --> 00:28:44,109 if we know the physical angular velocity of the 609 00:28:44,109 --> 00:28:47,190 camera, perhaps from the robot's joint encoders 610 00:28:47,190 --> 00:28:50,710 or an IMU, we can calculate exactly how much 611 00:28:50,710 --> 00:28:52,869 of the pixel movement on the screen was caused 612 00:28:52,869 --> 00:28:55,769 by that rotation. We calculate the expected rotational 613 00:28:55,769 --> 00:28:58,309 flow. And we simply subtract it from the total 614 00:28:58,309 --> 00:29:00,529 observed flow on the sensor. We mathematically 615 00:29:00,529 --> 00:29:03,329 erase the effects of rotation. whatever pixel 616 00:29:03,329 --> 00:29:05,789 motion is left over must have been caused exclusively 617 00:29:05,789 --> 00:29:08,970 by the camera's translation. And since that isolated 618 00:29:08,970 --> 00:29:11,490 translational flow is directly coupled to depth 619 00:29:11,490 --> 00:29:14,210 via the Z variable in the Jacobian, we can reverse 620 00:29:14,210 --> 00:29:17,170 the algebra. We use the magnitude of the isolated 621 00:29:17,170 --> 00:29:19,910 translational pixel shift to continuously calculate 622 00:29:19,910 --> 00:29:22,869 a highly accurate estimate of Z at every time 623 00:29:22,869 --> 00:29:26,549 step. We get a dynamic depth map for free just 624 00:29:26,549 --> 00:29:30,490 by analyzing the optical flow field. So IBVS 625 00:29:30,650 --> 00:29:34,170 effortlessly survives depth errors. It explicitly 626 00:29:34,170 --> 00:29:36,089 controls where the features are on the screen 627 00:29:36,089 --> 00:29:38,289 so they never leave the camera's field of view. 628 00:29:38,809 --> 00:29:40,589 I mean, it seems like it completely solves the 629 00:29:40,589 --> 00:29:44,470 fatal flaws of PBVS. It does. But by forcefully 630 00:29:44,470 --> 00:29:47,069 constraining the control problem to the 2D image 631 00:29:47,069 --> 00:29:50,309 plane, IBVS introduces a fatal flaw of its own. 632 00:29:50,990 --> 00:29:53,289 It's a phenomenon known in the literature as 633 00:29:53,289 --> 00:29:56,150 camera retreat, and it exposes the danger of 634 00:29:56,150 --> 00:29:59,170 ignoring 3D reality entirely. Okay, when does 635 00:29:59,170 --> 00:30:02,269 camera retreat happen? It manifests most aggressively 636 00:30:02,269 --> 00:30:05,309 during tasks that require a large rotation around 637 00:30:05,309 --> 00:30:07,970 the optical axis. Let's imagine a task where 638 00:30:07,970 --> 00:30:10,569 the robot needs to unscrew a large flat lid. 639 00:30:10,970 --> 00:30:13,390 The camera is hovering directly above a target 640 00:30:13,390 --> 00:30:15,930 with four tracking dots. Okay, looking down at 641 00:30:15,930 --> 00:30:18,809 a square? You define the desired state for the 642 00:30:18,809 --> 00:30:21,710 IBVS controller. You want the robot to execute 643 00:30:21,710 --> 00:30:25,190 a 180 degree pure rotation around the z -axis. 644 00:30:25,609 --> 00:30:27,630 You want the top left dot to become the bottom 645 00:30:27,630 --> 00:30:30,000 right dot. the top right to become the bottom 646 00:30:30,000 --> 00:30:32,779 left. You want the pixel coordinates to completely 647 00:30:32,779 --> 00:30:35,920 swap sides. If we were running position -based 648 00:30:35,920 --> 00:30:38,319 visual servoing, the controller would calculate 649 00:30:38,319 --> 00:30:41,319 the 3D geometry, realize it just needs a pure 650 00:30:41,319 --> 00:30:44,299 rotation, and simply twist the robot's wrist 651 00:30:44,299 --> 00:30:47,279 180 degrees while keeping the camera at the exact 652 00:30:47,279 --> 00:30:50,259 same depth. A perfect, efficient physical motion. 653 00:30:50,410 --> 00:30:53,329 But the IBVS controller doesn't know about 3D 654 00:30:53,329 --> 00:30:57,109 risks or 3D depth. It only processes arrays of 655 00:30:57,109 --> 00:31:00,250 pixel coordinates. The linear control law dictates 656 00:31:00,250 --> 00:31:03,809 that the error must decay exponentially. It mathematically 657 00:31:03,809 --> 00:31:06,269 forces every single pixel to take the shortest 658 00:31:06,269 --> 00:31:09,230 possible path across the 2D image sensor to reach 659 00:31:09,230 --> 00:31:11,950 its new coordinate. And the shortest path between 660 00:31:11,950 --> 00:31:15,329 two coordinates on a 2D plane is a straight line. 661 00:31:15,769 --> 00:31:18,430 Exactly. So instead of the four dots traveling 662 00:31:18,430 --> 00:31:20,789 in sweeping circular arcs around the center of 663 00:31:20,789 --> 00:31:22,529 the image, which is what they would do during 664 00:31:22,529 --> 00:31:26,029 a physical wrist twist, the IBVS controller forces 665 00:31:26,029 --> 00:31:29,089 the pixels to travel in straight linear chords 666 00:31:29,089 --> 00:31:31,450 directly across the center of the screen to swap 667 00:31:31,450 --> 00:31:33,970 sides. Think about the geometry of that. If you 668 00:31:33,970 --> 00:31:36,329 have a square formed by four dots and you force 669 00:31:36,329 --> 00:31:38,569 all four dots to travel in straight lines toward 670 00:31:38,569 --> 00:31:40,329 the center of the screen to cross over to the 671 00:31:40,329 --> 00:31:43,430 other side, the distance between the dots is 672 00:31:43,430 --> 00:31:46,049 shrinking. The area of the polygon they form 673 00:31:46,049 --> 00:31:48,509 on the sensor rapidly collapses as they converge 674 00:31:48,509 --> 00:31:50,930 on the center. And what does the visual controller 675 00:31:50,930 --> 00:31:53,329 deduce when the physical area of the tracked 676 00:31:53,329 --> 00:31:55,670 features begins to rapidly shrink on the image 677 00:31:55,670 --> 00:31:58,650 sensor? The Jacobian math interprets a shrinking 678 00:31:58,650 --> 00:32:01,210 feature set as the camera physically moving too 679 00:32:01,210 --> 00:32:03,230 close to the target object. It assumes the depth 680 00:32:03,230 --> 00:32:05,990 z is decreasing dramatically. So the control 681 00:32:05,990 --> 00:32:08,700 law panics. It thinks the target is getting too 682 00:32:08,700 --> 00:32:11,640 small. I must be too close. And to counteract 683 00:32:11,640 --> 00:32:13,279 that perceived error and maintain the target 684 00:32:13,279 --> 00:32:16,400 size, it commands a massive positive velocity 685 00:32:16,400 --> 00:32:19,619 along the Z axis. It commands the robot to pull 686 00:32:19,619 --> 00:32:22,339 the camera violently backward. It retreats. As 687 00:32:22,339 --> 00:32:24,420 the camera physically pulls away from the object, 688 00:32:24,920 --> 00:32:27,940 the IBVS controller continues to force the wrist 689 00:32:27,940 --> 00:32:30,619 to twist, dragging the pixels along their straight 690 00:32:30,619 --> 00:32:34,339 2D paths. The arm pulls way back, executes the 691 00:32:34,339 --> 00:32:37,069 twisted crossover at maximum distance. as the 692 00:32:37,069 --> 00:32:38,849 dots pass through the center of the image, and 693 00:32:38,849 --> 00:32:41,230 then as the dots continue on their straight paths 694 00:32:41,230 --> 00:32:43,509 and expand outward toward the corners of the 695 00:32:43,509 --> 00:32:46,029 sensor. The controller interprets the expanding 696 00:32:46,029 --> 00:32:48,990 polygon as the target getting too far away, and 697 00:32:48,990 --> 00:32:51,269 it aggressively plunges the camera back down 698 00:32:51,269 --> 00:32:53,569 along the negative z -axis to the original depth. 699 00:32:53,849 --> 00:32:57,069 That's wild. Quark simulation of a simple 180 700 00:32:57,069 --> 00:32:59,829 -degree twist results in a bizarre parabolic 701 00:32:59,829 --> 00:33:02,289 3D trajectory where the camera retreats a full 702 00:33:02,289 --> 00:33:05,450 meter away from the workspace, twists, and then 703 00:33:05,450 --> 00:33:08,069 dives back in. Then it's absurdly inefficient. 704 00:33:08,490 --> 00:33:10,529 And frankly, in a real industrial environment, 705 00:33:10,869 --> 00:33:13,609 it's a catastrophic collision hazard. If this 706 00:33:13,609 --> 00:33:16,170 robot is operating inside a cramped CNC machining 707 00:33:16,170 --> 00:33:18,849 cell, and it decides to arbitrarily pull its 708 00:33:18,849 --> 00:33:21,190 arm back a full meter just to twist its wrist, 709 00:33:21,710 --> 00:33:23,529 it's going to smash straight through a safety 710 00:33:23,529 --> 00:33:25,589 glass enclosure or destroy an adjacent machine. 711 00:33:25,869 --> 00:33:28,269 The pure image -based approach guarantees the 712 00:33:28,269 --> 00:33:30,809 features stay in the field of view, but it routinely 713 00:33:30,809 --> 00:33:33,730 generates wild, dangerous, and often physically 714 00:33:33,730 --> 00:33:37,569 impossible 3D trajectories to satisfy its 2D 715 00:33:37,569 --> 00:33:39,950 mathematical constraints. So we are stuck between 716 00:33:39,950 --> 00:33:43,369 two extremes. PBVS gives us perfect, efficient 717 00:33:43,369 --> 00:33:45,950 3D paths, but goes blind and crashes when the 718 00:33:45,950 --> 00:33:50,170 camera rotates. IBVS never goes blind, but executes 719 00:33:50,170 --> 00:33:52,769 insane 3D trajectories like the camera retreat. 720 00:33:53,130 --> 00:33:56,049 How do we engineer a compromise? We need the 721 00:33:56,049 --> 00:33:59,029 robustness of the 2D image plane combined with 722 00:33:59,029 --> 00:34:02,430 the sanity of 3D task space. That leads us to 723 00:34:02,430 --> 00:34:04,710 the advanced visual servoing architectures detailed 724 00:34:04,710 --> 00:34:07,490 in Chapter 16. The most prominent solution to 725 00:34:07,490 --> 00:34:09,730 the camera retreat problem is a hybrid approach 726 00:34:09,730 --> 00:34:12,590 called partitioned IBVS, often referred to as 727 00:34:12,590 --> 00:34:15,750 2 .5D visual servoing. Partitioning the matrix. 728 00:34:16,510 --> 00:34:19,320 How do we decouple the problem? We restructure 729 00:34:19,320 --> 00:34:22,019 the Jacobian to isolate the z -axis translation 730 00:34:22,019 --> 00:34:24,159 from the rest of the control law. We allow the 731 00:34:24,159 --> 00:34:27,059 standard IBVS mechanics to handle the x and y 732 00:34:27,059 --> 00:34:31,000 planar translation and the xyz rotations. Those 733 00:34:31,000 --> 00:34:32,739 five degrees of freedom continue to be driven 734 00:34:32,739 --> 00:34:35,400 by driving pixel coordinates u and v to their 735 00:34:35,400 --> 00:34:39,030 targets. But we sever the depth control. The 736 00:34:39,030 --> 00:34:41,250 z velocity is no longer dictated by the linear 737 00:34:41,250 --> 00:34:44,820 path of the individual pixels. Exactly. We control 738 00:34:44,820 --> 00:34:47,619 the z -axis velocity using an entirely different, 739 00:34:47,920 --> 00:34:50,900 decoupled visual feature. The scalar area of 740 00:34:50,900 --> 00:34:53,099 the polygon formed by the tracked points on the 741 00:34:53,099 --> 00:34:55,639 sensor, or more precisely, often the square root 742 00:34:55,639 --> 00:34:58,179 of that area as it scales linearly with depth. 743 00:34:58,880 --> 00:35:02,099 That is a brilliant architectural shift. We define 744 00:35:02,099 --> 00:35:04,280 a desired polygon area based on our reference 745 00:35:04,280 --> 00:35:06,840 image. If the current polygon area on the sensor 746 00:35:06,840 --> 00:35:09,519 is smaller than the desired area, the decoupled 747 00:35:09,519 --> 00:35:11,639 z -controller commands a forward velocity to 748 00:35:11,639 --> 00:35:14,929 zoom in. If the area is larger, it commands a 749 00:35:14,929 --> 00:35:18,110 backward velocity to zoom out. So now let's run 750 00:35:18,110 --> 00:35:21,389 that 180 -degree twisting task through the partitioned 751 00:35:21,389 --> 00:35:24,789 IBVS controller. The standard IBVS loop still 752 00:35:24,789 --> 00:35:26,530 tries to force the pixels and straight lines 753 00:35:26,530 --> 00:35:29,150 across the screen. The polygon begins to shrink. 754 00:35:29,650 --> 00:35:33,130 But the moment the area begins to decrease, the 755 00:35:33,130 --> 00:35:34,949 decoupled Z -controller registers the error. 756 00:35:35,510 --> 00:35:38,650 But instead of interpreting it as, I am too close, 757 00:35:39,110 --> 00:35:41,760 simply says, the area must remain constant to 758 00:35:41,760 --> 00:35:44,320 match the reference. It actively overrides the 759 00:35:44,320 --> 00:35:46,980 Z velocity commanded by the pixel tracks, forcing 760 00:35:46,980 --> 00:35:50,519 the Z velocity to remain at zero. It clamps the 761 00:35:50,519 --> 00:35:53,519 depth. The camera physically remains at the exact 762 00:35:53,519 --> 00:35:55,539 same distance from the target while the wrist 763 00:35:55,539 --> 00:35:58,400 twists. And the pixels are forced to travel in 764 00:35:58,400 --> 00:36:01,019 circular arcs on the sensor because the physical 765 00:36:01,019 --> 00:36:05,139 Z axis refuses to budge. Camera retreat is completely 766 00:36:05,139 --> 00:36:08,780 eliminated, yielding a highly efficient 3D trajectory 767 00:36:08,780 --> 00:36:11,699 while maintaining the robustness of 2D feature 768 00:36:11,699 --> 00:36:14,579 tracking. It's an elegant mathematical patch. 769 00:36:15,199 --> 00:36:17,300 But what about the field of view problem? Even 770 00:36:17,300 --> 00:36:20,260 with partition control, if we are doing complex 771 00:36:20,260 --> 00:36:22,940 planar translations, features might still drift 772 00:36:22,940 --> 00:36:24,760 dangerously close to the edge of the physical 773 00:36:24,760 --> 00:36:26,940 image sensor. If it crossed the boundary of the 774 00:36:26,940 --> 00:36:30,570 pixel array, the system still crashes. To guarantee 775 00:36:30,570 --> 00:36:33,150 that features never leave the frame, Cork details 776 00:36:33,150 --> 00:36:35,969 the integration of potential fields into the 777 00:36:35,969 --> 00:36:38,650 control law. We construct mathematical repulsive 778 00:36:38,650 --> 00:36:41,269 functions. It's like applying a virtual force 779 00:36:41,269 --> 00:36:43,590 field around the perimeter of the camera frame. 780 00:36:44,050 --> 00:36:47,579 Precisely. We define a buffer zone, perhaps 15 781 00:36:47,579 --> 00:36:50,360 pixels wide, around the absolute maximum and 782 00:36:50,360 --> 00:36:53,440 minimum U and V boundaries of the sensor. We 783 00:36:53,440 --> 00:36:55,619 modify the cost function of the controller so 784 00:36:55,619 --> 00:36:58,079 that when a tracked feature point enters that 785 00:36:58,079 --> 00:37:00,760 buffer zone, it generates an artificial error 786 00:37:00,760 --> 00:37:04,139 signal, a repulsive velocity vector. The closer 787 00:37:04,139 --> 00:37:06,320 the pixel gets to the physical edge of the silicon 788 00:37:06,320 --> 00:37:09,320 chip, the exponentially stronger that virtual 789 00:37:09,320 --> 00:37:12,239 repulsive force becomes. This repulsive vector 790 00:37:12,239 --> 00:37:15,139 is mapped through the Jacobian to command a physical 791 00:37:15,139 --> 00:37:18,179 camera velocity, usually pushing the camera backward 792 00:37:18,179 --> 00:37:20,480 along the z -axis to widen the field of view 793 00:37:20,480 --> 00:37:23,400 or inducing a slight pan to center the feature. 794 00:37:23,880 --> 00:37:26,420 It acts as a high -priority automated reflex. 795 00:37:27,219 --> 00:37:29,500 It's a self -preservation instinct for the robot. 796 00:37:29,940 --> 00:37:32,880 I am about to go blind, execute immediate evasive 797 00:37:32,880 --> 00:37:35,679 maneuver to keep the target in sight. That guarantees 798 00:37:35,679 --> 00:37:39,039 absolute continuous tracking. It's fascinating 799 00:37:39,039 --> 00:37:41,940 how much we have to hack the Cartesian pixel 800 00:37:41,940 --> 00:37:44,280 array to make it work. We have to partition the 801 00:37:44,280 --> 00:37:47,119 axes. We have to invent virtual force fields. 802 00:37:47,760 --> 00:37:49,719 It makes me wonder if representing the image 803 00:37:49,719 --> 00:37:53,579 as a flat grid of XY pixels is the root cause 804 00:37:53,579 --> 00:37:56,380 of the friction. Cork explores that exact question. 805 00:37:56,920 --> 00:37:59,440 The Cartesian UV coordinate system is merely 806 00:37:59,440 --> 00:38:02,539 a convention. We can entirely reformulate the 807 00:38:02,539 --> 00:38:05,280 interaction matrix using polar coordinates. Mapping 808 00:38:05,280 --> 00:38:08,059 the image using radius rho and angle theta from 809 00:38:08,059 --> 00:38:10,440 the center of the sensor instead of horizontal 810 00:38:10,440 --> 00:38:13,989 and vertical position. Yes, and mapping in polar 811 00:38:13,989 --> 00:38:16,550 coordinates completely restructures the optical 812 00:38:16,550 --> 00:38:19,489 flow dynamics. Think about rotation again. If 813 00:38:19,489 --> 00:38:22,349 you rotate a camera around its z -axis, the angle 814 00:38:22,349 --> 00:38:25,130 theta of every tracked feature changes at a constant 815 00:38:25,130 --> 00:38:27,849 rate, but the radial distance rho from the center 816 00:38:27,849 --> 00:38:31,030 remains absolutely identical. The rotation is 817 00:38:31,030 --> 00:38:34,280 completely decoupled from the radius. So if you 818 00:38:34,280 --> 00:38:37,480 use a polar IBVS controller to execute the 180 819 00:38:37,480 --> 00:38:40,400 degree jar lid twist, the control law simply 820 00:38:40,400 --> 00:38:42,460 commands a linear change in the angle theta. 821 00:38:43,159 --> 00:38:45,619 The math naturally dictates that the pixels must 822 00:38:45,619 --> 00:38:48,579 travel in perfect circular arcs. It elegantly 823 00:38:48,579 --> 00:38:50,679 solves the camera retreat problem without needing 824 00:38:50,679 --> 00:38:53,480 to artificially partition the depth axis. The 825 00:38:53,480 --> 00:38:55,679 geometry of the coordinate system itself solves 826 00:38:55,679 --> 00:38:58,280 the rotational constraints. But there has to 827 00:38:58,280 --> 00:39:00,559 be a trade -off. If polar coordinates are so 828 00:39:00,559 --> 00:39:03,920 perfect for rotation, Where did they fail? They 829 00:39:03,920 --> 00:39:06,500 degrade the performance of translation. If you 830 00:39:06,500 --> 00:39:08,920 want the camera to simply slide laterally along 831 00:39:08,920 --> 00:39:11,820 the x -axis, a Cartesian IBVS controller maps 832 00:39:11,820 --> 00:39:14,880 that perfectly to a linear change in the u -coordinate. 833 00:39:15,480 --> 00:39:18,219 But in a polar system, sliding laterally causes 834 00:39:18,219 --> 00:39:21,539 highly complex non -linear changes to both the 835 00:39:21,539 --> 00:39:24,099 radius rho and the angle theta simultaneously. 836 00:39:24,539 --> 00:39:26,780 The resulting physical trajectory for a simple 837 00:39:26,780 --> 00:39:29,639 sidestep becomes curved and suboptimal. Right. 838 00:39:29,909 --> 00:39:32,570 The choice of coordinate geometry depends entirely 839 00:39:32,570 --> 00:39:34,909 on whether the robot's primary task is highly 840 00:39:34,909 --> 00:39:37,909 rotational or highly translational. What if we 841 00:39:37,909 --> 00:39:39,730 take it a step further and change not just the 842 00:39:39,730 --> 00:39:42,250 coordinate system, but the fundamental projection 843 00:39:42,250 --> 00:39:45,409 geometry of the camera itself? Cork discusses 844 00:39:45,409 --> 00:39:48,639 spherical cameras. This is perhaps the most robust 845 00:39:48,639 --> 00:39:51,420 theoretical framework for visual servoing. Instead 846 00:39:51,420 --> 00:39:53,780 of modeling the camera as projecting light through 847 00:39:53,780 --> 00:39:57,340 a pinhole onto a flat 2D focal plane, we model 848 00:39:57,340 --> 00:40:00,119 it projecting onto the surface of a virtual unit 849 00:40:00,119 --> 00:40:02,739 sphere, s -squared. Imagine a camera equipped 850 00:40:02,739 --> 00:40:05,739 with a hyper -white fisheye lens that can perceive 851 00:40:05,739 --> 00:40:08,380 its entire surrounding environment simultaneously. 852 00:40:08,659 --> 00:40:11,079 Wrapping the image sensor into a complete globe. 853 00:40:11,760 --> 00:40:13,739 That instantly solves the field of view problem 854 00:40:13,739 --> 00:40:16,510 permanently. A sphere has no edges. it eliminates 855 00:40:16,510 --> 00:40:19,050 the boundary condition. No matter how wildly 856 00:40:19,050 --> 00:40:21,809 the robot swings its arm, no matter how extreme 857 00:40:21,809 --> 00:40:24,449 the rotation, the tracked features simply glide 858 00:40:24,449 --> 00:40:26,630 along the surface of the sphere. They can never 859 00:40:26,630 --> 00:40:28,989 fall off the sensor because the sensor encompasses 860 00:40:28,989 --> 00:40:32,210 the entire 360 -degree environment. You don't 861 00:40:32,210 --> 00:40:34,789 need repulsive force fields. The hardware geometry 862 00:40:34,789 --> 00:40:37,130 guarantees you never lose tracking. And think 863 00:40:37,130 --> 00:40:39,630 about what that 360 -degree field of view does 864 00:40:39,630 --> 00:40:41,429 to the motion ambiguity problem we discussed 865 00:40:41,429 --> 00:40:44,750 earlier. The narrow train window problem, where 866 00:40:44,750 --> 00:40:46,550 we couldn't tell the difference between sliding 867 00:40:46,550 --> 00:40:48,570 the camera left and panning the camera left? 868 00:40:48,909 --> 00:40:51,730 The ambiguity is completely destroyed. If a spherical 869 00:40:51,730 --> 00:40:54,349 camera slides translationally along the x -axis, 870 00:40:54,929 --> 00:40:56,869 the features in the front hemisphere flow to 871 00:40:56,869 --> 00:40:59,309 the left, but the features in the rear hemisphere 872 00:40:59,309 --> 00:41:01,710 flow to the right, converging on the antipole. 873 00:41:01,880 --> 00:41:04,400 But if the spherical camera pans rotationally 874 00:41:04,400 --> 00:41:07,440 around the y -axis, every single feature on the 875 00:41:07,440 --> 00:41:10,420 entire globe flows in the exact same circular 876 00:41:10,420 --> 00:41:13,039 direction along the lines of latitude. Because 877 00:41:13,039 --> 00:41:15,599 the matrix has access to the global optical flow 878 00:41:15,599 --> 00:41:18,199 field, the mathematical mapping between translation 879 00:41:18,199 --> 00:41:20,980 and rotation is instantly and definitively decoupled. 880 00:41:21,539 --> 00:41:23,780 The columns of the spherical image Jacobian are 881 00:41:23,780 --> 00:41:26,260 perfectly conditioned. It's mathematically bulletproof. 882 00:41:26,519 --> 00:41:28,929 So we've dissected the theory. We understand 883 00:41:28,929 --> 00:41:31,889 the Jacobian, the null space, the depth paradox, 884 00:41:32,090 --> 00:41:35,039 and how to stabilize the matrices. The final 885 00:41:35,039 --> 00:41:39,760 step is application. We need to take this highly 886 00:41:39,760 --> 00:41:42,679 abstract math -heavy visual control theory and 887 00:41:42,679 --> 00:41:45,019 physically integrate it into the actual hardware 888 00:41:45,019 --> 00:41:47,079 we studied back in parts two and three of the 889 00:41:47,079 --> 00:41:49,960 series. Let's start with the classic industrial 890 00:41:49,960 --> 00:41:53,400 manipulator, the six -axis arm. The most critical 891 00:41:53,400 --> 00:41:55,719 engineering reality when implementing visual 892 00:41:55,719 --> 00:41:58,460 surveilling on a physical robotic arm is understanding 893 00:41:58,460 --> 00:42:01,599 control bandwidth. You cannot simply take the 894 00:42:01,599 --> 00:42:04,000 commanded spatial velocity vector from the image 895 00:42:04,000 --> 00:42:06,800 Jacobian and dump it directly into the raw motor 896 00:42:06,800 --> 00:42:09,500 current controllers. Because the math assumes 897 00:42:09,500 --> 00:42:13,119 a massless, frictionless, ideal kinematic system. 898 00:42:13,599 --> 00:42:16,079 But a real robot arm is composed of massive steel 899 00:42:16,079 --> 00:42:19,480 castings, heavy gearboxes, and significant inertial 900 00:42:19,480 --> 00:42:22,460 dynamics. If the IBVS loop detects a sudden pixel 901 00:42:22,460 --> 00:42:25,019 error and instantly commands a massive velocity 902 00:42:25,019 --> 00:42:27,460 step change, the raw physics of the arm simply 903 00:42:27,460 --> 00:42:29,960 can't comply. If you try to force it, the arm 904 00:42:29,960 --> 00:42:32,260 will violently jerk, potentially stripping the 905 00:42:32,260 --> 00:42:34,719 gears, and the camera will shake so violently 906 00:42:34,719 --> 00:42:37,960 the image will blur into useless noise. To bridge 907 00:42:37,960 --> 00:42:40,019 the gap between kinematic theory and dynamic 908 00:42:40,019 --> 00:42:43,219 reality, we utilize a nested control architecture. 909 00:42:43,480 --> 00:42:45,300 We separate the high -level decision -making 910 00:42:45,300 --> 00:42:49,119 from the low -level physical execution. Exactly. 911 00:42:49,280 --> 00:42:52,300 At the lowest level, deeply embedded inside each 912 00:42:52,300 --> 00:42:54,840 individual joint of the robot is a high bandwidth 913 00:42:54,840 --> 00:42:57,599 dynamic control loop. This loop runs incredibly 914 00:42:57,599 --> 00:42:59,719 fast, typically at a thousand hertz, meaning 915 00:42:59,719 --> 00:43:02,400 it updates a thousand times every second. Wow. 916 00:43:02,800 --> 00:43:04,860 It reads the high resolution optical encoders 917 00:43:04,860 --> 00:43:07,579 on the motor shaft, calculates the exact torque 918 00:43:07,579 --> 00:43:10,300 required to overcome friction and inertia, and 919 00:43:10,300 --> 00:43:12,239 modulates the electrical current to ensure the 920 00:43:12,239 --> 00:43:14,900 joint achieves and maintains a very specific 921 00:43:14,900 --> 00:43:18,210 commanded rotational velocity. That 1000 Hertz 922 00:43:18,210 --> 00:43:20,809 loop completely abstracts the way the messy physics. 923 00:43:21,369 --> 00:43:23,869 It allows us to mathematically treat the robotic 924 00:43:23,869 --> 00:43:28,929 joint as an ideal, perfect velocity source. With 925 00:43:28,929 --> 00:43:31,530 the physical dynamics handled, we wrap our visual 926 00:43:31,530 --> 00:43:33,710 servoing system around the outside of those joints 927 00:43:33,710 --> 00:43:36,719 as an outer kinematic control loop. The wrist 928 00:43:36,719 --> 00:43:39,139 camera captures an image, extracts the features, 929 00:43:39,519 --> 00:43:42,199 computes the pixel error, multiplies it by the 930 00:43:42,199 --> 00:43:44,699 pseudo -inverse of the Jacobian, and outputs 931 00:43:44,699 --> 00:43:47,619 a desired 3D spatial velocity for the end effector. 932 00:43:48,039 --> 00:43:50,860 But the camera processing pipeline is computationally 933 00:43:50,860 --> 00:43:54,280 heavy. It might only run at 30 hertz, or 30 frames 934 00:43:54,280 --> 00:43:57,380 per second. It is a low -frequency, high -latency 935 00:43:57,380 --> 00:44:00,119 control loop. The visual servo system acts as 936 00:44:00,119 --> 00:44:02,940 a strategic director. Every 33 milliseconds, 937 00:44:02,980 --> 00:44:05,239 it hands down a new spatial velocity target. 938 00:44:05,800 --> 00:44:09,119 The robot controller then takes that 3D Cartesian 939 00:44:09,119 --> 00:44:11,679 velocity vector, pushes it through the inverse 940 00:44:11,679 --> 00:44:13,579 Jacobian of the manipulator, which we covered 941 00:44:13,579 --> 00:44:16,460 in part three, to resolve it into six individual 942 00:44:16,460 --> 00:44:18,800 joint velocities. And it hands those individual 943 00:44:18,800 --> 00:44:21,159 velocity commands down to the 1,000 Hertz inner 944 00:44:21,159 --> 00:44:24,099 loops to execute. It's exactly like the corporate 945 00:44:24,099 --> 00:44:26,079 structure of a manufacturing plant. Oh, that's 946 00:44:26,079 --> 00:44:28,300 a good way to look at it. The visual servo system 947 00:44:28,300 --> 00:44:31,519 is the CEO. The CEO operates on a slower time 948 00:44:31,519 --> 00:44:35,070 frame, analyzing the broad market trends. the 949 00:44:35,070 --> 00:44:37,750 visual scene, and issues a strategic directive 950 00:44:37,750 --> 00:44:40,170 once a week. We need to shift production in this 951 00:44:40,170 --> 00:44:43,070 direction at this overall speed. The CEO doesn't 952 00:44:43,070 --> 00:44:45,730 know how to run a lathe or tune a conveyor belt. 953 00:44:46,190 --> 00:44:49,010 Right. The CEO's directive is handed down to 954 00:44:49,010 --> 00:44:51,889 the factory floor manager, the 1,000 Hertz joint 955 00:44:51,889 --> 00:44:54,510 controller. The floor manager operates in real 956 00:44:54,510 --> 00:44:56,929 time. making thousands of micro adjustments every 957 00:44:56,929 --> 00:45:00,150 hour, speeding up a motor here, increasing torque 958 00:45:00,150 --> 00:45:02,750 there, fighting the physical friction of the 959 00:45:02,750 --> 00:45:05,789 factory to ensure the CEO's overarching strategic 960 00:45:05,789 --> 00:45:08,190 velocity is achieved. That nested hierarchy, 961 00:45:08,510 --> 00:45:10,710 a low bandwidth outer visual loop generating 962 00:45:10,710 --> 00:45:13,210 set points for high bandwidth inner dynamic loop, 963 00:45:13,730 --> 00:45:16,670 is the industry standard for stable visual manipulation. 964 00:45:16,989 --> 00:45:20,010 But that architecture assumes the robot is a 965 00:45:20,010 --> 00:45:22,960 holonomic manipulator. Meaning it assumes the 966 00:45:22,960 --> 00:45:25,099 arm can instantly generate a velocity vector 967 00:45:25,099 --> 00:45:30,219 in any arbitrary XYZ direction. Exactly. What 968 00:45:30,219 --> 00:45:32,480 happens when we apply visual surveilling to a 969 00:45:32,480 --> 00:45:35,639 platform that cannot move freely? Mobile robotics. 970 00:45:36,440 --> 00:45:38,599 The non -holonomic platforms we explored in part 971 00:45:38,599 --> 00:45:42,199 two. Like a differential drive robot or a car 972 00:45:42,199 --> 00:45:45,239 like Ackerman's steering vehicle. A car cannot 973 00:45:45,239 --> 00:45:48,460 simply command a lateral y -axis velocity to 974 00:45:48,460 --> 00:45:50,699 slide sideways into a parallel parking spot. 975 00:45:51,219 --> 00:45:54,360 The physical kinematics forbid it. The nonholonomic 976 00:45:54,360 --> 00:45:56,400 constraints fundamentally break the standard 977 00:45:56,400 --> 00:45:59,440 IBVS jucopian we just discussed. If the visual 978 00:45:59,440 --> 00:46:01,320 error dictates that the camera needs to move 979 00:46:01,320 --> 00:46:04,039 purely to the left, a car cannot execute that 980 00:46:04,039 --> 00:46:07,170 velocity command. To solve this, Cork demonstrates 981 00:46:07,170 --> 00:46:09,289 how to reformulate the visual control problem 982 00:46:09,289 --> 00:46:12,690 using specific polar error states that respect 983 00:46:12,690 --> 00:46:15,349 the vehicle's kinematic constraints. Let's trace 984 00:46:15,349 --> 00:46:17,670 the math for visually parking a differential 985 00:46:17,670 --> 00:46:20,710 drive robot. How do we structure the error? We 986 00:46:20,710 --> 00:46:24,469 utilize a position -based PBVS approach to extract 987 00:46:24,469 --> 00:46:27,170 three highly specific geometric variables from 988 00:46:27,170 --> 00:46:29,969 the visual scene. The robot identifies landmarks 989 00:46:29,969 --> 00:46:32,480 in the environment. It compares its current view 990 00:46:32,480 --> 00:46:34,820 of those landmarks to a stored reference image 991 00:46:34,820 --> 00:46:36,599 of what the landmarks should look like when the 992 00:46:36,599 --> 00:46:38,920 robot is perfectly parked at the goal location. 993 00:46:39,639 --> 00:46:41,480 From that visual discrepancy, it runs a pose 994 00:46:41,480 --> 00:46:43,599 estimation to determine its relative position 995 00:46:43,599 --> 00:46:46,179 to the parking spot. But instead of generating 996 00:46:46,179 --> 00:46:49,880 an XY Cartesian error, it calculates the polar 997 00:46:49,880 --> 00:46:53,750 states. It extracts rho, which is the straight 998 00:46:53,750 --> 00:46:56,429 line Euclidean distance from the robot's current 999 00:46:56,429 --> 00:47:00,050 position to the goal location. It extracts alpha, 1000 00:47:00,369 --> 00:47:02,650 which is the angle between the robot's current 1001 00:47:02,650 --> 00:47:04,869 heading and the vector pointing toward the goal. 1002 00:47:05,570 --> 00:47:08,469 And it extracts beta, which is the orientation 1003 00:47:08,469 --> 00:47:10,809 error, the difference between the robot's current 1004 00:47:10,809 --> 00:47:12,869 heading and the desired final parking angle. 1005 00:47:13,409 --> 00:47:16,030 So the vision system continuously feeds the controller 1006 00:47:16,030 --> 00:47:18,820 those three variables. You are five meters away, 1007 00:47:19,039 --> 00:47:21,360 rho. You need to steer 15 degrees to the left 1008 00:47:21,360 --> 00:47:23,860 to face the spot, alpha. And when you arrive, 1009 00:47:24,079 --> 00:47:26,079 you need to be parked at a 90 degree angle to 1010 00:47:26,079 --> 00:47:28,739 the street, beta. We then feed those visually 1011 00:47:28,739 --> 00:47:31,460 derived states into a highly specific nonlinear 1012 00:47:31,460 --> 00:47:34,380 control law tailored for non -holonomic platforms. 1013 00:47:35,159 --> 00:47:37,760 The forward velocity of the robot, V, is controlled 1014 00:47:37,760 --> 00:47:39,860 by a simple proportional gain on the distance. 1015 00:47:40,280 --> 00:47:43,760 V equals k sub rho times rho. As the robot gets 1016 00:47:43,760 --> 00:47:46,079 closer to the goal, the forward speed naturally 1017 00:47:46,079 --> 00:47:49,000 decays to zero. But the steering control is the 1018 00:47:49,000 --> 00:47:51,679 clever part. The angular velocity of the robot, 1019 00:47:51,820 --> 00:47:54,380 omega, is a coupled function of both the heading 1020 00:47:54,380 --> 00:47:57,300 error and the final orientation error. Omega 1021 00:47:57,300 --> 00:48:01,059 equals k sub alpha times alpha plus k sub beta 1022 00:48:01,059 --> 00:48:05,010 times beta. It simultaneously tries to point 1023 00:48:05,010 --> 00:48:07,809 the robot toward the goal while aggressively 1024 00:48:07,809 --> 00:48:10,369 correcting for the final parking angle. As it 1025 00:48:10,369 --> 00:48:12,949 drives, the vision system continuously updates 1026 00:48:12,949 --> 00:48:16,010 rho, alpha, and beta, closing the loop. The robot 1027 00:48:16,010 --> 00:48:18,309 elegantly arcs its way into the parking space, 1028 00:48:18,530 --> 00:48:20,849 continually adjusting its steering based purely 1029 00:48:20,849 --> 00:48:22,849 on matching its current camera feed to a target 1030 00:48:22,849 --> 00:48:25,429 photograph. Perfectly respecting the fact that 1031 00:48:25,429 --> 00:48:29,110 it cannot move sideways. It's a beautiful integration 1032 00:48:29,110 --> 00:48:32,530 of visual perception and constrained differential 1033 00:48:32,530 --> 00:48:37,500 kinematics. Now, for the final and perhaps most 1034 00:48:37,500 --> 00:48:42,320 complex application. Aerial robotics. The quadrotors 1035 00:48:42,320 --> 00:48:45,599 from part two. Drones represent the extreme edge 1036 00:48:45,599 --> 00:48:48,260 of control difficulty because they are severely 1037 00:48:48,260 --> 00:48:51,260 underactuated. They operate in six degrees of 1038 00:48:51,260 --> 00:48:54,559 freedom, translation in XYZ, and rotation in 1039 00:48:54,559 --> 00:48:57,579 roll pitch yaw. But a quadrotor only possesses 1040 00:48:57,579 --> 00:49:00,300 four control inputs, the rotational speeds of 1041 00:49:00,300 --> 00:49:02,940 its four propellers, which map to overall thrust, 1042 00:49:03,320 --> 00:49:05,849 roll, pitch, and yaw torques. It lacks the hardware 1043 00:49:05,849 --> 00:49:07,909 to directly command a force along the horizontal 1044 00:49:07,909 --> 00:49:12,309 X or Y axes. A drone cannot just slide forward 1045 00:49:12,309 --> 00:49:15,010 while remaining perfectly level. The translational 1046 00:49:15,010 --> 00:49:17,570 dynamics are inextricably coupled to the attitude 1047 00:49:17,570 --> 00:49:20,070 dynamics. If a drone wants to translate forward 1048 00:49:20,070 --> 00:49:22,730 along the X axis, it must increase the thrust 1049 00:49:22,730 --> 00:49:25,349 on the rear rotors and decrease thrust on the 1050 00:49:25,349 --> 00:49:28,070 front rotors. It must physically pitch its entire 1051 00:49:28,070 --> 00:49:31,409 chassis forward to direct the thrust vector horizontally. 1052 00:49:31,739 --> 00:49:33,780 And if the camera is rigidly strapped to the 1053 00:49:33,780 --> 00:49:36,320 bottom of the drone chassis, when the drone pitches 1054 00:49:36,320 --> 00:49:38,920 forward 20 degrees to initiate a forward translation, 1055 00:49:39,579 --> 00:49:41,960 the camera violently pitches 20 degrees as well. 1056 00:49:42,460 --> 00:49:44,579 Which creates a catastrophic problem for the 1057 00:49:44,579 --> 00:49:46,739 visual servo controller. Imagine the drone is 1058 00:49:46,739 --> 00:49:48,539 trying to hover perfectly over a target on the 1059 00:49:48,539 --> 00:49:51,500 ground using IBVS. A gust of wind pushes the 1060 00:49:51,500 --> 00:49:53,900 drone backward. The visual controller registers 1061 00:49:53,900 --> 00:49:56,340 the pixels sliding down the sensor and commands 1062 00:49:56,340 --> 00:49:59,210 a forward X velocity to correct it. The flight 1063 00:49:59,210 --> 00:50:01,150 controller pitches the drone forward to execute 1064 00:50:01,150 --> 00:50:03,949 that X velocity. The camera tilts toward the 1065 00:50:03,949 --> 00:50:07,250 ground. Instantly the entire visual field on 1066 00:50:07,250 --> 00:50:09,570 the camera sensor sweeps aggressively upward. 1067 00:50:10,190 --> 00:50:13,349 The IBVS controller registers this massive sudden 1068 00:50:13,349 --> 00:50:16,769 optical flow and interprets it as a massive physical 1069 00:50:16,769 --> 00:50:19,170 translation error. It thinks the drone just surged 1070 00:50:19,170 --> 00:50:21,610 wildly in the wrong direction. The controller 1071 00:50:21,610 --> 00:50:23,960 fights its own movement. The tilt required to 1072 00:50:23,960 --> 00:50:26,739 move creates a visual illusion of moving in the 1073 00:50:26,739 --> 00:50:29,320 opposite direction. It completely destabilizes 1074 00:50:29,320 --> 00:50:32,179 the feedback loop. The standard hardware solution 1075 00:50:32,179 --> 00:50:34,539 to this is to mount the camera on a stabilized 1076 00:50:34,539 --> 00:50:37,539 motorized three -axis gimbal. The gimbal uses 1077 00:50:37,539 --> 00:50:39,920 gyroscopes to aggressively counteract the drone's 1078 00:50:39,920 --> 00:50:41,780 pitch and roll, keeping the camera perfectly 1079 00:50:41,780 --> 00:50:44,639 level with the horizon at all times. If the camera 1080 00:50:44,639 --> 00:50:47,559 stays level, A forward translation doesn't induce 1081 00:50:47,559 --> 00:50:50,880 any rotational optical flow. The IBVS matrix 1082 00:50:50,880 --> 00:50:54,159 works perfectly. But gimbals are heavy, fragile, 1083 00:50:54,659 --> 00:50:56,960 they consume battery power, and they add a massive 1084 00:50:56,960 --> 00:50:59,739 layer of mechanical complexity. Which is why 1085 00:50:59,739 --> 00:51:02,619 Cork details a far more elegant software architecture. 1086 00:51:03,139 --> 00:51:06,099 Feature derotation. It's essentially a virtual 1087 00:51:06,099 --> 00:51:09,000 mathematical gimbal. How do you build a gimbal 1088 00:51:09,000 --> 00:51:12,909 in software? Through sensor fusion. Modern quadrotors 1089 00:51:12,909 --> 00:51:15,090 are equipped with high -frequency inertial measurement 1090 00:51:15,090 --> 00:51:17,929 units, IMUs. These sensors track the precise 1091 00:51:17,929 --> 00:51:20,510 pitch and roll angles of the drone chassis at 1092 00:51:20,510 --> 00:51:23,650 thousands of times a second. We take that instantaneous 1093 00:51:23,650 --> 00:51:26,030 attitude data and feed it directly into our visual 1094 00:51:26,030 --> 00:51:28,989 control loop. We mathematically calculate the 1095 00:51:28,989 --> 00:51:31,650 exact rotational optical flow that the pitch 1096 00:51:31,650 --> 00:51:34,349 and roll should be causing on the image sensor. 1097 00:51:35,050 --> 00:51:38,329 We use the rotational columns of the image Jacobian 1098 00:51:38,590 --> 00:51:41,349 the columns that don't depend on depth, and multiply 1099 00:51:41,349 --> 00:51:45,389 them by the IMU's angular velocity data. We compute 1100 00:51:45,389 --> 00:51:48,469 the fake attitude -induced pixel movement. We 1101 00:51:48,469 --> 00:51:50,949 then simply subtract that expected rotational 1102 00:51:50,949 --> 00:51:54,090 flow vector from the actual total optical flow 1103 00:51:54,090 --> 00:51:57,329 observed by the camera. So P dot translational 1104 00:51:57,329 --> 00:52:01,329 equals P dot total minus P dot rotational. We 1105 00:52:01,329 --> 00:52:03,570 mathematically erase the effects of the drone's 1106 00:52:03,570 --> 00:52:06,059 tilt from the visual data stream. The resulting 1107 00:52:06,059 --> 00:52:08,699 array of pixel velocities is perfectly clean. 1108 00:52:09,159 --> 00:52:11,519 It represents only the optical flow caused by 1109 00:52:11,519 --> 00:52:13,400 the physical translation of the drone through 1110 00:52:13,400 --> 00:52:16,360 3D space, completely isolated from its attitude 1111 00:52:16,360 --> 00:52:20,059 wobbles. The IBVS controller is fed this purified 1112 00:52:20,059 --> 00:52:22,780 translational data. It can just focus on driving 1113 00:52:22,780 --> 00:52:24,860 the drone to perfectly hover over the target, 1114 00:52:25,380 --> 00:52:27,199 entirely ignorant of the fact that the physical 1115 00:52:27,199 --> 00:52:29,920 drone chassis is aggressively pitching and rolling 1116 00:52:29,920 --> 00:52:32,920 to fight the wind. The software handles the decoupling. 1117 00:52:33,170 --> 00:52:36,329 It's a masterpiece of integrating high -speed 1118 00:52:36,329 --> 00:52:40,510 proprioceptive sensors, the IMU, with low -speed 1119 00:52:40,510 --> 00:52:43,630 exteroceptive sensors, the camera, to stabilize 1120 00:52:43,630 --> 00:52:46,750 an underactuated system. It perfectly illustrates 1121 00:52:46,750 --> 00:52:49,250 the overarching theme of this entire five -part 1122 00:52:49,250 --> 00:52:51,389 series. It really does. We've reached the end 1123 00:52:51,389 --> 00:52:54,829 of the line, part five of five. We started weeks 1124 00:52:54,829 --> 00:52:57,590 ago with the raw mathematical abstractions of 1125 00:52:57,590 --> 00:53:01,429 SE3 geometry. We learned how to move wheels and 1126 00:53:01,429 --> 00:53:04,340 calculate joint torques. We built the theoretical 1127 00:53:04,340 --> 00:53:08,039 camera to process photons, and now we finally 1128 00:53:08,039 --> 00:53:11,219 close the loop. Visual servoing is the ultimate 1129 00:53:11,219 --> 00:53:13,659 synthesis. It demonstrates that you don't need 1130 00:53:13,659 --> 00:53:17,119 a rigid, pre -calculated, infinitely precise 1131 00:53:17,119 --> 00:53:19,440 model of the universe to interact with it successfully. 1132 00:53:20,340 --> 00:53:22,500 By feeding pixel data directly into the velocity 1133 00:53:22,500 --> 00:53:25,460 kinematics, we invert the classic robotic paradigm. 1134 00:53:25,739 --> 00:53:28,599 We stop forcing the environment to conform to 1135 00:53:28,599 --> 00:53:31,050 the limitations of the machine. We allow the 1136 00:53:31,050 --> 00:53:33,789 machine to adapt to the chaos of the environment. 1137 00:53:34,889 --> 00:53:37,190 The continuous feedback loop inherently absorbs 1138 00:53:37,190 --> 00:53:40,650 calibration errors, tolerates messy depth estimations, 1139 00:53:41,050 --> 00:53:44,050 and reacts instantly to moving targets. Visual 1140 00:53:44,050 --> 00:53:46,849 control is what allows robotics to finally escape 1141 00:53:46,849 --> 00:53:49,570 the highly structured, multi -million dollar 1142 00:53:49,570 --> 00:53:52,389 cages of the automotive assembly line and step 1143 00:53:52,389 --> 00:53:56,010 into unstructured, dynamic reality. It transforms 1144 00:53:56,010 --> 00:53:59,179 the robot from a blind brittle automaton into 1145 00:53:59,179 --> 00:54:03,000 a reactive, adaptable agent. And it accomplishes 1146 00:54:03,000 --> 00:54:06,199 that with a mathematical elegance that is genuinely 1147 00:54:06,199 --> 00:54:09,099 thought -provoking. Which leaves me with one 1148 00:54:09,099 --> 00:54:11,960 final concept for you, the listener, to mull 1149 00:54:11,960 --> 00:54:14,679 over as we officially close this Marginalia series 1150 00:54:14,679 --> 00:54:17,780 on Peter Corke's textbook. I suspect we are moving 1151 00:54:17,780 --> 00:54:20,500 from engineering back to biology. The parallels 1152 00:54:20,500 --> 00:54:23,389 are unavoidable. We just spent an hour mathematically 1153 00:54:23,389 --> 00:54:26,010 proving that image -based visual servoing allows 1154 00:54:26,010 --> 00:54:28,909 a machine to successfully, reliably grasp an 1155 00:54:28,909 --> 00:54:31,769 object using wild mathematical shortcuts. It 1156 00:54:31,769 --> 00:54:34,010 doesn't need a high resolution 3D CAD model. 1157 00:54:34,510 --> 00:54:37,610 It can survive using wildly incorrect depth assumptions. 1158 00:54:38,369 --> 00:54:40,849 It fundamentally relies on a 2D feedback loop 1159 00:54:40,849 --> 00:54:43,389 to continuously drag feature points across a 1160 00:54:43,389 --> 00:54:46,670 sensor array until the error reaches zero. It 1161 00:54:46,670 --> 00:54:48,949 prioritizes the continuous feedback over the 1162 00:54:48,949 --> 00:54:51,690 fidelity of the model. It achieves the goal without 1163 00:54:51,690 --> 00:54:53,929 truly understanding the 3D space it operates 1164 00:54:53,929 --> 00:54:57,250 in. Right. So it forces you to question the mechanics 1165 00:54:57,250 --> 00:54:59,829 of human biology. When you rapidly reach across 1166 00:54:59,829 --> 00:55:02,349 your desk to grab a falling coffee cup, is your 1167 00:55:02,349 --> 00:55:04,789 brain actually taking the computational time 1168 00:55:04,789 --> 00:55:08,230 to reconstruct a perfect, high -resolution 3D 1169 00:55:08,230 --> 00:55:11,190 Cartesian pose estimation of the room? Or is 1170 00:55:11,190 --> 00:55:13,789 human hand -eye coordination just running a deeply 1171 00:55:13,789 --> 00:55:16,849 optimized, highly error -tolerant, partitioned 1172 00:55:16,849 --> 00:55:20,800 2D IBVS loop? Do we actually think in 3D when 1173 00:55:20,800 --> 00:55:24,079 we react? Or are we just biologically dragging 1174 00:55:24,079 --> 00:55:26,139 pixels across our retinas until our hand hits 1175 00:55:26,139 --> 00:55:29,500 the cup? The computational efficiency of IBVS 1176 00:55:29,500 --> 00:55:31,420 certainly makes a compelling argument for the 1177 00:55:31,420 --> 00:55:34,139 latter. It's a question for the neuroscientists 1178 00:55:34,139 --> 00:55:36,280 to debate, but the robotics framework proves 1179 00:55:36,280 --> 00:55:39,039 that the 2D shortcut is not only viable, it's 1180 00:55:39,039 --> 00:55:41,800 often superior. Something to ponder the next 1181 00:55:41,800 --> 00:55:43,800 time you successfully catch a set of keys thrown 1182 00:55:43,800 --> 00:55:46,400 at you. Thank you for taking this deep dive into 1183 00:55:46,400 --> 00:55:48,789 the math, the matrices, and the machines with 1184 00:55:48,789 --> 00:55:49,030 us.