1 00:00:00,000 --> 00:00:03,640 You know, it's funny, for decades now, our baseline 2 00:00:03,640 --> 00:00:06,219 for digital communication has essentially just 3 00:00:06,219 --> 00:00:08,099 been a walkie -talkie. Oh, yeah, completely. 4 00:00:08,380 --> 00:00:10,599 Like, you press the button, you speak your piece, 5 00:00:10,800 --> 00:00:12,660 you let go, and then you just wait and listen. 6 00:00:13,000 --> 00:00:15,419 Right. It's very sequential. Exactly. It's a 7 00:00:15,419 --> 00:00:18,600 highly polite, entirely turn -taking system. 8 00:00:19,260 --> 00:00:22,039 Information flows one way, it completely stops, 9 00:00:22,100 --> 00:00:24,940 and then it flows back. Which is clean. I mean, 10 00:00:25,079 --> 00:00:27,300 from a computer science perspective, it's incredibly 11 00:00:27,300 --> 00:00:29,879 easy to manage, because... The system knows exactly 12 00:00:29,879 --> 00:00:32,100 when you're done. Because you literally give 13 00:00:32,100 --> 00:00:34,579 it a, like, an end of input signal. Exactly. 14 00:00:35,280 --> 00:00:37,340 But then you think about how two human beings 15 00:00:37,340 --> 00:00:40,179 actually talk, right? Like, imagine sitting across 16 00:00:40,179 --> 00:00:42,740 a table at a coffee shop. Yeah, it's chaotic. 17 00:00:42,880 --> 00:00:44,359 It really is. You're nodding. You're saying, 18 00:00:44,600 --> 00:00:46,280 mm -hmm, maybe you both start talking at the 19 00:00:46,280 --> 00:00:48,700 exact same time. And one of you has to back down. 20 00:00:48,799 --> 00:00:51,200 Right. Or you interrupt because, I don't know, 21 00:00:51,200 --> 00:00:52,640 the other person has spinach in their teeth or 22 00:00:52,640 --> 00:00:55,420 something. Genuine, full duplex communication 23 00:00:55,420 --> 00:00:58,560 is messy. It's continuous. And trying to map 24 00:00:58,560 --> 00:01:02,020 that continuous messy human reality onto a digital 25 00:01:02,020 --> 00:01:04,180 architecture, I mean, that creates an absolute 26 00:01:04,180 --> 00:01:06,620 logistical traffic jam. Oh, massive one. You're 27 00:01:06,620 --> 00:01:09,299 no longer waiting for a polite pause. The sheer 28 00:01:09,299 --> 00:01:12,640 volume of data moving in both directions simultaneously 29 00:01:12,640 --> 00:01:16,359 is just staggering. Because think about the core 30 00:01:16,359 --> 00:01:18,519 engineering problem here. We aren't just talking 31 00:01:18,519 --> 00:01:22,019 about transmitting an audio file back and forth. 32 00:01:22,019 --> 00:01:24,019 No, not at all. We're talking about building 33 00:01:24,019 --> 00:01:26,760 a system capable of taking in your live audio. 34 00:01:26,959 --> 00:01:30,280 your live video feed and the text transcription 35 00:01:30,280 --> 00:01:32,519 of what you're saying all at the exact same time. 36 00:01:32,760 --> 00:01:35,359 Right. And while it is absorbing all of that 37 00:01:35,359 --> 00:01:38,500 incoming data, it has to simultaneously in real 38 00:01:38,500 --> 00:01:41,680 time, generate its own audio and video responses. 39 00:01:41,780 --> 00:01:45,439 It has to decide whether to listen or speak or 40 00:01:45,439 --> 00:01:48,980 interrupt all while maintaining a coherent visual 41 00:01:48,980 --> 00:01:50,980 and auditory presence on your screen. And it 42 00:01:50,980 --> 00:01:53,200 has to do all of this without freezing, without 43 00:01:53,200 --> 00:01:55,540 dropping the connection. And, you know, without 44 00:01:55,540 --> 00:01:57,420 losing the actual thread of the conversation. 45 00:01:57,959 --> 00:02:00,579 Which is wild to even think about. But that brings 46 00:02:00,579 --> 00:02:02,780 us to the source material on our desk today. 47 00:02:03,099 --> 00:02:05,719 We are looking at a foundational text that attempts 48 00:02:05,719 --> 00:02:09,280 to solve this exact traffic jam. That's a really 49 00:02:09,280 --> 00:02:11,979 fascinating paper. It is. The technical report 50 00:02:11,979 --> 00:02:15,479 is titled Duo, a reciprocal world model for full 51 00:02:15,479 --> 00:02:17,939 duplex interaction. It comes to us from Heng 52 00:02:17,939 --> 00:02:20,740 Yuan Kao and a really large team of colleagues 53 00:02:20,740 --> 00:02:23,449 from Ant Group. Yeah. And our goal today is to 54 00:02:23,449 --> 00:02:26,150 impact this framework meticulously, like chapter 55 00:02:26,150 --> 00:02:28,710 by chapter, concept by concept, to understand 56 00:02:28,710 --> 00:02:31,569 how they even attempted to pull this off. But 57 00:02:31,569 --> 00:02:34,030 before we get into the architecture, I think 58 00:02:34,030 --> 00:02:36,750 it's vital we set a boundary on what we are actually 59 00:02:36,750 --> 00:02:39,409 discussing today. Good idea. We have to separate 60 00:02:39,409 --> 00:02:42,669 the theoretical framework they propose from the 61 00:02:42,669 --> 00:02:45,090 actual software they built, and we need to stick 62 00:02:45,090 --> 00:02:47,990 strictly to the empirical results. Completely 63 00:02:47,990 --> 00:02:50,650 agreed. And for you listening right now, we are 64 00:02:50,650 --> 00:02:53,090 setting a hard ground rule. You aren't going 65 00:02:53,090 --> 00:02:55,509 to hear us talk about, like, imagined virtual 66 00:02:55,509 --> 00:02:59,349 rooms or digital souls or living beings waking 67 00:02:59,349 --> 00:03:01,930 up inside the machine. Absolutely not. We are 68 00:03:01,930 --> 00:03:04,409 looking entirely at the physical and software 69 00:03:04,409 --> 00:03:07,569 engineering realities. Mathematics, data pipelines, 70 00:03:07,990 --> 00:03:10,689 silicon. The best place to start, I think, is 71 00:03:10,689 --> 00:03:13,710 with the title itself. They use the phrase, reciprocal 72 00:03:13,710 --> 00:03:17,620 world model, or RWM. RWM, right. And to understand 73 00:03:17,620 --> 00:03:19,800 why that's a shift, we should probably define 74 00:03:19,800 --> 00:03:22,180 how the industry traditionally uses the term 75 00:03:22,180 --> 00:03:25,879 world model. OK, yeah. When I hear world model, 76 00:03:26,000 --> 00:03:28,879 I immediately think of physics engines or video 77 00:03:28,879 --> 00:03:32,060 games. Exactly. Like simulators. Yeah. Like if 78 00:03:32,060 --> 00:03:34,240 you're training an AI to drive a car, the world 79 00:03:34,240 --> 00:03:36,719 model is the simulation of the road. It knows 80 00:03:36,719 --> 00:03:38,900 that if the road is icy, the car slides. It knows 81 00:03:38,900 --> 00:03:41,400 a brick wall is solid. Right. That's the classical 82 00:03:41,400 --> 00:03:44,460 definition. The world model simulates the physical 83 00:03:44,460 --> 00:03:47,360 realities of the environment. But in that traditional 84 00:03:47,360 --> 00:03:50,360 setup, the intelligence. The agent making the 85 00:03:50,360 --> 00:03:52,879 decisions, right? Yes. The agent sits entirely 86 00:03:52,879 --> 00:03:55,840 outside of that simulated world. It's just looking 87 00:03:55,840 --> 00:03:58,580 at the simulation and pushing buttons, or turning 88 00:03:58,580 --> 00:04:01,139 a virtual steering wheel. It's like me playing 89 00:04:01,139 --> 00:04:03,639 a racing game on my couch. The game console holds 90 00:04:03,639 --> 00:04:06,020 the world model, but the intelligence is just 91 00:04:06,020 --> 00:04:07,780 my thumbs on the controller. I'm not part of 92 00:04:07,780 --> 00:04:10,159 the game's rendered code. It's a perfect analogy. 93 00:04:10,409 --> 00:04:13,210 The reciprocal world model, though, it turns 94 00:04:13,210 --> 00:04:17,149 that architecture inside out. How so? It treats 95 00:04:17,149 --> 00:04:19,310 both physical reality, which the authors call 96 00:04:19,310 --> 00:04:22,110 realization, and intelligence, which they call 97 00:04:22,110 --> 00:04:25,449 intent, as equally fundamental parts of the generated 98 00:04:25,449 --> 00:04:29,050 world. Oh, I see. So the cognition of the assistant 99 00:04:29,050 --> 00:04:31,829 and the physical manifestation of that cognition, 100 00:04:32,430 --> 00:04:34,410 like the movement of its digital mouth or the 101 00:04:34,410 --> 00:04:37,230 sound of its voice, they are intrinsic to the 102 00:04:37,230 --> 00:04:39,750 environment it's generating. Okay, so it's not 103 00:04:39,750 --> 00:04:42,329 just rendering a video of a digital avatar sitting 104 00:04:42,329 --> 00:04:45,069 in a room. It's generating the conversational 105 00:04:45,069 --> 00:04:47,850 intent and the visual realization of that intent 106 00:04:47,850 --> 00:04:51,790 simultaneously as one cohesive reality. Precisely. 107 00:04:52,129 --> 00:04:54,269 Now, the authors built a specific piece of software 108 00:04:54,269 --> 00:04:57,129 to test this, which they called Duo version 1. 109 00:04:57,329 --> 00:05:00,970 Duo V1? Right. And here's where we need to be 110 00:05:00,970 --> 00:05:02,930 very precise about their engineering choices. 111 00:05:03,410 --> 00:05:06,529 Duo V1 is a modular system. Right, they split 112 00:05:06,529 --> 00:05:09,439 the workload. They do. They use a multi -motor 113 00:05:09,439 --> 00:05:11,860 language model to handle the intent side of things, 114 00:05:12,259 --> 00:05:15,019 the thinking, deciding what to say. And they 115 00:05:15,019 --> 00:05:17,500 use a completely separate diffusion model to 116 00:05:17,500 --> 00:05:20,279 handle the realization, the actual drawing of 117 00:05:20,279 --> 00:05:22,980 the video frames and audio waves. But this modular 118 00:05:22,980 --> 00:05:25,680 split isn't like a fundamental law of the reciprocal 119 00:05:25,680 --> 00:05:28,100 world model, is it? No, not at all. Because the 120 00:05:28,100 --> 00:05:30,220 paper doesn't prove that you have to use separate 121 00:05:30,220 --> 00:05:32,220 modules to make full duplex interaction work, 122 00:05:32,240 --> 00:05:34,759 right? No, it doesn't. Theoretically, you could 123 00:05:34,759 --> 00:05:37,420 implement the reciprocal world model as one massive 124 00:05:37,420 --> 00:05:40,319 unified neural network that ingests pixels and 125 00:05:40,319 --> 00:05:42,600 outputs pixels and decisions all in one breath. 126 00:05:42,879 --> 00:05:45,259 But they didn't do that. Right. Practically, 127 00:05:45,480 --> 00:05:48,500 for version one, using separate modules is just 128 00:05:48,500 --> 00:05:51,339 engineering pragmatism. It allows them to leverage 129 00:05:51,339 --> 00:05:53,779 existing highly optimized architectures for language 130 00:05:53,779 --> 00:05:56,060 and video generation separately. The modular 131 00:05:56,060 --> 00:05:58,339 split is just the specific choice they made for 132 00:05:58,339 --> 00:06:00,769 this iteration. Which makes sense. But making 133 00:06:00,769 --> 00:06:03,430 that choice creates a massive logistical headache. 134 00:06:03,529 --> 00:06:06,050 No, a huge one. Because if you split the brain, 135 00:06:06,310 --> 00:06:08,550 the intent module, and the body, the realization 136 00:06:08,550 --> 00:06:10,850 module, into two totally different pieces of 137 00:06:10,850 --> 00:06:13,470 software, how do they stay synced? That is the 138 00:06:13,470 --> 00:06:15,569 million dollar question. I mean, if I'm talking 139 00:06:15,569 --> 00:06:17,889 to this thing live, how do you prevent the intent 140 00:06:17,889 --> 00:06:20,589 module from racing three sentences ahead while 141 00:06:20,589 --> 00:06:22,589 the video generator is still trying to, like, 142 00:06:22,629 --> 00:06:25,089 draw the first word? That is the central problem 143 00:06:25,089 --> 00:06:27,209 of the entire paper. And to solve it, they had 144 00:06:27,209 --> 00:06:29,449 to fundamentally rethink how the system processes 145 00:06:29,449 --> 00:06:32,600 time. which brings us to the anatomy of the intent 146 00:06:32,600 --> 00:06:35,199 module and how it slices the conversation into 147 00:06:35,199 --> 00:06:37,300 chunks. Okay, let's get into the intent module. 148 00:06:37,720 --> 00:06:40,720 So this thing operates autoregressively. Yes. 149 00:06:41,040 --> 00:06:43,040 And just to break that down for everyone, it 150 00:06:43,040 --> 00:06:45,980 means it's constantly looking at the past to 151 00:06:45,980 --> 00:06:48,699 predict the immediate future. Exactly. It's ingesting 152 00:06:48,699 --> 00:06:51,589 the user's video feed. It's taking in the user's 153 00:06:51,589 --> 00:06:53,930 spoken words through an automatic speech recognition 154 00:06:53,930 --> 00:06:56,649 system. And it's looking at whatever text it 155 00:06:56,649 --> 00:06:59,290 previously generated itself. And based on all 156 00:06:59,290 --> 00:07:01,769 of that past context, it makes a decision. It 157 00:07:01,769 --> 00:07:04,069 outputs an action. And there are two main actions, 158 00:07:04,089 --> 00:07:07,350 right? Right. The two primary actions are speak, 159 00:07:07,649 --> 00:07:09,129 which would then be followed by the words it 160 00:07:09,129 --> 00:07:12,269 wants to say, or wait. And once it makes that 161 00:07:12,269 --> 00:07:14,550 decision, it has to tell the realization module 162 00:07:14,550 --> 00:07:16,930 what to do. But it doesn't just send over a text 163 00:07:16,930 --> 00:07:18,689 document saying, you know, draw me saying hello. 164 00:07:18,930 --> 00:07:22,199 No, no. It passes over what the paper calls hidden 165 00:07:22,199 --> 00:07:24,779 states. Hidden states. Yeah. And we need to be 166 00:07:24,779 --> 00:07:27,439 crystal clear on what a hidden state is. A hidden 167 00:07:27,439 --> 00:07:29,860 state is a learned mathematical representation. 168 00:07:30,459 --> 00:07:33,240 It is a highly compressed vector of numbers. 169 00:07:33,779 --> 00:07:35,740 So it's essentially handing over the math homework. 170 00:07:36,240 --> 00:07:38,990 Precisely. It is passing mathematical patterns 171 00:07:38,990 --> 00:07:42,110 that correlate with the intended action. We really 172 00:07:42,110 --> 00:07:44,430 shouldn't mistake this for the machine possessing 173 00:07:44,430 --> 00:07:47,470 true meaning or having a conscious thought that 174 00:07:47,470 --> 00:07:49,910 it shares with the video generator. Right, there's 175 00:07:49,910 --> 00:07:52,709 no actual soul or consciousness in there. Exactly. 176 00:07:53,110 --> 00:07:55,170 It's just a heavily optimized array of floats. 177 00:07:55,829 --> 00:07:59,069 Okay, so the math gets handed over, but how often 178 00:07:59,069 --> 00:08:01,370 does this handoff actually happen? This is where 179 00:08:01,370 --> 00:08:04,149 they introduce the concept of the 17 -frame chunk. 180 00:08:04,430 --> 00:08:07,110 Right. The system operates in discrete temporal 181 00:08:07,110 --> 00:08:09,949 buckets. It chunks the incoming video into groups 182 00:08:09,949 --> 00:08:13,629 of 17 frames. 17 frames. Yes. And if we assume 183 00:08:13,629 --> 00:08:15,930 a standard video frame rate of 24 frames per 184 00:08:15,930 --> 00:08:19,350 second, a chunk of 17 frames works out to approximately 185 00:08:19,350 --> 00:08:21,930 0 .71 seconds. Now I want to pause right here 186 00:08:21,930 --> 00:08:24,470 because this is the easiest place to get confused 187 00:08:24,470 --> 00:08:26,829 by the numbers. Definitely. If I hear that the 188 00:08:26,829 --> 00:08:30,230 system operates in 0 .71 second chunks, my immediate 189 00:08:30,230 --> 00:08:32,789 instinct is to think, oh, the AI has a reaction 190 00:08:32,789 --> 00:08:35,769 time of 0 .71 seconds. Right, which is totally 191 00:08:35,769 --> 00:08:38,570 wrong. Yeah, that is completely wrong. This 0 192 00:08:38,570 --> 00:08:41,769 .71 second window is purely a scheduling unit. 193 00:08:42,129 --> 00:08:44,009 That's a vital distinction. It is a scheduling 194 00:08:44,009 --> 00:08:46,289 unit used to organize the incoming and outgoing 195 00:08:46,289 --> 00:08:49,389 data. It is absolutely not a proof of a fixed 196 00:08:49,389 --> 00:08:51,909 end -to -end response latency. It reminds me 197 00:08:51,909 --> 00:08:55,139 of the postal system, actually. No, now so. Say 198 00:08:55,139 --> 00:08:57,120 a postal worker collects mail from a specific 199 00:08:57,120 --> 00:09:00,019 blue mailbox every hour on the dot. The one -hour 200 00:09:00,019 --> 00:09:02,179 interval is the scheduling unit. It organizes 201 00:09:02,179 --> 00:09:04,720 the workflow. OK, I follow. But if you drop a 202 00:09:04,720 --> 00:09:06,379 letter in that box, it doesn't mean your letter 203 00:09:06,379 --> 00:09:09,379 reaches its destination in one hour. The actual 204 00:09:09,379 --> 00:09:11,279 delivery time depends on the sorting facility, 205 00:09:11,500 --> 00:09:13,580 the trucks, the final carrier. That's a great 206 00:09:13,580 --> 00:09:16,399 way to visualize it. The intent module is making 207 00:09:16,399 --> 00:09:19,320 a fresh decision based on every 0 .71 second 208 00:09:19,320 --> 00:09:22,179 bucket of incoming data. Right. But the actual 209 00:09:22,179 --> 00:09:25,279 interaction latency The delay between you speaking 210 00:09:25,279 --> 00:09:28,419 and the avatar responding on your screen involves 211 00:09:28,419 --> 00:09:31,279 the processing time of the intent module, the 212 00:09:31,279 --> 00:09:33,860 generation time of the realization module, and 213 00:09:33,860 --> 00:09:36,120 the hardware decoding speeds. It's a whole pipeline. 214 00:09:36,500 --> 00:09:39,659 Okay, so inside this 0 .71 second bucket, there 215 00:09:39,659 --> 00:09:43,399 are 17 video frames, and there is a very specific, 216 00:09:43,679 --> 00:09:45,539 almost bizarre constraint about how these frames 217 00:09:45,539 --> 00:09:47,980 are grouped. You're referring to the causality 218 00:09:47,980 --> 00:09:51,110 constraint and the repeated frame? Yes. Because 219 00:09:51,110 --> 00:09:53,769 if the system is chunking data, it has to fill 220 00:09:53,769 --> 00:09:56,269 certain tensor dimensions natively, like the 221 00:09:56,269 --> 00:09:58,669 math requires a specific shape of data. Right. 222 00:09:58,830 --> 00:10:01,049 But if they group the frames in a way that accidentally 223 00:10:01,049 --> 00:10:03,370 pulls in the 18th frame to finish out a data 224 00:10:03,370 --> 00:10:05,610 block, they have a massive problem on their hands. 225 00:10:06,049 --> 00:10:08,730 Because they've broken causality. Exactly. Frame 226 00:10:08,730 --> 00:10:11,730 18 is technically the first frame of the next 227 00:10:11,730 --> 00:10:14,490 chunk. It represents a moment in time that hasn't 228 00:10:14,490 --> 00:10:17,070 officially arrived yet in the system's schedule. 229 00:10:17,269 --> 00:10:19,809 It's the future. Right. If the intent module 230 00:10:19,809 --> 00:10:22,649 looks at frame 18 before it finalizes its decision 231 00:10:22,649 --> 00:10:25,330 for the current chunk, it is essentially looking 232 00:10:25,330 --> 00:10:27,590 into the future. Which completely ruins the whole 233 00:10:27,590 --> 00:10:30,330 experiment of real -time processing. So how do 234 00:10:30,330 --> 00:10:32,669 they pad the data without looking into the future? 235 00:10:33,269 --> 00:10:35,889 They just repeat a frame. They just repeat the 236 00:10:35,889 --> 00:10:38,129 last frame of the current chunk. And because 237 00:10:38,129 --> 00:10:40,769 programmers start counting at zero... Right, 238 00:10:40,870 --> 00:10:42,889 the frames in this chunk are numbered 0 through 239 00:10:42,889 --> 00:10:47,409 16. So frame 16 is the 17th frame. Exactly. When 240 00:10:47,409 --> 00:10:49,970 the system needs to fill out the grouping, frame 241 00:10:49,970 --> 00:10:52,669 index 16 is paired with a duplicate of itself, 242 00:10:53,070 --> 00:10:55,549 rather than pulling in frame 17 from the future 243 00:10:55,549 --> 00:10:58,590 chunk. That's so simple. It's a very simple, 244 00:10:58,769 --> 00:11:00,830 rigorous way to ensure the mathematical timeline 245 00:11:00,830 --> 00:11:03,629 remains honest. The model can never peek at data 246 00:11:03,629 --> 00:11:07,149 that hasn't arrived. It's a clever brute force 247 00:11:07,149 --> 00:11:10,009 fix. Now, going back to the actions this module 248 00:11:10,009 --> 00:11:13,029 takes, it outputs speak or it outputs weight. 249 00:11:13,330 --> 00:11:15,330 I actually find the weight command fascinating. 250 00:11:15,529 --> 00:11:18,090 Oh, it really is. Because waiting isn't just 251 00:11:18,090 --> 00:11:20,330 the system crashing or, like, failing to load. 252 00:11:20,370 --> 00:11:23,090 It's a choice. It's an active, calculated decision. 253 00:11:23,230 --> 00:11:25,090 It's like a musician reading a piece of sheet 254 00:11:25,090 --> 00:11:28,570 music. When a trumpet player sees a rest on the 255 00:11:28,570 --> 00:11:30,509 page, they don't just turn their brain off. They 256 00:11:30,509 --> 00:11:33,230 are actively observing the tempo, listening to 257 00:11:33,230 --> 00:11:35,490 the rest of the orchestra, holding their instrument 258 00:11:35,490 --> 00:11:39,090 ready, and making a continuous, calculated decision 259 00:11:39,090 --> 00:11:42,419 to not blow into the horn. That decision to observe 260 00:11:42,419 --> 00:11:44,519 and remain silent is just as critical to the 261 00:11:44,519 --> 00:11:46,899 music as playing a note. Metamalogy captures 262 00:11:46,899 --> 00:11:49,700 the essence of full -duplex beautifully. The 263 00:11:49,700 --> 00:11:52,559 weight action allows a system to coexist with 264 00:11:52,559 --> 00:11:54,860 its own ongoing playback. Right, so it doesn't 265 00:11:54,860 --> 00:11:57,340 talk over itself. Exactly. Or to let the user 266 00:11:57,340 --> 00:12:00,200 finish a long rambling thought without interrupting. 267 00:12:00,740 --> 00:12:03,120 Or to simply maintain a natural non -verbal presence. 268 00:12:03,299 --> 00:12:06,539 It is a generated choice to remain silent. Okay, 269 00:12:07,000 --> 00:12:09,600 so the intent module makes its choice. Let's 270 00:12:09,600 --> 00:12:12,639 say it decides to speak. It generates the text. 271 00:12:12,809 --> 00:12:15,370 It creates those mathematical hidden states and 272 00:12:15,370 --> 00:12:17,990 it hands them over. Which brings us to the realization 273 00:12:17,990 --> 00:12:20,610 module and joint generation. Right. And this 274 00:12:20,610 --> 00:12:22,730 module has a heavy lift. Oh, incredibly heavy. 275 00:12:23,149 --> 00:12:25,169 The realization module has the task of turning 276 00:12:25,169 --> 00:12:28,549 those mathematical vectors along with the static 277 00:12:28,549 --> 00:12:30,450 reference image of what the assistant is supposed 278 00:12:30,450 --> 00:12:33,590 to look like into actual moving pixels and audio 279 00:12:33,590 --> 00:12:36,350 waveforms. And they are using a joint causal 280 00:12:36,350 --> 00:12:39,450 diffusion transformer and variational autoencoders 281 00:12:39,450 --> 00:12:42,330 or VAEs to do this. We really need to talk about 282 00:12:42,330 --> 00:12:44,750 the VAEs first. Especially for the audio. Yeah, 283 00:12:44,850 --> 00:12:47,289 because the compression here is intense. Right. 284 00:12:47,529 --> 00:12:49,929 So a variational autoencoder essentially takes 285 00:12:49,929 --> 00:12:52,610 high dimensional data, compresses it down into 286 00:12:52,610 --> 00:12:55,720 a much smaller, dense representation, call it 287 00:12:55,720 --> 00:12:58,279 latent space, and then learns how to decompress 288 00:12:58,279 --> 00:13:00,480 it back to the original form. So imagine you 289 00:13:00,480 --> 00:13:03,299 have a massive, incredibly detailed oil painting. 290 00:13:03,639 --> 00:13:06,080 You can't just mail this painting quickly. No, 291 00:13:06,080 --> 00:13:08,779 it's too big. So an encoder acts like an artist 292 00:13:08,779 --> 00:13:10,899 who looks at the oil painting and draws a very 293 00:13:10,899 --> 00:13:13,480 quick, highly precise pencil sketch that captures 294 00:13:13,480 --> 00:13:16,019 the exact dimensions and colors just in shorthand. 295 00:13:16,139 --> 00:13:18,799 You mail the pencil sketch. The decoder on the 296 00:13:18,799 --> 00:13:20,879 other side looks at the sketch and paints the 297 00:13:20,879 --> 00:13:23,090 massive oil painting again. It's a way to move 298 00:13:23,090 --> 00:13:25,649 heavy data incredibly fast. It's a great way 299 00:13:25,649 --> 00:13:29,629 to put it. And in Duo V1, they compress the raw 300 00:13:29,629 --> 00:13:33,049 audio waveforms into 40 of these latent vectors 301 00:13:33,049 --> 00:13:36,450 per second. 40 per second. Yes. So if you do 302 00:13:36,450 --> 00:13:39,929 the math, each latent vector represents 25 milliseconds 303 00:13:39,929 --> 00:13:42,590 of audio. But again, let's stop and clarify the 304 00:13:42,590 --> 00:13:44,549 numbers here for everyone listening. Yes, please. 305 00:13:45,070 --> 00:13:48,330 That 25 millisecond latent. represents a duration 306 00:13:48,330 --> 00:13:50,809 of sound that is being processed. It does not 307 00:13:50,809 --> 00:13:53,690 mean the interaction latency is 25 milliseconds. 308 00:13:54,210 --> 00:13:56,409 Correct. It means they took one second of spoken 309 00:13:56,409 --> 00:13:59,730 audio and sliced it into 40 equal pieces. It's 310 00:13:59,730 --> 00:14:01,850 about the size of the data packet, not the speed 311 00:14:01,850 --> 00:14:04,570 of the conversation. It's the size of the slice 312 00:14:04,570 --> 00:14:06,830 of pie, not how long it takes to bake the pie. 313 00:14:06,990 --> 00:14:10,350 Exactly. Okay, so we have audio latency and video 314 00:14:10,350 --> 00:14:12,289 latency, but the big challenge of generating 315 00:14:12,289 --> 00:14:15,049 both at the same time is alignment. Right, synchronization. 316 00:14:15,169 --> 00:14:18,080 Yeah, because... If the audio says the word hello, 317 00:14:18,539 --> 00:14:20,559 the video better show the lips forming the word 318 00:14:20,559 --> 00:14:23,580 hello at the exact same millisecond. To manage 319 00:14:23,580 --> 00:14:25,460 this synchronization, they utilize something 320 00:14:25,460 --> 00:14:29,600 called 3D Rope. 3D Rope. Which stands for 3D 321 00:14:29,600 --> 00:14:32,639 Rotary Position Embedding. And how does that 322 00:14:32,639 --> 00:14:34,860 actually work in practice, like, mechanically? 323 00:14:35,440 --> 00:14:37,539 Well, you can think of positional embeddings 324 00:14:37,539 --> 00:14:41,340 as giving every single token of data a very specific 325 00:14:41,340 --> 00:14:44,659 set of GPS coordinates on the timeline. OK. Standard 326 00:14:44,659 --> 00:14:47,779 language models use 1D position embeddings like 327 00:14:47,779 --> 00:14:50,080 Word 1, Word 2, Word 3. Just a straight line. 328 00:14:50,299 --> 00:14:53,279 Right. But video and audio have spatial dimensions 329 00:14:53,279 --> 00:14:57,320 and temporal dimensions. So 3D Rope assigns coordinates 330 00:14:57,320 --> 00:15:00,220 to the tokens so that the audio tokens and the 331 00:15:00,220 --> 00:15:02,899 video tokens know exactly where they belong relative 332 00:15:02,899 --> 00:15:06,220 to each other on a shared unified playback timeline. 333 00:15:06,779 --> 00:15:09,379 So it mathematically forces the sound and the 334 00:15:09,379 --> 00:15:11,919 picture to hold hands basically. Yes. But, and 335 00:15:11,919 --> 00:15:14,899 we really need to emphasize this, while 3D Rope 336 00:15:14,899 --> 00:15:17,679 and this joint generation process aim to improve 337 00:15:17,679 --> 00:15:20,740 alignment, they are not magic. They absolutely 338 00:15:20,740 --> 00:15:22,879 do not guarantee perfect lip synchronization. 339 00:15:23,519 --> 00:15:25,379 They do not guarantee that the video won't drift 340 00:15:25,379 --> 00:15:27,860 out of sync over time. And they certainly do 341 00:15:27,860 --> 00:15:29,559 not guarantee that you could run this system 342 00:15:29,559 --> 00:15:32,919 for a reliable, flawless 20 -minute conversation. 343 00:15:33,320 --> 00:15:36,240 Exactly. It's a mitigation strategy, not a flawless 344 00:15:36,240 --> 00:15:39,730 cure. The physics of streaming generation mean 345 00:15:39,730 --> 00:15:43,389 that minor desynchronizations are basically inevitable 346 00:15:43,389 --> 00:15:46,929 right now. Now, how does the realization module 347 00:15:46,929 --> 00:15:49,830 actually draw the frames? The paper mentions 348 00:15:49,830 --> 00:15:52,529 it uses a technique called flow matching. Yes. 349 00:15:52,990 --> 00:15:55,409 So instead of formulating the diffusion process 350 00:15:55,409 --> 00:15:57,990 in the standard way, they use flow matching. 351 00:15:58,450 --> 00:16:00,269 And to understand this, we have to look at how 352 00:16:00,269 --> 00:16:03,350 generation actually works. OK. It starts with 353 00:16:03,350 --> 00:16:07,289 pure static noise. just random pixels. The goal 354 00:16:07,289 --> 00:16:10,450 is to turn that noise into a clear image. Now, 355 00:16:10,590 --> 00:16:13,710 a lot of people mistakenly describe standard 356 00:16:13,710 --> 00:16:16,889 diffusion as an unguided random walk, where the 357 00:16:16,889 --> 00:16:18,990 system sort of like stumbles around until it 358 00:16:18,990 --> 00:16:20,830 finds a picture. Right, and we need to strictly 359 00:16:20,830 --> 00:16:23,289 avoid that description. Yeah, it's not a random 360 00:16:23,289 --> 00:16:25,929 walk. Diffusion is a highly structured process 361 00:16:25,929 --> 00:16:29,360 of removing noise step by step. Flow matching, 362 00:16:29,580 --> 00:16:31,940 however, is a specific mathematical framework 363 00:16:31,940 --> 00:16:35,120 that focuses on predicting an optimal velocity. 364 00:16:35,559 --> 00:16:38,309 Optimal velocity. Yeah. Imagine the pure noise 365 00:16:38,309 --> 00:16:41,009 is a point in a massive high -dimensional space, 366 00:16:41,409 --> 00:16:43,590 and the perfect clean video frame is another 367 00:16:43,590 --> 00:16:46,629 point. Okay, two points. Flow matching trains 368 00:16:46,629 --> 00:16:49,110 the model to learn the vector field, the exact 369 00:16:49,110 --> 00:16:51,929 direction and speed required to travel from the 370 00:16:51,929 --> 00:16:55,029 noise directly to the target signal. So it's 371 00:16:55,029 --> 00:16:57,429 trying to find the straightest, most efficient 372 00:16:57,429 --> 00:16:59,909 highway between the static and the clear image 373 00:16:59,909 --> 00:17:02,470 rather than taking a winding back road. Exactly. 374 00:17:02,789 --> 00:17:05,130 It's a very efficient way to formulate the generative 375 00:17:05,130 --> 00:17:08,289 process. But this brings up a huge question for 376 00:17:08,289 --> 00:17:10,450 me regarding how they train this model. Okay, 377 00:17:10,589 --> 00:17:13,069 what is it? If it's learning to draw a path to 378 00:17:13,069 --> 00:17:15,829 a clean target signal, like a video of a person 379 00:17:15,829 --> 00:17:18,369 talking, how do you stop the model from cheating 380 00:17:18,369 --> 00:17:21,529 during training? Because if you feed it a recorded 381 00:17:21,529 --> 00:17:23,410 video clip to practice on, how do you prevent 382 00:17:23,410 --> 00:17:26,109 the AI from just peeking at the end of the video 383 00:17:26,109 --> 00:17:28,269 file to see what it's supposed to draw? Well, 384 00:17:28,269 --> 00:17:30,450 if you are training a human, you can just cover 385 00:17:30,450 --> 00:17:33,769 their eyes, right? Sure. To blindfold an AI, 386 00:17:34,049 --> 00:17:36,950 you need mathematical constraints. And this leads 387 00:17:36,950 --> 00:17:40,609 us to the incredibly complex three -stage training 388 00:17:40,609 --> 00:17:43,509 strategy the team employed. OK, let's break these 389 00:17:43,509 --> 00:17:45,230 stages down, because this is where the system 390 00:17:45,230 --> 00:17:47,410 actually learns the choreography of the full 391 00:17:47,410 --> 00:17:50,180 duplex dance. Right. Stage one is entirely focused 392 00:17:50,180 --> 00:17:52,980 on the intent module. They call it causal supervised 393 00:17:52,980 --> 00:17:56,660 fine tuning. Yes, in stage one, the multimodal 394 00:17:56,660 --> 00:17:59,059 language backbone is learning when to output 395 00:17:59,059 --> 00:18:01,880 speak and when to output weight. And one of the 396 00:18:01,880 --> 00:18:03,680 most critical interventions they make here is 397 00:18:03,680 --> 00:18:06,559 called action aware rebalancing. I found this 398 00:18:06,559 --> 00:18:09,339 fascinating because if you just scrape a bunch 399 00:18:09,339 --> 00:18:11,440 of conversations and feed them to a model, you 400 00:18:11,440 --> 00:18:14,000 run into a statistical problem. You do in a normal 401 00:18:14,000 --> 00:18:15,980 conversation while one person is talking, the 402 00:18:15,980 --> 00:18:17,819 other person is usually quiet and listening. 403 00:18:18,029 --> 00:18:20,710 So statistically, the most common action for 404 00:18:20,710 --> 00:18:24,109 an assistant is to do nothing. And neural networks 405 00:18:24,109 --> 00:18:27,190 are incredibly lazy. They really are. They will 406 00:18:27,190 --> 00:18:29,230 always try to optimize their loss function by 407 00:18:29,230 --> 00:18:31,869 finding the easiest path. So if the training 408 00:18:31,869 --> 00:18:34,069 data shows that 85 % of the time the correct 409 00:18:34,069 --> 00:18:36,910 answer is to wait, the model might realize, hey, 410 00:18:37,069 --> 00:18:40,250 if I just output wait permanently, I get a great 411 00:18:40,250 --> 00:18:42,450 score and I never have to risk saying the wrong 412 00:18:42,450 --> 00:18:44,869 word. It just learns to become a very polite 413 00:18:44,869 --> 00:18:47,789 mute. Exactly. It collapses into perpetual silence. 414 00:18:49,069 --> 00:18:51,690 So action -aware rebalancing manually adjusts 415 00:18:51,690 --> 00:18:53,609 the weights during the training process. It forces 416 00:18:53,609 --> 00:18:56,369 it to talk. Yes. It tells the mathematical loss 417 00:18:56,369 --> 00:18:58,809 function that the decision to open a conversation 418 00:18:58,809 --> 00:19:02,109 or the rare decision to interrupt the user carries 419 00:19:02,109 --> 00:19:04,710 much heavier importance than the baseline decision 420 00:19:04,710 --> 00:19:07,549 to weight. It forces the model to value speaking. 421 00:19:07,809 --> 00:19:09,549 But they also do something else in stage one 422 00:19:09,549 --> 00:19:12,869 that I think is brilliant engineering. They intentionally 423 00:19:12,869 --> 00:19:16,420 train the model on imperfect history. absolute 424 00:19:16,420 --> 00:19:18,500 necessity. Right, because if you only train a 425 00:19:18,500 --> 00:19:21,079 system on perfect, crisp, Aaron Sorkin -style 426 00:19:21,079 --> 00:19:23,640 alternating dialogue, it will completely shatter 427 00:19:23,640 --> 00:19:25,859 the first time it encounters the awkward reality 428 00:19:25,859 --> 00:19:27,900 of the live environment. Oh, yeah. Think about 429 00:19:27,900 --> 00:19:30,059 how many times on a Zoom call someone freezes 430 00:19:30,059 --> 00:19:32,619 or two people start talking, stop and say, no, 431 00:19:32,779 --> 00:19:35,579 you go ahead. Exactly. If the AI has never seen 432 00:19:35,579 --> 00:19:39,009 a mistake, it can't recover. So during stage 433 00:19:39,009 --> 00:19:41,329 one, the engineers actually perturb the training 434 00:19:41,329 --> 00:19:43,809 data. They artificially mess it up? They do. 435 00:19:44,009 --> 00:19:46,269 They might take a perfectly good conversational 436 00:19:46,269 --> 00:19:49,069 transcript and truncate a sentence halfway through. 437 00:19:49,349 --> 00:19:51,349 Or they might replace a natural opening with 438 00:19:51,349 --> 00:19:53,529 a wait command, essentially forcing the model 439 00:19:53,529 --> 00:19:56,450 into a state where it missed its cue. It's like 440 00:19:56,450 --> 00:19:58,769 pushing a robot while it's trying to learn to 441 00:19:58,769 --> 00:20:01,369 walk. Yes. You are forcing the model to realize, 442 00:20:01,569 --> 00:20:03,829 oh no, I messed up the timing, and then forcing 443 00:20:03,829 --> 00:20:06,089 it to mathematically recover from that missed 444 00:20:06,089 --> 00:20:09,069 opening in the very next chunk. You are literally 445 00:20:09,069 --> 00:20:11,890 training the system to be socially graceful when 446 00:20:11,890 --> 00:20:15,000 a conversational collision occurs. Because by 447 00:20:15,000 --> 00:20:17,059 exposing the model to its own potential failures 448 00:20:17,059 --> 00:20:19,480 during training, you ensure that during live 449 00:20:19,480 --> 00:20:22,480 inference, a minor delay or an unexpected interruption 450 00:20:22,480 --> 00:20:24,980 doesn't cause a cascading failure of the entire 451 00:20:24,980 --> 00:20:27,180 interaction. It knows how to pick up the pieces. 452 00:20:27,579 --> 00:20:29,720 OK, so that's stage one for the brain. Stage 453 00:20:29,720 --> 00:20:32,799 two is the realization training, teaching the 454 00:20:32,799 --> 00:20:35,779 body how to render the video and audio. And this 455 00:20:35,779 --> 00:20:37,740 answers my question about how you stop the model 456 00:20:37,740 --> 00:20:39,640 from cheating and looking ahead at the video 457 00:20:39,640 --> 00:20:43,089 file. Yes. In stage two, they implement causal 458 00:20:43,089 --> 00:20:45,809 attention masks. What does that mask actually 459 00:20:45,809 --> 00:20:48,049 look like in the architecture, like mathematically? 460 00:20:49,190 --> 00:20:52,190 Conceptually, imagine a massive matrix of ones 461 00:20:52,190 --> 00:20:54,730 and zeros that acts as a filter over the model's 462 00:20:54,730 --> 00:20:57,509 attention mechanism. Okay. When the model is 463 00:20:57,509 --> 00:21:00,049 looking at the noisy media tokens, the static 464 00:21:00,049 --> 00:21:02,390 that it is trying to resolve into a clean video 465 00:21:02,390 --> 00:21:05,529 frame, the mask allows those tokens to look backward. 466 00:21:05,970 --> 00:21:07,920 So they can see the past. Right. They can see 467 00:21:07,920 --> 00:21:10,259 the previous video frames. They can see the user's 468 00:21:10,259 --> 00:21:12,539 input. They can see the global system instructions. 469 00:21:13,140 --> 00:21:15,140 But the mask puts up a brick wall when it comes 470 00:21:15,140 --> 00:21:18,299 to the present and the future. Exactly. The mask 471 00:21:18,299 --> 00:21:20,900 absolutely fundamentally blocks the noisy tokens 472 00:21:20,900 --> 00:21:23,259 from looking at their own clean target outputs. 473 00:21:23,960 --> 00:21:26,299 And it blocks them from looking at any data from 474 00:21:26,299 --> 00:21:29,619 future chunks. It's a mathematical blinder. It 475 00:21:29,619 --> 00:21:32,740 is. You are permitted to look at the past to 476 00:21:32,740 --> 00:21:35,180 maintain continuity, but you cannot look at the 477 00:21:35,180 --> 00:21:37,750 answer key. and you cannot look forward in time. 478 00:21:38,349 --> 00:21:40,109 Okay, so stage one teaches it when to speak. 479 00:21:40,190 --> 00:21:42,829 Stage two teaches it how to render video without 480 00:21:42,829 --> 00:21:45,869 cheating. Stage three is where it gets highly 481 00:21:45,869 --> 00:21:48,069 optimized. They call it distribution matching 482 00:21:48,069 --> 00:21:50,930 distillation or DMD. Right. Because up until 483 00:21:50,930 --> 00:21:52,869 this point, the flow matching generation process 484 00:21:52,869 --> 00:21:55,809 takes quite a few computational steps to turn 485 00:21:55,809 --> 00:21:58,630 pure noise into a clean video frame. Which is 486 00:21:58,630 --> 00:22:01,509 too slow. Way too slow for a live real -time 487 00:22:01,509 --> 00:22:04,710 conversation. DMD is the distillation process 488 00:22:04,710 --> 00:22:06,869 used to condense that generation down to just 489 00:22:06,869 --> 00:22:09,630 four steps. And to do this distillation in a 490 00:22:09,630 --> 00:22:11,869 way that mimics a real conversation, they use 491 00:22:11,869 --> 00:22:15,009 a technique called dual rollout. Dual rollout 492 00:22:15,009 --> 00:22:18,089 is fascinating because it forces the two modules 493 00:22:18,089 --> 00:22:20,890 to practice together. During this training phase, 494 00:22:21,150 --> 00:22:23,710 they freeze the intent module. Freeze it? Yeah. 495 00:22:23,849 --> 00:22:25,950 Its weights are locked. It is no longer learning. 496 00:22:26,049 --> 00:22:29,170 It is simply acting. So they let this frozen 497 00:22:29,170 --> 00:22:31,630 intent module play out the language of response 498 00:22:31,630 --> 00:22:34,170 autoregressively, word by word, chunk by chunk. 499 00:22:34,829 --> 00:22:36,710 And as the intent module is spitting out those 500 00:22:36,710 --> 00:22:39,450 decisions, the realization module has to generate 501 00:22:39,450 --> 00:22:41,950 the accompanying audio and video history in tandem. 502 00:22:42,490 --> 00:22:44,710 It's like taking a dancer who has been practicing 503 00:22:44,710 --> 00:22:47,390 in a studio with a recorded track and suddenly 504 00:22:47,390 --> 00:22:49,609 forcing them to perform live on stage with a 505 00:22:49,609 --> 00:22:52,009 jazz band that might change the tempo. That's 506 00:22:52,009 --> 00:22:54,509 a brilliant way to phrase it. You are forcing 507 00:22:54,509 --> 00:22:56,750 the modules to practice their live, unpredictable 508 00:22:56,750 --> 00:22:59,170 dance together during the training phase, rather 509 00:22:59,170 --> 00:23:01,430 than just feeding them perfectly synced, pre 510 00:23:01,430 --> 00:23:03,849 -recorded data. It ensures the training environment 511 00:23:03,849 --> 00:23:06,349 mirrors the chaotic conditions of live inference. 512 00:23:06,970 --> 00:23:09,369 But when you distill video generation down to 513 00:23:09,369 --> 00:23:12,309 just four steps, you introduce a new problem. 514 00:23:12,690 --> 00:23:15,829 Visual drift. Visual drift. Because you are taking 515 00:23:15,829 --> 00:23:18,529 fewer steps to get to the final image, the model 516 00:23:18,529 --> 00:23:22,029 can get slightly sloppy with the geometry. Right. 517 00:23:22,410 --> 00:23:25,190 The background might start to warp or the avatar's 518 00:23:25,190 --> 00:23:28,089 head might slowly start to rotate unnaturally 519 00:23:28,089 --> 00:23:31,069 over time because the distribution matching math 520 00:23:31,069 --> 00:23:33,930 doesn't explicitly penalize that kind of subtle 521 00:23:33,930 --> 00:23:36,910 geometric drift. To counteract this, they introduce 522 00:23:36,910 --> 00:23:40,990 an auxiliary metric, camera rotation aware weighting. 523 00:23:41,190 --> 00:23:43,730 And how does that work? They use an external 524 00:23:43,730 --> 00:23:47,240 frozen geometry tracking model. As the realization 525 00:23:47,240 --> 00:23:50,000 module generates frames during the rollout, this 526 00:23:50,000 --> 00:23:52,559 tracker analyzes them and estimates the camera 527 00:23:52,559 --> 00:23:55,640 rotation. If it detects that the generated sequence 528 00:23:55,640 --> 00:23:58,519 is exhibiting sustained, unnatural rotational 529 00:23:58,519 --> 00:24:01,039 drift, like if the room is starting to spin, 530 00:24:01,660 --> 00:24:04,140 this metric penalizes the training update. It 531 00:24:04,140 --> 00:24:07,359 corrects it. Essentially, yes. It rebalances 532 00:24:07,359 --> 00:24:09,759 the distillation process to heavily favor visually 533 00:24:09,759 --> 00:24:12,720 stable, geometrically sound windows. It's like 534 00:24:12,720 --> 00:24:14,920 an automated director yelling, keep the camera 535 00:24:14,920 --> 00:24:17,339 steady. during the training process. Exactly. 536 00:24:17,799 --> 00:24:19,880 All right, let's zoom out for a second. We've 537 00:24:19,880 --> 00:24:21,480 talked extensively about the architecture and 538 00:24:21,480 --> 00:24:23,660 how they train it, but we haven't talked about 539 00:24:23,660 --> 00:24:26,779 what they are actually feeding it. Ah, the data. 540 00:24:27,059 --> 00:24:30,160 Right. Where does a lab get enough data to train 541 00:24:30,160 --> 00:24:33,140 a full duplex conversational model? Let's look 542 00:24:33,140 --> 00:24:36,039 at data construction in the Luma baseline. The 543 00:24:36,039 --> 00:24:39,579 data set is a meticulously engineered mix. They 544 00:24:39,579 --> 00:24:42,700 do use real recorded data sets, things like live 545 00:24:42,700 --> 00:24:45,480 streams, podcasts and interviews. Which are crucial. 546 00:24:45,960 --> 00:24:48,319 They are because they provide the messy natural 547 00:24:48,319 --> 00:24:50,980 timing of human speech, the overlapping dialogue, 548 00:24:51,160 --> 00:24:54,140 the natural pauses. But there isn't enough high 549 00:24:54,140 --> 00:24:56,440 quality perfectly framed conversational video 550 00:24:56,440 --> 00:24:58,920 in the world to train this purely on real data. 551 00:24:59,220 --> 00:25:01,720 No, nowhere near enough. So they rely heavily 552 00:25:01,720 --> 00:25:04,000 on synthetic data, and the pipeline they built 553 00:25:04,000 --> 00:25:07,119 to generate the synthetic data is wild. They 554 00:25:07,119 --> 00:25:09,859 basically built an automated Hollywood studio. 555 00:25:10,680 --> 00:25:13,059 They use large language models to act as screenwriters. 556 00:25:13,500 --> 00:25:15,420 The engineers provide the LLM with what they 557 00:25:15,420 --> 00:25:18,920 call control cards to force variety into the 558 00:25:18,920 --> 00:25:21,859 generated scripts. Let's invent an example of 559 00:25:21,859 --> 00:25:24,819 a control card to visualize this. Imagine the 560 00:25:24,819 --> 00:25:28,059 engineers hand the LLM a prompt that says, write 561 00:25:28,059 --> 00:25:30,819 a screenplay for a video call. Characters, an 562 00:25:30,819 --> 00:25:33,319 architect, and a frustrated client. Duration, 563 00:25:33,660 --> 00:25:37,059 60 seconds. Required micro actions at second 564 00:25:37,059 --> 00:25:39,799 15, the client must squint their eyes. At second 565 00:25:39,799 --> 00:25:43,359 45, the architect must nod reassuringly. So the 566 00:25:43,359 --> 00:25:46,259 LLM processes those constraints and outputs a 567 00:25:46,259 --> 00:25:49,900 highly structured time stamped script. Then a 568 00:25:49,900 --> 00:25:52,400 templating system converts that text script into 569 00:25:52,400 --> 00:25:55,420 a massive complex prompt for a text to video 570 00:25:55,420 --> 00:25:58,210 generation model. And the video model renders 571 00:25:58,210 --> 00:26:01,390 the entire 60 -second interaction, creating the 572 00:26:01,390 --> 00:26:04,509 user and the assistant on a unified visual timeline, 573 00:26:04,690 --> 00:26:07,289 guaranteeing that the visual reactions, the squinting, 574 00:26:07,390 --> 00:26:10,910 the nodding, match the audio perfectly. But generating 575 00:26:10,910 --> 00:26:14,269 long, continuous video introduces a severe issue 576 00:26:14,269 --> 00:26:17,130 known as error accumulation, which leads us to 577 00:26:17,130 --> 00:26:20,210 the Luma baseline problem. Right. When you are 578 00:26:20,210 --> 00:26:23,490 generating video block by block, say you generate 579 00:26:23,490 --> 00:26:25,609 the first five seconds, and then you try to generate 580 00:26:25,609 --> 00:26:27,769 the next five seconds, conditioned on the final 581 00:26:27,769 --> 00:26:30,490 frames of the first block, errors start to pile 582 00:26:30,490 --> 00:26:32,910 up. In the baseline method they tested, where 583 00:26:32,910 --> 00:26:35,769 each new block only looks at the immediate previous 584 00:26:35,769 --> 00:26:39,650 frames, the visual quality degrades incredibly 585 00:26:39,650 --> 00:26:43,069 fast. It is exactly like making a photocopy of 586 00:26:43,069 --> 00:26:45,690 a photocopy of a photocopy. Yes. And by the time 587 00:26:45,690 --> 00:26:48,059 you get to the tenth copy, All the crisp lines 588 00:26:48,059 --> 00:26:51,000 are gone, the contrast is blown out, and the 589 00:26:51,000 --> 00:26:53,980 image is just a dark, shadowy mess. In their 590 00:26:53,980 --> 00:26:57,400 testing, the baseline method lost 34 .2 luma 591 00:26:57,400 --> 00:26:59,940 levels over the course of the generation. And 592 00:26:59,940 --> 00:27:02,180 here we need to explicitly define our terms for 593 00:27:02,180 --> 00:27:06,200 everyone. What is luma? Luma is strictly a brightness 594 00:27:06,200 --> 00:27:08,799 value. Right. In digital video, it is usually 595 00:27:08,799 --> 00:27:13,000 measured on a standard scale from 0 to 255. 0 596 00:27:13,000 --> 00:27:17,019 is pitch black, 255 is pure white. Yes. So when 597 00:27:17,019 --> 00:27:19,839 the paper states that the baseline lost 34 .2 598 00:27:19,839 --> 00:27:22,599 luma levels, it literally means the generated 599 00:27:22,599 --> 00:27:25,519 video became drastically darker over time. It 600 00:27:25,519 --> 00:27:28,819 was degrading into shadows. Exactly. And it is 601 00:27:28,819 --> 00:27:30,819 absolutely crucial that we do not misinterpret 602 00:27:30,819 --> 00:27:33,380 this metric. A luma value is not a percentage. 603 00:27:33,660 --> 00:27:36,039 Nope. It does not mean the video retained 99 604 00:27:36,039 --> 00:27:39,539 % quality or 99 % accuracy or 99 % fidelity. 605 00:27:39,980 --> 00:27:42,539 It is strictly a measure of brightness and exposure. 606 00:27:43,240 --> 00:27:46,700 That is a vital distinction. So the baseline 607 00:27:46,700 --> 00:27:49,099 video gets progressively darker and more distorted. 608 00:27:49,740 --> 00:27:52,980 How did the Duo V1 team fix the photocopy problem? 609 00:27:53,299 --> 00:27:56,869 They used a fixed block zero anchor. Instead 610 00:27:56,869 --> 00:27:59,549 of only conditioning the next block on the immediately 611 00:27:59,549 --> 00:28:02,329 preceding frames, they take the very first video 612 00:28:02,329 --> 00:28:04,990 segment, block zero, and they use it as a persistent 613 00:28:04,990 --> 00:28:07,049 fixed reference for picture quality throughout 614 00:28:07,049 --> 00:28:10,009 the entire generation process. So it's like keeping 615 00:28:10,009 --> 00:28:12,640 the original pristine photographed? tape to the 616 00:28:12,640 --> 00:28:14,579 corner of the scanner glass every single time 617 00:28:14,579 --> 00:28:16,599 you make a copy. Right. The machine can always 618 00:28:16,599 --> 00:28:19,180 look at the original to remember what the lighting 619 00:28:19,180 --> 00:28:21,299 and the exposure are actually supposed to look 620 00:28:21,299 --> 00:28:24,000 like. Precisely. By anchoring every subsequent 621 00:28:24,000 --> 00:28:26,759 generation block to that original block zero 622 00:28:26,759 --> 00:28:29,680 reference, they stabilize the visual consistency. 623 00:28:29,980 --> 00:28:32,700 The paper notes that with this method, the Luma 624 00:28:32,700 --> 00:28:37,539 value stabilizes near 99 on that 0 to 255 scale, 625 00:28:37,839 --> 00:28:40,539 completely preventing the accumulated darkening 626 00:28:40,539 --> 00:28:43,710 effect. It's a very elegant solution to a very 627 00:28:43,710 --> 00:28:47,109 messy problem. OK. We've explored the world model 628 00:28:47,109 --> 00:28:49,750 framework, the intent chunking, the realization 629 00:28:49,750 --> 00:28:52,589 flow matching, the training stages, and the data 630 00:28:52,589 --> 00:28:54,910 pipeline. That's a lot of ground. It is. But 631 00:28:54,910 --> 00:28:57,049 now we have to talk about the physical reality 632 00:28:57,049 --> 00:29:00,829 of running this software. Let's look at the orchestrated 633 00:29:00,829 --> 00:29:03,589 streaming pipeline. Because you can't just run 634 00:29:03,589 --> 00:29:06,049 this on a standard laptop. They used a massive 635 00:29:06,049 --> 00:29:09,819 8 GPU server. And getting eight GPUs to act as 636 00:29:09,819 --> 00:29:12,640 a single cohesive brain for real -time interaction 637 00:29:12,640 --> 00:29:15,920 is an exercise in extreme hardware orchestration. 638 00:29:16,059 --> 00:29:18,720 I can imagine. The bottleneck isn't just compute 639 00:29:18,720 --> 00:29:21,279 power, it's the physical transfer of data. Right, 640 00:29:21,359 --> 00:29:24,240 because if GPU 1 is processing the intent and 641 00:29:24,240 --> 00:29:27,460 GPU 5 is supposed to draw the video, GPU 1 has 642 00:29:27,460 --> 00:29:30,019 to physically send massive amounts of data across 643 00:29:30,019 --> 00:29:32,779 the motherboard's wires to GPU 5 in a fraction 644 00:29:32,779 --> 00:29:35,529 of a second. To achieve low latency, They have 645 00:29:35,529 --> 00:29:38,509 to overlap tasks in a beautifully choreographed 646 00:29:38,509 --> 00:29:41,829 relay race. So they split the server in half? 647 00:29:42,250 --> 00:29:45,829 They do. While GPUs 0 through 3 are running the 648 00:29:45,829 --> 00:29:48,589 intent module, they are analyzing the incoming 649 00:29:48,589 --> 00:29:51,150 data and thinking about chunk k plus 1, which 650 00:29:51,150 --> 00:29:54,630 is the future chunk. GPUs 4 through 7 are simultaneously 651 00:29:54,630 --> 00:29:57,470 decoding the generated video latents for chunk 652 00:29:57,470 --> 00:30:00,960 k, which is the current chunk. The brain is calculating 653 00:30:00,960 --> 00:30:03,180 the next sentence on half the server while the 654 00:30:03,180 --> 00:30:05,099 body is rendering the current sentence on the 655 00:30:05,099 --> 00:30:08,460 other half. Wow. But to make that overlap work, 656 00:30:08,799 --> 00:30:11,480 they rely on what they call Unified KV Management. 657 00:30:12,059 --> 00:30:15,359 KV stands for Key Value Cache. For anyone unfamiliar, 658 00:30:15,619 --> 00:30:17,660 you can think of the KV cache as the model's 659 00:30:17,660 --> 00:30:20,000 short -term working memory. It stores the context 660 00:30:20,000 --> 00:30:22,480 of the conversation so far. And because the intent 661 00:30:22,480 --> 00:30:25,160 module and the realization module both need access 662 00:30:25,160 --> 00:30:27,859 to this memory, the system has to carefully manage 663 00:30:27,859 --> 00:30:30,579 swapping this massive cache across the memory 664 00:30:30,579 --> 00:30:33,660 banks of all eight GPUs. So it's constantly shuffling 665 00:30:33,660 --> 00:30:36,440 data. Exactly. The orchestration ensures that 666 00:30:36,440 --> 00:30:39,339 neither module is ever sitting idle waiting for 667 00:30:39,339 --> 00:30:43,019 data to traverse the PCIe bus. The speed of the 668 00:30:43,019 --> 00:30:45,519 decoding is the ultimate deadline here. The streaming 669 00:30:45,519 --> 00:30:48,880 VAE, the specific decoder that turns the mathematical 670 00:30:48,880 --> 00:30:52,559 latents back into actual visible pixels, decodes 671 00:30:52,559 --> 00:30:55,319 one latent into four video frames in about 30 672 00:30:55,319 --> 00:30:57,720 milliseconds. And that 30 millisecond benchmark 673 00:30:57,720 --> 00:31:00,940 is vital. Why is that? Because if the video is 674 00:31:00,940 --> 00:31:03,720 playing at 24 frames per second, the absolute 675 00:31:03,720 --> 00:31:05,960 hard deadline to render those frames before the 676 00:31:05,960 --> 00:31:09,880 video stutters is approximately 41 .7 milliseconds. 677 00:31:10,180 --> 00:31:12,980 Okay, math checks out. Because 30 milliseconds 678 00:31:12,980 --> 00:31:15,859 is comfortably less than 41 .7 milliseconds, 679 00:31:16,400 --> 00:31:18,380 the decoder stays ahead of the playback deadline. 680 00:31:19,140 --> 00:31:22,359 The video remains smooth. But I have to push 681 00:31:22,359 --> 00:31:25,140 back on this perfect orchestration scenario. 682 00:31:25,240 --> 00:31:27,380 Alright, go for it. It's one thing to run a relay 683 00:31:27,380 --> 00:31:30,019 race on an empty track, but what happens if I, 684 00:31:30,140 --> 00:31:33,349 the human user, act unpredictably? Like what? 685 00:31:33,549 --> 00:31:36,329 What if right in the middle of a tight 30 millisecond 686 00:31:36,329 --> 00:31:38,730 decoding window, I sneeze loudly, interrupt the 687 00:31:38,730 --> 00:31:40,369 assistant, and completely change the subject? 688 00:31:40,650 --> 00:31:44,490 Does the 8 GPU pipeline just crash? No. And that 689 00:31:44,490 --> 00:31:46,430 is the true achievement of this architecture. 690 00:31:47,009 --> 00:31:49,730 If a sudden interruption occurs, the intent module 691 00:31:49,730 --> 00:31:52,410 processes that new audio and video in the very 692 00:31:52,410 --> 00:31:56,190 next 0 .71 second scheduling chunk. It catches 693 00:31:56,190 --> 00:31:59,549 it that fast. It does. It recognizes the interruption, 694 00:31:59,730 --> 00:32:02,230 and it may immediately output a wait action. 695 00:32:02,380 --> 00:32:05,299 ceasing the generation of new speech. And it 696 00:32:05,299 --> 00:32:07,960 passes those new hidden states over to the realization 697 00:32:07,960 --> 00:32:11,420 module. Yes. The realization module, which is 698 00:32:11,420 --> 00:32:14,140 currently rendering the assistant talking, receives 699 00:32:14,140 --> 00:32:17,579 the updated hidden states and smoothly transitions 700 00:32:17,579 --> 00:32:20,180 into generating the visual behavior of stopping 701 00:32:20,180 --> 00:32:23,119 and listening. That's wild. The pipeline doesn't 702 00:32:23,119 --> 00:32:25,920 break. It simply redirects its generation path 703 00:32:25,920 --> 00:32:29,069 based on the newly updated causal history. It's 704 00:32:29,069 --> 00:32:31,289 like a train that can instantly lay down new 705 00:32:31,289 --> 00:32:33,230 tracks in response to a boulder falling on the 706 00:32:33,230 --> 00:32:35,009 path without having to stop the engine. It's 707 00:32:35,009 --> 00:32:37,470 a great visual. It's an incredible feat of engineering. 708 00:32:37,869 --> 00:32:41,910 But having said all of that, how well does it 709 00:32:41,910 --> 00:32:44,410 actually work in practice? We need to look at 710 00:32:44,410 --> 00:32:47,529 the evaluation, the showcases, and recognizing 711 00:32:47,529 --> 00:32:49,789 the limits of what they built. And this is where 712 00:32:49,789 --> 00:32:52,309 we must apply the most rigorous scrutiny of the 713 00:32:52,309 --> 00:32:55,869 entire discussion. Section 7 .1 of the report, 714 00:32:55,869 --> 00:32:58,289 it details the human evaluation of the model. 715 00:32:58,430 --> 00:33:00,430 And we have to look closely at the sample size 716 00:33:00,430 --> 00:33:02,990 here. Very closely. The evaluation was limited 717 00:33:02,990 --> 00:33:06,829 to exactly 30 real -time duplex cases. 3 -0. 718 00:33:07,049 --> 00:33:10,130 That is an incredibly small pool of data to draw 719 00:33:10,130 --> 00:33:13,309 sweeping conclusions from. It is. They used six 720 00:33:13,309 --> 00:33:16,369 binary criteria for each of those 30 cases, essentially 721 00:33:16,369 --> 00:33:19,869 a pass or fail checklist. Out of 180 total checks 722 00:33:19,869 --> 00:33:24,250 across the board, 168 passed. Which sounds fantastic 723 00:33:24,250 --> 00:33:27,890 on paper. It does. But you have to dig into which 724 00:33:27,890 --> 00:33:30,950 specific criteria failed. The metric called content 725 00:33:30,950 --> 00:33:33,970 responsiveness, which measures if the model actually 726 00:33:33,970 --> 00:33:36,630 responded substantively and appropriately to 727 00:33:36,630 --> 00:33:38,970 the specific things the user said and showed, 728 00:33:39,549 --> 00:33:43,109 passed in 22 out of 30 cases. Exactly. That means 729 00:33:43,109 --> 00:33:46,250 in 8 out of 30 live interactions, the model failed 730 00:33:46,250 --> 00:33:49,009 to respond adequately to the content of the user's 731 00:33:49,009 --> 00:33:51,839 input. And just missed the point. Right. It might 732 00:33:51,839 --> 00:33:53,619 have kept the video stable, it might have spoken 733 00:33:53,619 --> 00:33:56,259 in a clear voice, but it didn't actually grasp 734 00:33:56,259 --> 00:33:59,140 the conversational thread. It is so tempting 735 00:33:59,140 --> 00:34:02,519 to look at a high passing score on visual identity 736 00:34:02,519 --> 00:34:05,339 stability and assume the AI is a genius, but 737 00:34:05,339 --> 00:34:07,859 we have to remember what we are actually looking 738 00:34:07,859 --> 00:34:10,579 at. We really do. Because it passed baseline 739 00:34:10,579 --> 00:34:13,469 visual consistency checks on a 30 case sample, 740 00:34:13,670 --> 00:34:17,170 we absolutely cannot infer that this model possesses 741 00:34:17,170 --> 00:34:19,710 flawless perception. We cannot infer that it 742 00:34:19,710 --> 00:34:22,130 has deep conversational understanding or any 743 00:34:22,130 --> 00:34:24,829 form of general intelligence. It passed a very 744 00:34:24,829 --> 00:34:27,730 narrow, highly constrained test. Which naturally 745 00:34:27,730 --> 00:34:29,889 brings us to the showcase figures in the paper. 746 00:34:30,369 --> 00:34:34,230 Specifically, figure 12. Oh, the showcases. Showcases 747 00:34:34,230 --> 00:34:37,010 in research papers are notoriously easy to misinterpret 748 00:34:37,010 --> 00:34:39,349 if you don't read the methodology. Right. You 749 00:34:39,349 --> 00:34:41,789 look at figure 12 and you see these amazing images. 750 00:34:42,059 --> 00:34:44,320 There's an example of the AI giving watering 751 00:34:44,320 --> 00:34:46,199 advice to a real plant that someone is holding 752 00:34:46,199 --> 00:34:50,239 up to the camera. It is so easy to mentally leap 753 00:34:50,239 --> 00:34:53,260 to the conclusion that this AI has perfect live 754 00:34:53,260 --> 00:34:56,519 vision and botanical expertise. But if we examine 755 00:34:56,519 --> 00:34:59,280 the text, the first four categories in Figure 756 00:34:59,280 --> 00:35:02,380 12, which include plan revision, visual guidance, 757 00:35:02,960 --> 00:35:05,500 collaborative design, and social support, were 758 00:35:05,500 --> 00:35:08,079 generated using script condition training data. 759 00:35:08,639 --> 00:35:11,179 Meaning they aren't live, unpredictable tests. 760 00:35:11,650 --> 00:35:14,750 They are offline examples demonstrating the model's 761 00:35:14,750 --> 00:35:16,829 rendering capabilities given a highly controlled 762 00:35:16,829 --> 00:35:20,239 script. Contrast that with the interactive customization 763 00:35:20,239 --> 00:35:22,659 example in the same figure, which was derived 764 00:35:22,659 --> 00:35:25,599 from an actual user demonstration. The critical 765 00:35:25,599 --> 00:35:28,019 takeaway is that we cannot describe every single 766 00:35:28,019 --> 00:35:31,000 showcase image as an independently evaluated 767 00:35:31,000 --> 00:35:34,059 live capability. It is vital to separate the 768 00:35:34,059 --> 00:35:36,780 system's ability to render a realistic -looking 769 00:35:36,780 --> 00:35:39,840 response from its ability to autonomously reason 770 00:35:39,840 --> 00:35:42,699 about complex, unpredictable real -world inputs 771 00:35:42,699 --> 00:35:45,019 in real time. It can draw a great picture of 772 00:35:45,019 --> 00:35:47,079 an assistant talking about a plant, but that 773 00:35:47,079 --> 00:35:48,760 doesn't mean it actually understands the plant 774 00:35:48,760 --> 00:35:51,440 in a live setting 100 % of the time. We have 775 00:35:51,440 --> 00:35:53,480 to maintain that boundary between presentation 776 00:35:53,480 --> 00:35:57,920 and cognition. Okay. We have covered an immense 777 00:35:57,920 --> 00:36:00,090 amount of ground. We've gone from the theoretical 778 00:36:00,090 --> 00:36:02,849 framework of the reciprocal world model through 779 00:36:02,849 --> 00:36:06,710 the modular architecture of Duo V1, the autoregressive 780 00:36:06,710 --> 00:36:08,909 intent chunking, the flow matching realization, 781 00:36:09,130 --> 00:36:11,550 the three stages of training, the synthetic data 782 00:36:11,550 --> 00:36:15,570 pipelines, right, the data, the eight GPU orchestration 783 00:36:15,570 --> 00:36:18,349 and the strict limits of the 30 case evaluation. 784 00:36:18,769 --> 00:36:21,130 Let's pull this all together. What has this paper 785 00:36:21,130 --> 00:36:24,090 actually established and what remains untested? 786 00:36:24,369 --> 00:36:26,929 Well, what is firmly established is a monumental 787 00:36:26,929 --> 00:36:29,150 engineering contribution to the field of human 788 00:36:29,150 --> 00:36:31,869 -computer interaction. Definitely. The Ant Group 789 00:36:31,869 --> 00:36:34,570 team has successfully built a reciprocal, multi 790 00:36:34,570 --> 00:36:36,849 -module, full -duplex pipeline that actually 791 00:36:36,849 --> 00:36:39,929 functions. They prove that it is physically possible 792 00:36:39,929 --> 00:36:42,130 to synchronize an autoregressive language model 793 00:36:42,130 --> 00:36:44,530 and a streaming diffusion model on a unified 794 00:36:44,530 --> 00:36:47,309 timeline. To handle concurrent audiovisual input 795 00:36:47,309 --> 00:36:49,750 and output without the system crashing. Yes. 796 00:36:49,980 --> 00:36:51,920 But the authors themselves are very clear that 797 00:36:51,920 --> 00:36:54,480 Duo V1 is a preliminary release. They aren't 798 00:36:54,480 --> 00:36:56,320 claiming they've solved the problem permanently. 799 00:36:56,539 --> 00:36:59,639 Far from it. There are massive crucial elements 800 00:36:59,639 --> 00:37:02,179 that require much broader evaluation before this 801 00:37:02,179 --> 00:37:05,139 kind of technology can be deployed widely. Absolutely. 802 00:37:05,579 --> 00:37:08,000 For example, end -to -end response timing in 803 00:37:08,000 --> 00:37:10,539 the wild. Testing this on a controlled server 804 00:37:10,539 --> 00:37:13,099 is one thing. Testing it over fluctuating Wi 805 00:37:13,099 --> 00:37:15,840 -Fi networks across the globe is another. and 806 00:37:15,840 --> 00:37:18,679 consistency over much longer sessions. We talked 807 00:37:18,679 --> 00:37:20,980 about error accumulation earlier. How does the 808 00:37:20,980 --> 00:37:23,360 visual and conversational coherence hold up after 809 00:37:23,360 --> 00:37:27,420 a 45 -minute unstructured conversation compared 810 00:37:27,420 --> 00:37:30,139 to a 60 -second carefully evaluated clip? And 811 00:37:30,139 --> 00:37:33,340 perhaps the most difficult hurdle, the system's 812 00:37:33,340 --> 00:37:36,360 ability to recover from genuine deep misunderstandings. 813 00:37:36,400 --> 00:37:38,639 That's a big one. It's one thing to recover from 814 00:37:38,639 --> 00:37:41,150 a dropped word. It's another to realize you've 815 00:37:41,150 --> 00:37:43,570 completely misinterpreted the user's emotional 816 00:37:43,570 --> 00:37:46,269 state or intent for the last five minutes and 817 00:37:46,269 --> 00:37:48,570 trying to steer the conversation back on track. 818 00:37:48,949 --> 00:37:50,969 We have to frame these limitations clearly and 819 00:37:50,969 --> 00:37:53,010 fairly. Acknowledging that the system struggles 820 00:37:53,010 --> 00:37:55,730 with deep semantic understanding or that it might 821 00:37:55,730 --> 00:37:58,190 fail to recognize a subtle visual cue in eight 822 00:37:58,190 --> 00:38:01,190 out of 30 cases, it doesn't diminish the achievement. 823 00:38:02,050 --> 00:38:04,250 No, not at all. Getting the pipeline to flow 824 00:38:04,250 --> 00:38:07,269 smoothly is a massive victory. It simply highlights 825 00:38:07,269 --> 00:38:09,190 that the frontier of the science has shifted. 826 00:38:09,570 --> 00:38:12,010 The bottleneck is no longer just raw throughput. 827 00:38:12,449 --> 00:38:15,550 It is now cognitive alignment. And that shift 828 00:38:15,550 --> 00:38:19,090 from throughput to cognition leaves me with a 829 00:38:19,090 --> 00:38:21,409 final slightly provocative thought for you listening 830 00:38:21,409 --> 00:38:24,909 to them all over. OK, let's hear it. If the reciprocal 831 00:38:24,909 --> 00:38:28,449 world model paradigm succeeds, we are fundamentally 832 00:38:28,449 --> 00:38:31,210 moving away from computers that sit silently 833 00:38:31,210 --> 00:38:33,820 waiting patiently for us to hit enter or say 834 00:38:33,820 --> 00:38:37,639 over. We're entering an era of continuous reciprocal 835 00:38:37,639 --> 00:38:41,199 observation. Exactly. We are looking at future 836 00:38:41,199 --> 00:38:44,179 systems that will continuously perceive our micro 837 00:38:44,179 --> 00:38:47,840 expressions, our hesitations, the exact moment 838 00:38:47,840 --> 00:38:50,239 we take a breath to interrupt. It's a lot to 839 00:38:50,239 --> 00:38:53,039 process. If a machine is deciding in real time 840 00:38:53,039 --> 00:38:55,139 whether to speak or wait based on the subtle 841 00:38:55,139 --> 00:38:57,320 twitch of your eyebrow or the shift in your posture, 842 00:38:57,900 --> 00:39:00,920 how will that change our own human communication 843 00:39:00,920 --> 00:39:04,199 habits? That's a fascinating question. When we 844 00:39:04,199 --> 00:39:06,500 know the machine is always watching and always 845 00:39:06,500 --> 00:39:09,380 listening, even while it's talking, will we become 846 00:39:09,380 --> 00:39:12,619 hyper aware of our own physical presence? Will 847 00:39:12,619 --> 00:39:15,699 we find ourselves constantly performing for the 848 00:39:15,699 --> 00:39:18,900 lens? It's the psychological weight of interacting 849 00:39:18,900 --> 00:39:21,579 with a physical realization of an ever -present 850 00:39:21,579 --> 00:39:24,820 intent. When the engineering traffic jam finally 851 00:39:24,820 --> 00:39:27,000 clears, we are going to have to figure out how 852 00:39:27,000 --> 00:39:29,739 to drive on an entirely new kind of road.