EPISODE TRANSCRIPT
Speech and Language Processing (Daniel Jurafsky & James H. Martin) — Part 1
Automatically transcribed from the episode audio. Some words and names may be inaccurate.
Listen to the episodeDownload textDownload captions
You know, it's funny. If you really look at a modern microprocessor, it is at its core just a piece of rock. Right. A very highly processed rock, but yes. Yeah, like we literally took some silicon, flattened it out, carved these microscopic lines into it, and essentially tricked it into doing math by... you know, trapping lightning inside. That is a, well, a very poetic and slightly terrifying way to describe modern computing hardware. But you're right. Fundamentally, at the deepest physical layer, a computer is just a really, really fast calculator. It only understands high voltage and low voltage, ones and zeros. It operates entirely on binary logic. Which makes total sense for math, right? Like getting that rock to add two numbers together seems completely logical to me. Sure. And even getting it to render a massive 3D video game, which is obviously way harder and requires millions more calculations, that still feels like it lives in the realm of numbers. But then I look at my phone. Where are we going with this? Well, I type a text message and I use some weird, highly nuanced sarcasm and my keyboard instantly suggests like a laughing, crying emoji. Ah, I see. And that just feels like we crossed some kind of boundary. You know, like we took this hyper logical rock and somehow taught it human empathy. Or at least the illusion of it. It honestly feels like magic. It does feel like magic, yeah. Because human language is really the complete antithesis of binary logic.
It's messy. It's ambiguous. It constantly evolves. Messy. Right. And it relies on this immense amount of shared cultural context that we never explicitly say out loud. We leave so much unsaid. And yet over the last few decades, computer scientists have built these just astonishing architectural bridges between the rigidity of math and the fluidity of human thought. So it's not magic. There is zero magic. It is just incredibly complex, deeply fascinating architecture. Well, today we are taking a sledgehammer to that magic trick.
Welcome to a brand new Deep Dive. Glad to be here. We have our hands on the absolute definitive text in the field today. It's Daniel Jarafsky and James H. Martin's speech and language processing. We've got an enormous stack of excerpts from their third edition draft, and our mission today is to just tear this subject down to the studs. It's a fantastic resource. If you are listening to this right now and you've ever wondered how Siri... parses your voice, or how Google Translate flips Japanese into English, or how a large language model actually generates a paragraph of text, we are going to meticulously unpack the mechanics behind every bit of it. And it is a massive subject.
So you should really consider this part one of our journey. We are going to go concept by concept, starting from the very dawn of natural language processing. all the way through chapter 14 of the text. A lot of ground to cover. Yeah. We'll start with how a machine even looks at a raw letter and then move into how it calculates the probability of your next keystroke. We'll explore how researchers figured out a way to turn the actual meaning of a word into a geometric coordinate. That part is wild. It really is. And eventually we'll look at the deep neural networks that hold entire conversations in their memory. Now, I know it sounds intimidating when we start throwing around terms like, you know, neural architectures and vector semantics. It feels like this huge wall of jargon. It definitely can. But reading through these sources, I realized something.
Every single one of these complex systems was built simply to answer a few beautifully basic questions like. What is a word? How does one word relate to another word? And how do we use the past to predict the future? Right. If we just hold onto those questions, the math just becomes a tool to answer them. That's a really perfect way to frame it. And the desire to answer those questions actually goes back way further than many people realize. In the text, Juraski and Martin take us all the way back to 1966. Yes. To MIT, to a chatbot named Eliza. This was totally mind -blowing to me. 1966, they built a system that simulated a Rogerian psychotherapist.
And the transcript of people talking to this computer is just... wild he's famous for a reason yeah like the user types i need some help and eliza responds what would it mean to you if you got some help and then the user types perhaps i could learn to get along with my mother and eliza says tell me more about your family all in caps too right and people were stunned by this the text says some users genuinely believe eliza possessed deep emotional understanding Well, the irony there is that Rogerian therapy, which is a real framework where the therapist basically reflects the patient's statements back at them to encourage them to keep talking, is the absolute perfect persona for a machine that knows literally nothing about the world. Right, because it doesn't have to offer any original insight. Exactly. Eliza did not possess a single shred of actual comprehension. It was purely a parlor trick. But the mechanism of that trick is something we still use every single day. It relied on something called regular expressions.
Yes, regular expressions form the absolute bedrock of string processing in computer science. A regular expression is essentially an algebraic notation for characterizing a set of strings. You are handing the computer a highly, highly structured search pattern. Okay, let's bring this down to earth for a second. Because reading the reject section, it felt exactly like giving instructions to a bouncer at a comically exclusive nightclub. I like that analogy. You can't just tell the computer bouncer, hey don't let anyone wearing sneakers in you have to give it a hyper specific list of physical parameters you have to say only admit a person if their name begins with a capital letter they're wearing exactly three rings and their shoe size is between eight and ten point one and if a guest deviates by one millimeter they are rejected that captures the literal minded nature of the machine perfectly because if you ask a computer to search a document for the word woodchuck all lowercase it will scan the text but if it encounters woodchuck at the beginning of a sentence with a capital w it just ignores it it doesn't even see it as the same word not at all to the machine a lowercase w and an uppercase w are entirely different entities their underlying binary codes are completely different so to get around that we have to use regex syntax we basically give the bouncer options. The book shows how we use square brackets for this. You type an opening bracket, a lowercase w, an undercase w, a closing bracket, and then oodchuck. So it looks like bracket w, capital dash w, bracket oodchuck. Right. And those brackets define what's called a disjunction of characters. you are telling the system match any single character that exists inside these brackets. So it makes your search just slightly more flexible. And you can set up ranges too, which is super useful. Instead of typing out every single letter of the alphabet, you just type bracket, a dash Z bracket, and the computer knows you mean literally any lowercase letter. Or zero through nine for any single digit.
Yeah. I also saw you can build negative rules. If you put a caret symbol. you know, the little upward pointing arrow inside the brackets, it acts as a negation. So if I write caret A -Z inside the brackets, that tells the bouncer, let this character pass as long as it is not an uppercase letter. The real power of regex, though, emerges when we introduce counters and optionality, because human language is just full of variations in spelling or emphasis. Oh, for sure. Consider the word color. In America, it is C -O -L -O -R, but in Britain, it's C -O -L -O -U -R. Now, if you are searching a global database, you want to catch both spellings. In Rejects, you would place a question mark after the U, and the question mark dictates that the preceding character is optional, meaning it can appear zero times or exactly one time. Oh, so it catches both automatically. The textbook uses a really great example of a sheep bleeding to explain the other counters. The sheep example is a classic. Yeah, because a sheep could say bah with two A's or bah with three. or ba with five. And if I want the computer to catch that sheep noise, regardless of how long the vowel is dragged out, I use the clean star, the asterisk.
So if I write ba star exclamation point, the star means the preceding of 2a can occur zero times, or basically an infinite number of times. Alternatively, you might use the clean plus, which demands that the character appears at least one time, but then allows for infinite repetitions after that. And we combine all these counters with anchors. Anchors. Yeah, so the carrot symbol, when it's placed outside of the brackets, means the pattern must start at the very beginning of a line. And a dollar sign means the pattern must hit the very end of the line. Okay, so these are the basic tools. But how did the creator of Eliza actually use these tools to fake emotional intelligence? Like, how did it literally flip I need help with my mother into tell me more about your family? It utilized a feature of RageX called Capture Groups. When you place parentheses around a specific part of your regex pattern, you instruct the computer to not only find a match, but to actually save whatever text it matched into its active memory. Oh, it grabs it. Yes. We store it in what's called a numbered register. I really want to walk through the exact code the book provides here because it is just the perfect illusion. The code is s slash dot star space im. Let's demystify that whole sequence.
Please do. So, the is followed by a slash simply stands for substitute. It initiates a basic find and replace operation. The dot star at the very beginning means match any character any number of times. In plain English, it means... Ignore everything the user typed at the start of their sentence. Got it. So it's just scanning right past all the throat clearing. Like if I say, well, today I feel, it ignores that. Exactly. Next, it looks for the explicit string I am with spaces around it. Then we hit the parentheses, the capture group. Inside is depressed pipe sad.
The vertical pipe is a disjunction operator. It means O -R. Okay. So the machine is hunting for the word depressed or the word sad. And because those words are locked inside the parentheses, the exact moment the computer spots one of them, it snatches the word and holds it in its pocket. Right. Then there's another dot star to ignore whatever else the user typed at the end of the sentence. And now we reach the second half of the substitution string. The computer prints, I'm sorry to hear you are. And then it encounters the slash one. Right, the slash one. That slash one is the explicit command to reach into its pocket, pull out the exact word it captured in the first set of parentheses, and drop it directly into the sentence. It is so brilliant in its simplicity. I type, I think I'm depressed today.
Eliza ignores, I think, finds I am, captures depressed, ignores today, and spits out, I am sorry to hear you were depressed. And it feels like active listening. But it is just a microscopic find -and -replace script running in milliseconds. It runs through a whole cascading hierarchy of these scripts, actually. It swaps my for your, I for you. But constructing these regex patterns highlights the fundamental struggle of early natural language processing. Which is what? When you define a pattern, you are always engaging in a tug of war between two opposing metrics, precision and recall. Yes, I saw this mentioned constantly in the text. Let's break those down for the listener. Precision is a measure of exactness.
It is about minimizing false positives. Let's say you write a simple regex to find the word the in a text, T -H -E. If your pattern accidentally highlights the letters T -H -E inside the word theology or other, your precision plummets. Because you are catching garbage you didn't want. Exactly. So to fix that, I tighten up my bouncer's rules. I say, okay, only find T -H -E if it has a space before it and a space after it. Which is a very logical step. But by tightening the rules to increase your precision, you often damage your recall. Recall is a measure of completeness.
It is about minimizing false negatives. If you demand a space before the word the, you will suddenly miss the word the. It starts a paragraph because there is no space before it. You will miss the if it is immediately followed by a comma. You failed to recall all the valid instances. So building these systems is just this constant agonizing adjustment of the dial. You widen the net to catch more fish, but you start pulling up old boots. Right. Then you shrink the net to only kept salmon and half the salmon just swim right by. That tension never really goes away, even in modern systems. But regex is only useful if you know exactly what you are trying to find. And that leads us to a much deeper philosophical and technical hurdle. Before we can search for a word or count a word or translate a word, we have to figure out how to tell a machine what a word actually is. Which sounds ridiculously simple until you actually try to explain it to a rock with lightning in it. Exactly. To teach a machine language, we have to feed it text.
Massive amounts of it. The textbook calls a large collection of text a corpus or corpora for plural. And the first thing that jumped out at me is that, Not all corpora are created equal, like at all. They vary wildly. The text contrasts the brown corpus with the switchboard corpus. The brown corpus was assembled in the early 1960s. It contains one million words of printed American English, so newspapers, books, government documents. Very formal. Extremely. The sentences are structured, grammatically formal, and neatly punctuated. And then you look at the switchboard corpus, which is from the 1990s, and it's three million words of transcribed telephone conversations.
And it looks like absolute chaos. Because people don't speak in neat sentences. No, they stumble. They say and they interrupt each other constantly. They abandon sentences halfway through. There are no periods or commas in raw speech. Furthermore, the demographics of the speakers dictate the linguistic rules. The authors highlight African American language, or AAL, particularly as it appears on social media platforms like Twitter. Yeah, that was a fascinating section. A tweet might contain the string Iant instead of I don't or Talmud out instead of talking about. And to a machine trained only on, say, the Wall Street Journal, Iant looks like a catastrophic spelling error. It doesn't know how to process it. But the text makes a really strong point here. These aren't typos. They are systematic, rule -governed features of a specific dialect. This is a crucial data provenance issue in natural language processing.
The textbook heavily stresses the concept of data sheets for data sets. When you use a corpus to train a system, you absolutely must document the technical and demographic parameters of that data. Like an ingredients label. Precisely. Who generated this text? Under what conditions? For what purpose? If you train a customer service chatbot exclusively on formal written documents from the 1980s and then deploy it to process modern text messages from teenagers, it will fail technically. Because the underlying data does not match the deployment reality. The system is brittle. Exactly. So we have this massive, messy pile of text. Before the machine can do anything intelligent with it, it has to clean it up. It undergoes text normalization. And the first major step is tokenization. We have to take a long, continuous string of characters and basically chop it up into individual tokens, which usually represent words. The intuitive human approach is just to split the text wherever there is a white space. But white space is a trap. Right. What about a comma? If I write apples, oranges, the comma is attached directly to the word apples. I don't want a token that is apples comma. I want apples and I want the comma to be its own separate token. So you tell the machine, okay, separate all punctuation. But then you encounter an abbreviation like PhD or a price tag like $45 to 55 cents. If you aggressively separate all punctuation, you shatter the price into 45 and 55 and you completely destroy the semantic meaning of the number. And then you have multi -word expressions. New York is two words separated by a space, but it represents one single geographic concept. Treating it as two independent tokens, new in York, dilutes the meaning. Oh, and then the book points out that languages like Japanese and Chinese don't even use spaces between characters. So the entire concept of relying on white space just collapses entirely. For standard English, organizations have developed really extensive rule sets like the Penn Tree Bank tokenization standard. It has very specific protocols such as how to handle clitics. Remind me what a clitic is. A word like doesn't is actually split into two distinct tokens under this standard. Does and int. Because linguistically, those are the two functional components. Okay, that makes sense for words the system actually knows. But here is the thing that really fascinated me. No matter how big your training corpus is, eventually, out in the real world, the machine is going to encounter a word it has never seen before in its life. Inevitably. A totally novel string of characters. If it tries to tokenize a word that isn't in its vocabulary, doesn't it just crash? It used to. This was known as the out -of -vocabulary problem. But modern systems use an incredibly elegant solution called subword tokenization. The most famous algorithm for this is byte pair encoding, or BPE. I honestly read this section three times because it is so clever.
How does BPE actually figure out what a subword is? It is entirely data -driven. It does not rely on linguistic rules or human dictionaries at all. Instead, it looks at the raw character sequences in the training data. Initially, its vocabulary is literally just individual letters. A, B, C, D. Then it scans the entire corpus and finds the most frequently occurring adjacent pair of characters. Let's say E and R appear next to each other millions of times. Like in the, or her. Right. The algorithm merges them and adds er to its vocabulary as a brand new single token. Okay, so now I have individual letters. plus this new error block. Then it scans again. Maybe it notices that L and O appear together frequently. It merges them into low. It repeats this process thousands of times. Eventually, it starts merging the merged blocks. Oh, wow.
It might see low and W frequently adjacent, so it creates a low token. It might see low and error adjacent, and it creates a lower token. It's exactly like building words out of Lido bricks. It starts with the tiny single studs, clicks them together into bigger pieces, and just keeps those really common chunks in its toolbox. That's exactly it. The brilliance of BPE is how it handles the unknown. If the machine has seen the word unbelievable millions of times during training, it probably merged all those characters into one massive token. Unbelievable. But what if a user types a completely made -up word, like unflappable? The machine has no token for that full word. But it doesn't crash. It just breaks the unknown word down into the biggest Lego bricks it does recognize. Exactly. It might look at it and say, well... I don't know unflappable, but I have a brick for un, a brick for flap, and a brick for able. So I will just process it as those three tokens. The machine is never truly stumped. It creates a perfectly robust system. Now, alongside tokenization, normalization also involves case folding. Right, which is usually just converting all text to lowercase so the machine can match words regardless of capitalization.
Though, as the text notes, this can actually damage certain tasks. How so? Well, in sentiment analysis, a user typing in all cappy S is conveying anger or intense excitement. If you lowercase everything indiscriminately, you white out that emotional data. That's a great point. You also have to deal with word variations. Like if I have sang, sung, and sings in a document, I probably want the computer to realize those are all just versions of the same core concept. There are two distinct methods for doing that. Lemmatization is the careful, linguistically accurate method.
It uses a complex morphological analyzer and a dictionary to map a word back to its true dictionary root, or lemma. It knows that the lemma of better is actually good. Which I imagine takes a lot of time and processing power. It does, which is why we also have stemming. Stemming algorithms, like the widely used Porter stemmer, are basically blunt instruments. they rely on a cascade of simple regex style rules that aggressively chop suffixes off the ends of words just hacking the ends off yes a stemmer doesn't care if the resulting word is a real english word it might chop the word ponies down to pony with an eye and it might chop pony down to pony as well so it literally creates gibberish it does create gibberish but as long as both variations map to the exact same gibberish string the machine successfully groups them together for search purposes it is fast dirty and highly effective for things like search engines okay so We've run our text through the car wash. We've tokenized the clitics. We've stemmed the suffixes. We finally have clean data.
But humans are the ones supplying the data, which means there are typos. Always. If I am typing fast on my phone and I spell execution as E -X -E -C -U -T -O -N, omitting the I, my phone auto -corrects it in a microsecond. How does it know what I meant to type? It relies on a mathematical concept detailed in Chapter 2. Minimum edit distance. Minimum edit distance is a metric that quantifies how structurally similar two strings of text are. It calculates the absolute minimum number of editing operations required to physically transform the misspelled string into the target string. And the operations it is allowed to use are insertions, deletions, and substitutions.
Correct. The book walks through a specific, really tricky example. Transforming the word intention, spelled with a T, into execution with an X and a C. Let's trace how the machine actually thinks about this. Let's do it. We start with I -N -T -E -N -T -I -O -N. The target is E -X -E -C -U -T -I -O -N. Okay, so looking at the first letter, I have an I, I want an E, I can just substitute the I for an E. You could, but let's look at the length. Intention has nine letters. Execution has nine letters. I -N -T -E -N -T -I -O -N is nine. E -X -E -C -U -T -I -O -N is nine. Oh.
They are the exact same length. Okay, so substituting the first I for an E makes total sense. Actually, the optimal path the algorithm finds is slightly different because it looks at the whole board at once. The cheapest mathematical route is to delete the initial I entirely. Wait, really? Why would it do that? Because of how the subsequent letters align. If you delete the I, the N shifts over. You substitute... n for e then you substitute the t for x and then you leave the e alone because execution has an e in that exact spot oh i see by shifting everything over with that deletion it saved itself an operation down the line because that middle e suddenly lined up perfectly exactly after that free e You are looking at the second N. You substitute it with a C. You substitute the next T with a U. And then the rest of the word T -I -O -N lines up perfectly and requires zero operations. That is fascinating. So the operations were delete, substitute, substitute, substitute, substitute. Five operations in total.
Right. And if we assign a cost of one to each operation, the minimum edit distance, specifically what's called the Levenstein distance between those two words, is five. But wait a second. We just work that out by eyeballing it and jumping back and forth like humans. A computer can't eyeball things. No, it cannot. For a computer to be absolutely certain that five is the mathematical minimum, wouldn't it have to calculate every single possible combination of insertions and deletions to prove there isn't a secret four -step path? That would be millions of calculations for long words. If it used a naive brute force approach, yes. It would face an exponential explosion of possibilities, and it would take an eternity.
But computer scientists solved this using a paradigm called dynamic programming, specifically the Wagner -Fisher algorithm. How does that work? The philosophical core of dynamic programming is this. If you have a massive, complex problem, you break it down into tiny, overlapping subproblems. You solve the tiny problems, and crucially, you write the answers down so you never, ever have to solve them again. I was trying to picture this while reading. It felt like trying to navigate a massive hedge maze. If you take a left turn and hit a dead end, you don't walk all the way back to the entrance of the maze to try again. You just walk back to the last intersection, leave a little breadcrumb there that says dead end, and take the right turn instead. You are saving your intermediate progress. That is a very apt analogy. The Wagner -Fisher algorithm physically builds a grid, a matrix. The misspelled word runs down the left -hand column. The target word runs across the top row. The cell in the very top left corner represents the cost of transforming an empty string into an empty string, which is obviously zero. Okay, I have my grid. How does the computer actually fill it out? It moves systematically, left to right, top to bottom. To fill in any given empty cell, the algorithm only looks at three cells that it has already filled out. The cell immediately above it, the cell immediately to its left, and the cell diagonally up and left.
Because those three cells represent the three operations, the cell above represents a deletion. The cell to the left represents an insertion, and the diagonal cell represents a substitution. Precisely. The algorithm looks at those three neighboring numbers, calculates what it would cost to take each of those three local steps, picks the cheapest option, adds it to the running total, and writes that new number in the empty cell. Oh, wow. It doesn't look at the whole word. It only ever makes a tiny local decision based on the three squares right next to it. And because it does this row by row, by the time it reaches the very bottom right corner of the entire grid, that final cell contains the absolute minimum cost for the entire word. It solved a massive problem just by making thousands of tiny localized choices. And it does one more vital thing. Every time it writes a number in a cell, it leaves a breadcrumb. It draws a tiny pointer, a back pointer, aimed at whichever of the three adjacent cells it chose as the cheapest path. Once it reaches the bottom right corner, it simply follows those pointers backward through the grid all the way to the top left. That back trace gives the computer the exact step -by -step alignment of the two words. It is such an elegant piece of logic. And what really struck me is that the textbook points out this isn't just for autocorrect on my phone. This exact same matrix math is the foundation for huge swaths of computer science.
It is truly ubiquitous. If you're training a machine translation model... You use this algorithm to align sentences in parallel texts. In speech recognition, if a user speaks a sentence and the computer generates a transcript, how do we grade the computer? We use minimum edit distance to calculate the word error rate against a human transcript. Exactly. It is even used in bioinformatics to align sequences of human DNA to find genetic mutations. The math just scales beautifully. Okay, so we can clean up text and we can mathematically fix errors from the past. But what if we want to predict the future? If I pull out my phone right now and type, I'm going to... My keyboard instantly suggests three words. Store, park, and gym is guessing my intent. How do we move from fixing typos to actually anticipating thought?
This represents the major leap from processing text to modeling language. We are entering the domain of Chapter 3, Language Models. Let's dive in. The fundamental objective of an Engegram language model is to calculate the probability of a specific word appearing given the history of the words that came before it. Mathematically, what is the probability of word W given history H? Correct. Let's say your history is the phrase. Its water is so transparent. What is the probability that the very next word is that? Okay. To calculate this perfectly, strictly according to the laws of probability, you would use what's called the chain rule. You would calculate the probability of the word its, multiply it by the probability of water given its, multiply that by the probability of a is given its water and continue chaining those conditional probabilities until you reach the end of the sentence. But the textbook makes it very clear that doing that in reality is literally impossible. It is impossible because language is infinitely creative. Humans constantly generate sentences that have never been uttered before in the history of the universe. I mean, we're probably doing it right now. Very likely. So if your machine is trying to predict the next word by searching its corpus for an exact match of a unique 20 -word history, it will never find it. The count will be zero. You cannot calculate meaningful statistics from an event that has never occurred. So we can't look at the whole history. We have to make a compromise. We essentially have to cheat. We use a simplifying heuristic called the Markov assumption. We assume that the probability of a word does not depend on the entire history of the universe, but only on the local context. We restrict our vision.
An n -gram model looks n words into the past. So a bingram model where n is 2 only looks at the single word immediately preceding the target. It ignores everything else. If the sentence is, its water is so transparent, a bigogram model trying to predict the next word completely ignores its water is so. It only asks, what is the probability of the word that, given just the word transparent? It seems reductive, and it is, but it makes the math actually computable. How does the model calculate that probability? It uses maximum likelihood estimation, or MLE. MLE sounds complex, but it is literally just counting and dividing. It's just... Building giant lookup tables, right?
That is all it is. The machine scans the entire training corpus. It counts the total number of times the single word transparent appears. Let's say it appears 1 ,000 times. Then it counts how many times the specific two -word sequence transparent that appears. Let's say it finds it 500 times. To get the probability, you divide 500 by 1 ,000. The big round probability is 0 .5, 50%. So the model just has billions of these little fractions stored in memory. And if I want to find the probability of a whole sentence, I just multiply the bibliogram probability of every step together. Correct.
But when I read that, I immediately thought of a massive hardware problem. Probabilities are decimals between 0 and 1. If I multiply 0 .5 times 0 .1 times 0 .05 times 0 .01, the number shrinks incredibly fast. It shrinks exponentially. In computers, hardware processors only have a finite amount of memory allocated to store the decimal places of a floating point number. If you multiply enough small probabilities together, the resulting decimal becomes so microscopic that the computer physically cannot represent it anymore. It just gives up. It simply rounds the number down to absolute zero. Which destroys the entire calculation. The book calls this mathematical underflow. Yes. To bypass hardware limitations, language models do not multiply raw probabilities.
They convert every probability into its logarithm. Okay, log probabilities. In mathematics, if you add the logarithms of two numbers, it is functionally equivalent to multiplying the original numbers. By adding log probabilities together instead, the numbers stay in a manageable range, usually as negative numbers. Underflow is avoided entirely. If you need the raw probability at the very end, you just exponentiate the final sum. Okay, so we've dodged the hardware limits. We've built an Enneagram model. It has tables of log probabilities. How do we actually know if the model is smart?
Like, how do we grade it? We need an evaluation metric. The standard intrinsic metric for language models is perplexity. Perplexity basically measures how surprised a model is when it encounters real human language. I really love that framing. We want a model that is very hard to surprise. Precisely. You take a test, set a document the model was not allowed to look at during training. You feed this text into the model and ask it to calculate the probability of the text occurring. Perplexity is mathematically defined as the inverse probability of the test set normalized by the total number of words. Okay, inverse probability. Meaning, if the model assigns a very high probability to the text, if it thinks the text is totally normal and expected, the perplexity score will be very low. Yes. A lower perplexity score indicates a much better model. It means the model's internal statistical distribution closely matches the actual distribution of human language. But to truly grasp what an n -gram model has learned, the best method is to actually run it in reverse, to make it hallucinate. This was my absolute favorite visual in the chapter, the Shannon visualization from the 1950s. You basically use the probability tables to generate text. It is like a statistical game of Mad Libs. Let's walk through it. Imagine you have a bigogram model. You need to pick a starting word. You look at your unigram tables, the raw frequency of all words, and you treat it like a weighted roulette wheel. The word the occupies a huge chunk of the wheel. The word xylophone occupies a microscopic sliver. You spin the wheel. It lands on the. So you print the word they. Now you shift to your bigger gram tables. You pull up the specific table for the word the. This is a brand new roulette wheel.
It only contains words that followed the in the training data. Dog might have a 10 % chance. Man might have an 8 % chance. You spin this new wheel. It lands on dog. You print dog. And then because it is a bigger gram model, you completely throw away the word the. The model forgets it ever existed. You pull up the roulette wheel for dog, spin it, and maybe get barked. If you generate text using only a unigram model spinning the overall frequency wheel with zero context, the output is absolute gibberish. The textbook shows an example to him. Boyan. It's just statistical noise. Right. But if you step up to a bigogram model, suddenly local phrases start to make sense.
He stepped out into the hall. It feels like syntax. And if you step up to a trigram or a fourgram model, the generated sentences start to look eerily fluent. They sound like real English. However, if you read a full paragraph of it, you realize it lacks any overarching logical coherence. It is just wandering aimlessly. It sounds fluent because the statistics enforce local grammatical rules, but the Markov assumption prevents it from holding on to a global thought. But there is a fatal flaw in this entire counting method, isn't there? The MLE counting method.
We touched on it briefly before. Well, zero problem. It is the bane of all Enneagram models. Let's say I train my bigger model on the entire... Complete works of William Shakespeare. Millions of words. And then I ask it to evaluate the probability of the sentence, I want to eat Chinese food. Okay. The model looks at its tables. It finds the bigger M, eat Chinese. But Shakespeare never wrote the word Chinese. So the count in the training corpus is exactly zero. Which means the probability the model assigns to eat Chinese is zero. And because you are calculating the probability of the entire sentence by chaining these local probabilities together. If you multiply anything by zero, the entire equation drops to absolute zero. The model effectively declares, this sentence is a mathematical impossibility in the English language, which is absurd. Just because you haven't seen a specific combination of words doesn't mean it's literally impossible. To mitigate this, computer scientists developed techniques covered in Chapter 3 under the umbrella of smoothing. The goal of smoothing is to eliminate zeros without completely destroying the integrity of the real statistics. The most obvious solution is Laplace smoothing, also known as add -one smoothing. If you have a zero in your table, just artificially add the number one to it. Problem solved. But you can't just add one to the specific zeros you happen to encounter on the test. To keep the math valid, you have to add one to the count of every single possible big rem in your entire vocabulary matrix. Oh, and if my vocabulary is 20 ,000 words, that means my matrix of every possible two -word combination has 400 million slots. And the vast, vast majority of those combinations are absolute nonsense like the Tomodoro apple runs. If I add 1 to 399 million nonsense B -grams... You generate a massive ocean of fake probability mass. But the total probability of all events still has to sum to exactly 1 .0.
So where does that fake mass come from? You have to steal it from the bigograms that actually did occur in the text. It's like a Robin Hood algorithm, but it is disastrously aggressive. It steals probability mass from the rich, frequent words and gives it to the millions of poor, unseen words until the frequent words are virtually bankrupt. The statistics get completely distorted. Which is why Laplace is rarely used for engrams in practice. We need more nuanced approaches. The text introduces interpolation. What's that? Interpolation is highly pragmatic. If you are using a trigram model to predict a word, but the specific three -word history has a count of zero, why throw your hands up? Why not also consult the bigogram and the unigram models?
You just mix them all together? Yes. You calculate the trigram, the b -gram, and the unigram probabilities, and you combine them using specific weights, usually represented by the Greek letter lambda. You might give the trigram 60 % of the weight, the b -gram 30%, and the unigram 10%. You determine the optimal weights by testing them on a separate held -out dataset until you find the mix that yields the lowest perplexity. So even if the trigram count is a hard zero, the b -gram or unigram will almost certainly contribute some decimal. keeping the overall equation alive.
Exactly. The other method is backoff, which is similar, but instead of mixing them constantly, you only back off to the b -gram if the trigram is exactly zero. You only consult the lower order model in an emergency. But backoff introduces a mathematical complication. If you simply sub in the b -gram probability when the trigram fails, you are artificially inflating the total probability space. The numbers will add up to more than 1 .0. You must hold back some probability mass specifically to fund these back -off emergencies. You must discount the counts of the n -grams you did see. This leads to my absolute favorite piece of statistical trickery in the whole book.
Good Turing discounting. Specifically the realization by Church and Gale in 1991. That's very clever. They were trying to figure out how much to discount. They looked at a massive training corpus and asked a simple question. If a specific bigorum appears exactly four times in this training text, how many times should we expect it to appear in a brand new text of the exact same size? The intuitive human answer is four. If it happened four times here, it should happen four times there. But it doesn't. They empirically tested it and found the answer was 3 .23. Because of the mathematical distribution of rare words and language, things that appear four times in a training set are statistically slightly less likely to appear in a new text. It is a profound insight into the geometry of language distribution.
So Church and Gale use that insight. If they see a count of four, they artificially lower it to 3 .23. They take that leftover 0 .77 of probability mass and toss it into a savings account. And they use that savings account to fund all the zero -count n -grams. It is a much more accurate, empirically grounded Robin Hood. But the undisputed heavyweight champion of smoothing algorithms is Messernay. And then Messernay introduces a concept that... genuinely blew my mind, the probability of a word being a novel continuation. I really need to walk through this slowly to make sure I grasp it. Let's set up a scenario. You are building a biogram model. You have the history phrase I went to, and you want to predict the next word. Let's evaluate the word Francisco.
Okay, if I look at standard unigram frequencies. The word Francisco is actually incredibly common in an English corpus. San Francisco is a major city. It's in the news constantly. So a standard back -off model might look at its high overall frequency and guess, sure, Francisco has a pretty good chance of following I went to. But we know intuitively that is wrong. Why? Because Francisco essentially has only one friend in the English language. It almost exclusively appears immediately after the word San. It does not freely mingle. It never appears after the or a or two. It is not a versatile word. Exactly. Now contrast that with the word glasses. Glasses might have the exact same total unigram frequency as Francisco in the corpus. But glasses appears after dozens of different words. Reading glasses, water glasses, sunglasses, my glasses. It has a very high continuation probability. It is promiscuous. It hangs out with everybody. Kineserne smoothing fundamentally alters the back -off calculation. When the model drops down to a lower order distribution to evaluate a word, it does not ask how many times did this word appear in total. Instead, it asks how many different unique words did this word follow in the training data. It measures the word's versatility.
It knows that Francisco is a terrible novel continuation, but Glasses is a great one. It drastically improves the accuracy of the predictions. It is so incredibly elegant. We've pushed counting to its absolute mathematical limit here. But the textbook makes a hard pivot here. Even with Nezerne, even if we train an engram model on the entire internet, engrams have a fetal, insurmountable ceiling. They do. Because engrams possess absolutely zero understanding of semantics, they do not know what words actually mean. To an Enneagram model, the string of characters C -A -T and the string D -O -G are as completely unrelated as C -A -T and R -E -F -R -G -E -R -A -T -O -R. Exactly.
They are just distinct variables. The model does not realize that cats and dogs share properties. They are both pets. They both have fur. They both eat food. Because it cannot generalize, it is horribly inefficient. Right. If the model sees the sentence, the cat is fed a thousand times in training, it learns that fed follows cat. But if it never explicitly sees the phrase the dog is fed, it has no idea what to do. It cannot infer that dogs might also get fed. It treats every string as an isolated island of data. Which brings us to the real revolution. How do you teach a rock to understand the meaning of a word when the rock only understands numbers?
You turn the word into a location in space. We are diving into Chapter 6, Vector Semantics and Embeddings. The foundational linguistic philosophy behind this actually dates back to the 1950s, to structural linguists like J .R. Firth. Firth famously declared, you shall know a word by the company it keeps. I love that quote. The core theory is that the meaning of a word is not defined by a dictionary entry. It is defined by its distribution in language. It is defined by the neighbors that constantly surround it. The textbook uses a brilliant example to prove this. The word ongchoy. I had absolutely no idea what ongchoy was. But the authors say, imagine you read a corpus and you notice that the word ong choy frequently appears in sentences alongside words like rice, garlic, delicious, woke, and salty. What is your brain doing when you read that? I am instantly cross -referencing. I ask my brain, what other words do I know that hang out with rice, garlic, and woke? And my brain pulls up spinach, chard, and collard greens. Suddenly, without ever looking at a dictionary, I know with near certainty that ong choy must be a leafy green vegetable used in Asian cooking. I inferred its meaning purely from its neighborhood. That human intuition is the engine of vector semantics. We can represent a word as a vector, an array of numbers. In the early iterations, researchers used sparse vectors, specifically TF -IDF, which stands for Term Frequency Inverse Document Frequency. How does a sparse vector work? Imagine an array with 50 ,000 slots.
One slot for every word in the English language. To build the vector for Ong Choi, you scan the corpus. Every time ang shui appears within 10 words of garlic, you add a 1 to the garlic slot. Every time it appears near rice, you add a 1 to the rice slot. But because ang shui doesn't appear near refrigerator or spaceship, those slots just stay at zero. Exactly. You end up with a vector of 50 ,000 numbers, and 49 ,900 of them are zeros. It is incredibly sparse. But even with all those zeros, we suddenly have a mathematical way to compare meanings. If we plot these vectors in a 50 ,000 -dimensional space, which is obviously impossible to picture but mathematically trivial, we can measure the distance between them. We measure the similarity using a metric called cosine similarity. You treat the two vectors as arrows originating from the same central origin point. You calculate the geometric angle between the two arrows. If the angle is very small, if the arrows point in roughly the same direction, it means their vectors share high numbers in the same slots. It means they share the same neighbors. Therefore, they have similar meanings. And the brilliant thing about cosine similarity is that it measures the angle, not the length of the arrow. So if the word spinach appears 10 million times in the corpus and encoin only appears 100 times, their arrows will be vastly different lengths. But because they point in the same direction, the computer still knows they mean the same thing. The frequency doesn't distort the semantic comparison. Cosine similarity remains a vital tool. But the real explosion in AI capabilities occurred when researchers abandoned those massive, sparse vectors filled with zeros and moved to dense embeddings. Models like Word2Vec and GloVe. Instead of 50 ,000 slots of mostly zeros, we compress the vector down to maybe 300 slots. But every slot contains a dense, continuous decimal number. No zeros. But how on earth do you figure out what those 300 decimals should be? Word2Vec uses an architecture called Skipgram, and reading how it trains itself is just a revelation. It feels like a magic trick.
Well, it is a remarkable piece of indirect engineering. Skipgram does not try to explicitly calculate the meaning of a word. Instead, it sets up a fake localized task. It builds a very simple logistic regression classifier. Okay, let's slow down. A logistic regression classifier. What is that actually doing in plain English? It is simply a binary sorting machine. You feed it two words, and it outputs a probability between 0 and 1, attempting to answer a simple yes -no question. Are these two words likely to be neighbors in a real sentence? Okay, so how do we train this sorting machine? We use the corpus. We take a target word, say apricot, we look at the actual sentence it appears in, and we grab a real neighboring word, say jam. We feed apricot and jam into the classifier, and we mathematically force its weights to adjust so it outputs a higher probability.
We tell it, yes, these belong together. That makes sense. It's learning true neighbors. But to prevent the classifier from just guessing yes all the time, we introduce negative sampling. We take the target word apricot, but instead of a real neighbor, we reach into the dictionary and grab a completely random word, aardvark. We feed apricot and aardvark into the classifier and we force its weights to output a low probability. We tell it, no, these do not belong together. So the machine is just endlessly practicing this tiny specific task millions and millions of times.
Is apricot in your jam? Yes. Is apricot in your air fork? No. It slowly tweaks its internal numbers to get better and better at this game. And here's the magic trick. Once the classifier becomes highly accurate at distinguishing true neighbors from random noise, we completely throw the classifier away. We delete the sorting machine. We never actually cared about predicting neighbors. We only cared about the internal weights the network developed to win the game. To successfully predict that apricot and jam go together, the network had to compress the distributional properties of those words into 300 continuous numbers. Those 300 numbers are the dense embedding. The network inadvertently learned semantics by trying to win a proximity guessing game. And the resulting geometry is so robust that it captures abstract relational concepts. This leads to the most famous analogy in NLP, the parallelogram. Ah, the classic king and queen example. It is so cool. If you take the 300 -dimensional vector for the word king, and you mathematically subtract the vector for man, and then you add the vector for woman, The resulting coordinate in that 300 -dimensional space is incredibly close to the vector for queen.
It's like the vector space is a physical map of a city, and the concept of gender is just a specific street heading perfectly north. Royalty is a street heading east. You can navigate concepts like a cab driver. It is a stunning conceptual visualization. However, I must interject with the textbook's reality check on the parallelogram. It is heavily cited in pop science, but it is deeply flawed in practice. Wait, really? It doesn't work? Often, if you actually run that specific vector arithmetic, the algorithm just returns one of the input words, like king, or perhaps a pluralized version. The geometry is not always that perfectly delineated. Researchers have demonstrated that this simple additive analogy task fails on many complex relational categories, and it is likely far too simplistic to accurately model human cognitive analogies. Oh, man. It's a little heartbreaking. But do not lose the forest for the trees. While the parallelogram might be exaggerated, the fundamental underlying truth that dense, continuous geometry can capture rich semantic relationships completely revolutionized natural language processing. Embeddings are the fuel for everything that follows. Because once we have these dense, meaningful vectors, we can finally feed them into neural networks that process whole sequences over time. We are crossing the threshold into chapters 7 and 9. Deep learning for sequences. The first major step past n -grams was the feed -forward neural language model.
Instead of looking at exact strings of words, it uses a sliding window. It takes the embeddings of the last, say, three words, concatenates them, passes them through a hidden neural layer, and attempts to predict the embedding of the next word. And this instantly solves the sparsity problem that plagued engrams. Exactly. How? Because it operates on geometry, not specific spellings. If the network is trained on the sentence the dog gets fed, it adjusts its weights. Later... It encounters the incomplete phrase the cat gets. An engram model would freeze if it had never seen that exact phrase. But the neural network looks at the vector for cat, realizes it is physically sitting right next to the vector for dog in embedding space, and says, oh, these are practically the same thing. I bet the next word is fed. It generalizes to unseen data. It is a massive leap forward.
However, the feedforward network still suffers from a critical architectural flaw. The sliding window is a fixed size. It is still essentially making the Markov assumption. It can only see a few words into the past. Which is a huge problem because human language has long -distance dependencies. If I say, the keys that I left on the kitchen counter next to the overflowing fruit bowl are missing, the verb are is plural because it relies on the word keys, which is like 14 words away. A fixed window of five words would only see the overflowing fruit bowl, and you might incorrectly guess the singular verb is. To solve this, researchers developed recurrent neural networks, or RNNs.
An RNN abandons the fixed window. Instead, it maintains a hidden state, a vector of numbers that acts as a continuous memory. It processes the sentence sequentially. I try to picture this as a game of telephone. The network looks at the first word, processes its embedding, and writes a little summary in its hidden state. Then it moves to the second word, mixes that new word with the summary of the first word, and generates a new combined summary. It passes this evolving hidden state forward through time, step by step, until the end of the sentence.
A very functional visualization. And to train these temporal networks, we use a technique called teacher forcing. How does that work? At time step one, the network tries to guess time step two. We calculate its error using a metric called cross -entropy loss, which essentially measures how far its predicted probability distribution is from the actual truth. Now, normally, a network might feed its own prediction into the next step. But if its prediction was wrong, that error compounds and the network spirals into hallucinations. So instead, we force the true correct word from the training corpus into the next step. We keep it on the rails while it learns. But even with teacher forcing, the telephone game analogy reveals the fatal weakness of a basic RNN. If the sentence is 50 words long, by the time the hidden state reaches the final word, it has been mixed and remixed 50 times. The original signal from the first word is completely diluted. The network just forgets the beginning of the sentence. Mathematically, this manifests as the vanishing gradient problem.
Walk me through that. What is vanishing? When a neural network makes a mistake at the end of a sentence, it has to send an error signal backwards through time to update the weights of the earlier steps. This process involves multiplying gradients, which are essentially fractions together over and over. If you multiply 0 .5 by 0 .5, you get 0 .25. Multiply it again, 0 .125. Over 50 time steps, that error signal becomes so microscopically small that it vanishes. The network literally cannot send a learning signal across long temporal distances. It cannot learn long -term dependencies.
So to fix the telephone game, engineers added gates. We get LSTMs, which stands for long short -term memory networks, and GRUs, gated recurrent units. An LSTM is essentially an RNN with a highly sophisticated control mechanism. Instead of just blindly mixing the new word with the old memory, it has specific neural gates that learn what to keep and what to throw away. It's like the person playing the telephone game, has a notepad and an eraser. The forget gate decides if the old context is obsolete. The input gate decides if the new word is important enough to write down, so it can choose to write down the plural subject keys, stick it in its pocket, completely ignore the 14 words about the fruit bowl, and pull keys back out when it reaches the verb.
LSTMs and GRUs dominated natural language processing for years. They were the state of the art. But they still possessed a fundamental bottleneck. They were rigidly sequential. You physically could not process Word 50 until you had finished processing Word 49. You couldn't parallelize the training on modern GPUs efficiently. Which sets the stage for the absolute monumental breakthrough of Chapter 9, the architecture that currently powers ChatGPT, Claude, BERT, everything. The Transformer. The transformer abandons the sequential telephone game entirely. It does not process language left to right. It uses a mechanism called self -attention. Break down self -attention for me, because this really feels like the core of modern AI. In a transformer, every single word in the input sequence looks at every other word in the sequence simultaneously. It computes a mathematical compatibility score between itself and all its neighbors, regardless of distance.
So if the network is processing the word bank, it instantly casts a net across the entire sentence. If it sees the word river 12 words away, the attention mechanism calculates a high score between them, and the network dynamically updates the vector for bank to incorporate the river context. It instantly knows we are talking about a muddy bank, not a financial institution. Exactly. It creates a web of relationships. I like your previous analogy of a crowded room. An RNN is a bucket brigade passing a whisper down a line. A transformer is standing in the center of a crowded room where everyone is shouting at once. But through the mathematical weights of self -attention, the network instinctively knows exactly whose voice to tune into to get the necessary context. And it doesn't just do this once. It uses multi -hit attention. It's doing this dozens of times in parallel. One attention head might be specifically trained to look for subject -verb relationships.
Another head might be looking for semantic clusters. It parses all these different dimensions of the sentence simultaneously. The shift from Markov chains to LSTMs to transformers is profound. It isn't merely an engineering trick to speed up training. It represents a fundamental leap in capturing the non -linear hierarchical web of human thought. A sentence is not a ticker tape of isolated events. It is a complex structure of dependencies. The transformer architecture naturally maps to that reality. perfectly sets up the ultimate test. We have this architecture capable of holding a massive web of context in its head all at once. What happens if we force it to spit that context out in an entirely different language? We are tackling chapter 11, machine translation. Translation is perhaps the most difficult task in NLP because of language divergences.
Typology, the linguistic study of how languages differ structurally, outlines the massive hurdles. Give me an example of a structural hurdle. The most common is word order. English is predominantly a subject -verb -object language. or SVO, the cat ate the mouse. Japanese, however, is SOV, subject -object verb, the cat, the mouse ate. So you literally cannot do a word -for -word translation. The machine has to read the whole English sentence, hold the noun mouse in its memory, wait for the verb ate, and then flip their positions when generating the Japanese.
And that is a simple divergence. The textbook also discusses pro -drop languages like Spanish or Italian. In English, you must explicitly state the pronoun. The green witch arrived, she said. But in Spanish, you might say, And the next sentence might just be the verb dropping the she entirely because the verb conjugation implies the subject. But if the machine is translating that Spanish text back into English, it has to magically generate the word she out of thin air. It has to look backward, realize a female witch was the subject of the previous sentence, and synthesize the correct English pronoun. To solve these immense structural puzzles, We use the encoder -decoder architecture, also known as sequence -to -sequence models. How does the division of labor work there?
It is two separate networks working in tandem. The encoder network reads the entire source sentence, say, the Spanish text. It uses self -attention to process all the complex relationships, and it compresses all of that semantic and syntactic information into a dense mathematical representation. It creates a pure concept vector. It strips away the Spanish and just leaves the meaning. Essentially. Then it hands that concept over to the decoder network. The decoder's sole job is to take that pure concept and begin generating the target language, say English word by word, conditioned on that context. But the textbook notes a flaw with early models. If you just hand the decoder one single summary vector at the very beginning and ask it to generate a 30 -word English sentence, it forgets the details of the Spanish sentence halfway through. It's the bottleneck problem again. Which is why we introduced the attention mechanism specifically into the translation process.
The decoder does not rely on a single static summary. At every single step of generating the English sentence, the decoder gets to look back at all the hidden states of the Spanish encoder. It's like it had the original document open on the desk next to it. While it is trying to figure out the fifth English word, it asks the encoder, hey, which of the Spanish words are most heavily weighted for what I'm doing right now? It dynamically aligns the two languages. Yes, soft alignment. But once the decoder knows what it wants to say, how does it actually choose the words? The most straightforward method is greedy decoding. At each time step, the model looks at its probability distribution and simply picks the single most likely next word. That sounds logical, but the text says it's a trap. Why? Because of dead ends.
Imagine navigating a maze by always taking the widest hallway at every intersection. It seems like a good strategy locally, but it might lead you straight into a brick wall. If the model makes a slightly suboptimal word choice early in the sentence, it cannot go back and fix it. That early error might force the rest of the sentence into a grammatical corner that ruins the translation. So instead of being greedy, we use beam search. Beam search is like exploring the multiverse. Instead of committing to one single path, the computer keeps co -possible paths open simultaneously. Let's trace it. Say your beam width, k, is 3. At the first step, the decoder doesn't just pick the top word. It generates the top three most likely starting words. Path A, Path B, Path C. Then, for each of those three paths, it looks ahead to the second word and generates the top three options. You now have nine branching paths. It calculates the total probability for all nine paths. It throws away the worst six and keeps the top three survivors going. It is constantly pruning the multiverse tree, ensuring it finds a sequence that is globally optimal, not just locally optimal.
It is a profoundly effective search strategy. However, the true bottleneck in modern machine translation isn't the architecture, it's the data. To train these massive encoder -decoder models, you need a bitext, a parallel corpus containing millions of sentences flawlessly translated by human experts in both languages. Which is easy to find for English and French. The Canadian parliament has been translating everything for decades. But if you want to translate English to a low resource language, you don't have millions of human translated sentences. This scarcity led to one of the most ingenious hats detailed in the textbook, back translation. I read this and thought it was a joke at first. Okay, I want to build an incredible English or French translator.
But I am totally out of human -translated English -French data. You are telling me I can take a massive pile of monolingual French text, just random French Wikipedia articles, and I run them backwards through a terrible, mediocre French -to -English translator. Or re -reverse model, yes. And it generates completely garbage English. Like, the apple of the earth is good. Then I pair that garbage, synthetic English with a perfect French Wikipedia text, and I train my real... forward -facing English to French model on that synthetic pairing. How does training on garbage data make the model better? It seems entirely counterintuitive, but consider what the model is actually learning during training. The encoder processes the garbage English. It struggles, but it extracts a fuzzy concept. But the decoder, the part of the network generating the final French output, is being trained to match the perfect French Wikipedia text. Back translation acts as a massive language modeling exercise for the decoder side. It forces the decoder to practice generating exceptionally fluent, grammatically perfect target language, even when the input is noisy. Fluency on the target side dramatically improves the human perception of the translation quality. That is deviously clever. But speaking of human perception, how do we grade these machines?
We can't have humans sit and grade a million translated sentences. Historically, the industry standard was a metric called BLEU. BLEU is essentially an n -gram overlap counter. It compares the machine's generated translation to a human reference translation, and it counts how many exact word sequences match. But that has a glaring flaw, doesn't it? If the human translator wrote the fast car arrived and the machine translated it as the quick car arrived, BLEU penalizes the machine because the string quick does not exactly match the string fast.
It treats them as failures, even though the semantic translation is perfect. To rectify this, modern evaluation utilizes metrics like BERT score. Which relies on embeddings. Exactly. BERT score uses dense contextual embeddings. It calculates the cosine similarity between the vector of the machine's word and the vector of the human's word. In vector space, quick and fast have a very high similarity score. BERT score provides a soft match, granting the machine credit for capturing the true meaning regardless of the explicit spelling. This entire evolution in translation is staggering. The textbook outlines how, historically, linguists used the Volquois triangle. You had to parse the source text, write explicit transfer rules to map the syntax to the target language, and then generate words. It was hundreds of thousands of lines of hand -coded grammatical logic. And neural networks bypass almost all of it. They learn to translate purely from statistical associations in parallel corpora, mapping strings to vectors and back to strings. But that raises a ma - When linguists attempt to formally model the syntax of human language, they predominantly rely on two distinct frameworks. The first, covered in Chapter 12, is context -free grammars, or CFGs.
They are also known as phrase structure grammars. How does a CFG model language? By relying on abstract hierarchical rules. A CFG assumes that words naturally group into larger abstract structural units called constituents. For example, a core rule might state that a sentence consists of a noun phrase followed by a verb phrase. Another rule might state that a noun phrase consists of a determiner followed by a noun. The textbook uses the ISS domain as an ongoing example. It was an early 1990s system for booking airline flights. You would build CFG rules to parse a user saying, I'd like to fly to Atlanta. Right.
A parser would analyze that sentence and build an invisible hierarchical tree above the words. It groups to Atlanta into a prepositional phrase. It groups that phrase with the verb fly into a massive verb phrase. It connects that to the subject I to form the overarching sentence root. It reminds me of the biological taxonomy system, classifying animals. You have the kingdom, the phylum, the class, the order, down to the genus and species. You are placing specific instances, the words, into increasingly abstract overarching categories. CFGs are incredibly rigid, which makes them highly structured and computationally tractable. They are absolutely essential for tasks like deep semantic parsing. If you are building a system that takes a natural language request like Book me a flight to Atlanta under $300 and needs to convert it into a flawless SQL database query to actually execute the transaction, the rigid hierarchical tree of a CFG guarantees the structure is mapped correctly. But they have a weakness. A significant one. CFGs struggle immensely with free word order languages. In languages where the subject, verb, and object can freely move around the sentence for emphasis, trying to define strict structural rules for image. Noun phrase followed by verb phrase becomes a nightmare. The invisible trees get impossibly tangled. Which brings us to chapter 14 and the second major linguistic framework. Dependency grammars. I found this visual so much more intuitive.
Instead of building an invisible abstract tree of categories above the words, dependency parsing just draws direct arrows between the actual words physically on the page. It strips away the abstraction. Dependency grammar argues that syntactic structure is defined by binary, asymmetrical relations between the words themselves. You identify a head word, usually the main verb of the sentence, and you draw directed lines to its dependence. So in I'd Like to Fly to Atlanta, the word fly is the head. It draws an arrow directly to I and labels it the nominal subject. It draws an arrow to Atlanta and labels it the destination.
There are no invisible noun phrases. Just words axing on other words. I believe you had an analogy for this as well. Yes. If CFGs are the abstract biological classification of animals, dependency grammars are a food web. You don't say a hawk is an avian, which is a vertebrate. You just draw an arrow that says the hawk eats the mouse. It maps direct functional relationships. It shows exactly who is doing what to whom. That is exactly what it does. And machines parse these dependencies using a few different algorithms. Transition -based parsing processes the sentence left to right, maintaining a stack of words and making a rapid sequence of decisions. Do I link the word I'm looking at now to the word I just put in the stack? It is incredibly fast. And the alternative? Graph -based parsing. It evaluates the entire sentence at once, scores every possible combination of dependency arrows, and uses algorithms to search for the globally optimal tree. Regardless of the method, dependency grammars are incredibly robust for those free word order languages. Because the arrows just stretch directly from word to word, the structure remains clear no matter how the words are physically scrambled in the sentence.
Okay, let's take a collective breath here. Because the journey we just took through the first half of Jarafsky and Martin is staggering. We started in 1966 with Eliza. a machine blindly swapping out words using rigid regex bouncer rules, completely devoid of understanding. We moved from that to defining the boundaries of a word through tokenization and byte pair encoding. We learned to quantify typos with the dynamic programming matrix of minimum edit distance. We built n -gram probability tables to guess the next word based on historical counting, and we implemented Nezerne smoothing to ensure the zeros didn't destroy our math.
Then we abandoned mere counting for geometry. We mapped meaning into vector space with Word2Vec. We upgraded our temporal processing from the sliding windows of feedforward networks to the gated memory of LSTMs. And finally to the transformers, using self -attention to ingest entire context simultaneously, allowing us to build encoder -decoders that translate languages while we map the fundamental rules of human syntax using CFG trees and dependency arrows. It's a phenomenal... awe -inspiring amount of engineering just to get a rock to read a text message. It is the culmination of decades of human ingenuity. But as we close out part one of this deep dive, I want to pose a provocative thought. It hit me while we were discussing those final grammar chapters. We spent all that time looking at CFGs and dependency trees, the human linguistic way of mapping syntax. We spent centuries going back to Panini in ancient India, writing down these formal rules of grammar, but... Modern transformers, like the ones doing flawless machine translation, aren't explicitly taught these CFG rules. They are never shown a dependency tree. They are not.
They ingest raw text. They learn to generate perfect syntax purely through the mathematical weights of self -attention and the objective of predicting the next word in a massive corpus. Which forces a very uncomfortable question. Does this mean the formal grammar rules we've spent centuries obsessing over are actually just a human illusion? Are CFGs and dependency arrows just... crude, low -resolution maps of a statistical territory that the machine, with its billions of dense vectors, actually understands far better than we do? Are we just projecting our own biological need for neat, visible rules onto a system that is fundamentally fluid? It is a profound philosophical question. Have we discovered the true structure of language, or merely invented a useful fiction to explain the statistical patterns our brains recognize intuitively? The success of deep learning suggests the underlying reality of language is far more geometric and continuous than discrete and rule -based. It brings us right back to our opening. We tried to find the jagged white line on an X -ray of language. We wanted strict bones.
But it turns out language isn't bone. It's a cloud, a massive multidimensional cloud of probabilities. And that is something for you to mull over until next time. Thank you for taking this deep dive with us. The next time you look at your phone's autocomplete or use a translation app today, take a second to look past the screen. Appreciate the invisible magnificent math making it happen. We will see you for part two.