EPISODE TRANSCRIPT
A First Course in Causal Inference (Peng Ding)
Automatically transcribed from the episode audio. Some words and names may be inaccurate.
Listen to the episodeDownload textDownload captions
Usually when we talk about a medical diagnosis, there's this expectation of precision. It feels, I don't know, almost like engineering, right? Oh, absolutely. Very cut and dry. Yeah, like you break your arm, the x -ray shows that jagged white line, and the doctor just points and says, there it is. Right. It's binary, broken or not broken. It's clean. And frankly, it's comforting. We really like things to be visible. But then you step into the world of empirical data. Oh, yeah. And that comfort just... Vanishes. Completely vanishes, especially when you are trying to figure out if one specific variable actually like caused a specific outcome. Suddenly that X -ray machine is just broken. We're looking at a diagnostic landscape that is entirely murky. I take it a step further, actually. It isn't just murky.
Without the proper mathematical architecture, it's actively deceptive. Deceptive? Yeah. Wow. Yeah, because you were essentially looking at shadows and trying to infer the existence of a three -dimensional object that you can never, you know, actually touch. Which is exactly why we are plunging into paintings of first course in causal inference. Because for you listening, our mission today on this deep dive is to conquer this text. And it is quite a text. It really is. It's the distillation of seven years of teaching at UC Berkeley. It pulls together the statistical, the biostatistical, and econometric perspectives of figuring out basically what causes what. Right.
Getting to the actual root of it. Exactly. So we are going to unpack the heavy math and translate it into profound real -world analytical tools. Because that traditional aphorism we all learned in Stats 101, you know, correlation does not imply causation. Right. The golden rule. Yeah. It's safe, but it's... fundamentally a dead end. This text argues that statistics is absolutely crucial for understanding causality. We just need like a rigorous formal language for it. And that formal language has to start with a philosophical grounding. I mean, the author anchors the text initially not in equations, but actually in antiquity. Oh, right. The Democritus quote.
Exactly. Democritus said, I would rather discover one causal law than be king of Persia. Which is a massive statement. It is. And Aristotle argued that we do not have knowledge of a thing until we grasp its cause. It really highlights that causal inference isn't just like a niche statistical tool. It is the core engine of human epistemology. But classic statistics, the kind focused purely on drawing lines of best fit and finding associations, frequently falls into traps.
OK, let's unpack this idea of traps, because the text highlights a famous one right out of the gate, the Ewell -Simpson paradox. Ah, yes. I think a lot of people vaguely remember Simpson's paradox from school as a quirky math trick. Like a trend appears in different groups of data, but then magically disappears when you combine them. Yeah, and it is often taught as a quirk, which honestly does a massive disservice to how dangerous it actually is in empirical research. Dangerous how? Well, the Yule -Simpson paradox demonstrates that an unmeasured binary confounder can completely overturn an observed association.
I mean, it isn't just the magnitude of the effect that changes. The entire direction of the causal arrow can completely flip. Right. And the real -world example the book uses here is the Lalonde job training data, which, if you study econometrics, is legendary. Oh, totally foundational. So you have this data set evaluating a job training program designed to boost future earnings. If you run a naive, simple regression model, just regressing future earnings on a binary indicator of whether they took the training, doing absolutely nothing to control for other variables. Just a straight -up raw comparison. Yeah, exactly.
The model spits out a coefficient of negative $8 ,506. Which, if you take that at face value, suggests the program was a catastrophic failure. Like actively ruining lives. Exactly. It implies that participating in this training actively destroyed over $8 ,000 of a person's future earning potential. Right. But then the text demonstrates that if you take that exact same data and run a multiple regression controlling for covariates. Things like age, education. Yeah, and crucially prior earnings. If you control for those, the coefficient on the training program flips to a positive $1 ,067. The harmful program is suddenly beneficial. Because the mechanism at play here is omitted variable bias. Right. See, in the naive regression, the error term is heavily correlated with the treatment variable. Why? Because the people who opted into this job training program were predominantly individuals who were already economically disadvantaged. They had lower education or a history of lower earnings to begin with.
Exactly. So the naive regression wasn't measuring the causal effect of the training at all. It was simply measuring the structural disadvantage of the people who were enrolled. So relying on the raw correlation is basically like, it's like trying to deduce the shape of a complex object purely from its two -dimensional shadow on a wall. I like that. Depending on the angle of the light, which in our data represents the covariates we choose to look at, the shadow completely changes. A cylinder looks like a circle from the top down, but a rectangle from the side. That's a really helpful visualization. But let's push it further.
If you don't account for the angle of the light, the confounders, you aren't just seeing a distorted shape. You are drawing conclusions that will dictate real -world policy, and you will be completely, entirely wrong. Which is terrifying for policymakers. Exactly. The shadow is just the association. The true shape is the causal effect. To move from the shadow to the object, we have to formally define what a cause even is. which requires abandoning the purely associative view of statistics. Which brings us to the bedrock of modern causal inference, right? The potential outcomes framework. Yes.
Formalized by Nyman in 1923 and expanded by Rubin in 1974. And the text pairs this with an aphorism from Rubin in Holland, which is no causation without manipulation. Right. And that phrase is a hard boundary line for experimentalists. How so? It means if you cannot conceive of a mechanism to manipulate a treatment, even hypothetically, discussing its causal effect is mathematically incoherent. Wow. Okay. Yeah. You can talk about the causal effect of a drug or a tutoring program because you can intervene and change it. You cannot strictly talk about the causal effect of being tall or being born in 1990. Because those are immutable attributes. Exactly. Wait, I'm stuck on that limitation. People do studies all the time claiming things like height causes higher salaries. Are you saying the potential outcomes framework rejects that entirely? Well, it forces you to redefine what you are actually studying. Are you studying the immutable physical attribute of height? Or are you studying the perception of height by the employer? Oh. Because if it's the latter, you can manipulate that in an experiment, say by altering the height listed on a medical profile during a hypothetical hiring exercise. The manipulation constraint, it forces precision. Okay, that makes sense. It forces us to define the treatment clearly. So let's look at how potential outcomes actually works.
The core idea is that for every single unit in your population, there exist two parallel realities. Two potential outcomes. There is the outcome that would happen if they received the treatment, denoted as Y of 1. And there is the outcome if they received the control, denoted as Y of 0. So the individual causal effect is literally just the mathematical difference. Right. Y of 1 minus Y of 0. And if we scale that up, we get what the text calls the science table. Yeah. Imagine a massive spreadsheet. Every row is an individual unit. The columns are Y of 1 and Y of 0. If we had that complete table. causal inference would be a trivial exercise in subtraction. Just simple math. But we have a fundamental problem.
We only ever get to observe half of the science table. Half of it is always missing. Right. I can't take the headache medicine and simultaneously not take the headache medicine at the exact same moment in time under the exact same conditions. Exactly. This is known as the fundamental problem of causal inference. For every unit, one potential outcome is realized and observed while the other... The counterfactual is forever unobservable. So we are left with this massive missing data problem. And to even attempt to fill in those missing values mathematically, the framework demands a really rigid assumption called SATFA. SATFA.
The Stable Unit Treatment Value Assumption. Which is one of those acronyms that sounds incredibly dry. Oh, incredibly dry. But its implications literally dictate whether a study is actually valid or just statistical noise. So... SUTVA requires two specific conditions. First, no interference between units. Second, no hidden variations of treatments. Let's break down no interference first. The text uses the example of flu shots. Right. So interference occurs when the treatment assigned to unit A affects the potential outcomes of unit B. Okay. With vaccines, this is herd immunity. If I am studying the effect of a flu shot on whether you... contract the virus, your outcome doesn't solely depend on your treatment status. If I get the shot, I'm less likely to become a vector. Which lowers my probability of getting sick, even if I got the control. Exactly.
Another brilliant example the book highlights is social media advertising. If a platform is trying to measure the causal effect of an ad on my purchasing behavior, they can't just randomize who sees the ad. Because of the network effect. Right. Because if my friend sees the ad and buys the product, they might... text me about it. My potential outcome just changed based on my friend's treatment status. The math of basic causal inference assumes a vacuum. It assumes each unit is an isolated island. When you have network effects, which, by the way, happen constantly in epidemiology, economics, and sociology. Yeah, the real world. Right. SUDFA is violated. You have to move to much more complex models of spillover effects. And the second part of SUDFA, no hidden variations of treatments, is just as crucial.
If we are trying to measure the causal effect of a college education, Sutva demands that college education is a uniform treatment. Which we know it isn't. Far from it. A degree in computer science from MIT is a fundamentally different treatment than a degree in poetry from a small liberal arts college. If the treatment is a chaotic mixed bag, the Y of one column in our science table just loses its meaning. Which brings us to the necessity of precisely defining the experimental unit. And the text illustrates this beautifully with what we can call the headache pill paradox.
OK, here's where it gets really interesting, because this kind of broke my brain a little. Let's hear it. Let's say I have a headache at 2 p .m. I take a pill. At 3 p .m., my headache is gone. The sloppy everyday logic is I had a headache. I took a pill. Headache is gone. Therefore, the pill cured it. I observed my state before and my state after. Why is that logically invalid under this framework? Because you are conflating time with counterfactuals. The experimental unit is not you across all of time. The experimental unit is a specific slice of space -time. A slice?
Yes. You at 2 p .m. is one unit. You at 3 p .m. is a completely different unit. Oh, because my body has naturally metabolized things, my stress levels might have dropped. The ambient temperature changed. Literally anything could have happened. Precisely. You have four potential outcomes happening here. Four? Yeah. U before the pill has an observed Y of zero and an unobserved Y of one. U after the pill has an observed Y of one and an unobserved Y of zero. You cannot simply subtract your historical before state from your current after state and declare it a causal effect. Because I don't know what U... after the pill would have felt like if I had refrained from taking it. The headache might have simply resolved on its own. Exactly.
This level of strictness feels almost paralyzing. It forces you to acknowledge the vastness of the unobservable counterfactuals. And since we can't observe them, we have to rely on the mechanism that assigns people to the treatment or control in the first place, right? The treatment assignment mechanism? Right. And the book uses a fantastic hypothetical to contrast assignment mechanisms. The omniscient perfect doctor versus the clueless doctor. OK, let's talk about the perfect doctor. Let's say we have this doctor who knows the exact unobservable individual causal effect for every patient who walks in the door. A literal mind reader. Yeah. They know exactly who will benefit from the pill and who won't. So acting ethically, they only prescribe the pill to the people it actively helps. Okay, so now imagine you were a researcher looking at this perfect doctor's clinical data a year later. You look at the treated group. They all had fantastic outcomes. You look at the control group. They had worse outcomes.
Right. If you calculate the naive difference in means between the two groups, the drug looks like a miracle cure. But wait, doesn't that contradict what we just said? The doctor sorted everyone perfectly. Doesn't that prove the drug works? It proves the drug works for some, but... It completely ruins your ability to estimate the average causal effect for the whole population. Because the groups aren't comparable anymore. Exactly. By acting with perfect knowledge, the doctor introduced massive, insurmountable selection bias. The treated group and the control group are fundamentally different kinds of people.
Because the treatment assignment was perfectly correlated with the potential outcomes. Yep. Okay, so contrast that with the clueless doctor. This doctor doesn't know anything about medicine. They just flip a coin for every patient. Heads, you get the pill. Tails, you don't. From a medical ethics standpoint, it's a total nightmare. Sure. But from an epistemological standpoint, it is perfection. By assigning the pill randomly, the assignment mechanism is rendered entirely independent of the patient's potential outcomes. It's totally blind. Yes. And as your sample size grows...
The group that gets the pill and the group that gets the control will become statistically identical in their baseline characteristics. Both the things we can measure and, crucially, the things we cannot measure. So when we take the difference in means from the clueless doctor's clinic, we are actually uncovering the true average causal effect. By stripping away human choice and relying on random chance, we eliminate the bias. The clueless doctor is doing superior science. They really are. And this brings us out of the theoretical and into the foundational mathematics of part two of the text, which is all about randomized experiments. Right. When we formalize the clueless doctor's coin flip, we are utilizing the bedrock laid down by R .A. Fisher in his 1935 work, The Design of Experiments. Let's talk about Fisher's randomization test, because the text frames randomization as having this sort of mathematical superpower.
It really is a superpower. A completely randomized experiment neutralizes unmeasured confounding by design. Think back to the missing half of the science table. The unobservable counterfactuals. Right, because the treatment assignment is independent of the potential outcomes. The distribution of any test statistic under the null hypothesis is completely deterministically dictated by the randomization mechanism itself. We need to define Fisher's sharp null hypothesis here because it's an extreme stance. It isn't just saying the treatment has no average effect. Oh, no. Fisher's sharp null hypothesis posits that the treatment has absolutely zero effect on any individual unit in the study. For every single person, Y of 1 is exactly equal to Y of 0. So under Fisher's sharp null, this pill is essentially a sugar pill for everyone. It literally does nothing.
Yes. And if we assume that it's true, something mathematically magical happens. We suddenly don't have any missing data in our science table. Wait, I'm trying to visualize this. If I know for a fact that y of 1 equals y of 0 for me... Under the null hypothesis, yeah. And I observe that my headache went away after taking the pill, then I know with absolute certainty that my headache would have gone away even if I hadn't taken the pill. Precisely. Under the sharp null, whatever outcome you observe under your assigned treatment, is exactly the outcome you would have observed under the alternative. We can seamlessly impute the missing counterfactuals. Wow. And once we have a fully imputed, complete science table, we can run a permutation test. Okay, let's unpack the permutation test. Is this basically asking what are the odds I would see a result this extreme if the treatment assignment was just random noise, like shuffling a deck of cards over and over?
I like the card analogy. If I deal out a hand a thousand times and the hand the dealer originally gave me is better than 999 of those random shuffles, I reject the idea that the deck is fair. That's the exact logic. In the Fisher randomization test, you take the fully imputed science table and you computationally reassign the treatments based on the randomization mechanism thousands of times. For every permutation, you calculate the difference in means. This generates an exact null distribution. If your originally observed test statistic falls way out in the tail of the simulated distribution, you reject the sharp null hypothesis. You have proven that the treatment did something to someone. But the text draws a sharp distinction between Fisher's permutation approach from 1935 and Yersey -Naman's approach from 1923. What's the fundamental difference there? What's fascinating here is that they were solving slightly different problems. Fisher was the rigorous skeptic asking, is there any effect at all? Neyman, on the other hand, was focused on estimation. He wanted a number. He wanted to know the average causal effect across the population. So he introduced repeated sampling inference. Neyman proved that the simple difference in sample means is an unbiased estimator of the finite population average causal effect. But Neyman's math gets tricky when it comes to the variance, right? The text mentions that Neyman's estimator for the variance is inherently conservative.
Why is that? I'm really glad you brought that up because it hits right back at the core of the missing data problem. To calculate the exact true variance of the average causal effect, the mathematical formula requires knowing the covariance between y of 1 and y of 0 for each individual. But we can never know that. Because we never observe y of 1 and y of 0 for the same person. Exactly. The covariance of counterfactuals is fundamentally unidentifiable. Neyman recognized this, so he derived a variance estimator that essentially assumes a constant additive effect, which provides an upper bound. So it plays it safe. Very safe. It is a conservative estimate, meaning your confidence intervals might be slightly wider than they theoretically need to be, but they won't falsely claim precision you don't actually have. So Fisher proves the drug isn't a total placebo. And Neyman gives us a safe, unbiased estimate of how much it helps on average.
Right. And the text also bridges a critical gap here regarding populations, because Neyman's original 1923 math was strictly for a finite population, meaning it only mathematically applied to the exact people sitting in the room participating in the trial. Which is pretty limiting because researchers almost never care only about the people in the trial. They want to generalize to a superpopulation. The whole world. Or... at least a larger demographic. The book demonstrates that if you treat your experimental units as a random sample drawn from a broader superpopulation, Neyman's estimators still hold. The variance calculations shift to account for the sampling uncertainty from the superpopulation, but the core logic remains totally intact. It's really elegant. Yeah. When we can shuffle the deck, when we can rely on the clueless doctor to randomize, the math provides these clear answers. But that brings us to the threshold of part three of the book. The messy part. Very messy. The real world is messy. We can't run randomized controlled trials for everything. We can't force a random half of a town to smoke cigarettes for 20 years to see if they get cancer. Right. Highly unethical.
We can't force people to drop out of high school to measure the impact on their wages. So when people self -select their treatments, we are thrust into the Wild West of observational studies. And navigating observational studies requires a completely different mindset. It's anchored by what the text introduces as Dorn's Question from 1953, which is, how would the study be conducted if it were possible to do it by controlled experimentation? So whenever we are staring at messy, real -world data, our first conceptual step is to design the hypothetical perfect experiment we wish we could have run? Yes. Dorn's Question forces you to identify the target trial, because in observational data, you are immediately confronted with selection bias. As we saw with the Lalonde job training data, when people choose their own treatment, the potential outcomes, Y of 1 and Y of 0, are no longer independent of the treatment assignment.
Right. The people who take the medicine might be fundamentally sicker than the people who don't. Exactly. The book defines selection bias mathematically as the difference between the prima facie causal effect, which is just the naive difference in means between the treated and untreated groups, and the true average causal effect. And to close that gap, to eliminate that bias without randomization, causal inference relies on a monumental theoretical pillar, the ignorability assumption. Frequently called unconfoundedness. Unconfoundedness, right.
Okay, let's unpack ignorability. It states that conditional on a set of observed covariates, which we'll call X, the treatment assignment is independent of the potential outcomes. Translating that from mathematical notation to reality, we are assuming that if we measure enough characteristics about our subjects, age, gender, income, detailed medical history, we can essentially recreate a randomized experiment inside strata of the data. Inside the subgroups. Right. We are declaring that if two people have the exact same values across all measured X variables, the fact that one received the treatment and the other didn't is essentially a coin flip. It's random. So what does this all mean for a researcher? It means we are taking a massive leap of faith that we have measured everything that influences both the choice to take the treatment and the final outcome. We are assuming there are no invisible variables, no hidden confounders lurking in the shadows. It is a profound, untestable leap of faith. But if you are willing to make that assumption, the text provides an arsenal of estimation strategies.
What's the most basic one? The most basic is outcome regression. You simply regress the outcome on the treatment variable and all your covariates X. But the text warns that regression relies on parametric assumptions. It forces the data into specific shapes, usually assuming linear relationships. And if the true relationship between, say, age and income isn't a straight line, your regression will spit out a biased causal estimate. Which is why statisticians sought more flexible, elegant solutions. And this led to a paradigm shift in 1983 with Rosenbaum and Rubin's introduction of propensity scores. Oh, man, propensity scores.
Here is a concept that completely dominates modern research. What exactly is a propensity score? A propensity score is simply the conditional probability of receiving the treatment given your observed covariates. Mathematically, it's EX equals the probability that T equals 1 given X. OK. It's real genius lies in dimension reduction. Right, because if I have 50 different covariates, exact age, exact income, zip code, blood pressure, finding two people in my data set who match perfectly across all 50 dimensions is basically statistically impossible. It is. It's called the curse of dimensionality.
But Rosenbaum and Rubin proved a vital theorem. If treatment assignment is strongly ignorable given the full vector of covariates X, It is also strongly ignorable given just the scalar propensity score, EX. So I don't need to match people on all 50 variables. Exactly. You just run a model, usually a logistic regression, to calculate each person's probability of getting the treatment based on their traits. This collapses 50 dimensions down to a single number between 0 and 1. Wow. So if two people both have a propensity score of 0 .75, they have the exact same baseline likelihood of being treated. And once we have that score, one of the most powerful things we can do with it is inverse propensity score weighting, or IPW. Let's really drill down into the mechanics of IPW because it's brilliant but honestly counterintuitive.
It can be. IPW operates by constructing a pseudopopulation, where the treatment assignment is no longer confounded by the covariates. It achieves this by weighting each individual in your dataset by the inverse of the probability of receiving the treatment they actually received. Let me test an analogy here to see if I'm grasping the math. Go for it. Let's say we are evaluating a grueling extreme fitness boot camp. The covariates tell us that young elite athletes have a 95 % probability of signing up. Older sedentary individuals have a 5 % probability of signing up. A classic case of selection bias. The treated group will be overwhelmingly young and fit. The control group will be older and sedentary. So in my data set, I have hundreds of elite athletes who took the boot camp, but I only have one sedentary guy who accidentally wandered in and took it. To figure out the average effect of this boot camp for the general population, IPW acts like a massive mathematical megaphone for the unusual cases.
The athlete's weight is 1 divided by 0 .95, barely over 1. Their voice is quiet. But the sedentary guy's weight is 1 divided by 0 .05. He gets a weight of 20. His single outcome gets amplified 20 times to balance out the population. Does that track? That is an exceptionally precise analogy. By upweighting the treated individuals who were unlikely to be treated, and conversely upweighting the control individuals who were unlikely to be controls, IPW perfectly balances the covariate distribution across the two groups.
It's like magic. You have synthetically generated the clueless doctor's random coin flip. But wait, I see a glaring mathematical hazard here. What if the denominator is practically zero? What if the propensity score for that sedentary guy was 0 .001? His weight becomes 1 ,000. Doesn't that one single data point completely hijack the entire analysis? Ah, you've just identified the fatal flaw of IPW. Extreme propensity scores cause the variance of the estimator to absolutely explode. If your model for calculating the propensity score is misspecified, or if you have weights approaching infinity, your causal estimate becomes dangerously unstable. So IPW is powerful but fragile. How do we build a safety net? Well, the text introduces the pinnacle of these estimation strategies, doubly robust estimators. These estimators mathematically synthesize both outcome regression and inverse propensity score weighting into a single equation. Doubly robust. It sounds like wearing a belt and suspenders. It's exactly that.
The mathematical architecture of a doubly robust estimator is astonishing. It is designed so that if you get one of your models wrong, let's say your logistic regression for the propensity score is completely misspecified, but your outcome regression model is correct, the estimator still converges to the true average causal effect. And vice versa. If my outcome regression assumes a linear relationship, that is totally wrong, but my propensity score model correctly captures the treatment assignment mechanism, it still works? Yes. It gives you two chances to be right.
The math involves taking the IPW estimator and adding an augmentation term. If the outcome model is correct, the IPW error cancels out. If the propensity model is correct, the expected value of the augmentation term becomes zero. It is a brilliant mathematical safety net. The book also touches on matching algorithms, where you essentially search the data for statistical twins, and modern approaches using machine learning, like Bayesian additive regression trees, BART, or causal forests developed by researchers like Wager and Athe. Right, moving past standard regression.
They use machine learning to calculate these conditional expectations without relying on rigid parametric formulas. Machine learning is incredibly effective at identifying nonlinear relationships and complex interactions between covariates that a human researcher would never ever think to specify in a standard regression model. But, and this is a massive neon flashing but, whether we are using basic regression, the megaphone of IPW, the belt and suspenders of doubly robust estimators, or the most advanced machine learning AI, every single one of them still fundamentally relies on the ignorability assumption. Yes, all the mathematical gymnastics in the world cannot save you if you fail to measure a critical compounder.
Which hurls us into part four of the book, when ignorability fails. Chapters 16 through 20, because let's be honest, we can never definitively prove we've measured everything. There could always be a ghost in the machine. And acknowledging that ghost is the hallmark of rigorous science. To navigate this, the text introduces Judea Pearl's directed acyclic graphs, or DAGs. DAGs. For those who haven't used them, these are visual models. You draw nodes representing your variables and draw arrows representing causal pathways. They are visual, but they are underpinned by rigorous rules of conditional independence, specifically a concept called de -separation. DAGs are crucial because they prevent researchers from making a very common, very dangerous mistake. over -adjustment.
I think the instinct for most people doing data analysis is, I have this massive data set with 100 variables. I'm just going to throw every single one of them into my regression model as a control variable just to be safe. That is the kitchen sink approach. And DAGs prove mathematically why it is disastrous. The text highlights two specific structures that will ruin your analysis if you blindly control for them, M -bias and Z -bias. I really want to spend time on this because it's counterintuitive. Let's start with M bias. How does controlling for a variable actually create bias? Imagine a DAG shaped like the letter M. You have your treatment on the bottom left, your outcome on the bottom right. Now imagine two unmeasured hidden confounders at the top points of the M. And in the bottom middle, you have a measured variable, M. Okay, holding that shape in my head. The first hidden confounder causes both the treatment... and variable M.
The second hidden confounder causes both the outcome and variable M. Crucially, the treatment does not cause M, and M does not cause the outcome. M is what we call a collider variable. It is an effect of two separate independent causes. So because the treatment doesn't cause the outcome through M, and M is in a traditional confounder, what happens if I decide to include M in my regression model? If you condition on a collider, You open a spurious path. In the raw data, the treatment and the outcome are not confounded through this M pathway. But the moment you control for M, you mathematically induce a correlation between the two hidden confounders, which suddenly opens a backdoor path from the treatment to the outcome.
Wow. By trying to be safe and controlling for M, you literally injected bias into your estimate that wasn't there to begin with. That is terrifying. It's like trying to put out a small grease fire by throwing a bucket of water on it. You just spread the flames. And what about Z -bias? Z -bias occurs when you improperly control for an instrumental variable, a variable that strongly predicts the treatment but has no direct effect on the outcome. Including it in an outcome regression doesn't necessarily introduce confounding, like M -bias, but it can severely inflate the variance of your estimate. and actually amplify any residual confounding that you've failed to measure. The overarching lesson of DAGs is that you must think deeply about the causal mechanics generating your data before you ever touch a regression formula. But even with perfect DAGs, the ultimate threat is the unmeasured confounder, the ghost.
The ghost in the machine. And the book grounds this in a legendary historical debate. R .A. Fisher's 1957 critique of the link between smoking and lung cancer. This is one of the most... instructive episodes in the history of statistics. By the late 1950s, observational studies were showing a massive association between smoking and lung cancer. The medical community was ready to declare causation. Ready to close the book on it. But Fisher, the genius who invented the randomization test we discussed earlier, stepped in and said, not so fast. Fisher basically played the ultimate devil's advocate. He pointed out that we didn't randomize who smoked. People self -selected.
So, He hypothesized a ghost. What if there is a hidden, unmeasured genetic mutation, a gene that simultaneously makes a person crave nicotine and makes them highly susceptible to lung cancer? If that common cause gene exists, Fisher argued, then the entire association between smoking and cancer is spurious. Smoking doesn't cause cancer. The gene causes both. Which, I mean, sounds like corporate propaganda today, but from a strict causal inference standpoint, Fisher was mathematically justified in raising the objection. He pointed directly at the weakness of the ignorability assumption. So how did the field defeat Fisher's argument if they couldn't find the gene? They defeated him with math, specifically Jerome Kornfield in 1959. Kornfield didn't run a new biological study. He pioneered what we now call sensitivity analysis. He essentially said, Okay, Fisher, let's assume your hypothetical gene exists. How powerful would that gene have to be to completely explain away the massive association we observe in the data? And the data showed that smokers were roughly nine times more likely to develop lung cancer than nonsmokers. Exactly. Kornfield proved mathematically that for an unmeasured confounder to entirely explain away a nine -fold increase in risk, that hidden confounder itself would have to be at least nine times more prevalent in the smoking group than in the non -smoking group. So if smoking increases cancer risk nine times, this invisible cancer gene would have to be incredibly, almost impossibly concentrated specifically among people who chose to smoke.
Yes. Kornfield's inequality demonstrated that while Fisher's hypothesis was technically possible under the math, it was biologically absurd. A genetic factor that perfectly segregated itself among smokers at a 9 to 1 ratio defied all understanding of genetics. Kornfield shifted the burden of proof. And the text details how this historical concept was recently generalized by Vanderweel and the author, Peng Ding, into a modern metric called the E -value. The E -value is a formal quantification of Kornfield's logic. It calculates the minimum strength of association that an unmeasured confounder would need to have with both the treatment, and the outcome to fully explain away your observed causal effect. So if we connect this to the bigger picture, the E -value is essentially a mathematically rigorous BS detector. Well, I wouldn't use those terms in a peer -reviewed paper, but conceptually, yes. It changes the nature of peer review. Instead of a critic saying, you might have unmeasured confounding, which is a lazy critique because the answer is always yes, the researcher can publish their E -value and say, Yes, there might be a hidden confounder, but my e -value is 15.
Your hypothetical confounder would have to increase the likelihood of treatment 15 times and increase the outcome 15 times, beyond all the covariates I already controlled for. Does such a variable exist in reality? It grounds theoretical skepticism in numerical reality. Now, what happens when the assumptions fail, not because of hidden variables, but because of the structure of the treatment assignment itself? What if the overlap assumption fails entirely? Ah, overlap, or positivity, assumes that every individual has at least some non -zero probability of receiving either the treatment or the control, meaning the probability is strictly between 0 and 1. But in many policy settings, treatment is determined by a strict deterministic cutoff. The text dedicates Chapter 20 to this exact scenario, regression discontinuity. Let's paint the picture. Imagine a university scholarship that is automatically awarded to any student who scores an 80 or above on a standardized test.
If you score 79 .9, you get nothing. In this scenario, overlap is completely violated. The probability of treatment for someone with a 75 is 0%. For someone with an 85, it's 100%. You cannot use propensity score weighting. You cannot do standard matching because there are no untreated students with an 85 to compare to. It seems like a dead end, but economists realize the strict rule is actually hiding a secret experiment. Yes. We look at the forcing variable, the test score, and we zoom in microscopically on the threshold. The fundamental assumption of regression discontinuity is that potential outcomes are continuous at the cutoff. We assume that the student who scored a 79 .9 and the student who scored an 80 .1 are... on average identical in their unmeasured traits their motivation and their background the only reason one got an 80 .1 and the other got 79 .9 was just random noise a lucky guess on a multiple choice question a momentary distraction exactly right at that invisible boundary line nature has artificially generated a randomized experiment for us by comparing the outcomes of the students just barely above the line to those just barely below, we can estimate the causal effect of the scholarship. But I imagine the caveat here is that we are only estimating the effect for those specific boundary students, right? We aren't estimating the effect of the scholarship for the kid who scored up 40. You've hit on a critical limitation. Regression discontinuity yields a local average treatment effect, or LATE.
It is internally highly valid, but externally limited. Furthermore, it assumes researchers are dealing with a sharp discontinuity. If the cutoff can be manipulated, say, students near the boundary lobby the teachers for an extra fraction of a point, the random noise assumption breaks, and you have to use more complex, fuzzy RD models. Regression discontinuity is an incredible hack for finding causality in arbitrary rules. But what if we are dealing with messy observational data where there is no clean threshold and we know we have unmeasured confounders? This brings us to section six of our deep dive, instrumental variables. This is the absolute hallmark of econometric methodology covered in chapters 21 and 23. If DAGs and propensity scores are about measuring everything, instrumental variables, or IV, is a strategy for when you accept that you can't measure everything. It's a completely different paradigm. At the center of this paradigm is the instrument itself. Let's call it variable Z.
The text requires an instrument to meet two strict criteria. First, relevance. The instrument Z must actually affect whether you take the treatment D. Second, the exclusion restriction. The instrument Z must have absolutely zero direct causal effect on the outcome Y, except through its effect on the treatment. The exclusion restriction is a massive assumption. The instrument must be perfectly unconfounded with the outcome. To really understand how this math works, the book transitions from pure observational data back to randomized experiments, specifically focusing on Chapter 21, noncompliance, because human beings ruin perfect experiments.
They really do. In a textbook randomized controlled trial, you assign someone to take a drug and they take it. In reality, you suffer from noncompliance. Some people assigned to the treatment group experience side effects and stop taking the pill. Some people assigned to the control group get desperate, go to another doctor and secure the drug anyway. So the treatment assigned by the researcher is no longer the treatment received by the patient. We have a divergence. And this divergence forces us to define what we are actually measuring.
The most basic metric is the intention -to -treat effect, or ITT. The ITT is essentially the causal effect of the prescription, right? We just compare the average outcome of everyone we told to take the drug against everyone we told not to, ignoring what they actually put in their bodies. Yes. The ITT is mathematically pure because it relies solely on the original randomization. It's unbiased. But scientifically, it's often unsatisfying. A biological researcher doesn't care about the effect of a doctor's advice. They want to know the physiological effect of the molecule on the human body. But we can't just compare the people who actually swallowed the pill to the people who didn't because of selection bias. The people who complied and took the pill might be fundamentally more health conscious than the people who ignored the doctor. Exactly. If you just look at the treatment received, you are back in the murky waters of confounded observational data. This conundrum was elegantly solved by the Nobel Prize -winning framework developed by Angrist, Invens, and Rubin. They showed how to use the randomized assignment itself as an instrumental variable.
Okay, let's walk through this carefully. The random assignment, the doctor's prescription, is our instrument, Z. It is relevant because telling someone to take a pill definitely influences whether they take it. And it meets the exclusion restriction because merely hearing the words, take this pill, does not magically cure a disease. It only works if you swallow the pill. Perfect. To deploy the math, Angerist, Imbens, and Rubin conceptually divided the population into four latent strata based on how they respond to the instrument. It was before compliance groups.
First, the compliers. These are the rule followers. If assigned the pill, they take it. If assigned the control, they don't. Second, the always takers. They will take the pill no matter what group they are put in. Third, the never takers. They refuse the pill even if assigned to it. And fourth, the defiers. They do the exact opposite of whatever you tell them. Trolls, basically. We typically invoke an assumption called monotonicity, which posits that there are no defiers. We assume the instrument only pushes people in one direction.
So we are left with compliers, always takers, and never takers. Here is where the math feels like magic. We want the causal effect of the pill. The text explains that the IV method isolates the complier average causal effect, CACE, also known as the local average treatment effect, or LATE. Let's trace the logic. We observe the overall ITT, the effect of the assignment on the outcome. But we know the assignment had zero effect on the always takers. Why? Because they were going to take the pill regardless. The assignment didn't change their behavior, so... It couldn't have changed their outcome. And the exact same logic applies to the never -takers. The assignment didn't change their behavior either. They refused the pill in both alternate realities. Therefore, 100 % of the difference in outcomes we observe in the overall ITT is being driven exclusively by the compliers, the only group whose behavior was actually manipulated by the instrument. I want to use an analogy here to make the math concrete. Let's say I want to study the causal effect of exercising at a specific gym on weight loss. But gym attendance is horribly confounded by personal motivation. Highly motivated people go to the gym and lose weight. If I just run a regression, motivation acts as a massive, unmeasured confounder. Right. So I need an instrument. I randomly mail a massive 50 % off coupon to Half the Tone. The coupon is randomly assigned so it's unconfounded. Now I look at my data. The ITT is the difference in weight loss between the people who received the coupon in the mail and those who didn't. But the ITT just measures the effect of receiving a piece of mail. Exactly.
To find the late, the actual effect of going to the gym, I take that ITT and I divide it by the proportion of people who are actually influenced by the coupon. I divide by the compliance rate. Because the always -takers were already going. Right. The always -takers were already going to the gym at full price. The coupon didn't change their behavior. The never -takers aren't leaving their couch even with a discount. The IV math essentially strips away the always -takers and never -takers and calculates the exact causal effect of the gym specifically for the people who went only because they got the coupon. That is exactly how the Wald Estimator functions mathematically.
Late is the ITT of the outcome divided by the ITT of the treatment. But this raises an important question. It's a profound admission of limits. We are admitting that we can never know the average causal effect of the gym for the always takers. We can never know it for the never takers. We can only know it for this invisible subgroup of compliers. Yes, and that is a massive paradigm shift in statistics. It forces researchers to be incredibly specific about who their data is actually speaking for. You cannot generalize a late estimate. to the entire population, unless you make incredibly strong, usually unjustifiable assumptions. IV is a scalpel. It cuts precisely, but it only cuts a very small area. Instrumental variables elegantly handle the messiness that happens before the treatment is taken, the noncompliance, the self -selection. But as we move into the final leg of our journey, part six of the book, chapters 26 through 29, we encounter a totally different type of messiness. What happens when the complications occur? after the treatment is administered. We are entering the domain of post -treatment traps. This is where causal inference ceases to be just mathematics and bumps hard against the metaphysical limits of science. We are dealing with principal stratification and mediation analysis. Let's define the scenario.
You administer a treatment. That treatment affects a mediator variable, and that mediator variable subsequently affects the final outcome. A classic example, a doctor prescribes a new diet, the treatment. The diet lowers the patient's cholesterol, the mediator. The lower cholesterol reduces the risk of a heart attack, the outcome. Clinically, we want to know the mechanics of the treatment. Does the diet work exclusively because it lowers cholesterol? Or does the diet work through some other unmeasured pathway, like reducing inflammation, independent of cholesterol? We want to mathematically separate the indirect effect, through cholesterol, from the direct effect, the other pathways. It sounds totally reasonable. Just control for cholesterol in a regression, right? No. Remember the DAGs.
Cholesterol is a post -treatment variable. It is caused by the treatment. If you control for it in a standard regression, you are likely inducing Z -bias or M -bias. You will completely scramble the causal estimate. Okay, so how does the potential outcomes framework solve this without relying on standard regression? It attempts to solve it using nested potential outcomes. To calculate a natural direct effect, the math requires us to define a variable like y of 1 m sub 0. This is where the math gave me vertigo. Let's translate y of 1 comma m sub 0 into reality. The notation y of 1 comma m sub 0 represents the potential outcome, y, that an individual would experience if they were exposed to the treatment, the 1, but their mediator, m, was artificially held at the exact, specific level it would have been if they had received the control, the zero. Wait, I need to map this to the diet example. We want to measure the direct effect of the diet. The math says we must imagine a parallel universe where a patient is strictly adhering to this new, super healthy diet, so treatment equals one, but... Somehow, we reach into their biology and magically hold their cholesterol level exactly at the terrible high level it would have been if they had spent the last month eating fast food. So mediator equals zero. Yes, we are conceptually decoupling the treatment from its natural biological consequence to isolate the alternative pathways.
But that is physically impossible. We cannot biologically separate the healthy diet from the cholesterol response in that specific counterfactual way. And if we can't physically do it, even in a hypothetical experiment, some statisticians argue this entire exercise is basically science fiction. This raises the most profound philosophical clash in the book, found in chapter 27. Researchers like Frangakis and Rubin label these nested potential outcomes as a priori counterfactuals. They argue that because y of 1, m sub 0, cannot be observed in any conceivable physical experiment, Utilizing it crosses a hard line. You are no longer doing empirical science. You are doing metaphysics.
Metaphysics. The book literally brings the philosopher Karl Popper into the statistical debate. Yes. Popper's demarcation of science is based on falsifiability. A scientific hypothesis must be capable of being tested and proven false by physical observation. Because the state of y of 1 m sub 0 is fundamentally universally unobservable, any mathematical claim you make about it cannot be falsified. The strict Popperian statistician looks at natural direct effects and mediation analysis and says, this is mathematical poetry. It is not statistics.
That is a staggering critique. You have these elegant equations that work perfectly on paper, but the reality they describe cannot exist. It feels like a devastating blow to mediation analysis. Does the book offer a way out, or does it just leave us hanging in an existential crisis? Pingding does not leave us hanging. The text presents alternatives that satisfy the Popperian requirement for physical observation. Specifically, the controlled direct effect in Chapter 28. How does the controlled direct effect differ from the metaphysical natural direct effect? Instead of imagining a magical, unobservable, counterfactual world where cholesterol is held at what it would have been, the controlled direct effect asks a fundamentally different question. What is the effect of the diet if we physically, medically intervene and force everyone's cholesterol to a specific, observable, fixed level, say, exactly 150 milligrams per deciliter?
Oh, I... could theoretically run that experiment. I could put people on the diet and then use separate statin medications to clamp everyone's cholesterol at exactly 150. That is a physical observable intervention. Exactly. It grounds the mathematics back in observable reality. It forces the researcher to be incredibly precise about what they are actually claiming to know. If you cannot describe a physical experiment that corresponds to your mathematical equation, you must think very carefully about whether you should be claiming to have calculated it at all. It is a stunning reminder that causal inference isn't just about crunching numbers in a black box algorithm. It requires a deep philosophical interrogation of the physical reality those numbers are meant to represent.
Which is the core thesis of the entire textbook. And it brings us to the end of our journey today. We have covered an immense amount of intellectual ground, unpacking the architecture of Peng Ding's A First Course in Causal Inference. We really have. We've traversed the dangerous illusions of simple association and the Yule -Simpson paradox. We built the rigorous, demanding foundation of potential outcomes, counterfactuals, and the strict rules of sattva. We explored the mathematical magic of Fisher's randomization and Nyman's estimation and navigated the treacherous, unconfounded waters of observational studies using propensity scores and doubly robust safety nets. We deployed DAGs and E -values against hidden ghosts, leveraged the econometric hacks of instrumental variables in late. And finally, we brushed right up against the epistemological limits of science with mediation and post -treatment traps. It truly is a masterclass in how to think critically about the universe.
And I want to leave you, the listener, with a final provocative thought to mull over. The next time you are staring at a spreadsheet, using historical data to make a major decision in your life or your business, maybe you're evaluating the ROI of a massive marketing campaign or trying to trace the root cause of a health issue. Stop and ask yourself this question. It's an important one. Are you relying on an assumption that can actually be tested in the real physical world? Or are you hidden deep in your own logic, implicitly making a metaphysical claim about a parallel universe that can never exist? Because as we've learned today from Peng Ding, data isn't just a collection of numbers. It is a philosophy of reality. And mastering that philosophy is the only way we can ever truly understand what causes what. Thank you for joining us on this rigorous, custom -tailored deep dive into the architecture of causal inference. We hope your diagnostic x -ray machine for empirical data is looking a whole lot sharper. Until next time, keep questioning the causes.