Natural experiments
POL51
December 5, 2023
An empty-garbage-can model
Some studies don’t even control for confounds
From the dog ownership improves health article in the homework:
“A possible limitation was that the analyses were not adjusted for confounders.”
Huge confounds abound! A person who is too sick to care for a dog won’t get one
Causal inference
- More recently: a greater emphasis on worrying about causality
- Recognition that confounds are lurking everywhere, and we should worry about them
- Recognition that pipes and colliders means we shouldn’t just throw every variable into a model
Improvements but obvious limits
IF we can correctly specify our model (i.e., control for all backdoors, leave front-doors open) we will identify the effect of X on Y
Easier said than done! Often there is some confound (like W) that we know exists but can’t measure, or worse, don’t know exists at all
The causal revolution
Hard to correctly specify the DAG and measure all the variables; what can we do?
Ideal = an experiment – where treatment is randomly assigned – but this is rarely feasible
Natural experiments
Alternative - find moments, situations, or weird, freak occurrences where some group is exposed to some treatment by chance while another group was not
These weird moments in history are called natural experiments
As though nature accidentally created an experiment
A good year for causality
2021 Nobel Prize winners in economics pioneered quasi-experimental methods – using research designs that leverage natural experiments
Do gifted programs pay off?
- Millions of kids across the US placed in gifted programs
- Parents/society have to weigh pros and cons; is this worth doing?
- For instance: does being in gifted program increase your chances of success later in life?
The results
Big gifted program premium: $24,645.72 more in earnings
The problem
So many potential confounds, including ones that are really, really hard to measure (like ability) and others that are unknown
The solution
To get into gifted you typically need a certain score on an aptitude/IQ test
Imagine that the cutoff score is 75/100; above you’re in gifted, below you are not
This cutoff is known as the assignment mechanism = why some receive treatment (gifted program) or not
How was that cutoff chosen?
Visualize it
People with higher aptitude/training/ability/etc. better off today:
The cutoff
The natural experiment
The discontinuity
These are the students who, by chance, just barely made it (or not) into gifted
The discontinuity
On average, the students who by chance made it in make less money
The payoff
In the naive estimate we think gifted programs help students in the long-run;
With the discontinuity we see the opposite: gifted hurts
Arbitrary cutoff gets us clean estimates for free: no need to worry about backdoor paths
Diamond and Doyle
Finding
1 in 4 babies born after midnight spend an extra night in hospital (💵💵💵), even when there is no health benefit to doing so
McNamara and the Whiz Kids
- Decision to bomb based on threat scoring system (1, very secure, to 5, very insecure)
- Final score based on average of sub-scores, then rounded to whole number so:
- 4.4999 –> 4, whereas 5.0001 -> 5 (arbitrary cutoff)
- Paper shows long-term deleterious effects to bombing
How do we know we have a natural experiment?
In a true experiment, the treatment and control group should look similar or balanced demographically
i.e., if I put people into two groups at random, it should be very unlikely that the average person in the control is 22 years old and the average in treatment group is 55 years old
One thing we can do is compare the characteristics of the treatment and control group in our natural experiment; are they in fact similar?
Why some evidence?
If checking for balance is a way to see if we have a natural experiment, why can’t we always do this?
Want to know the effect of X on Y; test for balance on X; if balanced we are good to go / not confounded
Remember the problem is often we don’t know what confounds are out there! Or we know, but can’t measure them!
Checking for balance is suggestive, weak evidence that we have a natural experiment
No free lunch
Natural experiments are great but there are limitations:
Changing estimate If we find a good one, the thing we are trying to estimate likely changes (is this bad? depends)
Sorting at the cutoff If there’s a rule out there that has consequences, people will do what they can to end up on the right side of it
Limitations: changing estimates
By focusing on units near cutoff, our estimate subtly changes:
effect of putting a child in gifted on future earnings, on averageeffect of putting a child with roughly 75 IQ in gifted on earnings ✅
Is this a problem? Depends, but better to know one narrow(er) fact that nothing at all?
Sorting around the cut-off
If parents can force almost-there kids into gifted, parents become a backdoor path to earnings (since your parents likely affect your earnings directly, or in other ways)
