December 5, 2023
The causal revolution
Some studies don’t even control for confounds
From the dog ownership improves health article in the homework:
Huge confounds abound! A person who is too sick to care for a dog won’t get one
IF we can correctly specify our model (i.e., control for all backdoors, leave front-doors open) we will identify the effect of X on Y
Easier said than done! Often there is some confound (like W) that we know exists but can’t measure, or worse, don’t know exists at all
Hard to correctly specify the DAG and measure all the variables; what can we do?
Ideal = an experiment – where treatment is randomly assigned – but this is rarely feasible
Alternative - find moments, situations, or weird, freak occurrences where some group is exposed to some treatment by chance while another group was not
These weird moments in history are called natural experiments
As though nature accidentally created an experiment
2021 Nobel Prize winners in economics pioneered quasi-experimental methods – using research designs that leverage natural experiments
Go out and collect data on people who were and were not in gifted:
Big gifted program premium: $24,645.72 more in earnings
So many potential confounds, including ones that are really, really hard to measure (like ability) and others that are unknown
To get into gifted you typically need a certain score on an aptitude/IQ test
Imagine that the cutoff score is 75/100; above you’re in gifted, below you are not
This cutoff is known as the assignment mechanism = why some receive treatment (gifted program) or not
How was that cutoff chosen?
The precise cutoff is completely arbitrary; could have been 74.5, or 76, or 80, or whatever
The arbitrary nature of the cutoff creates a natural experiment
A student who a 76 and a student who got a 34 are probably very different;
if one makes more money than the other today, is that because of gifted? or other factors (ability?)
But a student who got a 76 and one who got a 74 are probably interchangeable
Few points difference might be function of: random error, luck, whatever
If we look only at students who scored close to the 75 cutoff we have the makings of a natural experiment
Around cutoff, essentially random whether you get treatment (gifted program) or not
Why? Score on test = ability + luck + measurement error
Example: Joe got a 74 and Amy got a 76; if they took the test many times, how often would Joe score worse than Amy?
“Natural experiments”: situations where some thing (often a rule) creates random assignment of treatment
In analysis terms, (can be) simple: look only at students who scored close to 75, let’s say +/- 2:
Ouch! Looks like being in gifted actually hurts students in the long-run:
|students near cutoff
|*** p < 0.001; ** p < 0.01; * p < 0.05.
People with higher aptitude/training/ability/etc. better off today:
These are the students who, by chance, just barely made it (or not) into gifted
On average, the students who by chance made it in make less money
In the naive estimate we think gifted programs help students in the long-run;
With the discontinuity we see the opposite: gifted hurts
Arbitrary cutoff gets us clean estimates for free: no need to worry about backdoor paths
How much moral hazard is there in insurance? E.g., if your insurance covers 10k in services, will provider charge you $10k even if you only need 8k in services?
lm(charge ~ insurance)
What’s wrong with the naive estimate?
Insurance will cover two nights in a hospital after delivery
But how long is “a night”?
Nights counted by number of midnights (arbitrary cutoff)
👶 born on Monday at 11:59pm = gets Tue + Wed in hospital
👶 born on Tuesday at 12:01am = gets Tue + Wed + Thur in hospital (by chance, get extra minimum coverage)
1 in 4 babies born after midnight spend an extra night in hospital (💵💵💵), even when there is no health benefit to doing so
What effect did bombing raids in Vietnam have long-term?
Problem: the places that were bombed are very different from the places that were never bombed
Biggest confound: if bombing accurate, places that were most bombed had most Vietcong presence
In a true experiment, the treatment and control group should look similar or balanced demographically
i.e., if I put people into two groups at random, it should be very unlikely that the average person in the control is 22 years old and the average in treatment group is 55 years old
One thing we can do is compare the characteristics of the treatment and control group in our natural experiment; are they in fact similar?
It looks like people close to the cutoff are pretty balanced
|Percent drug use
Some evidence we have a natural experiment
If checking for balance is a way to see if we have a natural experiment, why can’t we always do this?
Want to know the effect of X on Y; test for balance on X; if balanced we are good to go / not confounded
Remember the problem is often we don’t know what confounds are out there! Or we know, but can’t measure them!
Checking for balance is suggestive, weak evidence that we have a natural experiment
Natural experiments are “strongest” causal tool, and regression discontinuity designs are probably the most convincing version
Easy to implement; less worry about confounding; the tough part? Finding one! And collecting the data
Often the result of some rule, boundary, or some other random, arbitrary process having a sudden impact in the world
Premium on: specific knowledge of a particular region/area/issue
With the person to your right (or whatever):
Write down as many rules, demarcations, cutoffs, etc., out there in the world that you can think of.
Can any of these create an opportunity for a natural experiment, where something happens to some people/countries/places but not to others as a result of this rule/cutoff/thing?
What problem could that natural experiment help solve? In the gifted program case: the IQ test cutoff helps solve the problem of estimating the effect of gifted programs on future success.
Post a summary in the Slack and if the idea is good enough I will steal it and publish it.
Natural experiments are great but there are limitations:
Changing estimate If we find a good one, the thing we are trying to estimate likely changes (is this bad? depends)
Sorting at the cutoff If there’s a rule out there that has consequences, people will do what they can to end up on the right side of it
By focusing on units near cutoff, our estimate subtly changes:
effect of putting a child in gifted on future earnings, on average
effect of putting a child with roughly 75 IQ in gifted on earnings ✅
Is this a problem? Depends, but better to know one narrow(er) fact that nothing at all?
There is also a risk that if people know about the cutoff they will behave strategically in ways that confound our estimate
Imagine that there is room to pressure/bully the school psychologist into turning a 74.9 into a 75 so little Timmy can be gifted
And image wealthier/better-connected/etc parents are better bullies than poor parents
In that case, being gifted is no longer random
wealthy parents cause you to get over the cutoff
If parents can force almost-there kids into gifted, parents become a backdoor path to earnings (since your parents likely affect your earnings directly, or in other ways)
Think back to the arbitrary cutoff you came up with in the last exercise. Discuss again with your neighbor: