In my last post I tried to provide a decent intuitive discussion (with a little technical flavor) of an important empirical strategy often used in policy analysis: Difference-in-Differences estimation. Here I want to talk about a phenomenon important to public policy analysis: the natural experiment. D-i-D and natural experiments are related in the sense that D-i-D is the estimation technique most often used within a natural experiment.

Before we go any further here are some seminal papers related to topics that will come up in this post (I’ve tried to link directly to public .pdf versions wherever possible…but if you want access to one of the ones behind a journal wall shoot me an email and I can download a copy for you):

  • David Card’s 1980 paper using the Mariel Cuban Boatlift as a natural experiment to assess the impact of immigration on employment and wages.
  • Ashenfelter and Card (1985). They evaluate the effect of the Comprehensive Employment and Training Act that offered job skills training to low wage and unemployed workers with the intent of boosting long-run earnings.
  • James Heckman’s paper Shadow Prices, Market Wages, and Labour Supply. Generally considered one of the seminal papers on sample selection bias. Heckman noted that previous studies on hours worked and prevailing wage among married women in the workforce may were likely biased because they did not control for individual characteristics that may be correlated with a women’s decision to enter the labor force in the first place.
  • Josh Angrist (1990) used the Vietnam Era draft lottery as a quasi natural experiment to assess the effect of military service on civilian earnings.
  • David Card (1996) The Effect of Unions on the Structure of Wages: A Longitudinal Analysis.

What is a Natural Experiment?

A good way to start here is to think about what is NOT a natural experiment. Suppose we want to know whether a new fertilizer helps cotton grow faster in a certain area. We could plant a huge field with cotton, apply the fertilizer to some areas of the field but not others, expose the whole field to the same growing conditions (minus the special fertilizer), and see if the fertilizer cotton grows better/faster/whatever. Micro-Economists who deal with minimum wage policy or the impact of pollution can’t really do this for obvious ethical reasons: we can’t randomly select one group of people to make $5/hour and another to make $6/hour for the same work…and we can’t really intentionally expose one area to a shit-load of pollution while keeping a control group pristine.

Rather than controlled studies, economists use observational studies where we might observe

  • some individuals/states experienced a change in wage rates while another group was not exposed to this change.
  • some firms are subjected to a new type of environmental regulation while others are not
  • some individuals get a tax break that others don’t get (EITC)

and we’d like to make some inferences about the effects of these policy changes on behaviors.

A problem with observational studies arises when some units have a non-random probability of being included with the treatment group versus the control group.

A sample selection bias example

Suppose we are interested in the effect a job retraining program (like Ashenfelter and Card (1978)) aimed at boosting the earnings of low skilled or unemployed workers. More specifically, we want to know if this program leads to a long-run increase in earnings for participants. Suppose that the program is free but enrollment is voluntary. We observe one group of treated individuals (those enrolled in the program) and one group of non-treated individuals. The problem here is that the decision to enroll in the program may be correlated with unobserved features of the individuals that themselves are correlated with earning potential. In this case we won’t know if any earnings increase we might observe among the treatment group is the result of the program or just determined by the fact that workers with greater earning potential enrolled in the program.

A natural experiment solves the problem outlined above where some units might have a non-random probability of receiving the treatment. Basically, a natural experiment exposes some individuals (or clusters) to a treatment through force of nature (rather than by design of an experimenter).

A natural experiment example

In 1980 David Card published an awesome paper which analyzed the impact of immigration on wages in a local labor market. We’ve all probably heard the concerns in some forum or other that immigration pushes wages down because immigrants are generally willing to work cheaper than their native counterparts.

The problem with putting this claim to the empirical test by just comparing wages, or wage changes, in areas with high immigrant populations versus areas with low immigrant populations is that the decision to immigrate to a particular area might be a function of wages in that area…so if we just observe two areas (one with lots of immigrants and one with relatively few) and compare wages across the two, there is no guarantee we’d be measuring the ‘immigrant’ effect. The idea of a natural experiment says: it would be great if we could observe an immigration event where the people in question didn’t have a choice of where to locate (an alternative to this is a controlled experiment where you assign groups to different treatments…which, for obvious reasons, would be a pretty shitty thing to do with most life-altering processes that economists are interested in).

Card noted that we have observed such a natural experiment: The Mariel-Cuban Boatlift. Basically, Fidel Castro shipped a bunch of people out of Cuba (some were criminals and others were just people with means that want to GTFO Cuba). On April 20, 1980 the city of Miami experienced an immediate 7% increase in the size of it’s labor force from the Marielitos.

Card realized that, using this event, one could compare wages in Miami with wages in surrounding areas (that have similar characteristics minus the big, one-time influx of immigrants) before and after this mass immigration event and get a clearer idea of what the impact of immigration is on a local labor market.

And, since I covered DiD in my last post, it’s worth noting that Card’s analysis was based on difference-in-differences estimation. It’s also worth noting that Card found no statistically relevant decrease in wages resulting from the influx of immigrants:

The Mariel immigrants increased the Miami labor force by 7%, and the
percentage increase in labor supply to less-skilled occupations and
industries was even greater because most of the immigrants were
relatively unskilled. Nevertheless, the Mariel influx appears to have had
virtually no effect on the wages or unemployment rates of less-skilled
workers, even among Cubans who had immigrated earlier.

2 final points

1. Natural experiments are kind of like the Economist’s Holy Grail. They are very powerful phenomenon for estimating the marginal impact of a policy change…they are also exceedingly rare.

In the public policy arena what we encounter much more frequently than pure natural experiments are situations that, for lack of a better term, might be thought of as ‘quasi natural experiments.’ The minimum wage study by Card and Krueger that I talked about in my last post fits into this category. It’s not really a natural experiment in the sense that minimum wage hikes are not really a natural phenomenon – they are a product of legislation. And whether or not a state passes a minimum wage hike is somewhat endogenous in the sense that it is correlated with political environment and ideological make up of that state.

However, when it comes to evaluating the policy change, Card and Krueger’s set-up comes close enough to being a natural experiment. By that I mean, there’s probably not much reason to worry about sample selection bias. Individual restaurants in the study could not choose which group to be in. Treatment or control group assignment was made exogenous to the restaurants – if they were in Pennsylvania they were subject to the same minimum wage throughout the study period…and if they were in NJ they were subject to a change in the minimum wage.

2. If you are interested in the economic/behavioral impact of a policy and you do not have a natural or quasi-natural experiment (you have obvious non-random assignment into the groups) all is not lost. Read up The Heckman Correction for sample selection bias and other methods to correctly estimate marginal effects with non-random group assignment.