Natural Experiments

BIBLIOGRAPHY

Most empirical tests in the social sciences are motivated by the desire to estimate a potentially causal effect of an independent variable on a dependent outcome. For example, one might want to know whether a policy intervention, such as a training program for unemployed individuals or an additional year of schooling, has an effect on the outcome of interest, such as earnings or the likelihood of finding a job. To make any causal inference, a researcher must compare the outcome of individuals in a treatment condition with the outcome of individuals in a control condition. The two groups should differ only in that the former group has been subjected to an intervention and the latter has not. This is particularly important if one is concerned that a third, omitted factor may influence the outcome, or that the treatment group is a special selection of people.

Controlled experiments in which people are randomly assigned by the researcher to be part of a treatment condition are rarely possible in the social sciences. Natural experiments represent a second-best way to make causal inferences. A natural experiment does not rely on randomization initiated and controlled by the researcher; instead, it uses observational data involving transparent, naturally occurring random or pseudo-random variations in the treatment and control condition assignments. The treatment group and the control sample should be equivalent before the treatment. The randomization can then take care of any confounding factors (e.g., omitted variables or selection issues). For the natural experiment to be valid, the assignment must be uncorrelated with the measured outcome.

Previous studies have relied on various types of natural experiments. One group of studies uses random changes in public policy. These studies might analyze, for example, how the introduction of a higher minimum wage affects state-level employment, by comparing employment in one U.S. state that has raised the minimum wage with employment in a neighboring state that has not and is (assumed to be) otherwise equal. A second group of studies uses random biological or climate-related events, such as birth dates or weather conditions. These natural experiments are appealing, because the assignments are more plausibly seen as being uncorrelated with the outcomes to be explained than are the assignments in the first group. A third group uses naturally occurring random assignments to treatment and control groups. For example, a lottery might determine—randomly— whether people receive a sudden boost to their disposable income, are eligible to participate in a program, or are required to enter military service. The outcomes of individuals who “win” (treatment condition) can then be compared with those of individuals who participate but do not win (control condition).

Though controlled experiments have a long tradition in medicine and psychology, in the late twentieth century natural experiments became more prominent in the social sciences as a means of increasing the internal validity of empirical research. Indeed, the use of natural experiments has been extended to empirically test hypotheses where causality may be initially confusing. For example, it is not clear whether additional education leads to higher earnings or whether individuals who will earn a lot in the future seek additional education for other reasons. Natural experiments can disentangle these two lines of causality. One could measure the effect of schooling on earnings more generally by using the variation in years of schooling that is not directly correlated with earning but comes from the variation in the natural experiment, for example the date of birth or winning of a lottery.

Natural experiments, however, have limitations with respect to both internal and external validity. In experiments driven by policy changes, for example, the randomness of the intervention is often questionable, but hard to test. Changes in policy may be driven by political factors associated with the outcome. If this is the case, the change in policy is not independent of the outcome and cannot be assumed to be exogenous. What is labeled a natural experiment may then yield results no closer to true causal inference than would a simple observational study. Further, the source of the natural experiment may make it difficult to judge its external validity. The winning of a lottery to participate in a particular program may affect both the losers and the winners. The former may substitute another activity if they fail to win. This would bias the estimated effect of the treatment, even though assignment had been random. Nevertheless, natural experiments are still the best, and often the only, method available to obtain causal inferences in empirical studies in the social sciences when the researcher cannot control treatment assignment.

SEE ALSO Case Method; Experiments; Social Experiment

BIBLIOGRAPHY

Campbell, Donald T., and Julian C. Stanley. 1963. Experimental and Quasi-Experimental Designs for Research. Boston: Houghton Mifflin.

Meyer, Bruce D. 1995. Natural and Quasi-Experiments in Economics. Journal of Business and Economic Statistics 13 (2): 151–162.

Rosenzweig, Mark R., and Kenneth I. Wolpin. 2000. Natural “Natural Experiments” in Economics. Journal of Economic Literature 38 (4): 827–874.

Winship, Christopher, and Stephen L. Morgan. 1999. The Estimation of Causal Effects from Observational Data. Annual Review of Sociology 25: 659–706.

Stephan Meier