Randomized trials are rapidly becoming standard practice for evaluation research and applied social science. Hundreds, if not thousands of such trials have been conducted and many more are on the way. Most large trials involve multiple sites in order to provide the sample sizes that are required and to broaden the generalizability of their findings. But, typically, these trials have only focused on average program impacts and program impacts for common socio-demo subgroups. That is about to change, however, as researchers, policymakers and practitioners are beginning to see the value of learning about and from variation in program impacts across individuals, across theoretically- and policy-relevant subgroups of individuals, and across program sites.
To help promote this path-breaking research agenda, the William T. Grant Foundation, the Spencer Foundation and the U.S. Institute of Education Science are supporting a major project that will bring together prominent university-based methodologists and the three research firms (MDRC, Mathematica Policy Research, and Abt Associates, Inc.) that have conducted the most large multi-site trials in education, youth development, and related fields. This project grows out of a 2013 conference on studying impact variation sponsored by the William T. Grant Foundation and reflects key themes of a 2014 federal conference on unpacking the “black box” of program impacts sponsored by the Office of Policy Research and Evaluation of the U.S. Department of Health and Human Services.
In this working draft, Raudenbush and Bloom outline key features of this project, describe its statistical foundation, and identify its anticipated benefits. The project is organized in two parts: 1) developing and applying methods for learning about impact variation and 2) developing and applying methods for learning from impact variation.
