Raudenbush and Bloom outline key features of an ambitious project project that will bring together prominent university-based methodologists and the three research firms (MDRC, Mathematica Policy Research, and Abt Associates, Inc.) that have conducted the most large multi-site trials in education, youth development, and related fields. This paper describes the project's statistical foundation, and identifies its anticipated benefits. The project is organized in two parts: 1) developing and applying methods for learning about impact variation and 2) developing and applying methods for learning from impact variation.
This MDRC publication provides practical guidance for researchers who are designing and analyzing studies that randomize schools—which comprise three levels of clustering (students in classrooms in schools)—to measure intervention effects on student academic outcomes when information on the middle level (classrooms) is missing.
"Evaluations of public programs in many fields reveal that (1) different types of programs (or different versions of the same program) vary in their effectiveness, (2) a program that is effective for one group of people might not be effective for other groups of people, and (3) a program that is effective in one set of circumstances may not be effective in other circumstances. This paper presents a conceptual framework for research on such variation in program effects and the sources of this variation. The framework is intended to help researchers — both those who focus mainly on studying program implementation and those who focus mainly on estimating program effects — see how their respective pieces fit together in a way that helps to identify factors that explain variation in program effects and thereby support more systematic data collection on these factors. The ultimate goal of the framework is to enable researchers to offer better guidance to policymakers and program operators on the conditions and practices that are associated with larger and more positive effects."
Many youth development programs aim to improve youth outcomes by raising the quality of social interactions occurring in groups such as classrooms, athletic teams, therapy groups, after-school programs, or recreation centers. As a result, evaluators are increasingly interested in determining whether such programs significantly improve “group quality.” This paper consider methods for studying the reliability of measures of group quality, with implications for the design of evaluation studies, and illustrates these methods using a large-scale data set on classroom observations.
“This revised working paper examines strategies for interpreting and reporting estimates of intervention effects for subgroups of a study sample. The paper considers why and how subgroup findings are important for applied research, alternative ways to define subgroups, different research questions that motivate subgroup analyses, and the importance of prespecifying subgroups before analyses are conducted. […]