The Foundation supports a range of quantitative, qualitative, and mixed-methods research. In past years, we sought to build the capacity of social scientists, foundation leaders, and public officials to carry out credible evaluation studies of group-based interventions. Resources developed during this time focus on planning and analyzing this research.
Recently Added Resources
-
RCT-YES
RCT-YES is a free software tool that allows you to easily analyze data and report results on the effectiveness of programs in your context.
Research MethodsTools -
Harnessing Discovery: Writing a Strong Mixed-Methods Proposal
For researchers and their teams, it’s important to appreciate that when seeking to incorporate mixed methods, the forms and challenges vary from project to project.
Research Methods -
Identifying Responses to Inequality: The Potential of Qualitative and Mixed-Methods Research
We believe that qualitative and mixed-methods research is essential to building, understanding, testing, and improving responses to inequality—be they programs, policies, or practices.
Reducing InequalityDirections for Research: Reducing Inequality -
It’s No Longer All about the Mean: Using Multi-site Trials to Learn About and From Impact Variation
Raudenbush and Bloom outline key features of an ambitious project project that will bring together prominent university-based methodologists and the three research firms (MDRC, Mathematica Policy Research, and Abt Associates, Inc.) that have conducted the most large multi-site trials in education, youth development, and related fields. This paper describes the project's statistical foundation, and identifies its anticipated benefits. The project is organized in two parts: 1) developing and applying methods for learning about impact variation and 2) developing and applying methods for learning from impact variation.
Research MethodsAnalyzing Multilevel Trials -
Designing and Analyzing Studies That Randomize Schools to Estimate Intervention Effects on Student Academic Outcomes without Classroom-Level Information
This MDRC publication provides practical guidance for researchers who are designing and analyzing studies that randomize schools—which comprise three levels of clustering (students in classrooms in schools)—to measure intervention effects on student academic outcomes when information on the middle level (classrooms) is missing.
Research MethodsEducation -
Detecting Intervention Effects Across Context: An Examination of the Power of Cluster Randomized Trials
This abstract discusses how to calculate power for three types of treatment heterogeneity including 1) the variability in treatment effects across sites, 2) site-specific treatment effects, and 3) moderator effects at the cluster or student level. The sample includes the studies in the first wave of CRTs funded by IES, or those funded between 2002 and 2006 by NCER and NCEE. These studies represent a range of CRTs on various topics and with different research designs and sample sizes.
Research MethodsGroup Randomized Trials -
Improving the Quality of Classroom Interactions
The Foundation has funded several innovative classroom interventions designed to alter settings in ways that result in positive outcomes for the youth within them. In order to change a setting for the better, we need to first understand how that setting works. To that end, we have developed a theoretical framework to guide our intervention research and funded a number of studies on the development of reliable and valid measures of classroom processes. Strong theory and good measures are critical components in the development of a thorough understanding of settings.
Research MethodsEducation -
An Empirical Investigation of Design Parameters for Planning Cluster Randomized Trials of Science Achievement
This article presents empirical estimates of design parameters that can be used to appropriately power CRTs in science education and compares them to estimates using mathematics and reading.
Research MethodsEducation -
Confidence Bounds and Power for the Reliability of Observational Measures on the Quality of a Social Setting
"Social scientists are frequently interested in assessing the qualities of social settings such as classrooms, schools, neighborhoods, or day care centers. The most common procedure requires observers to rate social interactions within these settings on multiple items and then to combine the item responses to obtain a summary measure of setting quality. A key aspect of the quality of such a summary measure is its reliability. In this paper we derive a confidence interval for reliability, a test for the hypothesis that the reliability meets a minimum standard, and the power of this test against alternative hypotheses. Next, we consider the problem of using data from a preliminary field study of the measurement procedure to inform the design of a later study that will test substantive hypotheses about the correlates of setting quality."
Research Methods -
Intraclass Correlation Values for Planning Group Randomized Trials in Education
Experiments that assign intact groups to treatment conditions are increasingly common in social research. In educational research, the groups assigned are often schools. The design of group randomized experiments requires knowledge of the intraclass correlation structure to compute statistical power and sample sizes required to achieve adequate power. This paper provides a compilation of intraclass correlation values of academic achievement and related covariate effects that could be used for planning group randomized experiments in education. It also provides variance component information that is useful in planning experiments involving covariates, and illustrates the use of these values to compute statistical power of group randomized experiments.
Research MethodsGroup Randomized Trials -
A Conceptual Framework for Studying the Sources of Variation in Program Effects
"Evaluations of public programs in many fields reveal that (1) different types of programs (or different versions of the same program) vary in their effectiveness, (2) a program that is effective for one group of people might not be effective for other groups of people, and (3) a program that is effective in one set of circumstances may not be effective in other circumstances. This paper presents a conceptual framework for research on such variation in program effects and the sources of this variation. The framework is intended to help researchers — both those who focus mainly on studying program implementation and those who focus mainly on estimating program effects — see how their respective pieces fit together in a way that helps to identify factors that explain variation in program effects and thereby support more systematic data collection on these factors. The ultimate goal of the framework is to enable researchers to offer better guidance to policymakers and program operators on the conditions and practices that are associated with larger and more positive effects."
Research MethodsAnalyzing Multilevel Trials -
MET Database at the Institute for Social Research
This site enables users to apply for access to quantitative data and classroom videos created by the Measures of Effective Teaching (MET) project, funded by the Bill & Melinda Gates Foundation.
Research MethodsGroup Randomized Trials -
Studying the Reliability of Group-Level Measures with Implications for Statistical Power: A Six-Step Paradigm
Many youth development programs aim to improve youth outcomes by raising the quality of social interactions occurring in groups such as classrooms, athletic teams, therapy groups, after-school programs, or recreation centers. As a result, evaluators are increasingly interested in determining whether such programs significantly improve “group quality.” This paper consider methods for studying the reliability of measures of group quality, with implications for the design of evaluation studies, and illustrates these methods using a large-scale data set on classroom observations.
Research MethodsTools -
Best Practices in Conceptualizing and Measuring Social Class in Psychological Research
“An extensive body of research has documented the relation between social class, as indexed by socioeconomic status (SES) and subjective social status (SSS), and a host of outcomes, including physical and mental health, academic achievement, and educational attainment. Yet, there remains ambiguity regarding how best to conceptualize and measure social class. This article clarifies definitional […]
Reducing InequalityDirections for Research: Reducing Inequality -
Optimal Design with Empirical Information (OD+)
This software includes a series of empirical estimates of plausible parameter values for determining the minimum effect size that can be detected by a given number, size, and treatment/group mix of randomized groups.
Research MethodsGroup Randomized Trials
- 1
- 2
Categories
- Reducing Inequality (14)
- Research Methods (16)
- Use of Research Evidence (27)
- Youth Social Settings (9)
Tags
- After-school Programs (4)
- Analyzing Multilevel Trials (3)
- Child Mental Health (3)
- Child Welfare (5)
- Directions for Research: Reducing Inequality (10)
- Directions for Research: the Use of Research Evidence (4)
- Education (20)
- Evidence-Based Policy (9)
- Group Randomized Trials (10)
- Higher Education (1)
- Immigration (1)
- Justice System (1)
- Research-Practice Partnerships (8)
- Tools (7)