Abstract
Many youth development programs aim to improve youth outcomes by raising the quality of social interactions occurring in groups such as classrooms, athletic teams, therapy groups, after-school programs, or recreation centers. As a result, evaluators are increasingly interested in determining whether such programs significantly improve “group quality.” We consider methods for studying the reliability of measures of group quality, with implications for the design of evaluation studies, and we illustrate these methods using a large-scale data set on classroom observations. Our approach enables the analyst to compare options for improving reliability, including increasing the number of raters per classroom, increasing the number or length of occasions of measurement, or improving the training of raters. These inferences depend on model assumptions, and we develop and illustrate a method for testing the sensitivity of these inferences to errors of model misspecification. We then consider the implications of such investments for the statistical power of experiments that assess the impact of intervention on group quality. Our six-step approach extends generalizability theory and uses it to improve research on environments in which youth develop.
