APT Validation Study II: Improving rater reliability

Though there is widespread interest in after-school programs, it is difficult to effectively and reliably assess the quality of after-school practices. This study will focus on the Afterschool Program Practices Tool (APT) and the influence of two persistent sources of variation in ratings—different raters and different occurrences (e.g., when the site is observed). The APT is a widely used measure of after-school program quality and the Foundation previously funded Tracy to improve the instrument in ways that allow it to be scored more consistently. This is the next step in assuring that users of the APT can produce accurate scores. The investigators will examine eight after-school and extended learning programs that represent different age levels, quality, activities, and racial/ethnic composition. In addition, 32 raters will participate in the training and testing component of the study. In the first year, the eight programs will be recorded to establish an archive of training videos. Each site will be observed on four separate days by four expert raters. The project will then use information in the video clips to examine different strategies to improve rater reliability. The investigators will create a bank of expert-rated items and exemplars of behaviors and setting qualities. In the second year, the expertly coded items, exemplars, and ambiguous situations will be used to interactively train raters. Following a series of rating tests, trainees will complete self-directed, web-based exercises with feedback and complete a final test. In total, each trainee will complete four tests, and each test will include clips from two days at a specific program. The design will inform decisions about the optimal number of observation days needed to yield reliable ratings and the usefulness of different and sequential training strategies.