Conceptual use of research evidence deepens understanding of problems, offers new ideas and frameworks for acting, shifts beliefs, and informs priorities and resource allocation. Because conceptual use unfolds over time and is often indirect and difficult to observe, valid, broadscale assessments of this type of research use are challenging. Survey measures regularly focus on decision makers’ exposure to research, but not whether their thinking and understanding changed after such exposure. Walker and colleagues propose measurement work to address this challenge and focus on proximal indicators of research use and cognitive change. The study will be conducted with a cross-state network of juvenile courts, which are under increasing pressure to incorporate findings from research on adolescent development to inform decisions and service approaches. The study sample will include juvenile justice court leaders, including judges, administrators, midlevel managers, and probation supervisors from 80 jurisdictions in Washington State, Utah, and Los Angeles, California, with an estimated 520 survey respondents. Using a mixed-method approach, the study will be carried out in stages. First, the team will complete a review of relevant instruments and protocols and identify candidate survey items. Next, they will conduct cognitive interviews with ten justice leaders to ensure understanding of items and alignment with conceptual use. Following that, they will administer the survey to justice leaders each month for five months to assess concurrent validity and sensitivity to change. In addition to the survey, the team will assess related items on conceptual use, instrumental research use, symbolic research use, and measures of changes in thinking, exposure to research, and peer centrality. They will examine the associations between the doses of naturally occurring research exposure over the five months and changes in conceptual use of research using the newly developed instrument and related instruments, as well as study the association between an individual’s peer nominated conceptual research use and self-reported conceptual research use. Findings from this work will provide initial evidence about the measure’s internal reliability, construct, concurrent and discriminant validity, sensitivity to change, and feasibility of use.
How can we assess whether research has influenced the thinking and decision making orientations of policymakers and practitioners?
