During the COVID pandemic, policy debates about how to address the crisis have often been portrayed as scientific knowledge in competition with political rhetoric for public attention. Yet even in this unprecedented case, rarely has evidence of only a single type been used in policy arguments. Rather scientific research about virus transmission and vaccines has been combined with clinical findings from the daily work of health care providers, with stories about the effects of COVID-19 on families and businesses, and yes, with strong political ideologies represented by symbols such as decisions about wearing a mask or not.
Combining research and other types of evidence is typical of most policy areas. Understanding the conditions under which it occurs is especially important in areas of ongoing concern such as youth policy where, rather than speedy solutions to crises, efforts to translate research-based interventions into policies may take years to achieve. With their focus on the information needs and incentives of policymakers and the groups that provide such information, those who study policy recognize that the use of research evidence (URE) is best examined as one element in a strategic context, with different types of evidence used selectively at different points to pursue a political or policy objective.
Evidence serves multiple roles in the policy process. It can function as analytical information providing assessments about the scope and causes of identified problems, and about the conditions under which different policy options are likely to produce their intended effects. Different kinds of evidence may also provide political intelligence about the preferences and strategies of major actors, and about who wins and who loses under different policy options. In many cases, strategic use of the latter type of evidence may be more central to a policy outcome than the use of research for analytical purposes.
The what, who, and how of evidence use
In our recently-published book, Evidence, Politics, and Education Policy (Harvard Education Press, 2020), we present an analytical framework, based on the what, who, and how of evidence use in education policymaking, which assumes that the types of evidence used in policymaking depend on the incentives of those supplying and using the evidence, and on the stages of the policy process.
The what of evidence use includes peer-reviewed research, as well as statistical data, judgments based on professional expertise, the personal experience of policymakers and practitioners, and appeals to values articulated through stories and symbols. The who are participants in the policy process—elected officials, their staffs, and the interest groups and policy entrepreneurs who seek to persuade them. The how refers to how the purposes of evidence use vary across the policy process, from initial problem definition to implementation.
Insights from a broader perspective on evidence use
In our study, we examined two cases of youth-related policy: the development and early implementation of the Common Core State Standards (CCSS) and the two-decade legislative history of the Children’s Health Insurance Program (CHIP). By drawing on a broader conception of evidence that includes other types in addition to research, we gained insights that might have been missed in a traditional URE study.
Two examples are illustrative. The first highlights the individuals and groups who supply evidence. Those of us who study evidence use have increasingly recognized the central functions that intermediary organizations play in translating and communicating research results to policymakers and practitioners. Too often, though, differences among intermediaries are glossed over or simplified. In contrast, our framework differentiates among three categories of intermediaries:
- Translators and Disseminators have as their major function the translation and dissemination of research and other evidence. They typically customize it to the information needs of specific users, many of whom are practitioners rather than policymakers.
- Policy advocates are membership organizations that represent group interests in policy arenas, and do so on behalf of their members’ material or ideological interests. They use research strategically as one resource among others such as money and access to the media.
- Hybrids, in contrast to policy advocates with their focus on members’ interests, pursue a policy agenda that is often consistent with a broader ideological perspective (e.g., market reforms or greater equalization of educational resources). We use the term hybrid because these groups are similar to translators and disseminators in that they produce and disseminate research and other evidence as a primary means to promote their goals, but they use research strategically like policy advocates.
Distinguishing among intermediaries in this way directs attention not only to the information they are disseminating, but also to the purpose of their efforts. The varied purposes for which each type of intermediary uses research shape its incentives and the extent to which it adheres to the canons of scientific inquiry. These incentives may include professional recognition for improving education practice, the promotion of economic or ideological group interests, or the enactment of specific policy agendas.
The second example comes from our analysis of CCSS as it was implemented in four states. In recounting the history of the Common Core State Standards, academic and media commentators have emphasized that in several states political opponents mobilized and persuaded policymakers to abandon the CCSS and design alternative standards. However, what this portrayal misses is that once state officials agreed to replace the CCSS with revised state standards, political rhetoric largely ceased and professional judgment became the dominant source of evidence; most opponents backed off and allowed educators to assume responsibility. The revision process became an opportunity for educators to bring their classroom experience to bear in making the standards more accessible and coherent from their perspective, and to provide a stronger sense of local ownership than was possible with the Common Core.
With a more comprehensive conception of evidence, what was often portrayed as highly politicized arguments prevailing over research-based evidence appears quite different. Political rhetoric, with its images of “federal overreach” and “loss of local control,” was dominant at one point in the process. However, once the Common Core was vetoed, its role was limited. The rigor and focus that defined the development of the Common Core continued to influence the design of the new standards, but that evidence was also enhanced with practitioner judgments attuned to the local context.
Digging deeper in URE studies
These examples suggest at least two recommendations for those studying the use of research evidence in education policymaking. (We include others in our book.) The first speaks to how to assess the role of intermediary organizations. In an interview, the leader of a hybrid intermediary argued, “…If you want research to get legs in the policy process, give it to an organization that has a track record of accomplishments in policy.” More and more researchers are heeding this advice, and those studying URE have acknowledged the role of intermediaries as translators and disseminators. We recommend that such studies look more deeply at the intermediary’s purpose for using evidence, their underlying organizational incentives, and how those affect their choice of evidence. This knowledge can help in explaining why and how certain types of evidence are combined in the policy process. It can also assist research producers make more informed choices about collaboration with intermediary organizations based on how different types are likely to use research findings.
Second, we recommend that studies of the use of research evidence include policy analyses in addition to examining research focused on the functioning and effects of educational interventions. To be sure, knowing the school and classroom conditions under which an intervention affects student learning provides critical knowledge. But research is incomplete if analyses of costs and their distribution, professional capacity requirements, sources of variation across local jurisdictions, and levels of political support and opposition are lacking. These types of information are frequently critical pieces of evidence in policymaking, and they are best understood by those trained as policy analysts.
After a long period of dormancy, the study of research evidence use has made remarkable advances over the past decade. Consequently, it is an opportune time to strengthen the validity of URE policy studies by expanding the range of evidence analyzed.
