Fresh Insights on Measuring Research Use: Policymaker Perspectives on How Theory Falls Short

Social science has done well in providing empirical studies that depict how research is used in policymaking. Yet it has perfomed less well in another contribution science can make—developing explanatory theoretical frameworks that predict and promote future research use (Ness, 2010). For those who conduct theory-driven research studies, Carol Weiss (1999) raised a provocative question two decades ago that remains relevant today: Have researchers been groping around in the dark looking in the wrong places for the wrong purposes for the wrong reasons? Weiss claimed that some research contributions are not “easy to see” and may “not be visible to the naked eye” (Weiss, 1986, p. 217).

Existing studies have focused disproportionately on the supply side of research utilization—the conduct and communication of research to policymakers—with far less attention to the demand side—the uses to which policymakers put research (Gamoran, 2018; Tseng, 2012). To shed light on the demand side, Elizabeth Day, Emily Parrott, and I revisited existing theory by turning to policymakers themselves for their perspectives on how research contributes to policymaking. We compared how policymakers said they used research with the predictions of four prominent theories of research utilization. What we learned is that Weiss’s words may have been prophetic: If researchers rely only on existing theories to measure research use, they may be missing what policymakers see as important contributions to their decisions and to the policy process.

Turning to Policymakers

Between 2015 and 2017, we went inside the Statehouse and conducted semi-structured, face-to-face interviews with 123 legislators in Indiana and Wisconsin (60% response rate), 32 legislators nominated by colleagues as exemplar research users (84% response rate), and 13 key informants including governors, former legislators nominated as statesmen and stateswomen, nonpartisan legislative analysts, and lobbyists (100% response rate). We compared their reports of research use to the predictions of the following four theories, which are representative of four categories of theories that examine the interface between researchers and policymakers:

• The politico-administrative decision making model that moves away from an analysis of decision-making to an analysis of concrete action (Albaeck, 1995);
• A typology of the purposes of research use, specifically instrumental, conceptual, and tactical uses (Nutley, Walter, & Davies, 2007, Tseng, 2012; Weiss, 1999);
• Community dissonance theory that attributes the underutilization of research in policymaking and the lack of communication and trust between researchers and policymakers to a cultural disconnect between the communities of research and policy (Bogenschneider & Corbett, 2010; Bogenschneider, Corbett, & Parrott, 2019; and
• The agenda setting/multiple streams model that examines the role of research in problems, policies, and politics (Kingdom (2011).

What we Found

In some respects, policymakers’ responses were well-aligned with theoretical predictions about how research is used in policymaking, including the ways that research contributes to: 1) individual considerations of policymakers such as informing, explaining, and justifying their positions, and 2) policy issue considerations such as defining issues and designing legislation. Yet theories gave short shrift to four research contributions that were seldom or never predicted by theory:

1. Earning the trust of colleagues as a knowledgeable and credible information source was absent from theory, but mentioned frequently by policymakers as a contribution research makes to policymaking. Policymakers explained that research use runs on relationships. That is, research use influences relationships and relationships influence research use. Policymakers with a good grasp of research earn the respect and trust of colleagues as knowledgeable and credible information sources. Once policymakers have earned this reputation among colleagues, it parlays into trust and confidence in the research they use to advance their arguments into policy debate.

2. Educating others was not predicted by any theory, but was mentioned frequently by policymakers as a contribution of research. Policymakers reported using research for the purpose of educating their colleagues, constituents, and the press. Policymakers were explicit in distinguishing education from persuasion, yet noted a connection between the two. One policymaker described a two-step process, explaining that you need to educate your colleagues so that they understand the issue before you can persuade them to your point of view; sometimes if you do a good job of education and have a good relationship with a colleague who trusts you as an information source, they will accept your explanation and you won’t need to persuade them.

3. Asking important questions for policy or political purposes did not appear in any theory, but was mentioned frequently by policymakers. Policymakers explained that having a good grasp of research prepares them for asking questions, an important skill in the policy culture. Policymakers ask questions for policy purposes such as gaining a deeper understanding of the problem and of viable responses. Questions also can be asked for political purposes such as for casting aspersions on the credibility of opponent’s research or for creating “gotcha” moments that expose an opponent’s obscure motives or hidden agendas.

4. Improving the decision making process by enhancing debate, dialogue, collaboration, and compromise was mentioned in only one theory, but was mentioned frequently by policymakers. Research contributes to policymaking by moving issues away from political posturing and toward a problem-solving exercise. Policymakers gave several examples of how research can be used to help reach compromise in the politically charged policy process.

Rethinking Theory and Considering the Implications

Given that the predictions of research utilization theories fall short, what are the implications for future studies intended to measure research use in policymaking? Three sets of questions are mentioned here, and others are discussed in our article, “Revisiting Theory on Research Use: Turning to Policymakers for Fresh Insights.”

1. If studies are conceptualized using existing theory, could researchers be looking for research use in the wrong places? Could studies be focusing too much on the same-old, circumscribed uses of research, such as justifying pre-existing positions, rather than on fresh, inquisitive uses such as asking questions and educating others?

2. Have researchers been looking for research contributions for the wrong purposes? Has studies been slanted more to how research informs the end product—a specific policy or law—and less on its contributions to the policy process itself? Could understanding of research use be enhanced if studies incorporated the contributions of research to civil, problem-focused dialogue and to negotiation strategies that can enhance collaboration and compromise?

3. Have researchers been looking for research contributions for the wrong reasons? Has research too often been conceptualized as an influence on individual decision making rather that looking for its role in a relationship-driven culture?

Sitting across from dozens of state policymakers for extended interviews provided insights not easily visible to the naked eye. I came away from these interviews with a deeper respect for how hard a policymaker’s job is and a deeper appreciation for the many contributions research makes to policymaking, including many that I did not anticipate. Turning to policymakers can give us insights into the policy world—its inhabitants, institutions, and culture. And incorporating these insights into our measures of research use provides a deeper and more discerning understanding of the multi-dimensional ways that policymakers use research. Empirical understanding of the “how” of policymakers’ research use can inform our theoretical explanations of the “why”. Understanding the “why” is a manifestation of the mystique of science—its power to predict—on an important pursuit: promoting the use of research in policymaking (Pinker, 2018).

The full article, “Revisiting Theory on Research Use: Turning to Policymakers for Fresh Insights,” can be found in the October issue of the American Psychologist, 74 (7), 778-793.

References