When the William T. Grant Foundation began exploring research on the use of research evidence in 2007 (Tseng, 2008), the field was in a very different place than it is now. The U.S. was in the throes of an evidence-based policy movement focused on identifying “What Works” and incentivizing practitioners to adopt evidence-based interventions. Those attempts to push evidence into widespread practice had ambitious aims, but, ironically, were not informed by the growing body of research on research use, and subsequently “fell short of expectations” (DuMont, 2019).
At the Foundation, we see the puzzle of how to connect research with policy and practice as a vital area for scientific inquiry. We believe that attempts to improve the use of research evidence in policy and practice should be subject to rigorous theory building and empirical analysis. The stakes are high for the research community: If we do not understand what it takes for research to be used, then research will always stay on the proverbial shelf (or website)—far from the action of policy deliberations and decision-making. Our aspirations for research impact will remain limited to its academic, rather than social, impact (Gamoran, 2018).
In this essay, I reflect on the Foundation’s initiative on the use of research evidence: the progress made by grantees and others in the field and the challenges that remain in building a rigorous field of study that spans the globe. I end with reflections for the future, drawing inspiration from our collaboration with the Pew Charitable Trusts and the Transforming Evidence Funders Network, a collective of public and private funders working to make research more useful, used, and impactful in meeting today’s global challenges—from education to the environment, and from foreign policy to global health.
Building Theory and Empirical Evidence
A robust field of research on research use requires theory and empirical evidence on when, how, and under what conditions research is used and what it takes to improve research use (Tseng, 2008, 2012). Researchers and policymakers both have implicit theories about research use, but those theories are not often made explicit and are too rarely subjected to empirical testing. The research community can do better, simply by bringing the tools of science to bear on getting science used. If we do that, we can become more evidence-based in promoting evidence use.
Descriptive Studies
In the early days, many of the studies we funded focused on documenting what research use looks like and explaining when, how, and under what conditions research is used in policy and practice (DuMont, 2015; Tseng, 2012). These descriptive studies examining the realities of research use were critical, given popular notions of research use as the direct application of findings to a given decision, and research users as autonomous, rational individuals. Back in 1977, Carol Weiss was already critiquing the notion that research use is simply the matter of an individual decision-maker requiring more information to make a decision, finding that information in research, and using the research to make a choice. And although this misguided conceptualization still underlies many research dissemination and communication initiatives, we now have a broad body of evidence showing that research use is in fact much more nuanced.
Studies in recent decades generated descriptive theories of research use. They revealed the importance of conceptualizing research use as a process, not an isolated moment. Because decision-making itself is a process, not an event, understanding research use in decision-making requires grasping the decision-making process and the ways research gets interpreted in light of particular problems, contexts, and value orientations (Asen et al., 2013; Coburn, Honig & Stein, 2008; McDonnell & Weatherford, 2013; Weiss, 1977; Yanovitzky & Weber, 2018). Conceptualizing research use as a process also calls attention to the question of how to promote the routine use of research rather than aiming to create an instance of research use. As theory matured, our attention shifted to understanding ways to support agencies and other decision-making bodies in building their capacity to routinely draw upon research in their daily, weekly, and yearly work and as part of the flow of decision-making.
In the early days, many of the studies we funded focused on documenting what research use looks like and explaining when, how, and under what conditions research is used in policy and practice
Even what seems like a decision to adopt an evidence-based program is one piece of a long chain of decision-making that can begin with the identification of a problem to be solved and an assessment of resources, followed by a slew of decisions about implementation (Cohen, March, & Olson, 1972; Palinkas et al., 2009, 2018). Moreover, the decision-making process is rarely linear and sequential: It can start with the availability of funding and then backtrack to identify problems that need solving. Decisions can be made, then stall, then restart, then pause again. Taken in this light, focusing on an isolated decision-making moment is foolhardy.
Descriptive studies also revealed remarkable consistency in the mechanisms and conditions that support research use, showing that research use is not just about individuals, but about people embedded in complex social, organizational, and political contexts (Bogenschneider & Corbett, 2021; Finnigan et al., 2013; Mosley & Gibson, 2017; Palinkas et al., 2013; Scott et al., 2017; Tseng, 2012). Studies across governing bodies as diverse as state legislatures, child welfare agencies, school district central offices, and juvenile justice courts indicated that research use is a social process, heavily influenced by relationships, trust, social networks, and politics. Spurred in part by these advances, attention in the field broadened from the individual-level attributes of research users to the organizational capacity, institutional norms, leadership roles, and political conditions that support the use of research evidence (Boaz, Davies, Fraser, & Nutley, 2019; Parkhurst, 2017).
And despite an abundance of attention to instrumental uses of research (to make decisions), studies show that research is used conceptually (to understand problems and potential solutions) as often, if not more so, than instrumentally (Coburn, Toure, & Yamashita, 2009; Penuel et al., 2017; Yanovitzky & Weber, 2018). Karen Bogenschneider’s (2019) research, for instance, has elucidated the ways research supports policymakers’ relational goals. In a study of state legislatures, Bogenschneider describes legislative work as an inherently social process, wherein research use is about persuading others, educating constituents, building trust with colleagues, and improving the quality of policy dialogue. Under this conceptualization of research use, research has both informational value and relational value.
Intervention Studies
If our goal is to improve research use, knowing what influences research use is only the tip of the iceberg. Thus in 2016, we pivoted our funding focus from understanding research use to improving it (William T. Grant Foundation, 2015).
In the cumulative fashion of knowledge-building, studies to improve research use must build on what we know from descriptive studies about how research is used, who uses it, and what conditions influence its use. Without that grounding, interventions risk aiming at the wrong targets for change or neglecting the contextual conditions necessary to enhance research use. For example, we sometimes receive proposals that seek to improve research use in agencies but are not grounded in an understanding of who makes decisions within those agencies nor how and when decisions are made. Such proposals come across as ill-fitting to the realities of decision-making and thus unlikely to achieve impact.
Moving forward, we see a need for research and development activities to design robust interventions that can move the needle on research use. We are mindful that—although there are notable success stories—many tests of research use interventions have yielded null effects, often because the interventions were not sufficiently intensive to address the complexity of decision-making and the forces that maintain status-quo ways of working (Sharples, 2017; Dobbins et al., 2009). Creating meaningful change in any context is challenging, and the same is true when it comes to improving research use.
Studying improvement requires intervention theory. I use the term intervention broadly here to include any policy, program, or other intentional effort to create change in the status quo. I define intervention theory as an explanatory framework for what it takes to bring about that change. The theory part is important because even if a study focuses on one intervention, theory situates the intervention strategies within a generalizable explanatory framework that is relevant to other efforts. Intervention theories are available in many areas, but they are too seldom brought to the table in initiatives to improve research use. For example, we know from decades of professional development research in education that one-time workshops bear little hope of changing practice (Desimone et al., 2002), yet many attempts to promote research use consist of one-off workshops or trainings for practitioners. In another common example, when staff training on research use is not grounded in the day-to-day realities of staff work, it is challenging—if not impossible—for staff to turn any newfound knowledge into routine practice. Studies to improve research use need to harness what we know about effective professional development and training, as well as organizational and systems change, continuous improvement, and influencing public policy (Holmes et al., 2016; Desimone et al., 2002; Nutley, Walters, & Davies, 2003).
We see a need for research and development activities to design robust interventions that can move the needle on research use
An excellent example of theory-driven research is Max Crowley and Taylor Scott’s studies of the Research-to-Policy Collaborative. Building on a large body of descriptive evidence that researcher-policymaker relationships are key mechanisms for enabling research use, Crowley and Scott developed an intervention to foster ongoing interactions between academics and Congressional staff. They combined those relationship-building strategies with an assessment of the research Congress needed for their next session and trained researchers to understand the policy process and how to engage with Congressional staff. Their initial pilot testing indicated promising results, and the Foundation supported a randomized-controlled trial to test the impact of the intervention. The team’s latest publications finds that the intervention increased the use of research in legislation on child and family policy, legislative staff’s appreciation for the conceptual value of research, and researchers’ knowledge of how to engage policymakers (Crowley, Scott, Long, Green, Giray, Israel, et al., 2021; Crowley, Scott, Long, Green, Giray, Gay, et al., 2021).
Getting to Impact
Improving research use in policy and practice is an intermediate step toward the ultimate goal of benefitting society. To get to societal outcomes we need a clearer sense of what it means for research to be used well: what Rickinson et al. (2021) call the quality of research use.
Quality of research use is not synonymous with the quality of research, although the latter receives much more attention. For example, researchers often debate what constitutes rigorous research that is worthy of being used (Tseng & Gamoran, 2017). Researchers also pay increasing attention to the relevance of research to practice and policy, as well as to the body of research evidence in terms of consistency of findings. These aspects of research quality are important inputs for decision-making, but they tell us nothing about how well those inputs are used.
Quality use requires “thoughtful engagement” with research (Rickinson et al., 2020). Descriptive studies of research use consistently find that using research is an interpretive process. After all, research never speaks for itself and decision makers must always make sense of research findings in relation to their specific problems and contexts (Yanovitzky & Blitz, 2020). Moreover, research is never the only pertinent information on the table, and decision-makers must always make sense of research alongside other types of evidence such as data and prior experience. How research and other types of evidence are interpreted and integrated are key elements of quality research use.
With this in mind, we welcome proposals that theorize what it means to use research well and that empirically study quality research use. Because impactful research use should, by definition, lead to better outcomes, theoretical notions should be subject to empirical validation. Furthermore, the road from using research in decision-making to improving youth outcomes is long and hard to predict. Just because research informs a policy direction does not mean the policy will be implemented well, or that societal outcomes will improve. Just because high-quality research is used does not mean that its application to a different context, in a different time, will yield positive results. Instead, there are many intervening steps and supporting conditions that must be in place for upstream use of research in decision-making to lead to downstream impacts. I suspect the most illuminating studies will identify the mechanisms and conditions under which using research leads to positive youth outcomes.
Critical Race Lenses
Just as we cannot assume that using research leads to better outcomes, we cannot assume that using research is a race-neutral process in a society built on structural racism. In the past three years, the Foundation has been learning from Critical Race Theory, and we have been asking ourselves questions such as who defines whether research is useful, whose research gets used, and who determines how research is used (Doucet, 2019, 2021; Kirkland, 2019).
Hard truths have emerged. The use of research evidence is as embroiled in racial hierarchies as the rest of our society. The people who define whether research is useful are often society’s elites, and rarely do the communities who are supposed to benefit from research get to define what makes research useful (Doucet, 2019). Moreover, the research that gets promulgated and used is subject to the same racial biases that are embedded in our research and policy institutions. In education, for example, evidence-based policy to reduce race- and class-based achievement gaps has been dominated by economics—a field that is predominantly White and male (Scott, 2020). Too often absent from education policymaking about racial disparities are the many researchers of color who study the lived experiences of students of color. Furthermore, the users of research are usually those who sit in formal decision-making positions in government, foundations, and large nonprofits. In our racially stratified society, those decision makers are often White and (upper) middle class, with experiences that differ from racially and economically marginalized communities (Michener, 2020).
Critical lenses challenge us to step back and consider our goals for using research. These lenses help us see the ways research has served to support deficit narratives of Black communities in policy and practice (Doucet, 2021; Kirkland, 2019). They also highlight the limited range of policy goals that research has been mobilized to influence—goals that too often focus on mitigating the effects of racism rather than dismantling racism itself. We face the reality that using research to mitigate the effects of structural racism without concomitant attention to dismantling systems of oppression means that research fuels a cycle of paternalistic policies that leave racial hierarchies intact. Critical lenses enable us to reimagine our goals for using research, specifically what it means to use research toward just ends.
Critical lenses challenge us to step back and consider our goals for using research
As we move ahead, we welcome proposals from critical scholars who are interested in building theory and empirical evidence on improving research use. We are especially enthused about interdisciplinary teams that combine expertise in critical theories and research on research use. Michener offers a working definition of the critical uses of research evidence as those that “deploy research rooted in the perspectives and experiences of people of color and do so for purposes that intentionally connect to and benefit communities of color” (2020). Applicants might build off of this definition or offer other operationalizations of what it means for research to be used to advance anti-racist goals.
We hope studies that incorporate critical race perspectives will expand our existing theory and empirical evidence on research use (Doucet, 2021). For example, Farrell et al.’s (2021) recent landscape scan of research-practice partnerships (RPPs) reveals that many partnerships aspire to racial equity goals, but as Diamond (2021) argues, their efforts are thwarted if they fail to consider how racism is embedded within both research and practice institutions. As RPPs seek to align their practices with their equity aspirations, it is helpful for research on RPPs to adopt a critical lens. For example, researchers studying RPPs can examine what goes right and what goes awry as partners navigate racial dynamics in co-developing their research agendas, making sense of the findings, and using the findings (Doucet, 2021). RPP researchers can study how racism is manifested in universities, school districts, and their interactions with communities, as well as the strategies for undoing racism through partners’ perceptions, words, and actions (Ahn, 2021). Moreover, studies can investigate whether using research for antiracist goals shows up in policy and resource allocation decisions: Does research use go beyond rhetoric and become institutionalized (Quinn, 2021)?
Assessing Research Use
As theory and empirical findings have advanced in the past 15 years, so too have methods for assessing and measuring research use. When we first came to this field, people assessed research use by relying on proxies such as the number of times a research report is downloaded or the number of pageviews it receives online. While those are easy metrics to collect, they provide no information on whether or how the research is used. Back in 2007, we were also concerned with a reliance on self-report measures that elicited socially desirable responses about decision-makers’ attitudes toward research or how much they valued research. A poignant irony of this early work is that even surveys and interview protocols inquiring about research use too often posed decontextualized questions that did not ground research use in any particular policy problem or practice situation.
Today, a more rigorous and diverse set of methodological tools are available to assess research use in situ (Gitomer & Crouse, 2019; William T. Grant Foundation, 2020). Scholars have developed sophisticated methods for assessing research use through document analysis, discourse analysis, and observations—methods that do not rely on self-report. Surveys and interviews are still used, but they are designed with greater sophistication, by situating research use within the context of specific decision-making situations and by aggregating data from individuals to derive organization-level assessments of research use (see, for example, surveys by the Center for Research Use in Education). Social network analyses also provide stronger documentation of how research spreads across people and organizations, what characterizes robust diffusion networks, and who the key research brokers are within networks.
As with any area of research, research designs are strongest when they are informed by theory. The conceptualization of research use as a process, for example, has led investigators to pursue longitudinal research designs. Cynthia Coburn and colleagues have pursued studies in which they observe policy meetings, collect the policy documents coming into and out of those meetings, and interview decision-makers over a year to track how research is invoked and mobilized to influence “decision trajectories” over the course of policy deliberation (Coburn & Huguet, 2020). In contrast, studies that examine research use in an isolated moment in time can underestimate research influence because they cannot see the ways research influenced the course of decision making upstream nor identify how research-based ideas are carried through policy deliberations long after the research was introduced.
Gitomer and Crouse (2019) argue that the strongest research designs also pay careful attention to sampling decisions. When deploying surveys to assess research use in an organization, the sampling strategy must generate an adequate sample of people involved in using research. If individuals or a group are systematically missing, bias is introduced into the assessment. And when observing research use in policy deliberations, studies must sample the times and places where discussions occur. If, for example, investigators sample formal meetings, but the real decisions are made through email exchanges and hallway conversations, then the investigator will miss the story. In examining research use in policy formulation, collecting the various drafts of legislation or regulation may be more indicative of how research was used than only analyzing the final version.
Research designs for are strongest when they are informed by theory
Our grantees also demonstrate a keen awareness that all methods have strengths and limitations, and thus many investigators employ mixed-methods designs to triangulate findings and to paint a fuller picture of research use. A sole reliance on document analysis, for example, risks undercounting research use because it can go undocumented, whereas only conducting interviews suffers the limitations of recall bias and demand characteristics to provide socially desirable responses. Combining the two methods provides a more complete picture of what research was used, how it was used, and for what purpose.
At the same time, as our methodological tools advance, we need better ways to share new developments. A notable breakthrough is a new open-access repository of methods to assess research use. Led by Drew Gitomer, the repository spans a variety of methods, from document analysis to surveys to observations to social network analysis and more. Perhaps even more importantly, the repository includes specific measures, including survey instruments, interview protocols, and coding schemes, along with guidance on how to use or adapt those instruments in studies of research use. We hope that by anchoring research on research use in a set of strong methodological tools, we can better compare and aggregate findings across different contexts. We also hope that making the repository open-access will enable investigators to build on each other’s work rather than reinvent the wheel. (Learn more about the repository at https://uremethods.org).
Despite significant progress, the field still has further to go to achieve reliable, valid, and scalable assessments that can be employed to test research use interventions. A pressing challenge for the future is that while we have robust methods for assessing research use, those methods are costly and labor intensive. Farrell, Coburn, and Chong’s (2018) study of research use in school districts relied on hundreds of hours of observation, and Yanovitzky and Weber (2018) collected hundreds of bills and hundreds more legislative hearings for their study of Congressional use of research in child obesity policy. Both project teams devoted countless hours to manually code the data for analysis, tasks that are impractical for large-scale intervention studies. As we look to the future, it will be valuable to explore new techniques, such as machine learning, to code large amounts of data more efficiently.
Building the Field
Looking back at grantees’ work on the use of research evidence puts into relief how far we’ve come in 15 years. The fruits of this work—theory, evidence, interventions, methodological tools, and more—have contributed to our understanding of the very premise of research use and have revealed new avenues for exploration and discovery. Our grantees have joined leaders across disciplines and policy areas to light the way toward making research more useful, used, and impactful for society.
Fifty years ago Carol Weiss and others such as Michael Patton and Nathan Caplan were conducting empirical studies of research use in policy and developing some of the key theoretical frameworks that we are still building on today. It was a “golden age” for the field (Henry & Mark, 2003). But despite its promise, this work flourished for only a brief period and faded quietly soon after. This specter of fleeting bodies of work motivates us to ask what it would take to not only revitalize this field of study but to sustain it for the long term.
We now set our sights on ways to multiply our collective efforts and link the field globally and across policy sectors in the years ahead.
Funding
One condition for building and sustaining a field of study is a steady pool of funding, and so from the earliest days, we sought funding partners on this journey. We found enthusiastic colleagues in John Easton, Tom Brock, and Becky McGill-Wilkinson at the Institute of Education Sciences (IES) in the U.S. Department of Education. These leaders saw research on research use as key to their efforts to strengthen the connections between education research, practice, and policy. Under their leadership, IES supported two national research centers on knowledge utilization, The National Center for Research on Policy and Practice (Penuel, 2014-2019) and the Center for Research Use in Education (May, 2015-2020). At the National Institute of Justice in the U.S. Department of Justice, we also collaborated with John Laub, Dara Blachman-Demner, and Nadine Frederique as they pursued ways to advance “translational criminology” (Laub, 2012). Seeking ways to improve the usefulness and use of the research funded by NIJ, efforts at the Institute eventually led to an RFP seeking studies of “how criminal justice practitioners use research in their decision-making processes and how they implement evidence-based programs or practices” (NIJ, n.d.).
What would it take to not only revitalize this field of study but to sustain it for the long term?
Despite these investments, interest in funding research on research use has not taken off on a large scale. There are certainly other funders—such as the Economic and Social Research Council in the UK and Health Research BC in Canada—that have supported individual projects and grant portfolios. Other funders have supported evaluations in order to document the impact of their research funding. But altogether our investments have not added up to significant field-level impact (Bednarek, Tseng, & Jones, 2021; Oliver & Boaz, 2020). Building robust knowledge on research use remains frustratingly undercapitalized: Although many funders want to learn from research on research use in order to improve their work, few have invested in studies to produce those findings. Moreover, those of us that support research use are too often siloed in our policy sectors: Education research funders rarely interact with environmental funders, for example. This means that the studies we support are also siloed, and we miss opportunities to build a cumulative body of knowledge (Arnott, 2021).
But change is underfoot. Together with Angela Bednarek at The Pew Charitable Trusts, we are working to unite funders to build a more vibrant and sustainable field of research on research use. Working under the auspices of the Transforming Evidence Funders Network, we are convening a group of funders interested in field-building questions such as:
- What are innovative ways funders can promote research on research use beyond funding individual grants and grant programs?
- Can we unite around a shared research agenda and commit to filling key knowledge gaps?
- How can we connect scholars from different regions and policy sectors to accelerate knowledge building?
Together, we are exploring the creation of a global network of hubs for research on the use of research, anchored in different regions around the world. The network might serve as a meeting ground where investigators studying research use around the world can align their efforts around a shared research agenda, illuminating both generalizable knowledge and context-specific insights. Linked together, the regional hubs could draw together collective wisdom, enabling the field to build forward in strategic ways. We welcome funders to join us in aligning our efforts and collectively supporting activities to grow the field with diverse talent, identify key research questions, and seed studies with potential for high impact.
Community
In addition to funding, fields of scientific inquiry are buoyed by communities of people. That is why we invested for over a decade in annual learning community meetings focused on research on research use. Those meetings began as small grantee convenings, bringing together people we funded—largely in education and human services—to share what they were learning and to collectively build a stronger knowledge base. Over time, demand to join these convenings exceeded our ability to accommodate the many people who wanted to participate, and the growth soon outpaced our resources. Similarly, although the Foundation’s mission is focused on children and youth in the U.S., the community working on the use of research evidence is much broader, including colleagues around the globe working in international development, foreign policy, environmental sustainability, and entrepreneurship.
Given growing demand and a cross-sector community increasingly working toward similar aims, we realized that continuing to organize the meetings on our own would inhibit the field’s growth. In 2020, we launched a collaboration to spin off our learning community as part of a global initiative called the Transforming Evidence Network (TEN), with The Pew Charitable Trusts as a funding partner and Kathryn Oliver and Annette Boaz as scientific advisors. Boaz and Oliver had been regular participants in our meetings and were beginning to envision a conference modeled on ours that would bring together an even broader group of researchers across policy sectors and countries. They observed that the broader field needed a regular place to come together to stitch together what were otherwise scattered and disconnected projects (Oliver & Boaz, 2020). At the same time, Angela Bednarek was laying the groundwork for the Evidence Project at The Pew Charitable Trusts, which would serve as the institutional home for global networks, training, and collaborations focused on evidence use.
One of the Transforming Evidence Network’s signature activities will be hosting biennial international conferences focused on improving research use. The conferences will integrate both the research on and the practice of research use. On the research side, TEN will provide mechanisms to better share and synthesize what we already know, identify key gaps in knowledge, and deploy our resources and energy to fill those knowledge gaps. On the practice side, policymakers, intermediaries, and other decision makers working to improve evidence use will have opportunities to learn from each other and the research on research use, as well as shape future studies so that the findings meet their needs. Across the research and practice sides, the conferences will seek to break down silos; speed up our ability to learn across countries, disciplines, and policy sectors; and align efforts to transform the evidence ecosystem so that it is more evidence-informed, coordinated, and impactful. In the off-years, between the biennial meetings, smaller convenings will focus on issues such as research methods (e.g., measuring research use, etc.), work within regions (e.g., North America, Europe, Africa, etc.), or emerging issues (e.g., responding to the pandemic, building trust in science).
Looking Ahead
As I close this essay, we face a prolonged global pandemic, environmental crises, and expanding inequality. But I remain optimistic that research can improve decision-making, that decisions informed by research evidence can shape societal outcomes for the better, and that the community of people working to realize this vision is growing in size and strength.
And it’s happening all the time. Even as I drafted this essay, my colleagues and I were finalizing a collaborative partnership with the National Science Foundation, “Increasing the Use, Usefulness and Impact of Research about Youth” (Lupia, 2021). This partnership puts the weight of the U.S.’s largest social science funder behind the field.
Research can play a vital role in pointing policymakers, civil society, and communities toward a stronger, more sustainable, and just world. But getting there means building on what we know about what it takes for research to be useful, used, and impactful. We’ll need to give up some of our naivete, wade into the complexity of decision-making, and build relationships of trust. Extolling politicians and the public to “listen to scientists” is not enough. We have to gaze clear-eyed at the social and political realities of policymaking and grapple with the ways our values and the historical legacies of racism and colonialism pervade research and its use. We must imagine a better future, and we must put our scientific theorizing and methods behind the task of building that future. If we move forward with creativity and commitment, building on the work of others, we may just achieve our aspirations for research that serves society.
