This post builds on ideas we shared when we announced our updated focus area in early-2016.
Since January 2016, we have reviewed nearly 90 letters of inquiry for research grants in our focus area on improving the use of research evidence. These include some exciting strategies for getting research used and some clever study designs for considering its impact. In this post, we share some observations about the work that has been proposed thus far, as well as tips for potential applicants as they prepare their letters of inquiry.
Consider the full breadth of the focus area
Our focus on improving the use of research evidence emphasizes three specific areas of inquiry. First, we seek proposals to identify or test strategies to improve the use of existing research evidence. Second, we seek proposals to identify or test strategies to produce new research evidence that is useful to decision makers. And third, we seek proposals that test the assumption that if research is used it benefits decision making and the outcomes of young people.
The majority of research proposals up until now have focused on the first question outlined above: how to improve the use of existing research evidence. The ideas have ranged from offering instrumental supports to help mid-level managers apply research-based programs, such as coaching and technical assistance, to efforts to change how an organization, its leadership, and its workers value and use research. Most recently, we funded William Firestone and Jill Perry to investigate a possible strategy for bolstering school district leaders’ capacity to appraise and use research evidence. The team will examine an initiative to reform the education doctorate and create scholarly practitioners who use research and theory as tools for change in their work in school districts. This is a critical question given that the education doctorate is a primary mechanism for preparing future education leaders. We welcome other innovative projects that study ways to create the structural and social conditions that foster routine and constructive uses of research evidence among decision makers.
A smaller number of projects have focused on the third line of inquiry: testing whether and under what conditions research improves decision making and youth outcomes. Several research teams proposed evaluating whether incentives by federal and state governments were successful in encouraging the use of high quality research at the local level. Importantly, these studies also probed how incentives for research use changed behaviors among those local decision makers and impacted the educational and mental health outcomes of the youth they served. These teams have had to grapple with some tough challenges. In addition to having to think through how to conceptualize and measure the use research evidence, they also need to determine what it means to measure the impact of research use on decision -making and on youth outcomes. We invite others to take up this challenge and to consider a variety of conditions under which using research evidence improves youth outcomes. For example, this might involve examining how the use of high quality research for which there is a high degree of consensus affects the quality of services delivered and, in turn, youth outcomes.
Few investigators have taken up the second line of inquiry: ways to create the incentives, organizational structures, and relationships needed to generate a supply of evidence that is timely, relevant, and ready for use. We encourage researchers to investigate this gap in our knowledge and to identify ways to alter the organizational cultures of research institutions so that researchers have protected time to conduct more practice-oriented studies and are rewarded for doing so. A more modest strategy might involve cultivating the skills of motivated researchers’ to negotiate and effectively collaborate to produce research that benefits youth. Still others might evaluate whether funders’ efforts to incentivize joint work between community organizations and academic partners leads to the production of more policy-relevant research that carries weight in decision making.
Applicants also wonder whether projects are more appealing when they encompass all three interrelated lines of inquiry on improving the use of research evidence. While studying all three may be possible, there are no extra points for this approach. In fact, we have found that it is very challenging to address all three threads well in a single proposal. In most case, we recommend focusing on one or two key research questions and pursuing them comprehensively and with rigor.
Build a strong rationale for the study
Once you have selected your primary line of inquiry, it is essential to build a strong rationale for the study. We are looking for high-quality, systematic studies that investigate ways to improve the use of research evidence in ways that impact decision making, services, and youth outcomes. This means that in the span of a five-page letter of inquiry, investigators need to demonstrate their ability to think deeply, grapple with tough issues, and draw on existing empirical and theoretical work to lay out a compelling plan.
The most successful letters of inquiry excel in two important ways. They richly conceptualize and operationalize the use of research evidence and demonstrate how the project will advance the field. Both of these elements require connecting the proposed project to the broader literature. For example, improvement science can help us inform work on change management as it relates to shifts in organizational culture to facilitate research use, and the field of knowledge mobilization has a history of developing and testing strategies to advance knowledge. The Foundation has also supported nearly forty studies in this area, which have generated a body of evidence that offer empirical evidence, theory, and tools from a range of disciplines and methods.
Richly conceptualize and operationalize the use of research evidence
The Foundation is interested in building strong theory on strategies for improving the use of research evidence. We therefore encourage applicants to think critically about how they will conceptualize and measure the use of research evidence.
A first step is to define research evidence. The Foundation defines research evidence as empirical findings derived from systematic data collection and analyses. As we’ve written before, it is important to remember that data by themselves do not constitute research evidence. However when data are subjected to systematic tests to pursue a question, this construction aligns more closely with our definition of research evidence. Applicants should provide their definition of research evidence and a strong rational to support the definition proposed.
A second step is to convey how you are conceptualizing the use of research evidence. Successful letters of inquiry ground their conceptualization of research use in prior work on the use of research evidence. To meet this challenge, situate your ideas about the use of research within a larger theoretical framework. Share whether you are thinking about research use as part of a specific decision—what program to adopt, which problems to prioritize, how best to train workers. Or perhaps you are envisioning research use as something that is more strategic and intended to argue for or support an existing idea. Or whether you expect that research use will be more subtle and shape ideas. For example, Caitlin Farrell and Cynthia Coburn have given considerable thought what constitutes conceptual uses of research evidence. They suggest that knowledge from research seeps into discussions and shapes ideas that then inform “what leaders prioritize and focus on as they do their work.”
How you conceptualize research use provides a roadmap for thinking about how you will assess research use. Reviewers need enough detail to determine what and how you will measure research use, and if your approach is appropriate for the aims of the project. Successful letters of inquiry also lean on rich traditions to shape the research design as well as the measurement piece of the project. For example, Palinkas and colleagues (2014) developed a structured interview protocol to assess understand different stakeholders’ levels of engagement in acquiring, evaluating, and applying research evidence in social service settings. And Daly and Finnigan (2009) have developed network analysis techniques to examine how research evidence is exchanged within school districts. We have observed that while some measures exist, they are infrequently used. When appropriate, we encourage applicants to use existing measures and analytic tools so that findings can be compared across studies.
In sum, how you conceive of the use of research evidence is fundamentally tied to what systems you are studying, who the decision makers are, and why research use is important to those included in your sample or case.
Demonstrate how the project will advance the field.
As investigators make the shift from understanding the conditions that support the use of research evidence to identifying and testing strategies to improve research use, they will need to take some risks. Part of taking risks, though, is having a strong base on which to work. In your letter, be clear about what theoretical and empirical work undergirds the proposed study. As discussed above, this includes the foundations for your conceptualization of research use, your research questions, and, if relevant, your change effort.
One way teams can illustrate the rigor of their proposal is by explicitly showing how the proposed project adds value to or leverages prior work. For example, we know from prior empirical work that when research is designed with the end user in mind, the research evidence produced is more likely to be used. But we need more information about the incentives and social structures required for researchers to engage decision makers in the research processes.
A study might propose testing an existing strategy to build the relationships and processes needed to place researchers in closer proximity to decision makers and their needs, say a one year fellowship program that embeds researchers in a state department of juvenile justice. The letter of inquiry would need to provide the broad strokes of this effort and more detail about the key processes and the rationale for why these processes are expected to improve research. This would require conceptualizing what it means to facilitate the production of research that responds to decision makers’ needs and formulating an operational definition. Further, reviewers will want to know the implications of the researchers’ involvement in the fellowship, and evidence of a research design that convincingly and rigorously detects change. Would decision makers judge the fellows’ research as relevant and actionable? Did the strategies and outlets for sharing information change? Was the research produced by the fellows used? Did the fellows’ networks change in ways that would support longer-term improvements in their ability to produce relevant research? These are the kind of questions research teams will think about as they develop their letters.
Measurement work—either advancing existing measures of research use or rigorously developing new ones—is another way to push the work forward. It is an open question whether existing measures are sensitive enough to capture changes in research use over time or as a result of experimental manipulations. The three new lines of inquiry may necessitate modifying or supplementing existing measures. If proposing work to advance methods, applicants should make a strong case for the utility of the measures and analytic strategies beyond their proposed study, and offer a strong theoretical and empirical rational for the importance of the constructs that will be the focus of the measurement work.
Conclusion
Improving the use of research evidence in ways that benefit youth will require creating, identifying, and testing strategies to encourage the use of existing evidence, as well as strategies to produce more useful evidence. What’s more, we will need creative efforts to assess and understand the connections between use by decision makers and outcomes of the youth they serve. We have already received some exciting proposals, and, as we enter the second year of this initiative, are offering a few insights to help researchers be successful. We look forward to your best ideas for how to further our understanding and move the field ahead.
