Moving from Data to Research to Policy: What Does it Take?

These considerations are essential, but we know from Foundation-supported work that there is also a social side to research use. This includes consideration of how decision makers will use the resulting research and the relationships, incentives, and supportive structures it takes to bring these considerations into the research process. Thus, while promising efforts are underway to facilitate access for researchers, we need to do more to ensure that what is learned from these data is useful—that research is not just produced and pushed out for consumption without careful consideration of how and by whom the resulting research will be used.

But connecting decision makers’ interests with researchers’ pursuits is a significant challenge. Work from those studying the use of research evidence offers some promising paths for establishing important safeguards.

Working with decision makers

The move from data to use-ready research must consider how the evidence generated might be used. Input from decision makers on the types of questions they want answered could inform the types of data sets that need to be connected, the level of analysis (e.g., city, district, school, or student), and the identification of salient variables. For example, state education leaders may want to know how specific investments at a school level—say, technology, textbooks, and physical improvements—affect achievement. This list may reflect district leaders’ working hypotheses about competing expenditures and what matters for student success. Not all of these variables may be on the minds of researchers, however. The construction of data sets, then, begins the path to generating research evidence—and this beginning delimits the range of questions researchers are able to answer with big data.

Conversely, decision makers may be experts on their systems, but they may not have the capacity to anticipate the analytic structure of data sets, design research studies, or critically evaluate, prioritize, and interpret research evidence. But these skills can be nurtured through partnerships or other opportunities that allow users to engage with research evidence and make sense of how it might apply to their organizations.

The social side of research use includes consideration of how decision makers will use the resulting research, as well as the relationships, incentives, and supportive structures it takes to bring these considerations into the research process.

Research-practice partnerships and similar collaborations offer a sound strategy for involving decision makers. Reciprocal participation in the social process of building and using evidence strengthens the joint effort between researchers and decision makers. In partnerships, decision makers are provided with opportunities to articulate their needs, and this can improve the likelihood that the resulting research evidence will be useful to them. At the same time, frequent exchanges help researchers build familiarity with the needs and motivations of decision makers. This, in turn, can help researchers construct a process and communicate findings in ways that are accessible and oriented toward use.

The literature has strong examples of the value of partnerships when working with administrative data, especially at state and local levels. The University of Chicago Consortium on School Research, a long-standing partnership between researchers at the University of Chicago and within the Chicago Public School district, for instance, has used administrative data to identify indicators of success, chart improvement, and conduct theory-driven evaluations of within-district programs and policies. The Child and Adolescent Services Research Center at Rady Children’s Hospital in San Diego has also demonstrated the value of marrying research expertise and experience with national datasets with local knowledge of system operations and information needs. By embedding researchers within the local organization, they have linked local administrative systems, data from nationally representative longitudinal survey samples, and Medicaid data to show that court referrals were associated with racial disparities in the use of mental health services.

Thus, partnerships between decision makers and researchers, either within their organizations or at research institutions, may produce work that, from the outset, is likely to penetrate the realms of policy and practice.

Incentives to facilitate collaboration

Incentivizing collaboration and knowledge integration is fundamental to fostering broader uses of research in ways that benefit youth. For example, the recent reauthorization of the federal Every Student Succeeds Act includes provisions that call for local decisions about programs and activities to be “evidence-based.” Further, elements of the law link funding to these levels of evidence. This is just one example of how policy can encourage integration of research evidence in decision making. Creating these and other kinds of incentives may strengthen collaborations by structuring expectations and routines that promote use from beginning to end—from data set construction to evidence production to research use.

Beyond legislation, though, what structures might incentivize links between research institutions or trusted intermediaries and decision makers? To some extent, demand for program evaluations may encourage researchers and decision makers to work together. Still, more direct incentives may be needed to upend existing tendencies to focus on the demands and rewards of one’s own institution. This might include providing direct support for sustained research collaborations. On the research side, incentives might include course releases to allow time for partnership activity and recognition that partnering constitutes valuable service to the community. For decision makers, partnering may be more desirable if state and local agencies had expedient mechanisms to receive funds and award contracts, or if they could reallocate resources from programs deemed ineffective to those with a stronger theoretical or empirical evidence base.

Other strategies and supports for improving the use of research evidence

Another promising avenue to improve use includes interventions that are designed to develop skills related to accessing and appraising research. Since partnerships and other one-on-one collaborations are often not possible, technical assistance is an important alternative for cultivating such skills. Technical assistance providers often have deep knowledge of research and the decision makers’ system, as well as sustained contact with decision makers. It is important to imagine how this supportive infrastructure might be refined to develop both leaders’ capacity to access and process research evidence and opportunities to discuss how it might be integrated with existing evidence. In turn, technical assistance providers may also emerge as important conduits for sharing their knowledge with researchers or intermediaries engaged in analyzing data sets. This type of give and take can lead to new ways of seeing old problems and help inform action, thereby contributing to a culture in which it is normative for leaders to use research evidence in decision making processes.

Incentivizing collaboration and knowledge integration is fundamental to fostering broader uses of research in ways that benefit youth.

Lastly, because not all research will be interpreted with a researcher on hand, there need to be structures in place to make sense of the abundance of information that amasses from big data projects like the American Opportunity Study. Currently, numerous clearinghouses provide access to research evidence, but their structures are far from uniform. These outlets differ in terms of the level of rigor required for inclusion, elements reported about each study, supporting documentation, and synthesis across studies. What’s more, the kinds of information decision makers need—like user reviews from peers, implementation context, and infrastructure, training, and cost requirements—are often omitted. This variation, unfortunately, makes it difficult for decision makers to identify research evidence that they can use.

But there are promising examples of approaches that respond to these challenges. In the United Kingdom, for instance, systems (and funding) are in place to routinely conduct systematic reviews of emerging evidence bases, and centralized and developed frameworks ease the organization and sharing of research evidence. The London-based Education Endowment Foundation, for instance, rigorously evaluates strategies to improve the use of research evidence and is offering roadmaps for developing supportive structures for using research.

Moving between data, research evidence, and use: What does it look like?

Beyond generalities about building bridges and translating research to practice or policy, how do we create the real-world conditions for research use? What will the infrastructure actually look like? What strategies and incentives might it comprise?

As described, the potential of getting from data to useful science is realized only when the research is actually taken up by the decision makers. And this is more likely to happen when researchers are able to produce work that is relevant to the leaders who make decisions about policies and practices that affect young people.

One example of how administrative data can be leveraged to generate research evidence that informs decision making is CalYouth, a collaboration between researchers at the University of Chicago and members of the California Child-Welfare Co-Investment Partnership, including leaders from the California Department of Social Services, Child Welfare Directors Association of California, the Judicial Council of California and a group of philanthropic funders. Together, the collaboration is responding to a mandate from the state legislature to evaluate the implementation of extended foster care.

The potential of getting from data to useful science is realized only when the research is actually taken up by the decision makers.

Federal legislation gives states the option to extend services and receive reimbursement for youth in care until age 21, but policymakers in California and elsewhere want to know how and under what conditions the extended care is benefiting youth, and at what cost. Financial resources from a number of private funders have allowed CalYouth to link administrative data on youth foster care histories, employment insurance wage claims, use of public assistance programs, Medicaid, college engagement, and arrests. A sampling frame was generated from the administrative data, and led to extensive interviewing and web-based surveys. State and county administrators and supervisors, youth in foster care, and others provided input on the tools. And, over time, collaborators have used the assembled data to generate research evidence that describes youth and the services they receive, tackle concrete problems related to caseload sizes, and respond in real time to policy debates about the value of extended care. Thus, both the mandate to evaluate the implementation of extended services for youth and an influx of financial resources have helped to incentivize the production of research.

This example shows data being leveraged with an orientation toward use, but it also gets to a larger point about the use of research evidence. As we’ve said, linking and analyzing data sets can contribute to strong, useful bodies of evidence, as long as we’re looking to the data with purpose and remaining mindful that big data is, by itself, only a means to an end. But, as illustrated by the example, in order to increase the likelihood that the evidence we produce is used to inform policies, practices, and programs that benefit youth, we need an infrastructure that encourages opportunities for stakeholders and researchers to engage one another and benefit from each other’s perspectives. We need a framework that cultivates the use of research evidence by incentivizing collaboration and establishing structures that focus attention and resources on integrating the perspectives of decision makers into the research production process.

Work by Foundation-supported researchers and others suggest that merging research evidence in these processes requires having structured discussion about research. In the CalYouth example, research evidence—cost-benefit studies in particular—was included in deliberations about whether to extend care. And negotiations about what administrative data to link and what items were needed in the survey included input from decision makers and other stakeholders. In this way, the data extraction and subsequent data collection efforts were designed for use from the outset; the team solicited the stakeholders’ feedback, down to specific survey items and data fields.

Lastly, the CalYouth example also highlights some outstanding questions. Providing incentives for cross-sector work does not guarantee a productive collaboration. Although there was some evidence that the CalYouth cross-sector partners collaborated in critical deliberations, it is unclear whether these practices were happenstance or intentional and routine. We need to know more about the extent to which practices were codified. For example, were there formal agreements or memorandums of understanding that established guidelines for working with one another? Was there a coordinating body that instructed the process? Examining the extent and value of such formal structures may inform strategies for facilitating research use.

Conclusion

Linking big data sets is a promising first step to producing research evidence that is used by decision makers. But it is not enough by itself. Efforts to link large-scale data sets from diverse sources have the potential to rapidly enhance what can be learned from surveys, experiments, evaluations, and qualitative data about the lives of young people, as well as ways to improve their outcomes. Yet we must be aware that “many factors shape what research is sought, how it is shared, and the ways in which it is evaluated, used, contorted, or dismissed.”

This knowledge demands intentional efforts to orchestrate and incentivize the move from big data to research evidence to use. And, we have accumulated promising lessons about what is required to design and promote research use. For instance, research is more likely to inform decision making when it comes from a trusted source and is deeply understood. Participation in the research process and sustained relationships may enhance trust. Structured opportunities to discuss and push back on research appear to deepen knowledge. And these processes are hypothesized to increase appropriate uses of research evidence. When it comes to harnessing the potential of big data, these lessons should not sit idle.

At the same time, we need to know how to create these conditions; here, the knowledge base is relatively weak. Partnerships between researchers or intermediary organizations and decision makers present one potentially promising vehicle. But we need to create and test these and other mechanisms for improving the use of research evidence. Toward that end, our Foundation welcomes studies that identify, build, and test new strategies for improving the use of research evidence. These proposals should describe the body of evidence that is ripe for use; how use is being conceptualized, operationalized, and measured; and why use is expected to improve decision making and, ultimately, youth outcomes. These questions are a key line of inquiry for our Foundation. We suspect these pursuits will lead to new insights about the infrastructure needed to produce and use research in ways that benefit youth.