Digest, Issue 1: Summer 2016

Using Data to Produce Useful Research Evidence

Access to data doesn’t guarantee the production of useful research evidence, nor does it guarantee that evidence will be used. But linking large data sets and facilitating researcher access creates opportunities to answer different sets of questions than those allowed by survey, experimental, or qualitative data alone. Data linkages set the stage for scientific discoveries that can inform smarter policies and programs.1

Both researchers and policymakers have been calling for access to big data—large volumes of digital information about individuals, their activities, and their interactions with different systems, including government. Others have gone so far as to argue that the “big data revolution,” which facilitates analysis of large-scale data sets drawn from administrative records or linked records from multiple sources, will form the future of evidence based policymaking.

Thanks to efforts underway at the federal, state, and local levels to integrate and allow access to a variety of data sets, some researchers are already tracking individuals over long periods of time and across multiple systems and institutions, yielding findings with the potential to challenge assumptions and generate new insights. For example, researchers have accessed, linked, and analyzed state education and federal labor statistics to learn about the effects of state-wide economic downturns on school achievement, and others have used state-level accountability and test score data to better understand shifts and correlates of racial and ethnic achievement gaps.

Others have used big data for research to advance an understanding of how and when to respond to problems. At the William T. Grant Foundation, we are convinced that social science research, including that which taps into the potential of big data, can play an important role in addressing the challenge of inequality. We think that both the extent of inequality and its effects on youth outcomes are not inevitable, but amenable to social policy. And we believe that high-quality research can help identify and build understanding of approaches that help reduce short and long term inequalities.

Access to administrative data provides researchers with a unique opportunity to generate research that informs these efforts. Some have argued that this access will also facilitate faster and lower-cost evaluations of federal, state, and local programs and provide better estimates of program costs. But to fully maximize the value of data, we need to ensure that the research evidence produced from analyses of big data is useful and used. We think this requires:

  • linking administrative data to other sources of information,
  • involving decision makers in the process from the outset, and
  • establishing strategies and structures to initiate and sustain such efforts.

These ideas are consistent with the Foundation’s focus on improving the use of research evidence. We are interested in investigations that identify and test how to connect decision makers with research evidence—including research resulting from thoughtful uses of administrative data, as well as studies that identify and test the incentive structures that encourage the production of research that answers decision makers’ most pressing questions. Ultimately, we hope such efforts will foster a culture of evidence wherein what is learned about programs and policies to reduce inequality is moved into action.

Linking different data sources

The potential of big data would be stronger if the data were integrated with other sources of information. Many also argue that more researchers need access to these combined data. These are central aims of a new project underway at the National Research Council known as the American Opportunity Study (AOS).2 AOS would create a structure that allows all qualified researchers—instead of only a select group—regular access to public record data in a safe, protected environment.

Figure 1: AOS Data Sources (Grusky et al., 2015; 2016).

Data that comprise the AOS are extensive, and would offer new opportunities to study social mobility. The project aims to develop a comprehensive link between survey and evaluation studies and administrative data from the Census and agencies such as the Social Security Administration. The AOS panel would provide repeated observations on individual income, education, occupation, and other demographic variables for individuals. Parent–child or intergenerational matches are also possible with access to data from the IRS, the American Community Survey, and the decennial census.

“…it would open up new fields of social science inquiry; increase opportunities for evidence-based policy on poverty, mobility, child development, and labor markets; and otherwise constitute a new social science resource with much reach and impact.”

The structure of AOS is also being designed to inform program and policy evaluations aimed at contributing knowledge about upward mobility for disadvantaged youth. Figure 2 presents the three tiers of the broad plan. The top layer shows data links across decennial Censuses, a long-term, ten year panel of everyone who completes a Census form; the middle layer includes surveys, studies, or evaluations of social programs; and the bottom layer represents linked administrative datasets.

If successfully developed, AOS would provide a resource of unparalleled statistical power and an opportunity for large-scale causal research. One can easily imagine how survey or evaluation supplements to AOS might reveal the contours and correlates of social mobility as well as potential responses. One might investigate intergenerational issues and look at long-run outcomes of early life circumstances. Researchers could also link state data on educational experiences to national sources of earnings and income data (like that in the IRS sample) for both parents and children, and look forward and backward to see how they fare on a range of outcomes. As Grusky and colleagues write, “If an AOS of this sort were assembled, it would open up new fields of social science inquiry; increase opportunities for evidence-based policy on poverty, mobility, child development, and labor markets; and otherwise constitute a new social science resource with much reach and impact.”

Figure 2: Three Tiered Plan for Linking Census and Survey Data with Administrative Records (Grusky et al., 2015; 2016).

AOS might also be leveraged to answer questions about long term outcomes of policies. This would build on prior work that used administrative data to evaluate the educational and economic payoffs of federal social programs for cash transfers and refundable tax credits and examined the unintended consequences of policies designed to support families. With AOS, a research team might expand on these findings and use new sources of administrative data or survey work to examine areas of research that are currently lacking, such as the effects of policies on behavior and mental health. For example, a team might use linkages to access IRS data and incarceration statistics from the Bureau of Justice Statistics to learn about the effects of youth incarceration on longer term outcomes, such as earnings post release and recidivism. Another team might examine Veteran’s Administrative data to assess the effects of military service on health and economic outcomes later in life. Alternatively a team might use linked survey data from the Panel Study of Income Dynamics or the National Longitudinal Study of Adolescent to Adult Health to probe potential mechanisms or competing hypotheses about why the observed relationships exist. This range of findings could encourage further study on implementation and context, and, in turn, inform smarter policies and programs.

“Unless we use this data to produce research evidence that addresses the needs of decision makers, the promise of the moment may go unrealized.”

Ultimately, looking across data sets may build comprehensive evidence about how young people fare in the systems through which they live and grow, and provide new understanding about the intersections of these systems. Andrews, Imberman, and Lovenheim, for example, have linked administrative records about K–12 education, postsecondary education, and earnings to examine the impact of two programs in one state for students from low-income high schools. The programs provided additional financial aid and enhanced supports for students once enrolled at one of two highly regarded public universities in Texas. The team’s study demonstrated the potential of such programs to yield long-term earning benefits, and highlighted how differences in the design of these programs may have long-term implications.

While these lessons are compelling, questions remain, however, about what it takes to move from data to “a new social science resource with much reach and impact.” Unless we use this data to produce research evidence that addresses the needs of decision makers, the promise of the moment may go unrealized. We suspect this requires knowing how to foreground decision makers’ information needs at the front end of the process and employing strategies and structures to improve the likelihood that the research evidence produced is ultimately used.

Footnotes
  1. For full citations and a complete reference list, download the PDF of this essay.
  2. See Grusky, Smeeding, & Snipp 2015; 2016 for a more complete description.
Downloads

In this issue

For researchers and their teams, it’s important to appreciate that when seeking to incorporate mixed methods, the forms and challenges vary from project to project.
Harnessing Discovery: Writing a Strong Mixed-Methods Proposal
The potential of big data is multiplied when researchers are able to use it to produce work that is relevant to the leaders who make decisions about policies and practices that affect young people.
Moving from Data to Research to Policy: What Does it Take?
A key approach in our efforts to support impactful research is to invest in the development of tools that enhance the work of many researchers engaged in a common enterprise.
Investing in Tools to Create Evidence and Improve Policy
Taken together, these studies produced deep insights that would not have been accessible through quantitative research alone.
The Value of Qualitative and Mixed-Methods Research: Examples from our Portfolio on the Social Settings of Youth Development
Understanding the social processes that involve interactions between individuals, or between individuals and their contexts, is essential to responding to inequality.
How and Why: Questions that are Well-suited for Qualitative and Mixed Methods
Digest, Issue 1: Summer 2016
We believe that qualitative and mixed-methods research is essential to building, understanding, testing, and improving responses to inequality.
Why Qualitative Research?

More Digest Issues

The Digest

Issue 9: Winter 2023-24

The ninth issue of the Digest features an update on the Institutional Challenge Grant program five years after its launch, with insights from a Foundation stock-taking effort and reflections from two grantee teams. We also share new thinking on the use of research evidence, including ways for researchers to leverage existing findings to bolster studies of effective strategies to improve research use.
The Digest

Issue 8: Winter 2022-23

The eighth issue of the Digest features insights about where new research might help identify ways to reduce inequality in youth outcomes after COVID-19, as well as how the fields of implementation science and research on research use might learn from each other in ways that yield more transformative research in years ahead.
The Digest

Issue 7: Winter 2021-22

The seventh issue of the Digest includes fresh insights from program staff and grantees on how researchers can successfully confront the challenges of studying causal mechanisms for reducing inequality, as well as how the global community of researchers and funders focused on improving the use of research evidence can continue to break new ground in the years ahead.
The Digest

Issue 6: Winter 2020-21

Essays in this issue of the Digest focus on the importance of looking beyond individual action and customary metrics in research on reducing inequality, as well as how strengths-based, race-conscious research can be produced and used to uplift communities of color.
The Digest

Issue 5: Winter 2019-20

The fifth issue of the William T. Grant Foundation Digest features insights from program staff and grantees on the importance of looking beneath the surface to consider the underlying factors that create and shape challenges that social scientists seek to address, whether they be related to reducing inequality in youth outcomes or improving the use of research evidence in policy and practice.
The Digest

Issue 4: Winter 2018/19

The fourth issue of the William T. Grant Foundation Digest features insights that may point the way toward a more nuanced understanding of evidence use and inspire new and more wide-ranging examinations of ways to reduce inequality in youth outcomes.
The Digest

Issue 3: Winter 2017/18

The third issue of the William T. Grant Foundation Digest features insights on how research on ability tracking can inform studies to improve the outcomes of English learners, as well as how researchers and school districts can partner to build learning systems based on research evidence.
The Digest

Issue 2: Spring 2017

The second issue of the William T. Grant Foundation Digest features writing on research rigor and relevance, as well as the potential for a new research agenda for improving the outcomes of English learners under the Every Student Succeeds Act.