Digest, Issue 2: Spring 2017

Bringing Rigor to Relevant Questions

Social scientists too often view the concepts of relevance and rigor as competing. Through this lens, however, a rich spectrum of research methods, approaches, and purposes is reduced to monochrome binaries: either the work builds theory or practical knowledge; either it is for academics or for users in policy and practice; either it is scientific or practical. But for researchers in the social sciences to make greater contributions to the economic and social well-being of our nation, we need to complicate these binaries.1

Put simply, the dichotomy of rigor versus relevance is false. There is no inevitable trade-off between producing rigorous research and producing research with relevance for the real world. Researchers who want their work to matter in policy and practice should begin by identifying the questions of greatest relevance and then bring the highest standards of theoretical and methodological rigor to those questions. In this essay, we argue for the critical need to reach for both rigor and relevance in research to improve youth outcomes, and we urge universities and professional associations to create structures and incentives to support theoretically and methodologically rigorous research that addresses pressing policy and practice issues.

This Is The First Post In A Two-part Essay. Read Part Two Next: “promoting Research That Is Both Rigorous And Relevant.”

Defining Rigor

The term “rigor” is sometimes used as a code word for randomized controlled trials, but that is not our meaning. It is important that all types of research designs and methods be rigorous; what constitutes the rigor of a given study depends on the research questions being asked. Put another way, rigor is not a characteristic of a particular research method, but a matter of the “quality of mind in which evidence and (theoretical) insights are carefully specified and intertwined.”

Put simply, the dichotomy of rigor versus relevance is false. There is no inevitable trade-off between producing rigorous research and producing research with relevance for the real world.

In 2002, the National Academy of Education released the report Scientific Research in Education, which encapsulates much of our thinking about what constitutes rigor in social science research. Education research is an instructive example for defining rigor because it is a field focused on studying problems relevant to policy and practice; it is also a field that is sometimes accused of lacking rigor. The report identified six guiding principles for scientific inquiry:

  • Pose significant questions that can be investigated empirically
  • Link research to relevant theory
  • Use methods that permit direct investigation of the question
  • Provide a coherent and explicit chain of reasoning
  • Replicate and generalize across studies
  • Disclose research to encourage professional scrutiny and critique

Like the National Academy of Education, we define rigor as including attention to theory as well as to methods. Rather than considering theory to be antithetical to practical applications, we maintain that theoretical insight is essential for carrying out a rigorous study. At its best, theory provides a framework for explaining real-world phenomena. As Kurt Lewin articulated as early as 1943, “there is nothing as practical as a good theory.” What’s more, theory is not solely the province of researchers: policymakers and practitioners also have theories of action for how the world works and what it takes to bring about change.

What sets theory apart in academia is a dual emphasis on explicitly articulating theory so that it can be openly discussed and debated, and on systematically and empirically testing theory. As the research community seeks to bring the best that social science has to offer to the task of addressing real world problems, we should not leave theory behind. Rather, we should embrace theory and bring it to the table in our discussions with policymakers and practitioners. We should also do more to elicit their theories of action and line them up against theory generated in the social sciences. And, ultimately, we should empirically test theories from different sources so that we are building more robust explanatory frameworks to understand and respond to social problems.

We should empirically test theories from different sources so that we are building more robust explanatory frameworks to understand and respond to social problems.

This is not to say that theory is always useful. A common complaint lodged by policymakers and practitioners is that research is “too theoretical,” with the implication being that the work is too abstract to be applied to their work. In our view the problem lies not with theory per se, but with “theoretical talk” that is too distanced from the real-world problems faced by practitioners and policymakers. This challenge is heightened when social science jargon is divorced from the commonplace language used by those professionals, and when theoretical concepts are not brought down to earth in ways they perceive as relevant to their work.

Defining Relevance

All good research starts with asking the right questions, and this is no less true when it comes to producing relevant research. By relevance, we mean “a problem or question that is important, whose answers matter” to persons outside the confines of the research community. To develop research agendas that are useful to practitioners or policymakers, it is important to understand the problems that they are trying to solve. Researchers can start by identifying which decision makers they want to engage, and then mapping backwards to the questions and dilemmas that are most relevant to those decision makers. Policy-oriented researchers often seek to inform federal and state policymakers: some focus more on the development of policies by legislatures while others focus on the implementation and regulation of policy by executive branch agencies. Still other researchers focus on local policymakers and the agency leaders and administrators who play a critical role in designing professional development for frontline staff, administering programs, shaping implementation, and allocating resources for child and youth services.

Read More About Conceptual Uses Of Research In Policy And Practice:

Relevant research does not need to align with the particular ways decision makers have defined problems of practice or policy. As Carol Weiss argued, social science research can serve an “enlightenment” function by influencing the ways problems are formulated and framed, thereby pointing policymakers to certain types of solutions and not others. Relevant research can challenge prevailing policy or practice ideas, but to succeed, studies must grapple with existing notions. Relevant research must be in dialogue with existing ways of defining problems and solutions, not in a separate academic dialogue. For example, the University of Chicago Consortium on School Research’s work on high school dropout was powerful because it contradicted popular notions of why students drop out of school. Dropout was often attributed to students’ background (poverty, family factors, etc.) and those explanations for dropout shaped discussions about what schools could do. The research, however, found that two indicators—number of course credits and number of course failures in ninth grade—were stronger predictors of graduation than students’ background characteristics and even their prior achievement scores. This analysis challenged the prevailing ways that educators and other stakeholders understood the problem and caused them to focus their improvement efforts on those two indicators.

Combining Rigor and Relevance

Fortunately, we are not without exemplars of research that is both rigorous and relevant. Robert Pianta, Bridget Hamre, Joseph Allen, and their colleagues at the University of Virginia, for instance, have produced a program of research that has spanned descriptive, measurement, and intervention studies to improve student outcomes and in some cases narrow racial and socioeconomic achievement gaps.

A centerpiece of the research is the Classroom Assessment Scoring System (CLASS), a measurement tool that assesses the quality of classroom instruction along three dimensions: emotional support, instructional support, and classroom management (Pianta, LaParo & Hamre, 2008). This measure, which has consistently demonstrated strong psychometric properties, drew on prior theoretical and empirical work on the nature of supportive interactions between young children and their adult caregivers and then later adapted for adolescents based on developmental science about their needs.

In later years, the program of research was extended to intervention studies to empirically test theoretical notions about what it takes to improve teaching and learning. First in the early grades and then later in high school, Pianta, Hamre, Allen, and their colleagues used the CLASS dimensions as anchors for a professional development program for teachers, including a web-mediated coaching intervention, MyTeachingPartner (MTP: Allen et al; 2011; Pianta, Mashburn, Downer, Hamre, Justice, 2008) and a college course. MTP teachers video-record their classrooms and receive feedback from coaches who use the CLASS as a framework to help teachers identify areas of their practice that need improvement and guide them in changing their daily interactions with students. In multiple randomized-controlled trials, the MTP and MTP-Secondary interventions improved teaching quality and academic achievement, reduced racial disparities in disciple referrals, and yielded more positive peer interactions. Efforts to test these interventions at scale by practitioners have shown positive impacts and have demonstrated the potential value of these approaches beyond university-initiated randomized trials.

The Current Climate and a Look Ahead

Around the globe, a focus on research relevance has risen to the surface over the past decade. In a time of economic austerity, governments are asking tougher questions about the impact of public investments in research. In the United Kingdom, these concerns have generated attention to assessing research impact, a concept that overlaps with what our Foundation has talked about as the use of research evidence. The U.K.-based Economic and Social Research Council (similar to the National Science Foundation in the U.S.) requires that grant applications include a section that describes a pathway to impact.”

In that section of the grant application, “researchers are encouraged to:

  • identify and actively engage relevant users of research and stakeholders at appropriate stages;
  • articulate a clear understanding of the context and needs of users and consider ways for the proposed research to meet these needs or impact upon understandings of these needs;
  • outline the planning and management of associated activities, including timing, personnel, skills, budget, deliverables, and feasibility;
  • [provide] evidence of any existing engagement with partners or consultees.”

In the U.S., similar forces have pressed the National Science Foundation to develop its Innovation Corps program. Informed by the movement of technological innovations into the marketplace, the program seeks to “guide the output of scientific discoveries closer to the development of technologies, products, and processes that benefit society.” The National Science Foundation has long included “broader impact” as a criterion for research funding decisions, but just what constitutes impact is not well defined. At the Institute of Education Sciences, two national centers on knowledge utilization have been established to address the use of research evidence in practice and policy. And for two years, the National Institute of Justice funded a grants program that similarly supported projects to promote the use of research evidence. Under the George W. Bush and Barack Obama administrations, the Office of Management and Budget pressed federal agencies to develop and use evidence on intervention effectiveness. Currently, the Commission on Evidence-Based Policymaking, enacted in a bipartisan congressional act, aims to bring data, evidence, and policy into greater proximity.

In essence, pursuing relevance may end up making research more rigorous.

One challenge for improving research relevance is that the criteria for judging relevance are not clearly defined. When it comes to rigor, researchers have solid guidance to draw upon in judging the rigor of different types of research designs and methods. These include principles derived from different epistemological traditions, texts on research design and methods, and recommendations published by respected scientific bodies.2 In research training, doctoral students are required to go through a sequence of research design and methods courses. No parallel requirements exist to train students on how to develop policy- or practice-relevant studies. Some universities and some instructors incorporate such training into their courses or research mentoring, but these are more the exception than the rule.

At the level of institutions and the research profession, researchers and research funders need a better means of identifying relevant and high-priority research questions. As Tseng, Fleischman, and Quintero (in press) observe,

Research questions often arise out of researchers’ discussions and debates with each other in academic journals and at scholarly conferences. Imagine instead a world in which the research questions arose from vibrant back-and-forth exchanges between researchers and educators, as they jointly addressed the roadblocks to teaching and learning. Imagine, too, that parents, students, and community stakeholders had a say in determining the unanswered questions that future research should address. And what if the demand for evaluation was not driven primarily and punitively by policymakers, but by educators seeking knowledge to enhance their professional work and by parents and community groups invested in improving education? Setting research goals and priorities would become less an academic exercise, and more a matter of deliberation, negotiation, and compromise among diverse stakeholders. The process would likely be messier and less efficient, but it would also yield more meaningful agendas.

As more meaningful research agendas emerge, we suspect that they will reveal the need for methodological and theoretical innovations to address the new areas of inquiry. In the research community, we are accustomed to extending and testing each other’s ideas. Addressing less tread topics may push our scientific frontiers, revealing the need for new methods and stronger explanatory and predictive frameworks. In essence, pursuing relevance may end up making research more rigorous.

This is the first post in a two-part essay. Read part two next: “Promoting Research That is Both Rigorous and Relevant.”

  1. For full citations and a complete reference list, download the PDF of this essay.

In this issue

As we approach policy changes that present important opportunities for researchers, I would like to share some “meditations” about language, as well as the worlds of education policy and practice, which I have repeated to myself over the ...
Meditations on Language and Learning: ESSA in Policy and Practice
How can research to improve the lives of youth in the United States also benefit the other 95 percent of the world’s youth? Questions like this highlight the urgency of bringing the highest levels of rigor to relevant research.
Integrating Rigor and Relevance in Research: Three Questions
Digest, Issue 2: Spring 2017
Strengthening practice and policy in state and local education systems will be an essential part of reducing inequality for young people in the coming years. And a fundamental piece of the effort to raise student outcomes will be building and ...
Toward New Research and New Data
By moving accountability systems from the federal to the state level and highlighting the importance of English learner progress in accountability, ESSA provides an opportunity to develop state accountability systems that better support the learning of ELs. Assessments and data ...
Learning English in the Era of the Every Student Succeeds Act
We recognize that no single effort will be transformative, but we hope that our collective efforts as researchers, research funders, universities, and professional associations can support research that, over the long term, improves the lives of young people.
Promoting Research That Is Both Rigorous and Relevant

More Digest Issues

The Digest

Issue 9: Winter 2023-24

The ninth issue of the Digest features an update on the Institutional Challenge Grant program five years after its launch, with insights from a Foundation stock-taking effort and reflections from two grantee teams. We also share new thinking on the use of research evidence, including ways for researchers to leverage existing findings to bolster studies of effective strategies to improve research use.
The Digest

Issue 8: Winter 2022-23

The eighth issue of the Digest features insights about where new research might help identify ways to reduce inequality in youth outcomes after COVID-19, as well as how the fields of implementation science and research on research use might learn from each other in ways that yield more transformative research in years ahead.
The Digest

Issue 7: Winter 2021-22

The seventh issue of the Digest includes fresh insights from program staff and grantees on how researchers can successfully confront the challenges of studying causal mechanisms for reducing inequality, as well as how the global community of researchers and funders focused on improving the use of research evidence can continue to break new ground in the years ahead.
The Digest

Issue 6: Winter 2020-21

Essays in this issue of the Digest focus on the importance of looking beyond individual action and customary metrics in research on reducing inequality, as well as how strengths-based, race-conscious research can be produced and used to uplift communities of color.
The Digest

Issue 5: Winter 2019-20

The fifth issue of the William T. Grant Foundation Digest features insights from program staff and grantees on the importance of looking beneath the surface to consider the underlying factors that create and shape challenges that social scientists seek to address, whether they be related to reducing inequality in youth outcomes or improving the use of research evidence in policy and practice.
The Digest

Issue 4: Winter 2018/19

The fourth issue of the William T. Grant Foundation Digest features insights that may point the way toward a more nuanced understanding of evidence use and inspire new and more wide-ranging examinations of ways to reduce inequality in youth outcomes.
The Digest

Issue 3: Winter 2017/18

The third issue of the William T. Grant Foundation Digest features insights on how research on ability tracking can inform studies to improve the outcomes of English learners, as well as how researchers and school districts can partner to build learning systems based on research evidence.
The Digest

Issue 1: Summer 2016

The introductory issue of the William T. Grant Foundation Digest features essays and commentary on the value of qualitative and mixed-methods research in reducing inequality and the potential for researcher access to big data to yield useful research evidence.

Subscribe for Updates