Social scientists too often view the concepts of relevance and rigor as competing. Through this lens, however, a rich spectrum of research methods, approaches, and purposes is reduced to monochrome binaries: either the work builds theory or practical knowledge; either it is for academics or for users in policy and practice; either it is scientific or practical. But for researchers in the social sciences to make greater contributions to the economic and social well-being of our nation, we need to complicate these binaries.1
Put simply, the dichotomy of rigor versus relevance is false. There is no inevitable trade-off between producing rigorous research and producing research with relevance for the real world. Researchers who want their work to matter in policy and practice should begin by identifying the questions of greatest relevance and then bring the highest standards of theoretical and methodological rigor to those questions. In this essay, we argue for the critical need to reach for both rigor and relevance in research to improve youth outcomes, and we urge universities and professional associations to create structures and incentives to support theoretically and methodologically rigorous research that addresses pressing policy and practice issues.
Defining Rigor
The term “rigor” is sometimes used as a code word for randomized controlled trials, but that is not our meaning. It is important that all types of research designs and methods be rigorous; what constitutes the rigor of a given study depends on the research questions being asked. Put another way, rigor is not a characteristic of a particular research method, but a matter of the “quality of mind in which evidence and (theoretical) insights are carefully specified and intertwined.”
Put simply, the dichotomy of rigor versus relevance is false. There is no inevitable trade-off between producing rigorous research and producing research with relevance for the real world.
In 2002, the National Academy of Education released the report Scientific Research in Education, which encapsulates much of our thinking about what constitutes rigor in social science research. Education research is an instructive example for defining rigor because it is a field focused on studying problems relevant to policy and practice; it is also a field that is sometimes accused of lacking rigor. The report identified six guiding principles for scientific inquiry:
- Pose significant questions that can be investigated empirically
- Link research to relevant theory
- Use methods that permit direct investigation of the question
- Provide a coherent and explicit chain of reasoning
- Replicate and generalize across studies
- Disclose research to encourage professional scrutiny and critique
Like the National Academy of Education, we define rigor as including attention to theory as well as to methods. Rather than considering theory to be antithetical to practical applications, we maintain that theoretical insight is essential for carrying out a rigorous study. At its best, theory provides a framework for explaining real-world phenomena. As Kurt Lewin articulated as early as 1943, “there is nothing as practical as a good theory.” What’s more, theory is not solely the province of researchers: policymakers and practitioners also have theories of action for how the world works and what it takes to bring about change.
What sets theory apart in academia is a dual emphasis on explicitly articulating theory so that it can be openly discussed and debated, and on systematically and empirically testing theory. As the research community seeks to bring the best that social science has to offer to the task of addressing real world problems, we should not leave theory behind. Rather, we should embrace theory and bring it to the table in our discussions with policymakers and practitioners. We should also do more to elicit their theories of action and line them up against theory generated in the social sciences. And, ultimately, we should empirically test theories from different sources so that we are building more robust explanatory frameworks to understand and respond to social problems.
We should empirically test theories from different sources so that we are building more robust explanatory frameworks to understand and respond to social problems.
This is not to say that theory is always useful. A common complaint lodged by policymakers and practitioners is that research is “too theoretical,” with the implication being that the work is too abstract to be applied to their work. In our view the problem lies not with theory per se, but with “theoretical talk” that is too distanced from the real-world problems faced by practitioners and policymakers. This challenge is heightened when social science jargon is divorced from the commonplace language used by those professionals, and when theoretical concepts are not brought down to earth in ways they perceive as relevant to their work.
Defining Relevance
All good research starts with asking the right questions, and this is no less true when it comes to producing relevant research. By relevance, we mean “a problem or question that is important, whose answers matter” to persons outside the confines of the research community. To develop research agendas that are useful to practitioners or policymakers, it is important to understand the problems that they are trying to solve. Researchers can start by identifying which decision makers they want to engage, and then mapping backwards to the questions and dilemmas that are most relevant to those decision makers. Policy-oriented researchers often seek to inform federal and state policymakers: some focus more on the development of policies by legislatures while others focus on the implementation and regulation of policy by executive branch agencies. Still other researchers focus on local policymakers and the agency leaders and administrators who play a critical role in designing professional development for frontline staff, administering programs, shaping implementation, and allocating resources for child and youth services.
Relevant research does not need to align with the particular ways decision makers have defined problems of practice or policy. As Carol Weiss argued, social science research can serve an “enlightenment” function by influencing the ways problems are formulated and framed, thereby pointing policymakers to certain types of solutions and not others. Relevant research can challenge prevailing policy or practice ideas, but to succeed, studies must grapple with existing notions. Relevant research must be in dialogue with existing ways of defining problems and solutions, not in a separate academic dialogue. For example, the University of Chicago Consortium on School Research’s work on high school dropout was powerful because it contradicted popular notions of why students drop out of school. Dropout was often attributed to students’ background (poverty, family factors, etc.) and those explanations for dropout shaped discussions about what schools could do. The research, however, found that two indicators—number of course credits and number of course failures in ninth grade—were stronger predictors of graduation than students’ background characteristics and even their prior achievement scores. This analysis challenged the prevailing ways that educators and other stakeholders understood the problem and caused them to focus their improvement efforts on those two indicators.
Combining Rigor and Relevance
Fortunately, we are not without exemplars of research that is both rigorous and relevant. Robert Pianta, Bridget Hamre, Joseph Allen, and their colleagues at the University of Virginia, for instance, have produced a program of research that has spanned descriptive, measurement, and intervention studies to improve student outcomes and in some cases narrow racial and socioeconomic achievement gaps.
A centerpiece of the research is the Classroom Assessment Scoring System (CLASS), a measurement tool that assesses the quality of classroom instruction along three dimensions: emotional support, instructional support, and classroom management (Pianta, LaParo & Hamre, 2008). This measure, which has consistently demonstrated strong psychometric properties, drew on prior theoretical and empirical work on the nature of supportive interactions between young children and their adult caregivers and then later adapted for adolescents based on developmental science about their needs.
In later years, the program of research was extended to intervention studies to empirically test theoretical notions about what it takes to improve teaching and learning. First in the early grades and then later in high school, Pianta, Hamre, Allen, and their colleagues used the CLASS dimensions as anchors for a professional development program for teachers, including a web-mediated coaching intervention, MyTeachingPartner (MTP: Allen et al; 2011; Pianta, Mashburn, Downer, Hamre, Justice, 2008) and a college course. MTP teachers video-record their classrooms and receive feedback from coaches who use the CLASS as a framework to help teachers identify areas of their practice that need improvement and guide them in changing their daily interactions with students. In multiple randomized-controlled trials, the MTP and MTP-Secondary interventions improved teaching quality and academic achievement, reduced racial disparities in disciple referrals, and yielded more positive peer interactions. Efforts to test these interventions at scale by practitioners have shown positive impacts and have demonstrated the potential value of these approaches beyond university-initiated randomized trials.
The Current Climate and a Look Ahead
Around the globe, a focus on research relevance has risen to the surface over the past decade. In a time of economic austerity, governments are asking tougher questions about the impact of public investments in research. In the United Kingdom, these concerns have generated attention to assessing research impact, a concept that overlaps with what our Foundation has talked about as the use of research evidence. The U.K.-based Economic and Social Research Council (similar to the National Science Foundation in the U.S.) requires that grant applications include a section that describes a pathway to impact.”
In that section of the grant application, “researchers are encouraged to:
- identify and actively engage relevant users of research and stakeholders at appropriate stages;
- articulate a clear understanding of the context and needs of users and consider ways for the proposed research to meet these needs or impact upon understandings of these needs;
- outline the planning and management of associated activities, including timing, personnel, skills, budget, deliverables, and feasibility;
- [provide] evidence of any existing engagement with partners or consultees.”
In the U.S., similar forces have pressed the National Science Foundation to develop its Innovation Corps program. Informed by the movement of technological innovations into the marketplace, the program seeks to “guide the output of scientific discoveries closer to the development of technologies, products, and processes that benefit society.” The National Science Foundation has long included “broader impact” as a criterion for research funding decisions, but just what constitutes impact is not well defined. At the Institute of Education Sciences, two national centers on knowledge utilization have been established to address the use of research evidence in practice and policy. And for two years, the National Institute of Justice funded a grants program that similarly supported projects to promote the use of research evidence. Under the George W. Bush and Barack Obama administrations, the Office of Management and Budget pressed federal agencies to develop and use evidence on intervention effectiveness. Currently, the Commission on Evidence-Based Policymaking, enacted in a bipartisan congressional act, aims to bring data, evidence, and policy into greater proximity.
In essence, pursuing relevance may end up making research more rigorous.
One challenge for improving research relevance is that the criteria for judging relevance are not clearly defined. When it comes to rigor, researchers have solid guidance to draw upon in judging the rigor of different types of research designs and methods. These include principles derived from different epistemological traditions, texts on research design and methods, and recommendations published by respected scientific bodies.2 In research training, doctoral students are required to go through a sequence of research design and methods courses. No parallel requirements exist to train students on how to develop policy- or practice-relevant studies. Some universities and some instructors incorporate such training into their courses or research mentoring, but these are more the exception than the rule.
At the level of institutions and the research profession, researchers and research funders need a better means of identifying relevant and high-priority research questions. As Tseng, Fleischman, and Quintero (in press) observe,
Research questions often arise out of researchers’ discussions and debates with each other in academic journals and at scholarly conferences. Imagine instead a world in which the research questions arose from vibrant back-and-forth exchanges between researchers and educators, as they jointly addressed the roadblocks to teaching and learning. Imagine, too, that parents, students, and community stakeholders had a say in determining the unanswered questions that future research should address. And what if the demand for evaluation was not driven primarily and punitively by policymakers, but by educators seeking knowledge to enhance their professional work and by parents and community groups invested in improving education? Setting research goals and priorities would become less an academic exercise, and more a matter of deliberation, negotiation, and compromise among diverse stakeholders. The process would likely be messier and less efficient, but it would also yield more meaningful agendas.
As more meaningful research agendas emerge, we suspect that they will reveal the need for methodological and theoretical innovations to address the new areas of inquiry. In the research community, we are accustomed to extending and testing each other’s ideas. Addressing less tread topics may push our scientific frontiers, revealing the need for new methods and stronger explanatory and predictive frameworks. In essence, pursuing relevance may end up making research more rigorous.
This is the first post in a two-part essay. Read part two next: “Promoting Research That is Both Rigorous and Relevant.”
