Given the persistence of the rigor versus relevance dichotomy, researchers engaged in highly relevant research bemoan the lack of rewards for their work in the university setting.1Bogenschneider and Corbett put it aptly:
Presidents and provosts in the academy issue hortatory statements encouraging faculty to engage in public service. University administrators assert that organizing, synthesizing, and transmitting knowledge for the public good is a legitimate academic endeavor and should be so rewarded. At the end of the day, no faculty member at a research university is fooled. At best, public service might well rank even lower than undergraduate teaching in the hierarchy of valued activities at top research universities. You can do it if it is done with the effortlessness of Gods, but don’t spend too much time at it or intimate that you actually enjoy the experience. If you do, your dedication to scholarship will be questioned.
Most often, universities define research contributions narrowly and attribute work with real-world benefits to “service” rather than “research.” As a result, much of the best applied social science is generated by private or nonprofit research organizations rather than by universities. However, universities stand much to gain from enhancing the practical relevance of their scholarship. Responsiveness to public needs is an essential element in the mission of most state universities, and many private universities are increasingly challenged to be more responsive to the communities in which they reside. Recently David Leebron, the president of Rice University, an esteemed private institution in Houston, spoke at the founding of the National Network of Education Research-Practice Partnerships, an organization based at Rice University that supports collaboration between research institutions and state and local education agencies. By his presence, President Leebron emphasized the importance of this highly relevant mission, and, by his words, this university leader emphasized the value of partnerships between scholars and educators in applying research findings to school improvement. His commitment to rigor and relevance has had a tangible payoff, as the Kinder Foundation has just awarded a grant of $10.7 million to support the Houston Education Research Consortium, a research-practice partnership housed at the Kinder Institute for Urban Research at Rice University.
Universities can rethink how they evaluate the research contributions of faculty who engage in research with practical value by assessing impact differently than the usual way.
To encourage more relevant research, universities will need to broaden their criteria for judging the significance of faculty research contributions. At the same time, researchers will need to maintain rigor in order to attain value within the university reward structure. Policy and practice leaders such as school district officials and child welfare officers can help researchers develop a relevant research agenda and identify important questions, but researchers will need to bring to bear the ideas and tools of the social sciences to respond to those questions. For the research to be rigorous and impactful, and thus to merit esteem in the university setting, it must be:
- theoretical: informed by social science ideas that provide a framework for responding to the research question;
- systematic: involving data collection and analysis that meet methodological standards and are well aligned with the question; able to offer the potential for generalizability beyond the local setting, either through comparative designs, scaling up, or contributions to theory that can be tested elsewhere; and able to contribute to practical solutions by building, testing, improving, or furthering understanding of approaches to improving outcomes for youth.
Universities can rethink how they evaluate the research contributions of faculty who engage in research with practical value by assessing impact differently than the usual way. Typically, research impact is assessed with quantitative measures such as citation counts in academic publications, and with qualitative measures such as statements from research experts who can recount the contributions of the work. For research intended to impact policy or practice, universities may wish to assess impact by considering the use of research in those contexts. The use of research may be conceptual, shaping ideas and the way problems or solutions are defined; or it can be instrumental, influencing particular policy or practice decisions. Oftentimes research does not lead to an immediate change in policy or practice, but it has nevertheless been considered and weighed in the decision process (evidence-informed decision making). Like contributions to a research literature, contributions to practice or policy settings can be measured quantitatively, for example, by counting mentions of the research in legislative hearings or news media, or qualitatively, by seeking letters from impartial experts who can speak to its impact in policy, practice, or broader public discourse. Thus, contributions to local, state, and national debates over policy decisions may be regarded as demonstrations of impact, complementary to the appearance of research findings in academic journals or books. These considerations speak not just to the ways university faculty members provide service to their communities, but to the importance and value of the research they conduct.
Professional associations often develop policies for evaluating research in their respective fields, and they could play a leadership role in helping universities understand how to evaluate research that is relevant as well as rigorous.
While some universities have already moved in this direction, an impetus for further advancement could come from professional associations within the social sciences, such as the American Psychological Association, the American Sociological Association, the Society for Research in Child Development, the American Educational Research Association, and the like. These organizations often develop policies for evaluating research in their respective fields, and they could play a leadership role in helping universities understand how to evaluate research that is relevant as well as rigorous. For example, Michael Burawoy, in his presidential address to the American Sociological Association, called for “public sociology,” arguing that sociology was compelled to engage with a broader public beyond the academy about the social issues that form the core content of sociology such as inequality and power. Burawoy distinguished between public sociology and “professional sociology,” the ordinary activities of the academic discipline, accepting that each has its value, and stating that public sociology must rest on a strong foundation of professional sociology. Burawoy did not, however, offer suggestions for how universities could strengthen their support for public sociology. An early critique of the notion of public sociology argued that it may fail in part because professional sociologists lacked incentives to pursue it and, unlike professional sociology, public sociology lacked “tangible measures of success.”
In another example, the American Educational Research Association has considered the question of how “engaged scholarship” should be evaluated as contributions to research in postsecondary education, and has dedicated its centennial anniversary conference to “public scholarship.” Engaged scholarship rests on principles of reciprocity, respect for mutual expertise, and research for the public good; it “is shaped by theoretical and practical understandings of community-based problems, and responds to problems of educational policy and practice.” AERA challenges departments in schools of education to identify what constitutes engaged scholarship and how to identify its quality of impact—an essential first step toward fostering the production of relevant research.
Recently, our Foundation launched the Institutional Challenge Grant, an annual award that will provide funding for a research institution to strengthen its engagement with policy and practice.
Universities, professional associations, and research funders have the opportunity to encourage and facilitate efforts to bring rigor to relevant questions. Recently, our Foundation launched the Institutional Challenge Grant, an annual award that will provide funding for a research institution to strengthen its engagement with policy and practice. Through this grant program, we hope to support leaders who will boldly tackle the challenge of creating research cultures that will bring the highest levels of rigor to tackle the most important social problems in our communities. The grant encourages research institutions to partner with state and local government agencies and local youth-serving non-profit organizations to work together to construct meaningful research agendas. It challenges leaders at research institutions to think more about how to incentivize researchers to engage in this work, how to evaluate the work in terms of both rigor and relevance, and how to build the capacities of agency partners to use research evidence.
Conclusions
Many researchers perceive they will need to sacrifice rigor if their research is to take on greater relevance. Our view is that it is not a zero sum game; rigor and relevance can be complementary. Indeed, one might argue that there is a moral imperative to bring as much rigor as we can muster to the most pressing social problems. If policymakers and practitioners are advised to heed rigorous research, then researchers must rise to the challenge of providing credible findings on policy and practice questions. It would be a shame if those who make decisions about public services are left to rely on research that is too often either of questionable credibility or of little relevance to their work. They should not have to choose between rigorous versus relevant research.
We do not have all the answers to the questions we have raised, but as a research funder, we also have a role to play. Over the years, the William T. Grant Foundation has frequently emphasized the importance of linking research, policy, and practice. For years, we’ve focused research funding on understanding and improving the use of research evidence; we’ve also supported a vibrant community of research-practice partnerships, and have now launched a program to encourage research institutions to create the incentives and structures to support rigorous and relevant research. We hope that leading research institutions as well as professional associations will join us in taking up these challenges. We recognize that no single effort will be transformative, but we hope that our collective efforts as researchers, research funders, universities, and professional associations can support research that, over the long term, improves the lives of young people.
