As a lifelong science geek, I’ve always thought research was fascinating, but I never thought it would inspire much political interest. Current events suggest that I may have been wrong.
Research evidence is increasingly at the center of political and policy debates, and much of the federal focus on building and using research evidence has embraced a “What Works” agenda. Even as Congress and the Obama administration are debating various initiatives to use research evidence of what works in administering federal programs, the administration is encouraging executive departments and agencies to apply behavioral science insights in their work. Advocates are proposing legislative language for defining “evidence-based” in the reauthorization of the Elementary and Secondary Education Act. And, in a rare alliance, Representative Paul Ryan (R-WI) and Senator Patty Murray (D-WA) have partnered to introduce the Evidence-based Policymaking Commission Act, which, if approved by the Senate, would identify ways that data and research can improve public policy.
How did we get here?
Numerous forces over the past 15 years have shaped both the supply and demand for What Works evidence. On the supply side, agency leaders, advocates, and researchers sought increased rigor in evaluating the effectiveness of social interventions. In leading the Institute of Education Sciences from 2002-2006, for example, Russ Whitehurst promoted standards of evidence that privileged the use of randomized controlled trials to evaluate the effectiveness of programs and practices in education. Similar evidence standards were promoted in child welfare, mental health, criminal justice, and youth development. At the same time, Congressional set-asides and agency investments expanded the funding for rigorous evaluations, and What Works clearinghouses were launched to disseminate the evaluation evidence in education, criminal justice, child welfare, mental health, and substance abuse.
On the demand side, both the Bush and Obama administrations sought to bolster the use of What Works evidence in executive branch agencies. “Spending more on what works and less on what doesn’t” became a catchphrase for policy initiatives that directed public dollars to programs with evidence of effectiveness, preferably generated from randomized-controlled trials. The financial crisis of 2008, and the period of austerity that followed, allowed policymakers to attach stronger incentives to What Works evidence. Since the recession, for instance, the federal government has invested over six billion dollars in “tiered evidence” grantmaking initiatives. In a tiered evidence design, interventions with more rigorous evidence of impact are eligible for the most funding, while interventions with less rigorous or emerging evidence are eligible for smaller grants, with the intention that interventions will ascend tiers as they demonstrate impacts. Federal tiered evidence initiatives include the Investing in Innovation Program, Social Innovation Fund, Evidence-based Home Visitation Program, Evidence-based Teen Pregnancy Prevention Program, Workforce Innovation Fund, Trade Adjustment Assistance Community College and Career Training Grants Program, and the First in the World Initiative.
Where are we going
Now is an opportune time to take stock of these tiered evidence initiatives and the broader What Works agenda. Much-anticipated evaluation findings from the initiatives will be released over the next year, and a new administration and Congress will need to decide whether to continue these initiatives and in what form. We are approaching a crossroads. In navigating through it, policymakers and researchers will need to confront several issues:
- What counts as evidence? The most contested issue is what is considered “good” evidence. What Works evidence—defined as evaluation findings gleaned from randomized controlled trials—is currently at the top of the evidence hierarchy. Stronger research evidence, more confidence in the findings—what could be wrong with that? In research, as in life, things are more complicated. Evidence on What Works typically reflects the average impact of an intervention and its effects in the places where it was evaluated. Local decision makers say that that evidence is only somewhat useful. They don’t just want to know whether an intervention works on average, or somewhere else. They need to determine whether a program, curricula, or professional development strategy will work in their local context and for their communities. In essence, they want to know what will work for whom and under what conditions.They also want strong evidence that can inform their implementation decisions. How much staffing is required? How much training? How should resources be allocated? What alignment is needed with existing programs? While rigorous What Works evidence may inspire confidence among the federal policymakers and agencies who invest in programs, knowledge about context and implementation is critical for making What Works work in local communities. Yet building rigorous evidence on context and implementation has received remarkably little attention.
- What about improvement? The federal evidence agenda has maintained a laser beam focus on using What Works evidence to make funding decisions. But with some analysts indicating that as many as 90% of RCTs generate weak or no positive effects, we face a dilemma. Will enough programs pass the bar? What happens to those that do not? And what should we make of evidence that is mixed or inconclusive? Rather than being used for a simple thumbs-up or thumbs-down decision, evidence could be harnessed to support improvement goals. To get there, however, we will need to determine what an improvement agenda would look like and how the federal government can support improvement.
- What about systems? While the federal government has focused largely on evidence-based programs, the leaders of state and local human service and education agencies are more often concerned with systems. A narrow focus on evidence-based programs encourages people to run after silver bullet solutions that are not necessarily aligned with the myriad other interventions that they are running. Moreover, little support exists to ensure that the evidence-based programs that are adopted fit the key problems systems confront, and don’t create unintended negative consequences.
- What is the federal role? What about states, localities, practitioners? Finally, while making remarkable strides, the What Works movement in the US has taken a largely top-down approach. The role for states, localities, and practitioners is not well articulated, and the existing approach has caused a not-insignificant amount of tension with those who feel at the mercy of decision makers in Washington DC. Teachers and service providers on the front lines often feel that evidence is something done “to” them, and here they have a point. The past 15 years have not created a meaningful role for practitioners in building evidence agendas. Instead evidence agendas have been largely under the province of policymakers and researchers. As we look to the future, we will need to consider the federal government’s role in building and using evidence. If there remains a significant federal role—which we believe there should—then we need to ensure that future initiatives better meet the needs of states, localities, and practitioners.
What is the future of evidence-based policy?
We are at a crossroads in evidence-based policy. Federal evidence initiatives can be strengthened, but doing so requires the will and the patience to learn from the work thus far. Otherwise, evidence-based policy will likely recede into the background as yet another policy fad that came and went. To move forward, let’s take a good hard look at the current evidence initiatives and identify what can be learned from them. We will need to come to terms with outsized expectations, develop ways to improve programs and systems, and determine how the federal evidence agenda can better align with state, local, and practice needs.
Please join us for Evidence at the Crossroads as the science geeks and the policy wonks debate how research can be leveraged to improve social policy and benefit the public. Read other posts in the series, and join the conversation by tweeting #EvidenceCrossroads.
