Say you wake up tomorrow as the leader of a school district. On your desk when you get to your office is an ambitious to-do list and a full schedule of meetings. With the school board, staff, teachers, students, and families counting on you to deliver, you and your colleagues are expected to make decisions that can help improve instruction, encourage student achievement, provide support and professional development to teachers, and, in general, create an environment rich with learning—fertile ground for positive outcomes. At the same time, policy mandates in your state mean that practices in your district must be based on evidence, and impacts must meet determined metrics.
Looking up from your desk, you see your research partner walk into your office. Yes, your district is part of a research-practice partnership with a top university, which is focused on producing evidence that is relevant to your context, your questions, and your priorities. With pressure to deliver results and institute programs and practices informed by research, you are relieved to have such a resource by your side. And yet, as you look to your calendar, to your to-do list, and to the messages in your inbox, you see that things aren’t exactly coming together as you thought. The research isn’t being discussed, questioned, or used. Implementation is challenging. You are being pulled into meetings to put out new fires. Your colleagues are unsure of which way to turn. Parents are wondering about your decisions. What do you do now?
###
There was a time when people thought that research would be used in policy and practice if it were methodologically sound—if it were rigorous. Over time, the conversation has gradually shifted toward research relevance, or the degree to which a project’s design, questions, methods, and findings are applicable to a specific set of issues. We at the Foundation, and others, including the Spencer Foundation and the Institute of Educational Sciences, have championed research-practice partnerships as a means of producing relevant research, and we’ve been pleased by the progress that the growing field of RPPs has made toward this goal. But, recently, we’ve come to the edge of an even longer horizon. As articulated by my colleague Vivian Tseng, “Creating and communicating research that is relevant, timely, and actionable is a feat unto itself. But, on its own, it is not enough to foster an education system that is able to learn from and incorporate research findings into policy changes, professional development, curricula, and teaching and learning.” With this in mind, we ask now, what does it take?
This question is at the heart of recent conversations I had with three current and former school district leaders about the lessons they’ve learned throughout careers that have centered on using research to improve education. I wanted to know: take for a given that the research is relevant and rigorous, that your research partner is accessible, that the findings are sound. What then? What are the actual conditions and processes of research use in school districts? Who is involved? What are some of the possible variables and bumps in the road? Below are excerpts from these conversations, the insights from which may prove instructive for current and future partnerships alike.
Designed for Users, Designed for Use: Starting with a Plan for Actionable Evidence
Ritu Khanna, Chief of Research, Planning, and Assessment, San Francisco Unified School District: The whole research question needs to be actionable from the start. It can’t be the sort of situation where the research is done, then you have findings, then you think about how to make the findings actionable. The range of possibilities has to be understood. At the very beginning, it has to be a matter of “let me know more; let me know about a complex practical world.”
A theory of action, or a theory of applicability within a school district, has to take into account, right from the beginning, a question that is actionable.
Tonya Wolford, Chief of Evaluation, Research and Accountability, the School District of Philadelphia: We’re contacted almost every day by someone who wants to do research. Ideas are constantly coming in. There’s always a chance that one of those ideas will happen to align with something that we’ve just discussed in a meeting, or something that we’ve been dealing with here, and it just comes together.
More often, though, it’s through relationships with researchers who are regularly inquiring about what the district’s priorities are or are looking into the priorities themselves in a very proactive way and proposing things that fit their experience and interests, but are also relevant to our concerns.
Michael Sorum, Former Deputy Superintendent of Leadership, Learning, and Student Support Services, Fort Worth Independent School District: A positive relationship and the alignment of research questions and the needs of the district: these factors are absolutely necessary.
TW: There is a point where a researcher will have a general idea, and we’ll let them know really where our interests lie at that time, and the research is then shaped through that iterative process, in collaboration with the district.
RK: On the district side, we need a research question that takes into account the whole scope of practice; a question that considers the whole range of actions that are available at the district and tells us which are possible and which are not.
MS: In my experience working with the MIST partnership, the design of the overall project greatly assisted with the findings being integrated into our work, more so than would have been typical of research, say, that we picked up off the shelf.
TW: It’s always high-quality research; it’s just done at a different cadence and with different kinds of questions in mind.
Beyond Production: The Processes and Interactions of Evidence Use in School Districts
RK: There’s complexity on both sides, and we’re speaking in parallel languages, really. Researchers work in a complex and technical language, and practitioners work in a complex and tactical language. But if we try to understand questions that arise from the tactical complexity of practice and can be answered technically, there is a lot more promise, a lot more ease of use.
TW: It goes two ways: both the research and practice side of a partnership have an outside entity stepping in to influence the way they work. And the work is shaped by the collaboration—it’s sort of a co-dependent relationship.
MS: The researchers were attuned to our needs and our contexts, and they had a plan for how to work with us, which involved, a lot of times, how to get us all to the table to have some hard conversations.
TW: The research work has to be integrated with the day-to-day steps—the day-to-day realities and ways of operating. That’s when it works the best: When the researchers can meet every week with the district; when they’re able to meet with teachers, or parents, or principals; when they’re willing to come to a professional development workshop and give a tailored presentation, not the usual talk they’ve given at a handful of conferences already.
It has to be long term—while we need things that are immediately actionable, we also need that long term engagement.
RK: In all of the research-practice partnerships I’ve participated in that have been successful, there have been structures and routines to elicit conversations. The more you have these conversations, and the more regular they are—say, between the research side, the central office, the individual departments, and the site level—the more you are reviewing evidence with the practitioner’s lens, the more you are reviewing results for sensibility. And with this, there is a sense of humility and understanding that guides the work.
TW: Yes, there are the rigor and relevance pieces, but then there’s the process of doing the work. Is there a district staff member available and interested in learning from the research? Is the researcher going to be accessible? Are they going to be able to integrate themselves into the day to day here? It’s not the end point that we’re looking to—we’re not thinking yet about findings—it’s really the process of the research that’s top of mind. That process, many times, has as much if not more value than the outcomes of the research.
RK: It’s an ongoing conversation, a long-term iterative process: the research continues throughout the implementation of the original equation or findings.
For example, most school districts are focused on graduation—on college and career readiness. With that, you have early warning indicators: Who are the kids that may not be ready to graduate? What is the profile? In a lot of the research that I’ve participated in, the researchers give you the equation that tells which kids are at risk, and then their work is done. The research piece is over once they give the district the equation. But, really, the process can’t stop there. Rather, it almost has to begin there.
The researcher has to become more invested in learning if their equation is practically valid. They have to be around to learn about what the district does with the findings. What actions do practitioners take, and which are more successful than others? Do they develop intervention programs? Do they develop preventative measures? To take the example I just gave you, if we have started to identify at-risk students based on the findings or the original equation, to what extent have we reduced the risk ratio? Has it changed at all?
It’s not just about coming up with findings or measures, it’s about learning how they are used. That follow-up is essential to research use, but many researchers have not found or pursued that avenue of work yet.
MS: Districts are large and complex, and accountability is spread around, and so you have different voices coming at the teachers, and this can cause traffic jams in the process of effectively implementing new strategies informed by research.
TW: The vast majority of research goes like this: a researcher establishes a question, and they come to the district to get data or get access to subjects. They go off and spend three years analyzing things, presenting at conferences, maybe writing a preliminary paper—and then four years later we have an analysis. Unfortunately, that has nothing to do with the rhythm of K-12 education.
Our district has a lot of moving parts, a lot of diversity, a lot of distance between schools. And we’re trying to enact and move on our anchor goals: early literacy, college and career readiness, and developing our teachers, school leaders, and staff. So, the scope is huge. Getting people on the same page across the system requires a ton of communication, coordination, information sharing, setting and checking in on milestones, and performance management.
MS: In Fort Worth, the project with MIST required a broad spectrum of district participants to actively discuss what the researchers were finding, and what our thoughts and ideas were around the findings. This ensured that the planning of any actions that grew out of these review sessions was grounded in specific deliverables.
As the district supervisor of the overall project, that helped me a lot—for instance, to ensure that I had some clear deliverables or actions to hold people accountable for, to keep the process moving, to be able to check in at points and see how, say, step A or step B were progressing. It just gave us some measurable landmarks, and that was part of the design.
RK: I regularly meet with school administrators and leadership teams to review end-of-year summative data. Together, we are expanding our thinking about a given situation. We are validating hypotheses, or adding depth to an insight. We are building on knowledge and inferences that the practitioners have already thought about, but we are calibrating or validating it with research. So when they tell me that the most essential thing, say, is the interaction between the student and the teacher, I can then say, actually, let’s look at the instructional core, the curriculum, and the teacher/student interaction, and let’s try to write down our practice for all three and how we monitor them. There, we can see how these things intersect in a complex system.
Research is most useful when it helps in a process of building on the practitioners’ ideas, and conforming and dis-conforming to one another’s hypotheses.
MS: For instance, we wanted to improve the quality of math instruction, which involves, to an extent, improving content knowledge among teachers. At that time, the way math was taught had grown beyond the way it was once taught and assessed, and it now required a certain depth of instruction. Teachers needed a base of knowledge not only to teach the concepts, but to identify where students were having difficulty and, ultimately, to give them the appropriate support. So, as a district, improving instruction was a big priority, and it was reflected in our curriculum. It also corresponded very nicely with what we were doing with the MIST partnership. Still, a couple years into our strategy, scores still weren’t going up, and implementation was messy, etc.
Some of the leadership directors had been principals back when teachers could largely teach math procedurally and then put kids in front of a test and have them do alright. At one point, our math director observed that a handful of leadership directors had sent out to district math teachers scripted lessons that would prepare students to perform well on the old tests, but were ultimately going to undermine the work that we were trying to do to improve instruction of complex math. It was obviously a big problem.
Everyone was upset with each other for different reasons, but working with MIST, we were able to put all of this on the table in a constructive way. We were able to put specific observations and findings from the schools on the table and discuss them across the board, and we had the kinds of conversations that led to learning. Our math director actually taught some lessons to instructional leaders, and showed them about new math instruction, and activities like this helped to bring people around. Of course, this took enormous facilitation on part of the senior leadership, and it took us a couple months to get through it. But today, I know that everyone in that process is smarter about their work today.
I know that this learning wouldn’t have been possible but for our partnership with the MIST team. It took them saying, “Hey, we saw this in these classrooms, and we saw that in those classrooms, etc. What’s happening here, and how is it related to what we’re trying to do?” Everyone, together, had to wrestle with those questions.
Looking Forward: Fostering Learning Systems at the District Level
RK: Using research to inform practice is revered, but not deeply understood. What tends to happen is that research is consulted after a decision has already been made, in order to support that decision. Because school practice is complex and localized, it is difficult to formulate one approach that works for multiple schools and contexts. We seldom get to study these nuances, which might help to better articulate what works under one set of conditions but not under another.
TW: Really, I think it begins with the district. The future of RPP success is predicated on more deliberate internal capacity building at the district level. Strengthening local research offices is a good place to start.
MS: The general capacity for districts to have staff on board that understand and value research—who know how to design interventions that are informed by that research, in addition to evaluations or accountability points around the implementation—that capacity just isn’t always there. In light of what all districts, especially urban districts, are faced with—in terms of limited resources and external pressures—it’s unfortunate.
If I were a superintendent and I wanted to set up the conditions where research would effectively be integrated into my district, the first thing I’d look for is someone who is able to get people to the table. Ideally, collaboration would be organic, and you’d have relationships, and you’d be able to convince people of the need to be there and be involved—and of course this is all necessary and part of what should go on—but there also needs to be somebody who can bring the players together. More than just corral people, that person really needs to understand how people work. At the same time, one person can’t have all the skills, so these people who are in leadership positions not only need to understand research, but be able to work with researchers who themselves are sharing that knowledge.
For instance, people can do things by compliance, but, if they don’t see the results, they’re going to go back to the way they’ve always operated. So whoever is leading and getting people together also has to simultaneously work with people and bring them around to understand the time and the resources it takes to conduct high-quality work, whether it’s research or implementation based on research. In my experience, it has to be someone who is not just giving people orders—it has to be someone who is both a leader and a partner.
TW: I started out in this space thinking that the academics were right: that we just needed to educate the practitioners so that they could take full advantage of the research. Now, I feel like I’ve done a full 180, and I really feel that research itself needs to change. So I’ve been evolving through this.
In our district, we have a well developed research office, which produces original research and evaluation work, and also pushes for the use of research in decision making across the district. Still, there’s pressure to hand a lot of the research work out to external organizations or researchers. And in one sense, that’s what RPPs are about, and it’s good to be able to leverage expertise where you can get it, but, at the same time, it’s extremely valuable to have both the data in house as well as the capacity to use the data. Only then can you really have an effective RPP.
MS: Learning requires the capacity to ask deep questions and investigate beyond the black and white. When we worked with MIST, they brought that capacity with them. And the processes that we put into place as part of that partnership led people through that thinking and the discussions that came out of it. And you can’t do that kind of work through decree, you have to be willing and able to work with people, and accept that it takes time. If you want it to be meaningful, that has to be a part of it.
There are a variety of sources that have to come together. It takes insight and smarts, on both the part of the researchers and the funders of the research. The funders, for instance, must have to be aware that there are multiple ways of finding answers; there has to be that kind of openness and context there.
TW: In the past, researchers might come in without that collaboration piece or without the interest in long term relationships. But the demand for relevant work has shaped some parts of the research field to the extent that there is some competition now to do this kind of work. In 20 years, academia might look a little different, especially K-12 education research.
MS: The responsibility goes both ways. The partner coming into the district has to be ready to play a significant role in teaching the district, or identifying someone in the district who can create the space for the conditions for learning to thrive. At the same time, the district has a responsibility to articulate their priorities or challenges. Someone needs to be informed and empowered to say “OK, there are no overnight solutions, and this is going to take some time, but if we follow through on this we’ll know more about how to do better.”
I am optimistic. I’ve seen it. I know for a fact that it can happen. People can come a long way in terms of their understanding of the complexity of what it takes to effect positive change.
