Digest, Issue 5: Winter 2019-20

Centering the Margins: (Re)defining Useful Research Evidence Through Critical Perspectives

For over a decade, the William T. Grant Foundation has sought to support new research on ways to promote the use and usefulness of research evidence to improve youth outcomes.1 Toward this end, we have funded numerous studies aimed at identifying and testing strategies to improve the use of research evidence (URE) in policy and practice decisions that affect young people. Another line of inquiry for studies in this focus area involves identifying and testing strategies that encourage the production of more useful research evidence.

In the URE focus area, definition is key. We have defined research evidence as a type of evidence derived from applying systematic methods and analyses to address a predefined question or hypothesis. This includes descriptive studies, intervention or evaluation studies, meta-analyses, and cost-effectiveness studies conducted within or outside research organizations. Following the theoretical contributions of Weiss (1977; Weiss, Murphy-Graham, & Birkeland, 2005) and Nutley, Walter, and Davies (2007), we have also defined multiple forms of research use (Tseng, 2012). Conceptual use, for example, describes the influence of research evidence on how decision makers think about problems and potential solutions. Instrumental use refers to the direct application of research evidence to a policy or practice. Political use, sometimes called tactical or strategic use, describes the deployment of research to support particular positions, which are sometimes but not always preexisting. Imposed use refers to use mandated by government initiatives to promote evidence-based programs and practices. And process use describes the learnings gleaned by practitioners when they engage in research production. Finally, our definition of usefulness has most often rested with the needs of decision makers—that is, with respect to usefulness, we have prioritized practice- and policy-relevant studies that take into consideration the needs of decision makers.

But who really gets to decide what it means to “use” research evidence, and who gets to decide what it means to “improve” the use of evidence? Relatedly, who determines what research is most “useful” to decision makers? Who are the other stakeholders for whose aims research evidence would be useful? Would the people whose lives are impacted by the research-informed policies and practices implemented by decision makers agree with the definitions of use, usefulness, and improvement we have proposed? These are the kinds of questions with which critical theoretical perspectives invite us to grapple, yet, as Gitomer and Crouse (2019) point out in their monograph, Studying the Use of Research Evidence: A Review of Methods, few studies of evidence use have adopted critical theories to frame research questions, methods, and interpretation of findings.

But who really gets to decide what it means to “use” research evidence, and who gets to decide what it means to “improve” the use of evidence? Relatedly, who determines what research is most “useful” to decision makers?

As a human endeavor, research is inextricably implicated in the societal structures and systems that have served to maintain power hierarchies and accept social inequity as a given. Indeed, research has been historically and contemporaneously (mis)used to justify a range of social harms from enslavement, colonial conquest, and genocide, to high-stakes testing, disproportionality in child welfare services, and “broken windows” policing (Au, 2016; Jerrim & de Vries, 2017; National Academies of Sciences, Engineering, and Medicine, 2018; Wells, Merritt, & Briggs, 2009). Critical perspectives offer possibilities for repairing these wrongs and for reimagining the possibilities of what research can accomplish. In this essay, I ask: What possibilities do such perspectives illuminate for rethinking the production of research with the objective of making research more useful and relevant? Here the idea of usefulness takes on a different quality than it has in the past because it interrogates and centers how well research evidence communicates the lived experiences of marginalized groups so that the understanding of the problem and its response is more likely to be impactful to the community in the ways the community itself would want. I argue that this shift requires focusing on the production of research and the ways it can be reimagined. Put another way, improving the use of research evidence will require ensuring usefulness—and use—toward ends that are congruent with the goals and visions that marginalized communities have for their self-determined benefits (Tuck & Yang, 2014).

What Are Critical Theories?

Broadly speaking, critical theories interrogate societal systems and structures with particular attention to the production and reproduction of power hierarchies, social inequities, and the taken-for-granted assumptions of ideological hegemonies such as settler colonialism, patriarchy, white supremacy, capitalism, and imperialism. Examples of critical theories include postcolonial theories, Black feminist theories, and postructuralism.

Here, I will focus on Critical Race Theory (CRT) to anchor an exploration of how critical perspectives might inform the field of evidence use. One of the better-known theories in the critical tradition, CRT emerged out of but departed from Critical Legal Studies by focusing specifically on the ways in which White supremacy shapes the law and legal institutions. Delgado and Stefancic (2001) point out that in the mid-1970s, legal scholars, lawyers, and activists like Derrick Bell, Alan Freeman, and Richard Delgado had become dissatisfied that advances toward equity for Black Americans during the civil rights era were being halted or reversed; they began criticizing the liberal race discourse of Critical Legal Studies for its colorblind ideology and uncritical framing of merit as foundational to unbiased legal decision making. Arguing that this perspective had no utility for recognizing the central role of race in shaping life in the U.S. or for combatting racism, these scholars drew from radical thinkers like Sojourner Truth, Frederick Douglass, and Cesar Chavez, as well as the Black Power and Chicano movements, to offer a framework for resisting anti-Black racism. This work, which comprised contentious interrogations of the taken-for-granted assumptions of Critical Legal Studies, gained momentum in the decade to follow as more legal scholars, like Linda Greene and Neil Gotanda, and law students, like Mari Matsuda and Kimberlé Crenshaw, began taking up and iterating on the early work of Bell and others. By 1989, Crenshaw and fellow legal scholar Stephanie Phillips would organize organic and informal gatherings of “Crits” insistent on giving racial analysis a place in Critical Legal Studies through the founding of the “Critical Race Theory Workshop,” thus crystallizing the CRT movement. Crenshaw recounted how she and the other workshop organizers settled on the terminology: “We would signify the specific political and intellectual location of the project through ‘critical,’ the substantive focus through ‘race,’ and the desire to develop a coherent account of race and law through the term ‘theory’” (Crenshaw, 2002, p. 1351).2

Critical theories interrogate societal systems and structures with particular attention to the production and reproduction of power hierarchies, social inequities, and the taken-for-granted assumptions of ideological hegemonies

CRT fundamentally assumes that race is a social construction3—meaning that it has no basis in biology, but that it is used as a tool to create and maintain a racial hierarchy wherein people who are defined as “White” are seen as superior to people who are “not White.” Another fundamental assumption of the theory is that racism is endemic to life in the United States. CRT has been adopted, expanded, and applied in such fields as education, political science, psychology, and sociology, becoming what Delgado and Stefanic call the “CRT Movement: . . . A collection of activists and scholars interested in studying and transforming the relationship among race, racism, and power” (p. 2). CRT has also inspired scholars to iterate focused intersectional analyses of socially constructed categories that have been used to essentialize differences between people, such as disability (DisCrit), gender and sexuality (QueerCrit), race and ethnicity (AsianCrit, BlackCrit, LatCrit, TribalCrit), intersectional feminism (Critical Race Feminism), as well as interrogating Whiteness (Critical White Studies).

An important contribution to the movement has been Solórzano and Yosso’s (2002; Yosso & Solórzano, 2005) articulations of the themes of CRT that have methodological implications for social science research, namely that:

  1. Research must be designed to account for the intercentricity of race and racism with other forms of subordination. Whereas concentric shapes share the same center, with the largest shape encompassing the others, intercentric shapes are inextricably linked: One does not subsume the others, nor do they share a center, but they are indivisible from one another.
  2. The challenge to dominant ideology in research means refusing claims of objectivity and neutrality in research and instead calling out “deficit-informed research that silences, ignores, and distorts epistemologies of people of color” (Yosso & Solórzano, 2005, p. 122).
  3. CRT’s commitment to social justice calls for research that seeks to eliminate oppressions and to emancipate and empower marginalized groups.
  4. By acknowledging the centrality of experiential knowledge, CRT insists that people of color have legitimate expertise in the function of oppression and subordination that offers counternarratives to the master narratives of White supremacy, and thus that their storytelling about these experiences is legitimate and essential to understanding society.
  5. By insisting on the utilization of transdisciplinary approaches, CRT pushes traditional research beyond disciplinary boundaries to draw from knowledge in “ethnic studies, women’s studies, sociology, history, law, psychology, film, theater, and other fields” (Yosso & Solórzano, 2005, p. 123) in order to better comprehend how oppression and subordination impacts the lived experiences of people of color and other marginalized groups.

CRT and research use

In what ways might the application of these themes have implications for the production and use of research evidence that is more useful to marginalized communities—that reflects the realities of and positively benefits their lives?

It could mean that research provides a richer conceptual understanding of how groups have been oppressed and the mechanisms at play, or perhaps that research reveals ideas about ways to disrupt oppression. A simulation study, for example, might create a backward map from a given concern—say, increasing the numbers of immigrant youth graduating from high school—and use CRT to examine the evidence base on this matter. If the evidence were to illuminate mechanisms involved in creating or exacerbating issues surrounding high school graduation among immigrant youth, the evidence could be used conceptually by decision makers to inform efforts toward disrupting those mechanisms—in which case, the evidence would also be useful and beneficial to immigrant youth and their well-being. If, on the other hand, an interrogation of the evidence base reveals the evidence relies solely on outcome metrics and omits experiential knowledge, it would be less useful and would signal the need for additional research that adheres to the tenets of CRT.

In what ways might the application of these themes have implications for the production and use of research evidence that is more useful to marginalized communities—that reflects the realities of and positively benefits their lives?

Alternatively, more useful evidence might be used in an instrumental way to empower groups, perhaps through the research process itself (also an example of process use), or in the direct application of a research-informed policy or practice. A study might compare the questions that interest school principals around school discipline, such as how to reduce the number of discipline referrals, versus the questions students and families might have around school discipline, such as how discipline referrals over the course of students’ academic careers impact their life trajectories. While both sets of questions may be valid, the questions posed by principals have different objectives—ones that may or may not empower students and families to make evidence-informed decisions to shape their own futures.

More useful research evidence might also be deployed for political use in efforts to advance an idea or set of actions meant to disrupt power or interrupt policies and practices that disproportionately harm marginalized groups. For example, research evidence that reveals how programs, policies, and practices intentionally or unintentionally impede minoritized youth from thriving can support the initiatives of youth advocates and youth advocacy organizations. Eve Tuck’s Youth Participatory Action Research (YPAR) initiatives have engaged young people in research intended to illuminate their experiences with systems, structures, and policies that have harmed them and their communities, even as the work also eschews damage narratives that would deny young people of their agency, goals, and desires. In one study conducted with undocumented migrant youth in rural New York State, the research team explored how youth envisioned and planned for their futures against a backdrop of vilified migrant labor, toxic anti-immigrant rhetoric, and shifting Deferred Action for Childhood Arrivals (DACA) policy (Del Vecchio, Tommey, & Tuck, 2017). In another study with the Youth Researchers for a New Education System (YRNES), Tuck and colleagues investigated the problems and possibilities of New York City public schools with the goal of instantiating systemic change (Bacha et al., 2008). The research process first generated a problem tree documenting the systems and ideologies (roots); attitudes, goals, and policies (trunk); and visible symptoms (leaves) of the dysfunctions in the NYC public school system (Figure 1).

They then deconstructed the problem tree to explore areas of possibility for their research inquiry and action, ultimately leading to a set of recommendations for policymakers and other stakeholders:

  • Diffuse the feeling of competition in schooling by fairly distributing material resources, and effective teachers and guidance counselors.
  • Develop multiple ways for meaningful mass youth participation in school decision making, rule and policy creation and enforcement, and a reassessment of the purposes of schooling.
  • Re-imagine a school leadership model that integrates local control that is based on meaningful youth, family, and community participation (Bacha et al., 2008, p. 18).
Figure 1. The Problem Tree (Bacha et al., 2008)

Given these ideas about what makes research more useful, I now turn to considering what it would take to enact these efforts to engage marginalized communities in effectively setting research agendas and producing more useful research evidence with the long-range goal of improving the use of evidence that they themselves find useful. Using the themes listed above as a point of departure, I offer three strategies for how CRT can be used as a conceptual framework for informing studies to improve the usefulness of research evidence as I have (re)defined it here. First, more useful evidence centers a racial analysis of the usefulness of research for policy and practice. Second, the democratization of research production leads to more useful evidence. And third, transdisciplinary approaches and methods can be leveraged to improve the usefulness of research.

Strategy One: Centering Race for New Insights about the Usefulness of Research Evidence

As noted above, one fundamental assumption of CRT is that racism is endemic to life in the U.S. As such, applying a CRT lens to studying how research evidence can be made more useful means putting race at the center of inquiries around the nature and content of what research evidence exists, how it has been used historically and to what ends, and how it might be used with more equitable goals in mind.

Michener’s (2019) racialized feedback framework, which offers a tool for thinking through the role race plays in the policy feedback process, also has important parallels for thinking about the role of race in how research is produced, interpreted, and applied. Policy feedback theory is an approach from political science that argues that the policymaking process is informed by existing policies and their effects on a range of actors from everyday citizens to political elites (e.g., politicians, agencies, interest groups) (Campbell, 2018). Rather than policies being the outcomes of political processes, policy feedback theory posits that policies themselves are inputs that “can confer resources on some interest groups over others, shape views about what constitute good policies, impose budget constraints, and affect institutional capacity” (Campbell, 2018, para. 1). With respect to race, Michener points out that policies often direct resources to various racial groups unequally; that race shapes how people understand and interpret their experiences with various policies; and that interest groups emerge, coalesce, and vary in power and influence along racial lines.

We might adopt a similar approach to thinking about how research production, interpretation, and use all likewise informed by existing practices, historical legacies, and power relations that confer benefits on some interest groups over others, such that WEIRD (Western, educated, industrialized, rich and democratic) populations become the normative standard against which marginalized groups are measured, and marginalized groups are seen through lenses of deficit, difference, and deviation. For example, decades of research on children’s language development in the U.S. has taken for granted that all children and families, regardless of race or ethnicity, social class, or other factors, should aspire to adopt the linguistic patterns of middle-class or wealthy, usually White, families. As Jennifer Adair and I have noted elsewhere, “Deficit thinking sets in motion deficit-framing instead of thinking of children, families, and communities as capable, interesting, complex, and knowledgeable. Deficit views justify mistreatment, oversimplification, and stereotyping that devalue home languages and practices” (Doucet & Adair, 2018, p. 4). A vital question for researchers seeking to improve the use and usefulness of research, then, is how can studies of research use or efforts to improve such use mitigate against these dynamics?

When does race matter?

The principles at the core of Michener’s racialized feedback framework can be especially helpful in confronting challenges such as those described above. The framework addresses, for example: When is race central to the policy feedback process? Are the policies heavily disproportionate in the degree to which they distribute benefits and burdens to certain racial groups? Are the mechanisms for distributing these benefits and burdens decentralized, such that they are more likely to be disproportionate? Researchers pursuing questions related to inter-relationships of race and research use may apply a parallel approach in their own work: When is race central to the production and use of research? Is a given body of research evidence disproportionate in the degree to which it attributes desirable and deficient characteristics to certain racial groups? Does such research disproportionately distribute burden and benefit to certain racial groups? One example of research directed at such questions is Sandra Graham’s 1992 content analysis of empirical studies on African Americans published in American Psychological Association journals between 1970 and 1989. The title of the article, “Most of the subjects were White and middle class,” reveals one important finding of the study, but Graham also flagged the problematic practices of using White participants as a comparison group, conflating race and social class effects, and not accounting for the race and potential associated biases of data collectors. More recently, Polo et al. (2019) conducted a systematic analysis of 342 randomized control trials of depression over 36 years. The authors found that while reporting for ethnicity and socio-economic indicators has risen in that time, demographic groups including Asian Americans and linguistic minorities have been under-represented and treatment effects across ethnic groups have been woefully under-reported.

When is race central to the production and use of research? Is a given body of research evidence disproportionate in the degree to which it attributes desirable and deficient characteristics to certain racial groups? Does such research disproportionately distribute burden and benefit to certain racial groups?

How does race matter?

The racialized feedback framework also addresses the question: How can race be incorporated into research on the policy feedback process? Or, put another way, how does race shape the relationship between policy and polity? Here, Michener offers a structure that lays out the types and levels of feedback traditionally emphasized in the policy feedback literature and maps these components to the “prevailing research on race, ethnicity, and politics” (p. 427; see Figure 2).

Figure 2. Conceptualizing Race, Policy Feedback, and the Political Process (Michener, 2019, p. 435)

The parallel question for URE researchers would be, how can race be incorporated into research on the use and usefulness of research evidence? Or, how does race shape the relationship between research and the researched? In response, producing more useful evidence might involve examining the populations used to establish a base of evidence, and then asking whether this evidence would still hold up if a different population were at the center of the inquiry. Or it might be a project that examines strategies for producing more CRT-informed research by policy researchers. How might researchers interested in improving the usefulness of research evidence study such attempts to infuse CRT into policy research in order to make it more resonant with youth and communities of color? What processes might they put into place for such efforts? What are the conditions under which these efforts might be more or less successful? These questions also implicate the need for critical examinations of the measures and other tools used to gather evidence (e.g., IQ tests, measures of social skills, etc.).

Strategy Two: Democratizing Research Production to Improve the Usefulness of Research Evidence

Previously, I noted that attending to how research is produced has implications for its usefulness. It also has implications for studies on improving the usefulness of research evidence. Collaborative research models, such as the Youth- or Community-Based Participatory Action Research illustrated above, which are aligned with the principles of democratizing evidence (Tseng, Fleischmann, & Quintero, 2018), have been proposed as promising approaches to producing more useful research.

Indeed, collaborative models exemplify the CRT tenets of challenging the dominant ideology and the centrality of experiential knowledge, insisting that the intended beneficiaries of interventions and policies be involved in the process of developing interventions and policies, from naming the issues that need to be addressed, to shaping research questions and being involved in the research process, to determining the implications of research findings for further action. This could mean including these stakeholders at every stage of the process. For example, in my research I have used the qualitative research tool of trustworthiness known as member checking as a way to ensure that the results of my analyses and my interpretation of findings are consistent with how my participants understand their lives and how they envision improving their lives. Researchers interested in these approaches might test these assumptions, and, using CRT as a framework, also test whether collaborative models with an explicit commitment to social justice produce research that communities see as more actionable, more usable, and that they are more likely to actually use to advocate for change. This also raises questions around what type of research is seen as credible and valuable by decision makers, and what strategies might be used to persuade potential cynics of the rigor or credibility of research produced in more democratic ways.

…collaborative models exemplify the CRT tenets of challenging the dominant ideology and the centrality of experiential knowledge

Confronting power dynamics in research production

For instance, Chicago Beyond, a philanthropy that invests in community-led initiatives, community leaders, and equity-centered research that supports positive change for young people in Chicago, recently published a guidebook addressing how the power dynamics involved in research creation can be mitigated to make it more useful and impactful. As they assert, “If we do not address the power dynamic in the creation of research, at best, we are driving decision making from partial truths. At worst, we are generating inaccurate information that ultimately does more harm than good in our communities. This is why we must care about how research is created” (Chicago Beyond, 2018, p. 6). They name seven inequities, rooted in power dynamics, that stand in the way of the potential impact research can have, and that might inspire studies of improving URE through a CRT lens. These inequities can serve as an analytic frame to assess a given evidence base, and/or can serve as a self-check mechanism for researchers as they design research studies that are as useful as possible:

  1. Access: Could we be missing out on community wisdom because conversations about research are happening without community meaningfully present at the table?
  2. Information: Can we effectively partner to get to the full truth if information about research options, methods, inputs, costs, benefits, and risks are not shared?
  3. Validity: Could we be accepting partial truths as the full picture, because we are not valuing community organizations and community members as valid experts?
  4. Ownership: Are we getting incomplete answers by valuing research processes that take from, rather than build up, community ownership?
  5. Value: What value is generated, for whom, and at what cost?
  6. Accountability: Are we holding funders and researchers accountable if research designs create harm or do not work?
  7. Authorship: Whose voice is shaping the narrative and is the community fully represented? (Chicago Beyond, 2018, p. 7)
  8. Research-practice partnerships

    Finally, research-practice partnerships (RPPs) have emerged as a promising model of collaboration between those who typically conduct research and those who typically use it, with the ultimate goal of producing research evidence that is more useful and used. Historically, RPPs have been identified as “long-term, mutually beneficial collaborations that promote the production and use of rigorous research about problems of practice” (Coburn, Penuel, & Geil, 2013). Central to the functioning of successful RPPs are authentic questions that serve the needs or interests of the actors on the “use” side. To the extent that RPPs pursue joint research agendas that reflect the perspectives and realities of multiple stakeholders, such partnerships can be a potentially powerful strategy for addressing inequities by incorporating the lived experience of communities traditionally excluded from the production and use of research.

    Highlighting this potential while also noting that issues of race and class have not been central themes in the RPP literature, Calabrese Barton & Bevan (2016) argue for RPPs to “create explicit mechanisms for participants to directly confront the historical and systemic sources of racism and inequality that underlie the educational challenges they have been formed to address” (para. 2). That is, while RPPs may prove effective at contributing to greater equity in the contexts where they operate, by directing attention toward dismantling structures of racism and power, such outcomes may be more routine and impactful, especially for communities at the center of the research.

    The authors outline three initial steps for systematically incorporating these considerations in RPP work:

    1. identifying how RPP-designed approaches will be used to support/challenge normative schooling practices;
    2. mapping social and cultural histories of inequities affecting the school and community population, as a way to identify otherwise hidden systemic issues or decision points that may pertain to the improvement efforts; and
    3. expanding understanding of important stakeholders to include in the design, conduct, or dissemination of RPP work (Calabrese Barton & Bevan, 2016, para. 12).

Strategy Three: Leveraging Transdisciplinary Approaches and Methods to Improve the Usefulness of Research Evidence

Opening the door for researchers to explore insights from disciplines outside the social sciences can shed light on the lived experiences of marginalized youth and the ways those experiences have been expressed and represented first-hand. The third strategy for informing research studies to improve the usefulness of research, then, speaks to the CRT tenet regarding the utilization of transdisciplinary approaches, or the import of drawing from knowledge produced in disciplines such as literature, film, history, or law.

Transdisciplinary knowledge

I agree with Jenna Sablan (2019) that quantitative methods are not antithetical to critical inquiry. Indeed, quantitative researchers explicitly have been applying critical theories and asking critical questions using quantitative methods for at least two decades (see, e.g., Baez, 2007; Cokley & Awad, 2013; Covarrubias, 2011; Covarrubias & Vélez, 2013; Darity, 2008; Solorzano & Villapando, 1998; Stage, 2007; Stage & Wells, 2013; Teranishi, 2007; Walter & Anderson, 2013; Westmarland, 2001; Zuberi, 2001). At the heart of these explorations is a critical theory-informed stance that “the numbers never ‘speak for themselves’ and that, in fact, the numbers are given voice largely by the theoretical underpinnings upon which they rest” (Covarrubias & Vélez, 2013, p. 270). Drawing from a spectrum of transdisciplinary insights to animate such context is part and parcel of a critical approach.

Transdisciplinary critical quantitative research can serve to convey lived experience by documenting, for example, the prevalence and pervasiveness of racism and its impacts. Economist William Darity studies intergroup disparities in economic stratification. Framed in sociological and social-psychological concepts of racial bias, Darity’s work refuses explanations rooted in human capital deficiencies and instead analyzes the racial underpinnings of the uneven distribution of structural resources (Darity, 2008; Darity, Hamilton, & Stewart, 2015). In psychology, critical examinations of microaggressions and racial battle fatigue have been conducted using quantitative methods, including using structural equation modeling to study the psychological, physiological, and behavioral stress in students of color exposed to racial microaggressions (Franklin, Smith, & Hung, 2014; Smith, Hung, & Franklin, 2011).

Opening the door for researchers to explore insights from disciplines outside the social sciences can shed light on the lived experiences of marginalized youth and the ways those experiences have been expressed and represented first-hand.

One exciting development in the application of transdisciplinary knowledge, and CRT principles in particular, to quantitative work is “QuantCrit,” which Gillborn, Warmington, and Demack (2018) call a “kind of toolkit that embodies the need to apply CRT understandings and insights whenever quantitative data [are] used in research and/or encountered in policy and practice” (p. 169).

This also has implications for studies looking to improve the usefulness of research evidence, from producing more useful original evidence by constructing measures and scales from a CRT lens (Knowles & Hawkman, 2019; Sablan, 2019) to taking a fresh look at existing data and analyzing these data with youth and community goals in mind. QuantCrit has been applied in educational research to ask, “Can quantitative methods, long critiqued for their inability to capture the nuance of everyday experience, support and further a critical race agenda in educational research?” (Garcia, López, & Vélez, 2017, p. 149). The answer, in a word, is yes, and a mounting body of research is demonstrating how so (Lopez, Erwin, Binder, & Chavez, 2017; Solorzano & Ornelas, 2002; Solorzano & Villapando, 1998; and for helpful summaries of this work to date, see Knowles & Hawkman, 2019 and Sablan, 2019). Rebecca Campbell-Montalvo, for instance, (2019a; 2019b) has applied a QuantCrit frame to interrogate the use of race and ethnicity data by a school district in Florida and to call out how aggregating data can result in the erasure of racial, ethnic, and linguistic groups. The importance of data disaggregation for more precise, and, therefore, better use was also called out by Carter, Skiba, Arredondo, and Pollock (2017), and Teranishi (2007). This point about disaggregation is also relevant to my broader argument here about producing research evidence that is more useful to the youth and communities that are often the target of interventions and reforms.

Transdisciplinary methods

Beyond tapping into the well of transdisciplinary literature or materials to inform research that nevertheless uses the traditional methods of social science, researchers may also seek to utilize methods that cross disciplinary boundaries. That is, researchers might explore whether and under what conditions methods from outside disciplines can produce more useful research evidence, and even, potentially, whether these forms of research evidence are more likely to be used by decision makers. For example:

Arts-based research methods, like poetic inquiry, ethnotheater and ethnodrama, performed research, research-based theater, research poems or research poetry, verbatim theater, and autobiographical theater, may illuminate lived experience in ways traditional interviews may not (see, e.g., Hones, 1998; Prendergast & Belliveau, 2013; Santoro, Kamler, & Reid, 2001; White & Belliveau, 2011). Evidence produced through such means might be more useful to advocates seeking to convey to practitioners and decision makers how given rules and regulations in the juvenile justice system increase the likelihood of recidivism because of the ways in which they shape young people’s perceptions of themselves, distance them from family and community, or foster learned helplessness.

Oral history methods, which researchers interested in producing more useful research to inform mental health policy for youth might employ to access a fuller view of a community’s conditions over time, or ways in which those conditions might reveal systemic sources of stress, generational trauma, limited access to resources, and unintended consequences of prior policy decisions that have served to exacerbate a community’s precarity. This could be complemented by narrative inquiry or narrative archival analysis (see, e.g., the work of Saidiya Hartman) as methods to unearth forgotten or unspoken histories with contemporary consequences.

Policy discourse analysis is described by Allan and Tolbert (2019) as a method that can be used to reveal how dominant discourses shape policy problems and solutions. One phase of the multi-phase analytic process they outline includes looking for “policy silences,” illustrated by a powerful example from Bertrand, Perez, and Rogers (2015), who, the authors describe, “found that the educational attainment gaps between Whites and students from minoritized groups (percentage of graduates at postsecondary institutions) are shaped by deficit discourses that assert . . . that those most negatively impacted by inequality cause inequality” (p. 2). That is, drawing upon dominant discourses, minorities are blamed for their own lack of college degree attainment. The silence Allan and Tolbert identify here is a lack of focus on structural inequality, “built through the history of slavery and segregation that produced the inequitable K-12 systems of education today” (p. 144).

Visual methods such as photo elicitation, photovoice, and autophotography, with roots in visual studies and cultural studies, are increasingly being used by social scientists to capture lived experience from the standpoint of youth and communities. This might also be integrated with or informed by the type of image analysis that scholars like Tina Campt and Christina Sharpe conduct to amplify marginalized voices (Campt, 2017; Campt, 2019; Sharpe, 2016). Researchers have also created ethnographic films or documentary films or employed digital storytelling with participants to relay lived experience. For example, psychologist Louis Kruger’s 2009 documentary Children Left Behind, considers the unintended consequences of high-stakes testing on the intended beneficiaries of education reforms, namely students with disabilities, students of color, emergent bilingual students, and students experiencing poverty, while Kelly Duane de la Vega and Katie Galloway explore the experiences of families and communities following the prison release of thousands of incarcerated people serving life sentences after California’s amendment of the “Three Strikes” law in their 2016 documentary The Return.

Concluding Thoughts

As a Foundation deeply concerned about inequality and injustice, we are eager for studies that use a critical lens to examine key questions surrounding the usefulness of research evidence and innovative strategies to improve the usefulness and use of research evidence to improve youth outcomes. The nature of the evidence used in decision making necessarily shapes understandings of a problem space and the design of responses that may facilitate or limit the potential for its impact. Leveraging research toward more just ends demands holding decision makers accountable for considering the full range of evidence that informs the decisions they make, including evidence produced by the communities that are the targets of programs, policies, and practices. It also calls for disrupting dominant narratives and elevating alternative visions and possibilities for the use of research evidence.

Of course, the ideas and strategies offered here are only a beginning. We hope to see these and other critical theoretical lenses seed new and different questions and strategies for exploring who benefits from the construction and interpretation of particular kinds of research evidence.

  1. For full citations and a complete reference list, download the PDF of this essay.
  2. This brief historicization only scratches the surface of the fascinating and instructive origins and evolution of CRT. I enthusiastically recommend the following to those interested in further exploring this history:
    Crenshaw, K. W. (1988). Race, reform, and retrenchment: Transformation and legitimation in antidiscrimination law. Harvard Law Review, 101(7), 1331-1387. doi: 10.2307/1341398
    Crenshaw, K. W. (2002). The first decade: Critical reflections, or “a foot in the closing door.” UCLA Law Review, 49(5), 1343-1372.
    Crenshaw, K. W., Gotanda, N., Peller, G., & Thomas, K., Eds. (1995). Critical Race Theory: The key writings that formed the movement. New York: New Press.
    Delgado, R., & Stefancic, J. (2001). Critical Race Theory: An introduction. New York: New York University Press.
  3. See, for example, Ignatiev, N. (2009). How the Irish became White. United Kingdom: Routledge.

In this issue

What is needed is not a one-off intervention, but an informed understanding of how racism makes its way into encounters between youth and police and a commitment to addressing the ...
The Long Shadow: Considering How Racism Shapes Justice System Inequality and Efforts to Disrupt it
We can no more afford methodologically rich research studies that lack racial consciousness than we can afford racially conscious studies that lack methodological rigor.
No Small Matters: Reimagining the Use of Research Evidence From A Racial Justice Perspective
To begin shifting the underlying structures of inequality, scholars can harness the power of rigorous research to produce innovative, ground-breaking research that might further upend the narratives that inequality is inevitable or explained by individual differences.
Shifting the Lens: Why Conceptualization Matters in Research on Reducing Inequality

More Digest Issues

The Digest

Issue 8: Winter 2022-23

The eighth issue of the Digest features insights about where new research might help identify ways to reduce inequality in youth outcomes after COVID-19, as well as how the fields of implementation science and research on research use might learn from each other in ways that yield more transformative research in years ahead.
The Digest

Issue 7: Winter 2021-22

The seventh issue of the Digest includes fresh insights from program staff and grantees on how researchers can successfully confront the challenges of studying causal mechanisms for reducing inequality, as well as how the global community of researchers and funders focused on improving the use of research evidence can continue to break new ground in the years ahead.
The Digest

Issue 6: Winter 2020-21

Essays in this issue of the Digest focus on the importance of looking beyond individual action and customary metrics in research on reducing inequality, as well as how strengths-based, race-conscious research can be produced and used to uplift communities of color.
The Digest

Issue 4: Winter 2018/19

The fourth issue of the William T. Grant Foundation Digest features insights that may point the way toward a more nuanced understanding of evidence use and inspire new and more wide-ranging examinations of ways to reduce inequality in youth outcomes.
The Digest

Issue 3: Winter 2017/18

The third issue of the William T. Grant Foundation Digest features insights on how research on ability tracking can inform studies to improve the outcomes of English learners, as well as how researchers and school districts can partner to build learning systems based on research evidence.
The Digest

Issue 2: Spring 2017

The second issue of the William T. Grant Foundation Digest features writing on research rigor and relevance, as well as the potential for a new research agenda for improving the outcomes of English learners under the Every Student Succeeds Act.
The Digest

Issue 1: Summer 2016

The introductory issue of the William T. Grant Foundation Digest features essays and commentary on the value of qualitative and mixed-methods research in reducing inequality and the potential for researcher access to big data to yield useful research evidence.