Digest, Issue 6: Winter 2020-21

Can Policy Interventions Reduce Inequality? Looking Beyond Test Scores for Evidence

Introduction

Among the many troubling legacies of centuries of slavery and discrimination in the United States are extraordinary race-based inequalities in life chances.1 Black children grow up in families with much lower income and wealth, on average, than White children. They are more likely than White children to live in neighborhoods of concentrated poverty and to attend underfunded schools in which their classmates are also from low-income families. One consequence of these inequalities is that Black children are less likely than White children to acquire the skills, educational credentials, and networks that lead to well-paying jobs. Another is that the inequalities that hampered their lives are passed on to their children (Chetty et al., 2020).

In this essay, I use evidence from three studies to show that, as a nation, we have learned a lot about what it takes to reduce race-based inequalities. I also point out that the primary public policy strategies for reducing race-based inequalities and for measuring progress are inconsistent with the evidence from these recent studies.

Historically, the United States has relied on public schools as the primary institution for equalizing opportunities in the United States. Horace Mann, the first Secretary of Education in Massachusetts described this view in an 1848 report: “Education, then, beyond all other devices of human origin, is the great equalizer of the conditions of men—the balance-wheel of the social machinery.” (Mann, 1848). A little time spent on the website of the federally funded What Works Clearinghouse (WWC) reveals the variety of programs, products, practices, and policies that have been tried to improve American schools (U.S. Department of Education Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, 2020). While not all of these initiatives were focused on reducing race-based inequalities in educational outcomes, many were. And since the creation in the late 1960s of the National Assessment of Educational Progress (NAEP), reports of trends in the Black–White test score gap have received a great deal of media attention.

From one perspective, relying on public schools to reduce race-based inequalities makes sense. Ninety percent of the nation’s children attend public schools and spend thousands of hours in them. As David Cohen and Susan Moffitt (2009) have explained, the nation’s schools have been remarkably responsive to demands for changes in programs and practices. There is also value in paying attention to students’ scores on reading and mathematics tests. Changes in the economy over the last 50 years have reduced labor market opportunities for workers who lack basic reading and mathematics skills (Murnane, Willett, & Levy, 1995; Goldhaber & Ozek, 2019). Consequently, it is a plausible theory of action that improving the reading and mathematics skills of Black children would increase their educational attainments and labor market earnings.

From another perspective, the reliance on schools to reduce race-based inequalities is a mistake. A significant limitation of most attempts to improve the education of Black children is that they take as given the historical structure of American schooling. Elements of this structure include heavy reliance on local property taxes to fund public schools in each of the country’s 13,000 school districts and rules requiring children to attend school in the school district in which they reside, and in most districts, in their neighborhood public school. Given the racial and economic segregation of housing in America, the structure of American schooling results in Black children attending racially and economically segregated public schools (Reardon et al., 2019).

A significant limitation of most attempts to improve the education of Black children is that they take as given the historical structure of American schooling.

Working within this structure, most new programs, practices, and policies aim to increase the quality and quantity of instruction. As a perusal of the WWC shows, common initiatives include new curricula and professional development to improve teaching and learning, and a longer school day and after-school programs to increase learning time. Some of these programs have had positive impacts, usually measured by scores on tests of reading and math skills (Cohen et al., 2014). However, as the evidence from three studies described below shows, the racial segregation of housing in America, coupled with the historical structure of American public education, dramatically limits the ability of public education to fulfill Horace Mann’s vision of education as “the great equalizer of the conditions of men.”

Sole reliance on test scores to measure the effectiveness of interventions aimed at reducing race-based inequalities is also a mistake. One reason is that scores on standardized reading tests may not measure accurately the literacy of Black children and youth (Kirkland, 2013; Steele, 2011). A second, even more important reason is that improving reading and math skills is by no means the only way to reduce inequality in youth outcomes (Irons, 2019). Indeed, academic skills by themselves are no guarantee of a good job because advances in computer-based technologies, especially in artificial intelligence, have made it possible to automate many jobs requiring significant academic skills (Deming, 2017). New research shows that it is the combination of academic skills and social skills that is increasingly important in labor markets (Atalay et al., 2020). Elements of social skills include reliability; persistence in the face of challenges; listening, negotiating, and communicating effectively; and the ability to work productively in groups with people of different backgrounds (Murnane & Levy, 1996; Levy & Murnane, 2013).

Three Interventions that Made a Difference

We turn now to a description of three interventions that affected the lives of Black children. These interventions did not improve their reading and math scores but did dramatically improve their life chances.

Moving to Opportunity

In 1993, the U.S. Department of Housing and Urban Development (HUD) sponsored an experiment in five large cities designed to test whether low-income families with children benefit from moving to neighborhoods with lower poverty rates. Over the next four years, each of the 4,604 families living in publicly subsidized project-based housing that agreed to participate in the Moving to Opportunity (MTO) experiment was randomly assigned to one of three groups: an experimental group given vouchers that families could use only to rent private housing in a neighborhood in which the poverty rate was less than 10 percent; a second treatment group given standard Section 8 housing vouchers they could use anywhere; or a control group that did not receiving a housing voucher but retained access to public housing. Approximately half of the families in the experimental group “took up” the voucher offer and moved to a low-poverty neighborhood. Children raised in families that did take up the offer grew up in Census tracts with 22 percentage point lower poverty rates than those in control group families.

Researchers followed the families participating in the experiment, two-thirds of which were Black, for the next 12 to 15 years. They paid special attention to differences between the experimental group and the control group in a variety of outcomes. Using a statistical technique known as “instrumental variables,” they were able to estimate whether moving to a lower poverty neighborhood as a result of being assigned to the experimental group had a causal impact on outcomes for family members. To the surprise of many social scientists, the researchers found that such moves did not affect labor market outcomes of family members who were adults at the time of random assignment (RA). In contrast, they found that moving to a lower-poverty neighborhood did improve both the physical health and mental health of these same adults. Given the importance of parental health to the well-being of young children, these results led researchers to predict that children in experimental group families who were quite young at RA would fare better academically than children in control group families. They were disappointed to find that this was not the case, at least as measured by reading and math scores four to seven years after RA (Sanbonmatsu et al., 2006). Fortunately, this was not the last word on the impact of MTO on children.

…moving did not improve scores on reading and math tests for Black children who were less than 13 years of age at random assignment, but it did dramatically improve early their outcomes as adults

In 2014, Lawrence Katz of the MTO research team joined forces with Raj Chetty and Nathaniel Hendren, who had obtained access to data from federal income tax records for almost all Americans over a three-decade period. By merging federal tax records with data from the MTO experiment, the researchers were able to examine adult outcomes between 1996 and 2012 for individuals whose families had participated in the MTO experiment. The results for children are striking. Using the same instrumental variables technique that the MTO researchers had used, Chetty, Hendren, and Katz (2016) found that for children who were less than 13 years of age at RA, moving to a lower-poverty neighborhood increased their earnings by their mid-20 by an average by 31 percent ($3,477). It also increased the probability that they enrolled in college, and that they attended a relatively high-quality college. This pattern held for males and females, all racial/ethnic groups, and across all five sites. In contrast, moving to a lower-poverty neighborhood did not improve adult outcomes for children who had been 13 years or older at RA.

In summary, moving from a high-poverty housing project to a lower-poverty neighborhood did not improve scores on reading and math tests for Black children who were less than 13 years of age at random assignment. However, it did dramatically improve early adult outcomes for these children, as measured by subsequent educational attainments and labor market earnings.

Chicago Public Housing Closures

In 1990 the Chicago Housing Authority (CHA) had the third largest public housing system in the nation, providing services to nearly five percent of the city’s population. CHA owned and managed 17 developments that provided housing for low-income families with children, each consisting of several apartment buildings, many of them high-rises. Ninety-five percent of the families in Chicago public housing projects were Black. As the result of a consent decree, severe maintenance problems in some buildings, and gang violence, CHA demolished over the next decade a subset of the high-rise buildings in each of these projects. Families living in buildings scheduled for demolition were given the option of transferring to a unit in another CHA development (contingent on availability) or receiving a Section 8 voucher that they could use to subsidize the cost of private housing

Economist Brian Jacob recognized that the CHA high-rise demolitions provided a natural experiment that would allow him to estimate the impact of the forced family moves on the academic achievement of affected children. He could do this by comparing reading and math scores for children whose families lived in buildings that were demolished (the treatment group) with those of children living in buildings in the same housing project that were not demolished (the control group). The availability of test score evidence led Jacob to focus on children who were less than 14 years of age at the time of test administration. He found no statistically significant differences between the average reading and mathematics scores of children in the treatment and control groups two years after treatment group families were forced to move (Jacob, 2004).

Fortunately, it was possible more than a dozen years after Jacob published his results to revisit the question of whether displacement from a public housing project located in a high-poverty neighborhood affected Black children’s life chances. Eric Chyn (2018) did this by collecting information from the CHA on children living in 53 high-rise buildings in seven projects. Twenty of the buildings were demolished between the years 1995 and 2000, while 33 remained in service. Chyn merged the information from the CHA on displaced and non-displaced children with information on a variety of schooling and adult outcomes from state administrative databases. He found that three years after demolition, displaced families (the treatment group) lived in neighborhoods with a 21 percent lower poverty rate and a 42 percent lower rate of violent crime than non-displaced families originally living in the same project (the control group).

In contrast to Brian Jacob’s disappointing findings, Chyn found that children who were aged 13 or less when their families were forced to move from public housing slated for demolition were nine percent more likely to be employed as adults in their mid-20s and had labor market earnings 16 percent higher than members of the control group. He also reported that the labor market results for the full sample were driven by large effects for females. In contrast, Chyn found that moving out of a public housing unit reduced the number of arrests for violent crimes by 14 percent, driven by large effects for males.

…three years after demolition, displaced families lived in neighborhoods with a 21 percent lower poverty rate and a 42 percent lower rate of violent crime than non-displaced families

Chyn’s study adds to the insights from the MTO long-term study in several important respects. First, it demonstrates that benefits of moving out of high-poverty public housing projects to lower-poverty neighborhoods are present for Black children whose families were compelled to move as well as for those in families who volunteered to move, as in the MTO experiment. Second, as with the MTO study, the moves did not result in improved student reading and math scores. Third, while Chyn also found that the largest benefits of moving to a lower-poverty neighborhood accrued to Black children who were less than 13 when their family moved, he also found that benefits accrued to Black youth who were 13 to 18 years old at the time of the move. Finally, Chyn found that the long-term benefits of moving to a lower-poverty neighborhood were largest for children from the most disadvantaged circumstances.

Opportunity to Choose a School

In 2001, the U.S. Supreme Court allowed the Charlotte-Mecklenberg (C-M) school district to end the mandatory school busing that had been part of its operation for the previous 30 years. In the wake of this change, the C-M district leadership introduced a school choice program in 2002, under which students could rank order the public schools they would like to attend. All students were guaranteed enrollment in their neighborhood school. If they ranked first a school other than their neighborhood school, they were enrolled in that school if it had vacant spaces in the relevant grade. Admission to schools for which applications in the relevant grade exceeded the number of available places was determined by lottery, with students from low-income families given priority.

David Deming recognized that he could study whether winning a lottery to attend an oversubscribed school made a difference to students. He found that almost all lottery winners were low-income Black students. The lottery winners attended middle schools and high schools that had more economically diverse student bodies and better prepared teachers than the lottery losers, almost all of whom attended their neighborhood school. Deming also found that the reading and math scores of lottery winners were no higher, on average, than those who lost out in a lottery.

Fortunately, Deming understood that the opportunity to attend a better school might affect outcomes other than test scores, including subsequent criminal activity and post-secondary education. In a 2011 paper, Deming demonstrated that winning a lottery to attend an oversubscribed school reduced arrests for adult crimes, especially serious crimes, over the next seven years. Impacts were concentrated among Black male students who, based on academic performance and behaviors in school measured prior to the lotteries, were at high risk of being arrested in the future. For this group, winning a lottery reduced the number of crimes committed by about 50 percent.

…winning a lottery to attend an oversubscribed school reduced arrests for adult crimes, especially serious crimes, over the next seven years

In a 2014 paper, Deming and three co-authors reported that Black females who attended their first choice high school as a result of winning a lottery were 17 percent more likely to attend a 4-year college than lottery losers who attended their neighborhood school, and were 17 percentage points more likely to complete a four-year college degree. In contrast, lottery outcomes did not improve college outcomes for Black boys.

Deming and colleagues offer compelling evidence that providing low-income Black students the opportunity to attend a middle school or high school with a more diverse student body and better prepared teachers improved their life chances, even though it did not improve their scores on state reading and math tests. An important aspect of the research findings is that the opportunity affected different outcomes for boys than for girls.

Implications for Research

Since research that follows individuals for many years takes considerable resources, one immediate question concerns which interventions offer real promise of long-term impacts. Insight about the answer to this question comes from commonalities among the three interventions described above. All three changed the social context of children’s and youths’ daily lives. The MTO experiment and the CHA demolitions resulted in affected children growing up in neighborhoods that were more socioeconomically diverse than the neighborhoods they left. The school lotteries in C-M resulted in many low-income children living in high-poverty neighborhoods attending schools with more diverse student bodies and better prepared teachers than the public schools in their neighborhoods.

The studies described above were not able to pin down the causal mechanisms through which the interventions affected early adult outcomes. Doing so would require a different kind of research, with an emphasis on qualitative methods. However, a great deal of research has documented the critical role of peers in influencing the behavior of children and especially adolescents. So changing the mix of peers with whom children interact on a daily basis could be an important part of the story.

A second common attribute of the interventions is that all were the direct result of government initiatives. In each case, the initiative ameliorated the effects of decades of conscious federal, state, and local government housing policies that had produced residential and school segregation (Rothstein, 2017). MTO required experimental group families that took up the offer to move from a high-poverty public housing project to a lower-poverty neighborhood. CHA demolished public high-rise housing projects in high-poverty neighborhoods that had been the dominant government strategy for providing housing for the poor. Introduction of the school choice system in C-M was an attempt to reduce the concentration of children of color from low-income families in struggling schools located in high-poverty neighborhoods.

A third common attribute of the interventions is support to enable families to overcome institutional and financial barriers to living in socially diverse neighborhoods and enrolling their children in schools that will serve them well. For example, MTO staff provided families in the experimental group not only with a voucher to help pay for housing in a low-poverty neighborhood, but also with help in overcoming discrimination and financial challenges to moving to a lower-poverty neighborhood. Qualitative research by Stefanie DeLuca on a subsequent experiment conducted in the Seattle area documented the critical role of these supports in enabling low-income families with Section 8 housing vouchers to move to lower-poverty neighborhoods (Bergman et al., 2019).

The C-M school district gave low-income families priority in school lotteries, which enabled them to enroll their children in schools with more diverse student bodies and better prepared teachers. In the CHA housing demolitions, the supports to families forced to move included paying for moving expenses and the cost of transferring telephone service, electricity, and other utilities. The minimal nature of the supports CHA provided helps to explain why families forced to move relocated to neighborhoods with only slightly lower poverty-rates than those in the public housing projects they left.

Importantly, the evaluations of all three interventions documented that the benefits to children were the greatest for those who experienced the greatest change in social context. In other words, it is not choice per se that matters. It is growing up in neighborhoods with lower crime rates and more economically advantaged neighbors and attending better schools with children from more economically advantaged families. The experiment conducted in the Seattle area documents that just providing low-income families living in high-poverty neighborhoods with Section 8 housing vouchers is not enough to enable them to move to lower-poverty neighborhoods: Supports to enable them to overcome financial barriers and discrimination are critical.

…the evaluations of all three interventions documented that the benefits to children were the greatest for those who experienced the greatest change in social context

The studies of schooling by David Deming and his colleagues as well as recent path-breaking research by C. Kirabo Jackson (2018) document that teachers and schools affect students’ lives not only by enhancing academic skills measured on standardized tests, but also by helping students to develop social skills. These social skills are important to success in college, in labor markets, and in personal relationships. Educators have long been aware of the importance of social skills, such as listening and communicating effectively, and working productively in groups. Indeed, these skills are part of the learning standards in most states. However, standardized tests used in state accountability systems do not measure social skills. Consequently, a concern is that schools under pressure to improve scores on standardized tests will focus on test-taking strategies to the exclusion of activities designed to develop social skills.

Until recently, cost was a significant barrier to using measures of a variety of outcomes in evaluating interventions to reduce inequality in youth outcomes. This is no longer the case. Advances in administrative data systems make it possible at very low cost to develop and use early indicators of students’ success after high school graduation. College-relevant measures include how long it takes students to pass a first-college level math class or to earn 30 credits. Labor market-relevant measures could include employment status and earnings. Criminal-justice outcomes include number of arrests for serious crimes. Civic outcomes include voter registration and voting regularity.

Of course, just as standardized test scores have limitations as outcome measures, each of these post-graduation outcomes does as well. For example, the employment rate of graduates depends not only on their academic and social skills, but also on the strength of the local economy. Racism influences number of arrests of Black and Latino young people. Nonetheless, using administrative data from a variety of sources provides a more rounded and complete picture of the success of interventions aimed at reducing inequality in youth outcomes.

Summing Up

Given the history of housing segregation in America and the funding structure of American schooling, a great many Black children grow up in high-poverty, unsafe neighborhoods and attend underfunded schools that do not prepare them for success in post-secondary education and the labor market. But the studies described above make a strong case that reducing the social isolation in which a great many low-income Black families live and their children attend school are powerful strategies for reducing race-based intergenerational inequalities. These studies also document that low-income Black families face many barriers to moving to lower-poverty neighborhoods and enrolling their children in better schools, but that purposeful government action can help low-income families to overcome these barriers.

Can policy interventions reduce inequality? Yes, but learning how particular government policies affect families requires well-designed and well-coordinated mixed-methods research that pays attention to a variety of indicators of success.

Footnotes
  1. For full citations and a complete reference list, download the PDF of this essay.
Downloads

In this issue

In the quest to address income inequality and intergenerational poverty, it is critical to seek answers to the questions “What works?” and “For whom?” Education is ...
Test Scores are Only a Symptom: The Challenge of Seeking to Close Educational Gaps While Ignoring Historical Legacies
After five years of supporting research on reducing inequality, we’re proud of the body of work that grantees are producing and we’re inspired by their efforts to inform thought and action beyond the university gates.
Forging Connections: Grantee Experiences Reaching Audiences Beyond the Academic World
Encouraging the use of evidence from antiracist research requires thinking differently about what is relevant, how antiracist perspectives are built into routines for using research, and what it means to ...
Identifying and Testing Strategies to Improve the Use of Antiracist Research Evidence through Critical Race Lenses

More Digest Issues

The Digest

Issue 9: Winter 2023-24

The ninth issue of the Digest features an update on the Institutional Challenge Grant program five years after its launch, with insights from a Foundation stock-taking effort and reflections from two grantee teams. We also share new thinking on the use of research evidence, including ways for researchers to leverage existing findings to bolster studies of effective strategies to improve research use.
The Digest

Issue 8: Winter 2022-23

The eighth issue of the Digest features insights about where new research might help identify ways to reduce inequality in youth outcomes after COVID-19, as well as how the fields of implementation science and research on research use might learn from each other in ways that yield more transformative research in years ahead.
The Digest

Issue 7: Winter 2021-22

The seventh issue of the Digest includes fresh insights from program staff and grantees on how researchers can successfully confront the challenges of studying causal mechanisms for reducing inequality, as well as how the global community of researchers and funders focused on improving the use of research evidence can continue to break new ground in the years ahead.
The Digest

Issue 5: Winter 2019-20

The fifth issue of the William T. Grant Foundation Digest features insights from program staff and grantees on the importance of looking beneath the surface to consider the underlying factors that create and shape challenges that social scientists seek to address, whether they be related to reducing inequality in youth outcomes or improving the use of research evidence in policy and practice.
The Digest

Issue 4: Winter 2018/19

The fourth issue of the William T. Grant Foundation Digest features insights that may point the way toward a more nuanced understanding of evidence use and inspire new and more wide-ranging examinations of ways to reduce inequality in youth outcomes.
The Digest

Issue 3: Winter 2017/18

The third issue of the William T. Grant Foundation Digest features insights on how research on ability tracking can inform studies to improve the outcomes of English learners, as well as how researchers and school districts can partner to build learning systems based on research evidence.
The Digest

Issue 2: Spring 2017

The second issue of the William T. Grant Foundation Digest features writing on research rigor and relevance, as well as the potential for a new research agenda for improving the outcomes of English learners under the Every Student Succeeds Act.
The Digest

Issue 1: Summer 2016

The introductory issue of the William T. Grant Foundation Digest features essays and commentary on the value of qualitative and mixed-methods research in reducing inequality and the potential for researcher access to big data to yield useful research evidence.