Facts to Refocus the Education Reform Debate

Like many states faced with making deep fiscal cuts in a declining economy, Massachusetts is looking at some difficult choices. One area under particular scrutiny is education, where funding for programs such as charter and pilot schools, mainstays in the effort to bring innovative teaching practices into broader use, has long been a political hot topic. Now, new research led by Professor Thomas Kane is redirecting the debate by focusing on facts instead of conflicting ideologies.

The distinctions between charter, pilot, and traditional schools are key factors in discussions about budget allocations for education in Massachusetts.

• Traditional schools receive public funding, are under local jurisdiction and maintain tenure and salary guidelines.
• Charter schools receive public funding, but operate outside supervision by local school committees and district collective bargaining agreements.
• Pilot schools remain under local jurisdiction and maintain the same basic tenure and salary guidelines as traditional schools, but have more flexibility to make operational and curricular decisions.

While giving some support to the pilot school concept, teachers' unions — with concerns about contract protections, declining enrollments, and the siphoning of public funds from traditional schools — have been among the most vocal critics of the charter school movement.

Teachers' concerns aside, many parents favor the longer school days, flexible scheduling, and innovative curricula that are offered, in varying degrees, by both charter and pilot schools. Reports of higher standardized test (MCAS) scores among students in non-traditional schools also have helped fuel the demand for more access. A recent Boston Globe article reported there are 20,000 students on the waiting list for the 25,000 charter school slots in Massachusetts.

Adding substance to the debate

As parents, teachers, and policymakers consider the wisest deployment of dwindling financial resources, a significant barrier to reaching consensus has been the lack of quantifiable data on school performance. Although there is a general impression that nontraditional schools perform better over a variety of measures, the absence of objective information that clearly establishes cause and effect raises questions about the underlying reasons for success.

“Critics maintain that kids who apply to pilot and charter schools are more likely to succeed anyway, because they and their parents are more motivated and engaged in schooling,” said Kane, professor of education and economics at the Harvard Graduate School of Education and director of the Center for Education Policy Research at Harvard University (CEPR), an initiative that partners with states and districts to evaluate innovative education policies. “They also say that because these schools have higher academic standards, students who might do poorly on achievement tests may drop out or flunk out before standardized exams are given.” Because of these arguments, said Kane, achievement test scores “have completely failed to persuade either side. Most of the debate has been purely ideological.”

The situation is particularly ironic, he said, given that “pilot and charter schools were intended to be laboratories where we could learn about new approaches.” Since these experimental schools educate only a small fraction of the state's students, the longer-term goal was to transfer approaches that worked over to the traditional schools. “Instead,” he said, “we're doing experiments where no one believes the results.”

To help resolve the impasse, Kane, along with Joshua Angrist, Ford Professor of Economics at MIT, and colleagues have completed a study, published by The Boston Foundation, that adds factual substance to the discussion. Entitled “Informing the Debate: Comparing Boston's Charter, Pilot and Traditional Schools,” the study was designed to establish the extent to which different kinds of schools have produced significant achievement gains.

The research draws on data from the Massachusetts Department of Elementary and Secondary Education documenting student performance from 2001-2007. Two specific design elements target concerns about the validity of MCAS test results as an indicator of program efficacy. “The study's observational results include statistics about students' achievement, demographics, and program participation before they attended a charter or pilot school,” said Kane. “That made it possible for us to compare their performance after enrollment in non-traditional programs with the test results of traditional school students who shared similar basic traits in an earlier school year.”

To address the possibility that there are differences in students from traditional and non-traditional schools that don't show up in their prior demographic or academic achievement histories, the study also includes data gathered from charter and pilot school admissions lotteries between 2003 and 2007. Kane said that an analysis of the thousands of middle and high school students who entered the lotteries showed no significant difference in race, ethnicity, gender, or standardized test scores between those who were admitted and those who were not chosen. “That suggests the lotteries were random and fair, first of all, and that when the paths of the charter and traditional school kids diverged, the two groups were similar in composition,” he said.

Unexpected Results

The results of the observational and lottery-based research have surprised many. Among middle and high school students, the study shows that each year of charter school attendance raises achievement scores .09 to .17 standard deviations in English and .18 to .54 standard deviations in math when compared to students in traditional public schools.

But the figures for pilot schools are less encouraging. In elementary grades, there was a gain of .09 in English, but no statistical gain in math. Among middle school students, the study's observational results indicate a decline in pilot school performance relative to traditional schools, with -.06 standard deviations per year in English and -.07 in math. The lottery-based research shows no statistically significant difference between pilot and traditional school performance at the middle or high school level. However, the observational results for high school students, although ambiguous, seem to be more in line with the gains seen among charter school students.

Looking Ahead

For Kane and his colleagues, the research has raised additional questions they are eager to pursue. Keeping his eye on the original goal of using the lessons from experimental schools to enhance traditional school systems, Kane is hopeful that future studies will be conducted to get at “what it is about charter schools that accounts for the successes we've measured.”

For now, however, he is sanguine about what he considers a major breakthrough in harnessing the power of an under-utilized resource. “There is a huge stockpile of data that's accumulating — probably the most valuable byproduct of the No Child Left Behind Act — that states and districts don't have the departments or staff to dig into,” Kane said. “That kind of analysis is something that academics do very well.

“This study shows the value of HGSE's role as a point of contact between social science researchers at Harvard and elsewhere and policymakers and school administrators. There is a wealth of valuable information in existing public education data, and we're just at the beginning of learning how to get at it and use it to best advantage.”