The Effectiveness of School Choice in Milwaukee: A Secondary Analysis of Data from the Program's Evaluation
American Political Science Association Meetings, San Francisco 1997

In 1990 Milwaukee became the site of the first publicly funded school choice program providing low-income parents with vouchers that could be used to send their children to secular, private schools. Milwaukee's school choice experiment was evaluated by a research team headed by political scientist John Witte at the University of Wisconsin at Madison. In five annual reports issued between 1991 and 1995, the researchers (hereinafter referred to simply as Witte) reported on the effectiveness of the Milwaukee experiment, as measured by the performance of students on standardized mathematics and reading tests. The senior author has summarized the results of his investigation as follows: "This school experiment . . . [has] not yet led to more effective schools. . .. Choice creates enormous enthusiasm among parents . . . but student achievement fails to rise."

Since this evaluation, until now, provided the only source of information on the test performance of choice students, many scholars, groups and foundations, drawing upon its findings, have concluded that school choice is not an effective way of improving the education of low-income, central-city students. The Carnegie Foundation for the Advancement of Teaching declared: "Milwaukee's plan has failed to demonstrate that vouchers... can spark school improvement." Albert Shanker, president of the American Federation of Teachers, claimed that the "private schools [in the Milwaukee choice plan] are not outperforming public schools."

For five years, the researchers did not release data from the evaluation for secondary analysis by other members of the scholarly community. But in February of 1996 they made the data available on the World Wide Web. Over the past several months the Center for Public Policy at the University of Houston (CPP) and the Program in Education Policy and Governance at Harvard University (PEPG) have accessed the data, cleaned them of identifiable errors, and organized them into a readable usable format.

Although the certainty with which conclusions may be drawn is restricted by certain data limitations, results based upon the highest quality information contained within the data set indicate that attendance at a choice school for three or more years enhances academic performance, as measured by standardized math and reading test scores. Correcting for errors in the dataset and using appropriate analytical techniques, the CPP/PEPG analysis of student performance finds that students enrolled in choice schools for three or more years, on average, do better on standardized tests, than a comparable group of students attending Milwaukee public schools.

The results indicate that the reading scores of choice students in their third and fourth years, were, on average, from 3 and 5 percentile points higher, respectively, than those of comparable public school students. Math scores, on average, were 5 and 12 percentile points higher for the third and fourth years, respectively. These differences are substantively significant. If similar success could be achieved for all minority students nationwide, it could close the gap separating white and minority test scores by somewhere between one-third and more than one-half.

CPP/PEPG results are based on data derived from a natural experiment that randomly assigned students to a test and control group. The natural experiment was the product of a mandate imposed on the program by the Wisconsin state legislature. It required choice schools, if over-subscribed, to admit applicants at random. This mandate created two randomly selected groups of students, one selected to participate in the choice program, the other not selected. The experimental situation is not unlike that widely practiced in medical research, where individuals are randomly allocated to treatment and control groups. The data are thus quite well suited for drawing scientific conclusions about the effectiveness of the choice program, provided they are analyzed correctly and interpreted cautiously.

The earlier analysis of the Milwaukee choice program did not give careful attention to this experimental data. On the one occasion when the experimental data were examined, the researchers failed to employ appropriate analytical techniques. The bulk of their research efforts focused instead on comparisons between choice students and a much less disadvantaged cross-section of public school students. No valid conclusions can be drawn from the comparisons they conducted.