Bill Honig is the former California superintendent of public education. In a recent piece for EdSource, he questions the new charter school research findings from the Hoover Institute’s CREDO.
ecently, an evaluation of charter school performance compared to public schools by CREDO has been widely reported as showing charters significantly outperforming public schools in reading but comparable results in math.
Macke Raymond, founder and director of CREDO, characterized the results as “remarkable.” EdSource reported that “charter school students in California significantly outperformed similar students in nearby traditional public schools in reading.”
These claims are highly misleading. The report uses “significant” in its statistical sense of not by chance. Research studies involving large populations will almost always find statistical significance. Most articles about the report confused statistical significance with the common meaning of significance as “making a substantial difference.”
Statistical significance doesn’t tell you anything about the magnitude of the difference between populations which is usually reported as “effect size” in terms of standard deviation, which is a measure of how far the data spreads from the average.)
CREDO’s own data (see pages 38, 48 and 50) show that the difference between public schools and charters is minuscule. They report the difference in days of learning and explain that 5.8 days of learning equals 0.01 (one hundredth) standard deviation. So, the additional 11 days of learning found for California students in reading is equivalent to 0.02 standard deviations in reading and the additional four days equals less than 0.01 in math.
The research community usually suggests that a much larger standard deviations of 0.2 be considered a “small” effect size, 0.5 represent a “medium” effect size and 0.8 a “large” effect size. Most regard any effect size below 0.2 standard deviations as negligible even if statistically significant. The CREDO findings in reading for California are one-tenth of that 0.2 standard. Even the better growth results for Black and Hispanic students are one-fourth of that 0.2 floor.
Additionally, these results measured growth, or improvement in results, but even after growth, the performance of large numbers of charter (and traditional public school) students remain significantly behind.
