Freddie deBoer takes a deep dive into CREDO, the Stanford think tank that is not going to let data dampen its love for certain ed reforms. In one part of this post, he addresses CREDO’s made-up metric beloved by too many.
“Days of Learning”: A Jury-Rigged Metric Invented for the Purpose of Inflating the Effect
If you share my confusion over that metric, don’t blame yourself. “Days of learning” is not merely an unusual measure of student performance; it’s a boutique metric that CREDO invented themselves for their charter studies. It’s like YouTube changing how views are counted so the numbers look bigger.
This should raise a number of red flags, most importantly when it comes to comparison with other research. We have standardized metrics in social science research for a reason. One core reason among them is mutual interpretability, the ability to compare outcomes involving different interventions that are measured with different instruments and in different units. That’s why effect size exists, and various expressions of effect size have served as the dominant measure of interest in all manner of human research for decades precisely because effect size enables ease of comparison. If you want to compare the influence of the same intervention when one study uses grades as the DV and another used test scores, effect size enables you to do that. If you want to compare two different types of intervention, effect size enables you to do that too. There are some caveats, as there always are, but the basic, elegant logic of effect size makes the measure remarkably flexible and portable: effect size tells us how a given intervention moves average outcomes along the distribution of outcomes. If you want to know more there are plenty of simple explainers of effect size out there; here’s mine. There are some fields where effect size is used less often (economics, for example) but the interpretability and flexibility of effect size metrics like Cohen’s d are powerful advantages. Not surprisingly, then, it’s the dominant reported measure in education research.
So if effect size is both useful and common, and is in fact the dominant measure of interest in the relevant field, why would CREDO come up with its own boutique measure? Why invite the potential for confusion with such consequential research? Occam’s razor suggests a simple answer: because CREDO is a pro-charter organization that’s intent on convincing stakeholders that charters have meaningfully superior outcomes when compared to traditional public schools… and their own research demonstrates the opposite. Their own numbers, in fact, suggest that the returns on charter schools are meager even if we ignore the selection problems that continue to plague charter advocacy. Here’s what sixteen additional “days of learning” in math and six days in reading looks like in terms of effect sizes, what we find when we pull back the curtain:
Those numbers, you may notice, are quite small! That’s not a .28 in reading, it’s a .028 in reading. That’s not a .11 in math, it’s a .011 in math. Effect size is always contextual, but these are very small effects no matter the context. If you have any familiarity with effect size, you might look at these numbers and be shocked by the various headlines that insisted that, finally, the pro-charter crowd had its silver bullet. The actual, real-world impact of the intervention being measured here, attending a charter school, is just about nil. CMO affiliated, for the record, stands for charter management organization, which are nonprofit organizations that run networks of charter schools. The numbers are slightly better but still close to zero in practical terms, and of course restricting to CMO schools reduces the sample size and degrades the generalizability of the study’s findings. The virtual charter finding aligns with similar research and contributes to the belief that virtual K-12 schools should not exist.
There’s lots more. Read the full post here.