Thomas Ultican: NPE Throws Cold Water on CREDO Paper
Thomas Ultican looks at a new debunking of CREDO charter research. Reposted with permission.
The Center for Research on Education Outcomes (CREDO) just released another pro-charter school study, “CREDO also acknowledges the Walton Family Foundation and The City Fund for supporting this research.” It is not a study submitted for peer review and so opaque that real scholars find the methodology and data sets difficult to understand. Carol Burris and her public school defenders at the Network for Public Education (NPE) have provided an in-depth critical review.
With the new CREDO study, Education Week’s Libby Stanford said that “charters have drastically improved, producing better reading and math scores than traditional public schools.’’ Rupert Murdoch’s Wall Street Journal declared charter schools are now “blowing away their traditional school competition.” Burris retorted with “despite the headlines, the only thing ‘blown away’ is the truth.” (Page 3)
Putting a CREDO Thumb on the Scale
CREDO uses massive data sets, unavailable to other researchers, getting minuscule differences which are statistically significant. No one can check their work. They employ a unique and highly discredited statistical approach called “virtual twins” to compare public school with charter school testing outcomes. Instead of reporting the statistical results in standard deviations, CREDO uses their “crazy pants” days of learning scheme.
NPE discovered that the “blowing away” public school results amounted to 0.011 standard deviations in math and 0.028 standard deviations in reading. The minuscule difference is “significant statistically but is meaningless from a practical standpoint” according to CREDO. In a 2009 report showing public schools with a small advantage, CREDO declared, “Differences of the magnitude described here could arise simply from the measurement error in the state achievement tests that make up the growth score, so considerable caution is needed in the use of these results.”
To give these almost non-existent differences more relevance, CREDO reports them as “days of learning” instead of standard deviation. “Days of learning” is a method unique to CREDO and generally not accepted by scholars. They claim charter school math students get 6 more “days of learning” and English students, 16 days.
CREDO Days of Learning Conversions
The above chart comes from the Technical Appendix of a previous CREDO study, which reveals that Eric Hanushek and Macke Raymond used NAEP data from 2017 to create the table. No justifications for the conversions are given. It appears to be sloppy science and headlines generated by its use are unfounded propaganda.
Bad Methodology
The CREDO method does not compare charter school performance to actual public schools. It creates mathematical simulations. Professor Andrew Maul of UC Santa-Barbara stated, “The study’s ‘virtual twin’ technique is insufficiently documented, and it remains unclear and puzzling why the researchers use this approach rather than the more accepted approach of propensity score matching.”
CREDO’s stipulation that “virtual twins” comes from “feeder schools,” favors charter schools. Management expert, Andrea Gabor, explained that CREDO used less than five student transfers to a charter school as the cutoff for a particular public school’s data. She notes the “study excludes public schools that do NOT send students to charters, thus introducing a bias against the best urban public schools, especially small public schools that may send few, if any, students to charters.”
This study is singularly focused on test results as determinate of school quality. Many charter systems, like IDEA and Success Academy, spend an inordinate amount of time preparing for standardized tests. This biases results.
Professor Mark Weber of Rutgers University adds a few more observations:
- “The matching variables that create the counterfactuals are far too crude to do the job properly.”
- “The definition of the treatment — enrolling in a charter school — does not account for factors such as increased spending, peer effects, and other advantages which have nothing to do with ‘charteriness.’”
- “The consistently small effect sizes have been pumped up by an unvalidated conversion into ‘days of learning’ which has never been properly justified by the authors.”
The NPE report highlights another source of bias for charter schools:
“In addition to their presence in a CREDO-identified feeder school, students are matched by gender, grade level, scores, race, and special education and English language learner status. Yet special education students are not a monolith. Research has consistently shown that charters take fewer special education students and enroll fewer students with more challenging disabilities than public schools.” (Page 9)
The combination of rejected methodologies, murky data studies and biases toward charter schools render the CREDO study useless. Minuscule advantages reported, favoring charter schools, indicate that in reality, public schools outperform charters.
The CREDO Story
In 1981, Massachusetts Institute of Technology trained economist, Eric Hanushek, wrote “Throwing Money at Schools.” Right-leaning philanthropies and institutions were drawn to his declaration:
“The conventional wisdom about public schools is that they face serious problems in terms of performance and that improving schools requires additional money. However, the available evidence suggests that there is no relationship between expenditures and the achievement of students and that such traditional remedies as reducing class sizes or hiring better trained teachers are unlikely to improve matters.”
In a 1981 Ed Week commentary referencing this paper, Hanushek points to SAT testing as the gold standard for judging school performance. He claims, “Advanced statistical techniques are employed to disentangle the influences on achievement of schools and teachers from those of other factors such as family backgrounds and student abilities.” This motivated him to push for teachers to be evaluated, using “value added measures”, that since have been thoroughly discredited.
At the time, he was teaching political science and economics at Rochester University and meeting his future wife, a significantly younger student, named Margret (Macke) Raymond. She completed her Rochester University political science PhD in 1985.
A 1999 announcement from the school said, “The Center for Research on Education Outcomes has been established at the University of Rochester’s Wallis Institute of Political Economy…” In the same posting, it revealed, “Two foundations have committed $1.25 million to fund a three-and one-half year initiative to address the current shortage of evaluation research in education policy matters.” CREDO never made the names of the two foundations public but a knowledgeable academic disclosed one of them was the Walton Family Foundation. It is documented that the Waltons give generously to CREDO.
The announcement listed Eric Hanushek first and Macke Raymond as founding Director.
CREDO moved to Stanford University’s Hoover Institute in July 2000 which made networking in conservative circles much easier. Their 2nd year report stated that moving to the Hoover Institute brought many new contacts, including the New Schools Venture Fund, the District of Columbia Charter School Board, the Teacher Union Reform Network and others.
Public Schools are Superior to Charter Schools
Staffing in public schools is made up of mostly college graduates with certified state teaching credentials. Before the appearance of the billionaire-created Teach For America (TFA), nearly 100% of public school teachers had a year of teacher training and a bachelors degree or higher. A significant percentage of charter school teachers come from TFA with just five weeks of education training. Charters are typically not required to use certificated teachers.
The depth of experience in the public school teaching corps is larger than that of charter schools. A Fordham Institute article states:
“That being said, there is a bona fide but often unaddressed teacher shortage: experienced teachers in charter schools. In the United States, a third of charter teachers have fewer than three years of teaching experience, compared to only a fifth of public school teachers.”
“Comparative inexperience and youth in front of classrooms carries costs. More than any other school-related factor, a teacher’s efficacy matters most to student learning. And especially in the early years, nothing improves a teacher’s efficacy quite like experience.”
Stability is important for school-aged children and especially for those growing up in difficult home environments. In 2020, NPE conducted an in-depth look at charter schools since their inception. They discovered that charters were closing at extremely high rates; 18% by year 3, 25% by year 5, 40% by year 10 and 50% by year 15. In some cases, charters closed their doors mid-semester without warning; this never happens in public schools.
Management in public schools must meet state credentialing requirements. They focus on good pedagogy, safe schools and parent engagement. In charter schools, supervisors are often untrained in education and make return on investment, a key goal.
Safety in public schools is state-mandated but charter schools can ignore some rules. In California, all public schools must be earthquake-safe facilities. Charter schools may not heed this requirement.
In 2013, Christopher A. Lubienski and Sarah Theule Lubienski published The Public School Advantage – Why Public Schools Outperform Private Schools. Nothing has happened over the last ten years that invalidates this scholarly work. Charter schools are private schools with a state contract, similar to garbage collection companies, contracting with a city. They are private companies, paid with taxpayer funds.
Charters are substandard education organizations that only survive because of marketing.