Thomas Ultican breaks down a study purporting to show the advantages of school choice, a study that turns out to be less than rigorous or accurate. Reposted with permission.
Another education study financed by Arnold Ventures and the Walton Family Foundation blurs education reality. Their 2022 model did not pass the laugh test so “researchers” from the University of Colorado Denver tried again. Unfortunately their claims still confuse correlation with causation. This error seems purposeful.
The study of school reform in Denver was conducted by the Center for Education Policy Analysis (CEPA). They state, “For the past three years CEPA has partnered with the Center on Reinventing Public Education to consider a paradigm-shifting approach to family and community engagement efforts in school districts.” It is a study apparently to justify and promote the portfolio model of school management, a system first proposed in 2009 by the founder of the Center for Reinventing Public Education (CRPE), Paul Hill.
In their 2022 study, this same team also used state testing data from years 2004/5 through 2018/19. They explained that the first 4-years of the research employed pre-reform data and the final 10-years were from the portfolio model reform period. The authors reported, “During the study period, the district opened 65 new schools, and closed, replaced, and restarted over 35 others.” (Page 7)
The National Education Policy Center contracted with Robert Shand to review the 2022 Denver study. Dr. Shand is Assistant Professor of Education Policy and Leadership at American University and an affiliated researcher with the Center for Benefit-Cost Studies of Education at Teachers College, Columbia University. Shand also did a review of the new 2024 study.
In his 2022 review, Shand agreed that the test scores for Denver Public Schools had gone up but he noted a few reasons why claiming these gains were because of the portfolio model was unreasonable:
- Demographics shifting to a larger percentage of white students in Denver coincided with the reforms.
- Per-student revenues increased in Denver by 22% but only 13% across Colorado.
- Student-to-teacher ratio in Denver dropped from 17.9 to 14.9.
- DPS was already showing academic improvement before implementation of the portfolio reforms.
- Black and Hispanic/Latinx students were growing at approximately 0.06 standard deviations per year pre-reform and 0.03-0.04 standard deviations per year post-reform. (Page 7)
The 2024 Redo
Professor Shand’s summary response to the 2024 report states:
“While the new report does convincingly demonstrate that the gains are not significantly due to changing demographics, it fails to address other critiques of the prior study, including (1) that the portfolio model was undertheorized, with unclear mechanisms of action and insufficient attention to potential drawbacks; and (2) that circumstances, events, and resources besides the portfolio reform and student demographics were changing concurrently with the reform. Additionally, the report’s sweeping conclusion—that Denver’s reform is the most effective in U.S. history—is unsupported. The improved outcomes in Denver during this time period are impressive, but the authors seem overly determined to cite a package of favored reforms as the cause.” (Page 3)
While Shand agrees that demographic changes are not the whole reason for the improved test scores, they are a significant input. The chart above from USAFacts.org shows the typically higher scoring groups Asians and Whites going from 54.2% of the population to 58.9% in the 14 years from 2005 to 2019. During the same period, the Hispanic and Black population shrunk from 42.9% to 38.1% which resulted in a 9.5% shift in the population from a lower scoring to a higher scoring racial mix.
An even bigger impact on the scoring in Denver was the change in economic circumstances. Standardized testing is useless because the results are dependent on one variable, family wealth. Statisticians assign r values between -1 and +1 to results tested. Plus 1 signifies certainty, zero shows no influence and -1 indicates certainty in the opposite direction of expectations. The only input ever found with more than 0.3 r-value is family wealth at a 0.9 r-value. The median family income in Denver is up significantly.
Two sources show how strongly Denver’s family income has grown. Neilsberg research shares that between 2010 and 2020 the median income grew from $61,394 to $82,335, a 25% growth. City-Data states:
“The median household income in Denver, CO in 2022 was $88,213, which was about the same as the median annual income of $89,302 across the entire state of Colorado. Compared to the median income of $39,500 in 2000 this represents an increase of 55.2%”
This kind of wealth growth over the 14 years the Denver researchers studied was bound to have a significant impact on testing results, but they ignored it. Add this to the 9% greater revenue for Denver schools and three less students per teacher compared to the rest of the state and of course Denver’s student made comparative testing gains.
Professor Shand mentions the damage caused by school turnaround efforts and closing schools noting the research indicates these are especially harmful events for students in low income or marginalized neighborhoods. (Page 6 and 7) Shand concluded:
“In sum, this report provides some additional supporting evidence in favor of the tentative conclusion that Denver’s portfolio reform was positive. Importantly, the report also grossly exaggerates both the magnitude of the success and certainty behind the evidence for it. The findings should thus be interpreted with extreme caution. (Page 8)
He is being nice. He should have concluded that this report is school choice propaganda.
About the Report Authors
The lead author, Parker Baxter, is Director of the Center for Education Policy Analysis at the University Of Colorado Denver School Of Public Affairs. He previously was Director of Knowledge at the National Association of Charter School Authorizers. Parker is also a Senior Research Affiliate at the CRPE, where he worked on the District-Charter Collaboration Compact Project and the Portfolio School District Project. He is a former alumnus of Teach for America.
Anna Nicotera is a Senior Researcher at Basis Policy Research specializing in quantitative and qualitative applied research methods. She worked six years as Senior Director, Research and Evaluation for the National Alliance for Public Charter Schools. Nicotera was a Graduate Research Assistant at the National Center on School Choice, Vanderbilt University for four years.
David Stuit holds a Ph.D. in Leadership and Policy Studies from Vanderbilt University. He is a former Emerging Education Policy Scholar at the Thomas B. Fordham Institute, fellow at the Friedman Foundation for Educational Choice (Rebranded EdChoice), and member of the American Enterprise Institute’s K–12 working group. He began his career as a classroom teacher in Denver, Colorado.
Expecting an unbiased piece of research from this group is like learning about the dangers of smoking from Phillip-Morris.
Conclusion
The report by Baxter et al. was dutifully promoted by The 74. It is dangerous propaganda in favor of school choice. This report is another example of using arithmetic and titles to sell a farce.
