John Thompson: Emily Oster and the Confidence of Non-experts

John Thompson thought the medical world would show more respect for the teaching world. Unfortunately…

In the early 1990’s, I naively believed that Silicon Valley and Big Data would collaborate with social and cognitive scientists, and practitioners, thus creating holistic, evidence-based solutions for social and education problems.  Later, I invested more than a decade trying to communicate with data-driven education researchers. I was told, again and again, that my critiques of their methodology might be right, but there wasn’t time to go back to the old-fashioned approach which included falsifiable hypotheses and peer review; instead, they would add more controls to their models in order to build better teaching staffs in high-poverty schools.

(The common response to the evidence that their algorithms were biased against teachers in high-challenge schools was that those errors only hurt about 15% of teachers’ evaluations, and that error rate was about average. I don’t recall any clear answers to the question why top teachers would move to, or remain longer in schools where there was more than a 15% chance per year that a statistical error could threaten their jobs.)

At any rate, when Covid hit, I remained hopeful that economists, true believers in data, and commentators would show more respect for medical science than they had for educators. Emily Oster was the first to prove me wrong. She went “viral” when arguing that educators’ fears are “overblown,” and that kids are “simply very unlikely to be infected.”  Oster was followed by other epidemiologists and opinion writers with simplistic headlines like, “We Have to Reopen Schools, Not Bars,” ignoring the fact that educators had no power over bars. And on the eve of the 2020 Thanksgiving and Christmas vacations, which were bound to spread infections to and from students, the New York Times published the opinion piece “When Trump Was Right and Many Democrats Wrong,” and on the eve of the Omicron surge, the Atlantic opined We Opened the Schools and … It Was Fine.”

I don’t recall any apologies by these authors for urging families to lower their guards of the eves of imminent surges.;

Seeming to ignore the complexities of our diverse schools and politics, Oster even cited Texas and Florida as evidence that schools aren’t super spreaders, raising the question of why she would trust numbers published in those states.

In fact,  a key to early surges in many states was young people infecting members of their multigenerational homes. And as Rachel Cohen explained, Oster’s data “reflected an extremely small and unrepresentative sample of schools.” Then, in November as more public health advocates pushed for more rapid reopenings, Texas became the first state to have a million infections.

Worse, Cohen reported, “Rebekah Jones, a former Florida Department of Health data scientist who says she was fired in May over a refusal to manipulate her state’s COVID-19 stats, has publicly pushed back on Oster’s claims.” Jones “offered Oster full and free access to their data. ‘But she [Oster] basically decided to just pick what data she wanted, not what’s available.’ Jones added, “It’s offensive to researchers, when you see something so unabashedly unscientific, and when the opportunity to do something scientific was there.”

So, why did Oster continue to lead the public and other analysts astray as Covid killed over a million Americans and mutated in ways that kept the pandemic going?

Ginia Bellafante’s New York Review of Books article, “Mothers Under Pressure,” provides the best explanation I’ve seen. Ironically Bellafante’s indictment of economists like Oster, who pontificate on crucial issues that are outside of their spheres of expertise, is embedded in her review of Oster’s The Family Firm: A Data-Driven Guide to Better Decision-Making in the Early School Years. Drawing on Oster’s previous work, Bellafante asks why mothers should trust an economist with the extremely important question of whether they should have an alcoholic drink per day during the second and third trimesters of their pregnancies, despite the scientific evidence that says Oster was wrong? Bellafante concludes, “Oster’s popularity is rooted to a great extent in how reliably her conclusions … align with what her cosmopolitan readership would like to hear.”

My original complaint with Oster was that she expressed such extreme confidence in simple answers to complex issues. Why would someone who is out of her field be so confident that she’s right and the experts were wrong? And, why would a so-called data expert use numbers that are clearly so unrepresentative of diverse situations?