Critics assail widely touted study on mask effectiveness against COVID

An acclaimed study on the effectiveness of masks in reducing symptomatic COVID-19 has been widely mischaracterized and suffers from serious design flaws, according to critics.

They include Harvard Medical School epidemiologist Martin Kulldorff, who was suspended from Twitter for a month for questioning the protective power of masks for unvaccinated elderly people.

The randomized controlled trial (RCT) of 600 Bangladeshi villages from November through April was led by a management professor at Yale, Jason Abaluck, with participation from medical and public health researchers at Yale, Stanford and the University of California Berkeley, among other institutions. 

The working paper was funded by a grant from charity assessment nonprofit GiveWell to the Institute for Poverty Action, which studies the effectiveness of programs to reduce global poverty, and has yet to be peer-reviewed. The nonprofit National Bureau of Economic Research published a shorter and somewhat different version.

Several variations were tried with the "treatment" villages. The researchers "cross-randomized mask promotion strategies at the village and household level, including cloth vs. surgical masks," and some were given window signs indicating the household wears masks.

A quarter of the villages were promised a "monetary incentive" for village leaders if they hit 75% mask compliance within the eight-week study. "Neither participants nor field staff were blinded to intervention assignment," the study said.

About 335,000 people provided data on symptoms, if any, by the end of the eight-week period. In the villages that "received free masks, information on the importance of masking, role modeling by community leaders, and in-person reminders," the interventions reduced "symptomatic seroprevalence" by 9.3% and the "prevalence of COVID-like symptoms" by 11.9%. 

Villages that got surgical masks reduced symptomatic infection by 11% — twice as high for those ages 50-60 and three times higher for those older than 60. Cloth masks, by contrast, had "an imprecise zero" effect. 

For surgical masks specifically, the differences between treatment and control groups were statistically insignificant for every age group under 50. This age-based finding was a red flag for some critics, as was the confounding variable of increased physical distancing observed among the masked group.

"Odd that mask advocates are excited by this study," Kulldorff, a pioneer in vaccine safety research, tweeted last week.

"As a vaccine advocate, I would be horrified if a vaccine trial showed 11% efficacy. Based on the 95% confidence intervals, we do not even know if surgical mask efficacy is more than 0%," he wrote.

The working paper came out the same week as a peer-reviewed study in the Southern Medical Association's journal. Researchers led by the U.S. Army Institute of Surgical Research looked at the effect of the July 3, 2020 Texas mask mandate in San Antonio's Bexar County, and the county's own mask mandate 12 days later.

They found "no reduction in per-population daily mortality, hospital bed, ICU bed, or ventilator occupancy of COVID-19-positive patients attributable" to either state or county mandate a month later.

The Bangladeshi study also contrasts with an earlier but much smaller peer-reviewed RCT by Danish researchers, published last fall in Annals of Internal Medicine.

They gave test participants in Denmark free masks and advice up front but otherwise left them alone, unlike the repeated interventions in the Bangladeshi study. Researchers did not find a statistically significant difference between those who wore masks and those who didn't. 

University of California San Francisco epidemiologist Vinay Prasad called the Danish study "well done, thoughtful, useful" at the time, and urged scientists to pursue many more RCTs "to test messaging strategies."

'Highly moralistic framing' 

"The overall effects of this [Bangladeshi] study are miniscule — 0.07% absolute reduction in seroprevalence," tweeted Emily Burns, former CEO of a marketplace for private instructors and doctoral dropout in neuroscience at Rockefeller University.

What the study really tested was "the impact of mask promotion on symptom reporting," she said, explaining that only people who reported symptoms were asked to participate in a serology study, "and only 40% of those with symptoms chose to have their blood taken."

Burns wondered whether the "highly moralistic framing and monetary incentives given to village elders for compliance might dissuade a person from reporting symptoms representing individual and collective moral failure — one that could cost the village money?"

The study "violates pretty much every single tenet of setting up and running a randomized controlled experiment," according to the COVID-19 data analyst known as "el gato malo," who was kicked off Twitter in January for purportedly spreading misinformation.

"[N]otice what is jarringly absent: any clear idea what prior disease exposure was in each of these villages and village cohorts at study commencement," the anonymous writer said in a lengthy analysis.

The researchers failed to isolate masks as a variable, running "a whole panoply of interventions, signage, hectoring, nudges, payments, and psychological games ... hundreds of known effects and who knows how many unknown ones," the analysis continued. 

They observed enormous gaps between self-reported mask-wearing (80%) in April 2020, when researchers started surveying Bangladeshis, and observed mask-wearing in late May (51%) and late June (26%), yet rely on self-reported symptoms for their findings, the writer said.

The incentive structure, which also included a non-monetary "certificate" for a quarter of villages, illustrates the core problem of studies that aren't blinded: The "subjects seek to please the researchers and it wrecks the data." 

Responding to the study, UCSF's Prasad blamed the lack of RCTs for Americans' choice of "the wrong mask" throughout the pandemic. "What a failure, a colossal failure," he said in a video.

Prasad was critical of attempts in the U.S. to extrapolate the Bangladeshi study to debates over school mask mandates, tweeting that American children "nearly exclusively" wear cloth masks.

A peer-reviewed study published in July, in the American Institute of Physics journal Physics of Fluids, casts doubt on whether surgical masks are that much better than cloth, however.

University of Waterloo researchers found they were 12% effective in filtering exhaled aerosols that carry the novel coronavirus, while cloth masks were 10% effective. They blamed the poor fit of most masks for the low numbers, though even tight-fitting N95 and KN95 masks only filtered a little over 50% of exhaled aerosols.

A glaring difference between the two studies: Waterloo researchers tested masks on mannequins in a controlled environment.