Over the weekend I received several news alerts on my phone from various media outlets about masks and the spread of COVID. The exact words in the bulletin were, “Study Confirms Masks Prevent Spread of COVID.” As the week progressed, I continued to see this bulletin on various outlets, posted on Facebook and retweeted on twitter. Given the momentum of the paper I decided to read it. After reading this atrocity I could not resist writing this post. This is not a referendum on masks. I believe masks are a widely available and inexpensive method to protect yourself and others. This is a referendum on bad science. Really, it’s not so much a referendum as a thorough necessary dissection.
When you first look at the article there are a couple of red flags right from the beginning. First, the authors are from the department of atmospheric studies at Texas A&M University. Nothing against atmospheric studies, but they are forecasters. In general, forecasters try very hard, but often make mistakes. Check the track record of your local weather man or your favorite stock broker. These researchers in particular study climate change. They are not epidemiologists, virologists, or researchers in any medical field. Second, the paper itself has no structure to it. Typically, a medical research paper has an introduction, methods, results, discussion, and reference section. This paper has an introduction, which is about 2 pages long. The intro has very little discussion about mask use. There is no formal methods section. The results section takes up half a page. The remainder of the paper is divided between a discussion section and policy recommendations for governments. When the introduction and discussion are the two largest sections of the paper, you can assume little actual science has been done. Third, the article was published in PNAS, which actually is a decent impact publication, but publishes few studies in the arena of epidemiology.
This study mostly resembles a retrospective cohort. Which is an observation study. Observational detect casual relationships or correlations, not causal relationships. This is very important to understand. A cohort is a group of people. Retrospective indicates they started after the exposure, or at the end, and looked back to the beginning to see if there was a change. When determining the validity of a retrospective cohort there are certain criteria that must be met. First, were the groups equal or was there an attempt to make the groups as equal as possible at the beginning. Second, was the chance of exposure equal in both groups. Third, was an attempt made to correct for any differences between the groups (confounders) by statistical analysis. This is necessary because you want to know that the prognosis for each group was the same at the beginning, the risk of being exposed was the same, and it is unlikely that any other variable was responsible for the result.
The article creates several groups and compares them. There is no comparison between a group that wears a mask and a group that doesn’t. There was a publicized mandate. To prove the hypothesis, they compared actual cases in New York to projected cases if no order had been issued. This is a forecast. It’s not real because it didn’t actually happen. They also compare cases in New York after the order and the rest of the US. It should be stated that New York and the rest of the country are quite different. The start date of the pandemic differed between the two. The virus spread differently in New York compared to the remainder of the US. There was greater access to testing in the New York. Right away you know the groups are not similar.
This variable of interest was detected cases. If you are following the data you may already know that only 1/8 cases are being captured. The prevalence of the virus in the rest of the country compared to New York is currently unknown. This is important because they are comparing these groups, so the underlying risk of exposure should be equal. States varied in the measures they implemented which differed from New York City. Regional heterogeneity existed. There was no attempt by the authors to compare the degree and effectiveness of the other preventative measures. There were no demographics analyzed or even recorded. There was no attempt to determine mask use prior to and after the mask order. There was simply the mandate and a record of before and after cases. Therefore, the risk of exposure between the two groups is unknown.
There was no attempt to determine if another variable could have impacted their forecast and to what degree. They should have looked at cases before and after stay at home orders, success of case tracking systems, testing rates, ages of cases, race of cases, zip codes, etc…There are so many reasons that could have interfered with their forecast. Maybe the virus was naturally leveling off, maybe most of the nursing home infections were completed, maybe stricter quarantine policies were to blame. They mention some of these things but there was no analysis. There was no data discussed. There was no attempt to correct for confounding.
You may be asking about why confounding matters. If you were to design a study looking at ice cream consumption and the risk of drowning you would find a strong correlation to ice cream and drowning. Should we stop eating ice cream? Of course not, that’s ridiculous. People eat more ice cream in the summer, and there is more swimming in the summer, therefore there is more drowning. It has nothing to do with ice cream. That’s confounding.
Once the methods for the study are determined to be invalid the end result cannot be trusted.
Even though they can’t be trusted, I will spend a short time discussing the results. The results section is broken down into three sections. The first is a before and after representation for Wuhan, Italy, and New York. They mark the date on the graph that the order was publicized in each region and the event curve begins to decelerate. Although compelling, it is hardly proof due to the issues discussed above. The authors state in this section that other measures likely contributed to flattening the curve, which contradicts their stated conclusion. The data from China is not reliable (period!). The second section is the one I have most issue with. In this section they compare a forecasted linear progression of cases to the actual case rate after the mask order. This graph was not built with data. They use words like “relative” and “reasonable” to describe the inputs for their model. This is not science, or a valid analysis. It is playing with numbers and forecasting a plausible difference. It’s nice but it doesn’t prove anything. The third section compares New York to the US. As discussed above this is not a valid comparison. Also, if you view the curves, they appear very similar, if anything the US looks a little better. The graphs are very similar except the US graph has a two-week lag time. This would be expected since New York was the epicenter and the virus has a 1-2-week incubation period.
You may be thinking…So what’s the big deal? If you have read previous posts, I did something similar with my post about a COVID vaccine. The difference is I was very clear that these numbers were projections and simulations. My post was based on a true story, not an actual true story. The language they use to describe their results is inaccurate and misleading. In the abstract they state masks alone significantly reduce the number of infections. This is a definitive statement that the study does not support. Then they had the arrogance to write a recommendation section for policy makers. It is important to understand that when a scientist uses the language such as this, they claim there is no doubt. There is plenty of doubt as evidenced by my dissection above. There is no truth. Only a correlation.
Despite this language this paper gets published. In a peer reviewed journal none the less. I really don’t care if they post it on Texas A&M website, or a blog, or whatever. The fact that it was peer reviewed and approved with this language is not becoming of the scientists, the reviewers that approved it, and the publisher. It discredits the entire peer review process, which has been losing credibility for quite some time due to the lack of an objective validity scoring system and the politicized distribution of information and publishers.
Then the media picked up on the article and used it for click bait. You would think there could be a science officer responsible for educating journalists and editors about the merits of the studies they choose to promote. Alas, I don’t think this position exists. Its misinformation run amok by individuals with a huge responsibility. This folly is repeated over and over again by main stream media. Mask wearing theoretically makes sense. There is anecdotal evidence it helps. I’m not saying it’s wrong, or we should abandon masks. My point is, it was not proven by this study. Not even close.
Social media allows so many people to fall into the trap as well. Confirmation bias accelerates the momentum exposing more individuals to misinformation than ever before. Individuals are easily influenced by their social peers and perpetuate the spread not realizing the underlying information is misleading and inaccurate. I doubt many of the retweets, and shared posts I saw took the time to review the paper presented by the article. My advice, seek to understand, do not seek to be right, and be a skeptic. When someone stands up and states “This is the truth, there is no doubt,” they are probably a charla