Association Between Universal Masking in a Health Care System and SARS-CoV-2 Positivity Among Health Care Workers
Over the weekend JAMA published the study above. It gained notoriety because the CDC has sited it as evidence to support a universal masking recommendation. I am going to spend the first couple of paragraphs describing the study and the results to provide a basic understanding then I will dive in.
The study was executed at Mass General Bingham Health System (MGB). MGB is the largest health system in Massachusetts. A universal masking order was issued on March 25th. It was a retrospective cohort, which means they created two groups (cohorts) and divided them based on a specific time or event. The study period was divided into the time prior to universal masking, a two-week transition phase, and universal masking. The endpoint was a positive COVID test in a healthcare worker. The group prior to masking was the control group, post masking is the experimental group.
Validity of Methods
For the results of an observational study to be valid the risk and circumstances of exposure needs to be equal in both groups. The methods for detecting the endpoint needs to be equal in both groups. Finally, an assessment for confounding needs to be performed.
In this study we are uncertain of the risk of exposure for the following reasons. During the pandemic there were many policy changes that were changing within the state of Massachusetts and likely MGB that may have affected the prevalence of the disease throughout the study period. The authors highlight the dates of some of these changes, but there were likely several more that were implemented. There was no attempt to control for the changing landscape. They do not provide prevalence data in the state of Massachusetts or number of treated COVID cases during the study period. Prevalence is a big deal when it comes to risk. The more times you are exposed, the more likely you are to get the virus. Given that the risk of exposure throughout the study period is unknown, it is difficult to trust the results.
The authors did not comment on the type of COVID-PCR test used to detect the virus. If you’ve followed the news you are aware that the accuracy of the available tests varies considerably. Also, early in the pandemic testing supplies were scarce. Hospitals had to use whatever test was available, which could have created variability in test accuracy. Also, as we learned more about COVID the criteria for testing changed. The authors do not provide the test used, or if testing assays changed throughout the period. It is uncertain if the criteria to be tested changed. Since we are uncertain that the circumstances and methods for testing were equal throughout the study period, it is difficult to trust that each group was treated equally.
There was no assessment for confounding.
Overall the methods of this study lacked validity for an observational study. The risk of exposure in each group is unknown. It is unknown if circumstances and methods to detect the outcome was consistent. There was no assessment for confounding. The results should be viewed with skepticism.
Results
During the entire study period they ran roughly 9000 tests and had around 1200 positives. The positivity rate in the control group (pre-mask) increased from 0% on day one to 21% at the end of observation. The rate increased 1.16% daily. The positivity rate in the experimental group decreased from 14.65% to 11.45% with a daily negative decline of 0.49%.
They only provide confidence intervals for the net slope change in the linear regression between the pre-mask and post-mask group. I’m sure you’re thinking “what the heck is that?” To simplify, it is supposed to represent the rate of change in correlation between positive cases and mask use. Remember the confidence interval is the number range in which the true result likely lies with 95% confidence. This value was -1.13%- -2.15%.
My Assessment
I really enjoyed this study because it demonstrates the difficulty of drawing conclusions from observational studies. It is particularly difficult to draw conclusions from retrospective cohort studies. Even though this study was significantly better than the original study I appraised from Texas A&M, it lacked strong validity and therefore the credibility of the results is compromised. Once again, the study circulated through the media touted by experts as strong evidence. Even though I know many people want it to be true, it just isn’t so. Its weak, if anything.
The results suggest that the universal mask order reduced positive tests on a daily basis. However, there was very little discussion about other possible explanations. Besides the pitfalls discussed in the methods section above there was no assessment for mask compliance. A survey, or security camera review of mask use prior to and after the order would have provided significant insight. It is likely that many workers were using masks prior to the mandate convoluting results. Also, they lumped universal masking of patients and workers into the same cohort even though these dates were two weeks apart. It would have been very interesting to see a rate of change between these orders. They likely implemented many other interventions during this time as well. Common places where healthcare workers gather, like the cafeteria and coffee stands, may have been closed. Negative pressure rooms may have been added. Elective procedures were stopped likely reducing the number of staff in the hospital. etc...There are many theories that could explain the results. The purpose of the scientific method is to vanquish alternative theories that support the null hypothesis. In my opinion, this hurdle was not cleared. It’s possible the study gets a pass because on the surface it appears to confirm a popular theory. (confirmation bias)
(This is going to get a little technical for a paragraph)
Then there is the statistics. A linear regression analysis was used to determine the slope of the infection curve. The first day post universal masking of workers had the highest number of infections. The day after that was one of the lowest. In the graph they appear to include that first day in the pre-order regression, while the next day is placed in the transition phase. The first day after the transition phase was the highest positive test day. This was included in the post mandate phase. They report change in slope of the regression curve as their confirmatory statistic. The greater the degree of change, the more convincing the correlation. Their results would be significantly different: 1) if these days did not happen,2) if they were excluded, or 3) if their dates were moved one day forward or one day backward (see figure below). These two data points greatly influenced the results. They may be outliers. The occurrence was likely random. Was the difference they observed simply from random chance from these two days alone? That’s my point
Also, they did not report the correlation coefficient. This is a reported value between 0-1. Zero is no correlation, 0.5 is maybe, and 1 is highly correlation. The strength of the correlation matters. Finally, the confidence interval in the change in slope is 1.13-2.15. This is quite wide considering the compounding effects of daily change in slope. Presenting a best and worst-case scenario would provide greater insight into the range of potential results. Also, no sensitivity analysis to evaluate the risk of confounding. The statistical metric choice is odd as well. Generally, results in observational studies are reported as odds ratio’s, hazard rations, or Wilcoxon rank sums for non-normally distributed data.
I say all that to say this…it is very difficult to prove a hypothesis with observational studies because of the risks stated. In fact, it is nearly impossible. The hope is to find a correlation. The validity in the results is in the details of the design. The small details make a big difference. Believe me. As a former researcher myself, (whom has produced mostly rubbish) I know most of my work, at best, was thought provoking. I never proved anything. This is after many grueling hours collecting and analyzing data.
The authors deserve credit for their stated conclusions. It was an appropriate representation of their results without overstating what they discovered. Unfortunately, many others overemphasized the findings. In the second sentence of their conclusion they mention the risk of confounding on their results. Also, they did not generalize their findings to outside of healthcare settings.
In fact, healthcare facilities are an ideal setting for masks. Not simply for the reasons everyone touts (cheap, available, easy to use?), but because of genuine authority and tangible threat. The authority is your administrator that can easily view you either wearing a mask or not. The threat are the patients, each one is a forced contact, whether they are there for COVID or not. Theoretically, you can expect compliance to be higher amongst workers and patients due to their environment. Further reducing the generalizability of the results.
The authors also comment that the quality of evidence is likely the best we will get since it is unethical to randomize individuals to unmasked or masked in the setting of a potential airborne threat. I agree, it would be unethical. We will have to accept observational studies as best evidence. That does not mean we should base decisions on poor observational data. Studies should be executed well and pitfalls should be mitigated as best they can. “Experts” should be able to discern between strong and weak evidence. The peer review process is in place to protect against this. This study was published in JAMA, one of medicines most impactful journals. It was cited by the director the CDC on television. The interview has 230k views on YouTube alone. It was highlighted in the Wall Street Journal. But it fails to answer any question definitively.
In summary, the researchers showed masks may be an effective tool to help prevent the spread of COVID amongst healthcare workers, but the extent of impact and the effect from external factors on the results remain in question.
The remainder of this post is a few comments about masks.
1) The other day while watching SportsCenter I had an interesting observation. They were highlighting a baseball game. The batter was wearing a cloth mask at the plate. He smacks a home-run to left center. They show him rounding the bases with his mask on. When he reaches the dugout, he pulls his mask down, walks past each one of his teammates, high-fiving them, in a 120 ft2 dugout.
2)
This is a picture of Torch Lake Michigan July 4th, 2020. Torch lake is beautiful. Most of the people you see there are likely on holiday for the 4th of July. They are going to return to Detroit, Chicago, Grand Rapids, Indianapolis, Ann Arbor, etc…You get the picture. Mask or no mask the virus is going to spread from these events.
3) I was in the grocery store the other day, and I happened to witness the gathering of a small group of people. Four to be precise. Two couples. Likely under the age of 40. They saw each other across some produce then congregated. They were all wearing masks until started talking. Eventually each of them pulled their mask down below their mouth likely to reduce the effort of having conversation.
4)
Obviously, these are anecdotes (except #4). But they help prove my point. Reality often deviates from theory. Masks may be helpful tool to slow the virus. However, they do not appear to be as easy to use as hoped. Also, the evidence being published is simply not reliable when you reviewed critically. As publicity and compliance of masks has increased, so have COVID cases. I’m not going to spend a bunch of time theorizing why that may be. And I’m not a conspiracy theorist. There are plenty of explanations for this. I’m just stating, its more complicated than wearing masks, or not wearing masks. I’m also not making a statement that they should be abandoned. We just don’t know how large an impact they have. I wear one every day at work, when I’m in a crowd, and when a sign on a store front says I have to wear one or I can’t enter. You probably should do the same. You probably don’t need to put one on your 5-year-old when you’re walking the dog. A lot of this is common sense.
So be kind to others. Even if you think their stance is silly. They are probably thinking the same about you. Ultimately nobody is an “expert” on this pandemic, or the factors driving it. Engage in thoughtful debate. Intelligently orchestrate your argument. Don’t compare people who don’t wear masks to drunk drivers on social media, or accuse them of wanting to kill Grandma. That moves the conversation nowhere and only drives us further apart.
Link to original article: