Skip to main content

COVID-19: underpowered randomised trials, or no randomised trials?

Abstract

A recently published trial of face mask use to protect against COVID-19 demonstrated a key barrier to carrying out randomised trials in public health: the need for unattainably large sample sizes. For many public health interventions, the choice is not between sufficiently powered trials and underpowered trials, but between underpowered trials and no trials at all. Underpowered trials should be viewed as contributions to the larger body of evidence, alongside other studies of various sizes and designs, collectively assessed and synthesized in systematic reviews. Overemphasis on sample size calculation is probably more of a hindrance than a help to scientific progress.

Peer Review reports

Background

The recently published Danish trial of face masks for COVID-19 prevention [1] has spurred controversy [2,3,4]. A major criticism is that the trial was powered for an overly large effect size, i.e. the researchers based their sample size calculation on the assumption that a simple recommendation to use masks for protection outdoors, would halve the risk of infection.

Consequently, the study was underpowered: the 95% confidence interval around the modest 18% relative risk reduction in the face mask group included both a substantial effect (46% relative risk reduction), as well as an increased risk of infection (23% relative risk increase). Consequently, the trial results are of limited direct value for decision makers.

So, was the trial a waste of time?

Few trials in public health

The COVID-19 pandemic serves as an illustration of a well-known problem: with the notable exception of vaccines, randomised trials of public health interventions are rare. The Danish face mask trial is one of very few conducted trials of interventions to curb the spread of SARS-CoV-2—in stark contrast to the abundance of trials of pharmaceuticals and other clinical interventions for COVID-19 [5].

The need for large sample sizes is a key barrier to carrying out randomised trials in public health, mainly due to low event rates. Even during the most intense phases of the COVID-19 pandemic, weekly incidence rates have rarely exceeded 1% in the general population—e.g. for England, the highest ever recorded is 0.7% [6]. Demonstrating a risk reduction from, say, 1.2% to 0.8% requires 20,000 participants. When we planned a randomized trial to evaluate the effect of school closures to limit the spread of the virus, the results of our sample size calculations meant that we needed to enroll nearly all schools in Norway [7].

In practice, we are often not able to choose between having sufficiently powered trials and underpowered trials. The realistic choice is between underpowered trials and no trials at all.

Surely, some trial evidence must be better than no trial evidence?

The larger body of evidence, not individual trials

The main risk with underpowered trials is that of type II errors, i.e. the study findings may be too imprecise to demonstrate an important, but real effect. The Danish face mask trial may be an example of that. Further, when trials do not provide clear evidence of an effect, the results may be erroneously interpreted as evidence of no effect. A recent example is from a headline reporting on the Danish study: “Face masks do NOT protect the wearer from coronavirus” [8].

Another challenge is it may be considered unethical to recruit participants to a study that is unlikely to yield conclusive results [9].

However, the idea that a single trial provides a definitive answer should have been abandoned long ago. Underpowered trials should be viewed as contributions to the larger body of evidence, alongside other studies of various sizes and designs, collectively assessed and synthesized in systematic reviews [9]. Pooling of findings from well-conducted but small trials in a meta-analysis can yield statistically robust results.

Face mask use in the community has been evaluated previously, e.g. in two trials with American college students during influenza seasons [10, 11]. The findings from those studies—both with inconclusive results—are included in the body of evidence that has informed recommendations about face mask use during the COVID-19 pandemic [12, 13]. The Danish face mask supplements the earlier trial findings and demonstrates that it is feasible to conduct such trials during a pandemic.

Another reason to tone down the emphasis on sample size calculations, in general, is that researchers would be less incentivised to game the system by adjusting assumptions about expected effect sizes and event rates in order to arrive at feasible sample sizes [9]. It might also discourage the use of composite endpoints or other inventions that mainly serve the purpose of artificially inflating the statistical power of trials [9].

Conclusion

Sufficiently powered trials are certainly preferable to underpowered ones, but overemphasis on sample size calculation is probably more of a hindrance than a help to scientific progress.

Underpowered trials should be viewed as contributions to the larger body of evidence, alongside other studies of various sizes and designs, collectively assessed and synthesized in systematic reviews.

Availability of data and materials

Not applicable

References

  1. 1.

    Bundgaard H, Bundgaard JS, Raaschou-Pedersen DET, von Buchwald C, Todsen T, Norsk JB, Pries-Heje MM, Vissing CR, Nielsen PB, Winslow UC, et al. Effectiveness of adding a mask recommendation to other public health measures to prevent SARS-CoV-2 infection in Danish mask wearers: a randomized controlled trial. Ann Intern Med. 2021;174(3):335–43. https://doi.org/10.7326/M20-6817. Epub 2020 Nov 18.

  2. 2.

    Laine C, Goodman SN, Guallar E. The role of masks in mitigating the SARS-CoV-2 pandemic: another piece of the puzzle. Ann Intern Med. 2020;174(3):419–20. https://doi.org/10.7326/M20-7448. Epub 2020 Nov 18.

  3. 3.

    Haber NA, Wieten SE, Smith ER. Letter of concern regarding “Reduction in COVID-19 infection using surgical facial masks outside the healthcare system”. Dan Med J. 2020;67(12):A205063.

    PubMed  Google Scholar 

  4. 4.

    Frieden TR, Cash-Goldwasser S. Of masks and methods. Ann Intern Med. 2021;174(3):421-2. https://doi.org/10.7326/M20-7499. Epub 2020 Nov 18.

  5. 5.

    McCartney M. We need better evidence on non-drug interventions for covid-19. BMJ. 2020;370:m3473.

    Article  Google Scholar 

  6. 6.

    GOV. UK Coronavirus (COVID-19) in the UK. https://coronavirus.data.gov.uk/details/cases. Accessed 16 Mar 2021.

  7. 7.

    Fretheim A, Flato M, Steens A, Flottorp SA, Rose CJ, Telle KE, Kinge JM, Schwarze PE. COVID-19: we need randomised trials of school closures (supplemental material: The School Opening in the Age of Pandemic (SOAP) study: a cluster-randomised reintroduction of school activities in Norway). J Epidemiol Community Health. 2020;74(12):1078–9. https://doi.org/10.1136/jech-2020-214262.

    Article  PubMed  Google Scholar 

  8. 8.

    Face masks do NOT protect the wearer from coronavirus, but will stop them from infecting other people, Danish study finds. https://www.dailymail.co.uk/health/article-8962397/Danish-study-finds-face-masks-provide-limited-protection-wearer.html. Accessed 21 Nov 2020.

  9. 9.

    Guyatt GH, Mills EJ, Elbourne D. In the era of systematic reviews, does the size of an individual trial still matter. PLoS Med. 2008;5(1):e4. https://doi.org/10.1371/journal.pmed.0050004.

    Article  PubMed  PubMed Central  Google Scholar 

  10. 10.

    Aiello AE, Murray GF, Perez V, Coulborn RM, Davis BM, Uddin M, Shay DK, Waterman SH, Monto AS. Mask use, hand hygiene, and seasonal influenza-like illness among young adults: a randomized intervention trial. J Infect Dis. 2010;201(4):491–8. https://doi.org/10.1086/650396.

    Article  PubMed  Google Scholar 

  11. 11.

    Aiello AE, Perez V, Coulborn RM, Davis BM, Uddin M, Monto AS. Facemasks, hand hygiene, and influenza among young adults: a randomized intervention trial. PLoS One. 2012;7(1):e29744. https://doi.org/10.1371/journal.pone.0029744.

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  12. 12.

    Advice on the use of masks in the context of COVID-19. Interim guidance 5 June 2020: World Health Organization; 2020. WHO reference number: WHO/2019-nCov/IPC_Masks/2020.4. https://apps.who.int/iris/rest/bitstreams/1279750/retrieve. Accessed 21 Nov 2020.

  13. 13.

    European Centre for Disease Prevention and Control. Using face masks in the community. Stockholm: ECDC; 2020.

Download references

Acknowledgements

Not applicable

Funding

No specific funding for this work.

Author information

Affiliations

Authors

Contributions

The author read and approved the final manuscript.

Corresponding author

Correspondence to Atle Fretheim.

Ethics declarations

Ethics approval and consent to participate

Not applicable

Consent for publication

Not applicable

Competing interests

The author declares that he has no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Fretheim, A. COVID-19: underpowered randomised trials, or no randomised trials?. Trials 22, 234 (2021). https://doi.org/10.1186/s13063-021-05209-5

Download citation