Skip to main content

Problems in the reporting of acne clinical trials: a spot check from the 2009 Annual Evidence Update on Acne Vulgaris

Abstract

In the course of producing the 2009 NHS Evidence - skin disorders Annual Evidence Update on Acne Vulgaris, 25 randomised controlled trials were examined. From these, at least 12 potentially serious problems of trial reporting were identified. Several trials concluded no effect of a treatment yet they were insufficiently powered to exclude potentially useful benefits. There were examples of duplicate publication and "salami publication", as well as two trials being combined and reported as one. In some cases, an incorrect "within-groups" statistical comparison was made and one trial report omitted original efficacy data and included only P values. Both of the non-inferiority studies examined failed to pre-specify a non-inferiority margin. Trials reported as "double-blind" compared treatments that were dissimilar in appearance or had differing adverse effect profiles. In one case an intention-to-treat analysis was not performed and there was a failure to account for all of the randomized participants. Trial results were made to sound more impressive by selective outcome reporting, emphasizing the statistical significance of treatment effects that were clinically insignificant, and by the use of larger-sounding odds ratios rather than rate ratios for common events. Most of the reporting problems could have been avoided by use of the CONSORT guidelines and prospective trial registration on a public clinical trials database.

Peer Review reports

Introduction

Each year, NHS Evidence - skin disorders (a national specialist library funded by NICE, available at http://www.library.nhs.uk/skin) publishes an Annual Evidence Update on Acne Vulgaris, which is a search for new evidence published or indexed in the last year [1]. NHS Evidence - skin disorders also produces Annual Evidence Updates on atopic eczema, psoriasis and skin cancer. The purpose is to make our community of clinical users (mainly dermatologists, general practitioners and nurses) aware of newly published research studies, to discuss their significance for clinical practice, and to warn of any methodological issues in their interpretation.

The Annual Evidence Updates normally search for systematic reviews and guidelines, because of the potential hazards in commenting on single randomized controlled trials or RCTs [2]. However, as only one systematic review on acne was found for our 2009 Annual Evidence Update, which was published on 2nd March 2009, we also searched for new RCTs published or indexed over the previous year since the last Annual Evidence Update [1, 3]. A full description of the methodology and search strategies used can be found on the Annual Evidence Update web pages [1].

The RCTs that were found for the 2009 Annual Evidence Update comprise a "spot check" of acne trials published over a one year period. In the course of putting together the Annual Evidence Update [1, 3] the authors were struck by a high frequency of problems in the reporting and interpretation of these acne RCTs, which are now highlighted in this article. Our perspective in this commentary is not to condemn well-intentioned authors but to highlight common problems that may not be immediately obvious to a wider readership in the hope of reducing bias, improving patient welfare and influencing the future conduct and reporting of clinical trials on acne. The problems highlighted in this commentary are not restricted to acne trials and we hope that the examples given will help to provide further evidence for the need to improve standards in the reporting of all clinical trials.

Discussion

From the 25 RCTs found for the 2009 Annual Evidence Update, at least 12 major problems of trial reporting were identified; these are listed in Table 1 in the order in which the trials appeared in the Update [1].

Table 1 Common problems in the reporting of acne trials

Lack of power

The first problem identified was RCTs being insufficiently powered to provide evidence of no difference between trial interventions. One study [4], designed to assess the effect of exercise on acne, randomized a total of 30 teenage boys to avoid or perform exercise, and the latter group was further divided into those who showered 1 hour or 4 hours later. The small numbers in the three groups produced very wide confidence intervals that illustrated the underpowered nature of the study. It was reported as a pilot study but a power calculation had been performed. A second study [5] which recruited 60 subjects claimed equivalence between an oral acne therapy and the same treatment in combination with topical agents. However, an equivalence margin was not determined in advance and the equivalence claim was made on the basis of non-significant tests for superiority, a problem frequently encountered in clinical trial reporting [6]. In essence, no evidence of an effect had been misinterpreted as evidence of no effect.

Duplicate publication

There were two sets of duplicate publications, in which the same trial was published more than once, identified in the 2009 Annual Evidence Update. The first [7] was an additional analysis in a subgroup of patients from a trial on low glycaemic load for treating acne that had already been reported twice. The original duplicate publications [8, 9] had been picked up by the 2008 Annual Evidence Update [10]; the papers reported the same trial but failed to cross-reference each other and the journal editors had not been informed. In the second set of duplicate publications, primary efficacy outcomes were presented in one paper [11] without indicating the presence of secondary efficacy outcomes, with the latter then being presented in a second paper four months later [12]. The secondary efficacy variables were similar to the primary variables and showed similar results. We believe that all relevant trial results (especially efficacy results) should be presented in one paper. If there are good reasons to split the results, the seminal index paper should make at least some reference to the measurement of other outcomes and whether there is a plan to publish them elsewhere. Several issues arise from duplicate publication. There could be distortion of any subsequent meta-analysis if the study results are counted twice - such a problem has already arisen with the duplicate publication on low glycaemic load [13]. In addition, journal copyright may be infringed, and multiple articles take up additional journal resources. It has also been demonstrated how duplicate publications result in higher citations [14].

Testing the wrong thing

Another pitfall that we picked up was the issue in a parallel group study of performing a "within-groups" comparison, rather than the correct "between-groups" analysis of change from baseline. In its abstract, a study that compared a computer presentation with a written information handout stated benefit in favour of the computer approach based on a within-groups comparison, despite a non-significant between-groups comparison in the main article text [15]. Another study of two topical treatments for active acne only performed a within-groups comparison [16], so no account was made for the effect of natural disease history, in particular regression to the mean. Whether such erroneous highlighting of results is deliberate or accidental is unclear - we suggest that it can be a ploy used by authors to try and "save face" in the light of an essentially inconclusive study, especially as some journal editors and clinicians will not spot the lack of a correct between-groups statistic.

"Salami publication" and absent inferiority margins

"Salami publication" of a clinical trial involves splitting the results from a single trial into several packages that are then published separately and may artificially increase the impact of the study [17]. This issue affected a three-armed parallel groups study registered as a single trial on the ClinicalTrials.gov database [18]. Two of the treatment arms were separately compared with the third arm and each comparison was published as a stand-alone trial [19, 20], albeit in the same journal supplement. It would have been straightforward to report the results of all three arms in a single publication. Neither publication referenced the other. Another problem with the trial is that it was reported as a non-inferiority study but details of the 15% non-inferiority margin were not stated in the ClinicalTrials.gov register entry, so it is uncertain whether this margin was chosen prospectively or retrospectively. We also found an acne study that compared the same antibiotic at a low dose compared with the standard dose for acne which was essentially a non-inferiority trial, but no non-inferiority margin was specified [21].

Reporting two independent studies as one

Almost the reverse of duplicate publication is pooling the results of more than one previously unpublished, independent clinical trial in a combined analysis, rather than reporting the results separately. Under such circumstances, the larger, combined analysis could produce a significant result when individually the trials fail to reach significance. Two pairs of RCTs combined in single analyses were spotted in the Annual Evidence Update [22, 23]. Results of the individual, independent studies were not presented separately. In both cases these were industry-funded studies of novel topical preparations conducted in North America. It is presumed that two identical RCTs were needed for FDA licensing approval. Whilst it is sometimes appropriate to combine similar studies using a formal meta-analytical approach, we suggest that it is inappropriate to only present combined results in the primary publication of two pivotal RCTs [24].

Were they really "double-blind"?

In RCTs of topical therapy, particular care is needed to ensure that the comparator preparations closely resemble each other, to prevent loss of participant or investigator masking. In placebo-controlled studies the ideal comparator is the vehicle used for the active treatment, but this is not necessarily possible in head-to-head studies of two active treatments. One trial was reported to be "double-blind" but it compared an acne cream with a gel [25], which would differ in appearance and properties on the skin. Another common reason for loss of blinding in RCTs is a frequent adverse effect associated with one of the treatments and not the other. In topical acne therapy, skin irritation often differs between preparations and this probably caused some loss of blinding in a topical retinoid trial reported to be double-blind [26].

Absent data and missing patients

Good practice in trial reporting is concerned with providing as much original trial data as possible. Confidence intervals are needed as well as just P values. Unfortunately, one efficacy study failed to provide any trial data and relied on stating P values along with a potentially unrepresentative selection of clinical photographs [27]. Another issue of good practice with RCT reporting is to account for all the patients randomized to prevent attrition bias, with an intention-to-treat analysis and a pre-specified method to deal with missing values. One trial randomized 45 participants but included data for only 30 of them at the final 8 week endpoint; no data or explanation were given about those participants who dropped out of the study [26].

Data fishing, impressive P values, and "plumped up" odds ratios

There are several ways in which a trial report can make the results appear more impressive than they really are. One of these is to "data fish" amongst a large number of outcomes, rather than focus on a single, pre-specified primary outcome. This was probably the case in an acne trial that displayed only its positive outcomes in the abstract [28]. Another issue is reliance on a statistically significant effect that may be insignificant in clinical practice. An impressive P value of 0.001 was used to justify the efficacy of an acne therapy [29], but this equated to only a modest 11% reduction in the acne lesion count, which probably would not be meaningful to a patient. Finally, use of more impressive sounding odds ratios rather than rate ratios was spotted [11] which will give an overestimate when event rates are frequent [30].

Conclusion

One of the foundations of evidence-based practice is the availability of high quality evidence on which to base clinical decisions. Although some of the trials found in the Annual Evidence Update were reported to a high standard, around a half contained potentially serious reporting problems and framing biases that could mislead the clinical readership.

Many of the problems outlined in this article could have been avoided by adherence to the CONSORT guidelines [31] and prospective trial registration. CONSORT has provided the gold standard for RCT reporting, and adoption of the guidelines by many, but not all, journals has ensured a standardized method of quality control. The CONSORT list can also be used to aid trial design at the planning stage. Prospective trial registration on a public clinical trials database, or publication of the study protocol, is also very helpful for subsequent users of research to ensure that primary endpoints are stated prospectively. In essence, the study designers are asked to "nail their flag to the mast" in advance in terms of their most important endpoint. Again, adoption of this as a requirement for publication by journals has helped to promote its use.

References

  1. 2009 Annual Evidence Update on Acne Vulgaris - Home Page. [http://www.library.nhs.uk/SKIN/ViewResource.aspx?resID=306605&tabID=289&catID=8275]

  2. Ioannidis JP: Contradicted and initially stronger effects in highly cited clinical research. JAMA. 2005, 294: 218-228. 10.1001/jama.294.2.218.

    Article  CAS  PubMed  Google Scholar 

  3. Ingram JR, Grindlay DJ, Williams HC: Management of acne vulgaris: an evidence-based update. Clin Exp Dermatol. 2010, 35: 351-4. 10.1111/j.1365-2230.2009.03683.x.

    Article  CAS  PubMed  Google Scholar 

  4. Short RW, Agredano YZ, Choi JM, Kimball AB: A single-blinded, randomized pilot study to evaluate the effect of exercise-induced sweat on truncal acne. Pediatr Dermatol. 2008, 25: 126-128. 10.1111/j.1525-1470.2007.00604.x.

    Article  PubMed  Google Scholar 

  5. Dhir R, Gehi NP, Agarwal R, More YE: Oral isotretinoin is as effective as a combination of oral isotretinoin and topical anti-acne agents in nodulocystic acne. Indian J Dermatol Venereol Leprol. 2008, 74: 187-10.4103/0378-6323.39727.

    Article  PubMed  Google Scholar 

  6. Greene WL, Concato J, Feinstein AR: Claims of equivalence in medical research: are they supported by the evidence?. Ann Intern Med. 2000, 132: 715-722.

    Article  CAS  PubMed  Google Scholar 

  7. Smith RN, Braue A, Varigos GA, Mann NJ: The effect of a low glycemic load diet on acne vulgaris and the fatty acid composition of skin surface triglycerides. J Dermatol Sci. 2008, 50: 41-52. 10.1016/j.jdermsci.2007.11.005.

    Article  CAS  PubMed  Google Scholar 

  8. Smith RN, Mann NJ, Braue A, Makelainen H, Varigos GA: A low-glycemic-load diet improves symptoms in acne vulgaris patients: a randomized controlled trial. Am J Clin Nutr. 2007, 86: 107-115.

    CAS  PubMed  Google Scholar 

  9. Smith RN, Mann NJ, Braue A, Makelainen H, Varigos GA: The effect of a high-protein, low glycemic-load diet versus a conventional, high glycemic-load diet on biochemical parameters associated with acne vulgaris: a randomized, investigator-masked, controlled trial. J Am Acad Dermatol. 2007, 57: 247-256. 10.1016/j.jaad.2007.01.046.

    Article  PubMed  Google Scholar 

  10. 2008 National Knowledge Week on Acne Vulgaris - Homepage. [http://www.library.nhs.uk/skin/Page.aspx?pagename=ACNEHOME]

  11. Koltun W, Lucky AW, Thiboutot D, Niknian M, Sampson-Landers C, Korner P, Marr J: Efficacy and safety of 3 mg drospirenone/20 mcg ethinylestradiol oral contraceptive administered in 24/4 regimen in the treatment of acne vulgaris: a randomized, double-blind, placebo-controlled trial. Contraception. 2008, 77: 249-256. 10.1016/j.contraception.2007.11.003.

    Article  CAS  PubMed  Google Scholar 

  12. Lucky AW, Koltun W, Thiboutot D, Niknian M, Sampson-Landers C, Korner P, Marr J: A combined oral contraceptive containing 3-mg drospirenone/20-microg ethinyl estradiol in the treatment of acne vulgaris: a randomized, double-blind, placebo-controlled study evaluating lesion counts and participant self-assessment. Cutis. 2008, 82: 143-150.

    PubMed  Google Scholar 

  13. Spencer EH, Ferdowsian HR, Barnard ND: Diet and acne: a review of the evidence. Int J Dermatol. 2009, 48: 339-347. 10.1111/j.1365-4632.2009.04002.x.

    Article  CAS  PubMed  Google Scholar 

  14. Wilhelmus KR: Redundant publication of clinical trials on herpetic keratitis. Am J Ophthalmol. 2007, 144: 222-226. 10.1016/j.ajo.2007.04.047.

    Article  PubMed  Google Scholar 

  15. Koch PE, Ryder HF, Dziura J, Njike V, Antaya RJ: Educating adolescents about acne vulgaris: a comparison of written handouts with audiovisual computerized presentations. Arch Dermatol. 2008, 144: 208-214. 10.1001/archdermatol.2007.35.

    Article  PubMed  Google Scholar 

  16. Sharquie KE, Noaimi AA, Al-Salih MM: Topical therapy of acne vulgaris using 2% tea lotion in comparison with 5% zinc sulphate solution. Saudi Med J. 2008, 29: 1757-1761.

    PubMed  Google Scholar 

  17. Jamrozik K: Of sausages and salami. Aust N Z J Public Health. 2004, 28: 5-6. 10.1111/j.1467-842X.2004.tb00623.x.

    Article  PubMed  Google Scholar 

  18. ClinicalTrials.gov. Trial Identifier NCT00469755. [http://www.clinicaltrials.gov/ct2/results?term=NCT00469755]

  19. Pariser D, Colon LE, Johnson LA, Gottschalk RW: Adapalene 0.1% gel compared to tazarotene 0.1% cream in the treatment of acne vulgaris. J Drugs Dermatol. 2008, 7: s18-23.

    PubMed  Google Scholar 

  20. Gold LS, Colon LE, Johnson LA, Gottschalk RW: Is switching retinoids a sound strategy for the treatment of acne vulgaris?. J Drugs Dermatol. 2008, 7: s11-17.

    PubMed  Google Scholar 

  21. Toossi P, Farshchian M, Malekzad F, Mohtasham N, Kimyai-Asadi A: Subantimicrobial-dose doxycycline in the treatment of moderate facial acne. J Drugs Dermatol. 2008, 7: 1149-1152.

    PubMed  Google Scholar 

  22. Raimer S, Maloney JM, Bourcier M, Wilson D, Papp K, Siegfried E, Garrett S: Efficacy and safety of dapsone gel 5% for the treatment of acne vulgaris in adolescents. Cutis. 2008, 81: 171-178.

    PubMed  Google Scholar 

  23. Thiboutot D, Zaenglein A, Weiss J, Webster G, Calvarese B, Chen D: An aqueous gel fixed combination of clindamycin phosphate 1.2% and benzoyl peroxide 2.5% for the once-daily treatment of moderate to severe acne vulgaris: assessment of efficacy and safety in 2813 patients. J Am Acad Dermatol. 2008, 59: 792-800. 10.1016/j.jaad.2008.06.040.

    Article  PubMed  Google Scholar 

  24. Katz KA, Kim CY, Williams HC: Reporting clinical trials: why one plus one does not equal two. J Am Acad Dermatol. 2009, 61: 1082-1083. 10.1016/j.jaad.2009.06.077.

    Article  PubMed  Google Scholar 

  25. Marcinkiewicz J, Wojas-Pelc A, Walczewska M, Lipko-Godlewska S, Jachowicz R, Maciejewska A, Bialecka A, Kasprowicz A: Topical taurine bromamine, a new candidate in the treatment of moderate inflammatory acne vulgaris: a pilot study. Eur J Dermatol. 2008, 18: 433-439.

    CAS  PubMed  Google Scholar 

  26. Ruamrak C, Lourith N, Natakankitkul S: Comparison of clinical efficacies of sodium ascorbyl phosphate, retinol and their combination in acne treatment. Int J Cosmet Sci. 2009, 31: 41-46. 10.1111/j.1468-2494.2008.00479.x.

    Article  CAS  PubMed  Google Scholar 

  27. Ghoshal L, Banerjee S, Ghosh SK, Gangopadhyay DN, Jana S: Comparative evaluation of effectiveness of adapalene and azithromycin, alone or in combination, in acne vulgaris. Indian Journal of Dermatology. 2007, 52: 179-183. 10.4103/0019-5154.37721.

    Article  Google Scholar 

  28. Tanghetti E, Kircik L, Wilson D, Dhawan S: Solubilized benzoyl peroxide versus benzoyl peroxide/clindamycin in the treatment of moderate acne. J Drugs Dermatol. 2008, 7: 534-538.

    PubMed  Google Scholar 

  29. Ansarin H, Savabynasab S, Behzadi AH, Sadigh N, Hasanloo J: Doxycycline plus levamisole: combination treatment for severe nodulocystic acne. J Drugs Dermatol. 2008, 7: 737-740.

    PubMed  Google Scholar 

  30. Katz KA: The (relative) risks of using odds ratios. Arch Dermatol. 2006, 142: 761-764. 10.1001/archderm.142.6.761.

    Article  PubMed  Google Scholar 

  31. Schulz KF, Altman DG, Moher D: CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials. Trials. 11: 32-10.1186/1745-6215-11-32.

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Hywel C Williams.

Additional information

Competing interests

The authors declare that they have no competing interests.

Authors' contributions

DJCG carried out the database searches and identified the relevant randomised controlled trials. JRI and HCW carried out the critical appraisal of trial quality and reporting. JRI took the lead in drafting the paper, with contributions and edits by HCW and DJCG. All authors read and approved the final manuscript.

Rights and permissions

Open Access This article is published under license to BioMed Central Ltd. This is an Open Access article is distributed under the terms of the Creative Commons Attribution License ( https://creativecommons.org/licenses/by/2.0 ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Reprints and permissions

About this article

Cite this article

Ingram, J.R., Grindlay, D.J. & Williams, H.C. Problems in the reporting of acne clinical trials: a spot check from the 2009 Annual Evidence Update on Acne Vulgaris. Trials 11, 77 (2010). https://doi.org/10.1186/1745-6215-11-77

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/1745-6215-11-77

Keywords