Open Access

Review and publication of protocol submissions to Trials – what have we learned in 10 years?

  • Tianjing Li1Email author,
  • Isabelle Boutron2, 3,
  • Rustam Al-Shahi Salman4,
  • Erik Cobo5,
  • Ella Flemyng6,
  • Jeremy M. Grimshaw7 and
  • Douglas G. Altman8
Trials201618:34

https://doi.org/10.1186/s13063-016-1743-0

Received: 5 December 2016

Accepted: 5 December 2016

Published: 16 December 2016

Abstract

Trials has 10 years of experience in providing open access publication of protocols for randomised controlled trials. In this editorial, the senior editors and editors-in-chief of Trials discuss editorial issues regarding managing trial protocol submissions, including the content and format of the protocol, timing of submission, approaches to tracking protocol amendments, and the purpose of peer reviewing a protocol submission. With the clarification and guidance provided, we hope we can make the process of publishing trial protocols more efficient and useful to trial investigators and readers.

The importance of establishing a protocol at the outset of every clinical trial is unarguable [19]; however, the availability and completeness of protocols and their consistency with the final study report have been poor [1017]. Making protocols publicly available will contribute to improving the transparency of research and will enable researchers and prospective participants to learn about trials that are underway. Protocol availability also enables systematic reviewers, sponsors, regulators and others to understand the scientific rigor of the design and results, as well as to compare what was intended with what was described in the reports of trials in order to assess possible reporting bias [720].

Journals and trial registries are part of the solution to these problems because they provide a platform for making trial protocols publicly available [8]. Beginning in 2006, Trials pioneered open access publication of protocols for randomised controlled trials. Other journals have taken similar approaches [2123], but more than half of published trial protocols have been published in Trials [24]. From 2006 to November 2016, Trials published over 2100 protocols. In general, though at the editor’s discretion, those protocols with previous full external peer review as part of the process of receiving a grant from a major external funding body or ethics approval were published with peer review from a member of the editorial board [25], with the remainder undergoing full peer review. Trials provides authors with guidance [25], but uncertainties remain.

The roles of journals, journal editors and peer reviewers are not well defined when a trial protocol, as a stand-alone document, is submitted to a journal for consideration for publication. How much detail should be included in the protocol? What is the ‘right’ moment that the protocol should be made publicly available? How should subsequent protocol amendments be documented and linked to the published version of the protocol? What are appropriate peer review and editorial comments and suggestions when the trial is already underway, especially if the protocol has already been approved by the sponsor and the institutional review board (IRB) or research ethics committee (REC)? Herein we, the senior editors and editors-in-chief of Trials, discuss editorial issues regarding managing protocols submitted for publication and provide guidance for future protocol submissions to Trials.

What is expected in a protocol submission?

Articles that describe protocols are different from traditional trial ‘design articles’ and from the Methods sections of articles that report trial results. Design articles typically follow the format of a research report, including Background, Methods, Results and Discussion sections. We are aware of two types of design articles. Design articles published in clinical journals typically include an abbreviated description of the protocol, are published after enrolment has been completed, and report baseline characteristics of participants in the Results section. Design articles published in methodological journals focus on new, uncommon or challenging aspects of the trial design or procedures; protocol details are included only as necessary to understand the issues [26]. Here readers can find examples of two design articles, one for each type, arising from the same trial [27, 28].

In contrast, a protocol, as defined in the Standard Protocol Items: Recommendations for Interventional Trials (SPIRIT) statement, is

a document that provides sufficient detail to enable understanding of the background, rationale, objectives, study population, interventions, methods, statistical analyses, ethical considerations, dissemination plans, and administration of the trial; replication of key aspects of trial methods and conduct; and appraisal of the trial’s scientific and ethical rigour from ethics approval to dissemination of results. ([29]; p. 203)

Typically, each element above is described in a distinct section, and no performance or scientific results are included.

So, a protocol should provide sufficient details for readers to understand the general design and major operating features of the trial. However, ‘this document is generally distinguished from the study manual of operations by the absence of detailed instructions needed for the actual application of treatment and data collection in the trial’ [30]. Following the SPIRIT guideline is one way to strike a balance between comprehensiveness and conciseness [29, 31]. Trials endorses SPIRIT and requires authors to submit a completed SPIRIT checklist as an Appendix file and also to include a SPIRIT figure that depicts the schedule of enrolment, interventions and assessments within the main body of the article. The Trials Editorial Office is asked to verify the fulfilment of this requirement before assigning a new submission to an editor.

When reviewing protocol submissions, we have noticed that the completeness and understanding of the reporting of items in the SPIRIT checklist could be better among authors. In some submissions, the authors have entered on the SPIRIT checklist ‘not applicable’ for many critical items that in our view are applicable to any randomised controlled trial (e.g., Item 6c. Explanation for choice of comparators [29]). One possibility is that what the item refers to was not done in the trial; another is that the authors see the description as unnecessary. When an item is truly ‘not applicable’, it is to the authors’ advantage to provide a succinct explanation so that the readers understand the rationale for not addressing the item on the SPIRIT checklist in their protocol. For items 16b and 16c concerning allocation concealment mechanism and implementation [29], some authors have mistakenly referred to the section of the protocol that describes blinding. We encourage authors to read the SPIRIT explanation and elaboration document [31], which will help them improve their trial design, save time in publishing the manuscript and enhance transparency. In addition, the items on the SPIRIT checklist can be used as a general guide to organising the contents of the protocol.

When to submit a protocol for publication

Procedures in trials almost always change over the course of the trial for scientific and logistic reasons. Some of these changes are trivial, such as modification of an item on a data collection form to improve clarity, but others might be important enough to trigger a protocol amendment. For example, the IRB/REC, the data monitoring committee or the sponsor may recommend collecting safety outcomes that were unknown to the investigators at the outset of the trial, adding a new clinical site to achieve the target recruitment goal, or adopting a new statistical method to analyse the data. In general, protocol amendments should be limited to the most relevant and critical ones that affect the trial design and data collection procedure. The final version of the protocol, which tracks all protocol amendments, provides a fuller and more accurate account of the trial design and implementation than the version submitted to the sponsor for funding, to the IRB/REC for ethics approval, or to the data monitoring committee before enrolment begins.

We encourage investigators to submit their trial protocol for publication at an appropriate stage during the conduct of the trial; almost always this should be before recruitment is complete. In a ‘Trial status’ section, the authors should report the protocol version number and date, the date on which recruitment began, and the approximate date when recruitment will be completed. Authors should also describe any protocol amendments that were made after trial commencement, along with the rationale for such changes.

When amendments are made to the protocol after its publication in Trials, we encourage the authors to submit an ‘Update’, an article type that enables authors to modify the published record. Depending on the particular amendments, the Update article is generally free to submit and is linked back to the original article. Authors should bear in mind that the credibility of a trial can be damaged when changes are not reported, which also complicates the assessment of a trial report at a later date if inconsistencies between the protocol and the report are identified.

As of November 2016, Trials had published 38 updates, two-thirds of which are statistical analysis plans (SAPs). We do not consider SAPs as protocol amendments. A SAP describes the planned analysis for a clinical trial. It may be written as a separate document after finishing the protocol [32]. SAPs are required for trials on regulated products, a subset of all trials. The International Conference on Harmonisation defines a SAP as ‘a document that contains a more technical and detailed elaboration of the principal features of the analysis described in the protocol, and includes detailed procedures for executing the statistical analysis of the primary and secondary variables and other data’ ([32]; p. 35). At most pharmaceutical companies, the trial statisticians write the SAP following a standardised template; general guidance is under development [33]. SAPs typically include sections providing a brief description of the study design, a description of the analysis population, data-handling rules, precise analyses to be done and statistical methods to be used for each, as well as shells for planned tables, figures and listings [32]. We encourage trial investigators to submit SAPs, regardless of funding source, and link them to the trial protocols when they become available.

What is the purpose of peer review of trial protocol submissions to Trials?

One essential purpose of peer review of a protocol submission is to improve clarity of reporting. Because the protocol is more than a list of items (e.g., as in ClinicalTrials.gov or the World Health Organisation International Clinical Trials Registry Platform), it is reasonable for peer reviewers and editors to ask the authors to provide a fuller and richer narrative of certain aspects of the design or, in some cases, to shorten overly long descriptions, to clarify unclear and seemingly inconsistent descriptions, and to provide justifications and rationales for aspects of the chosen design.

The comments made by peer reviewers and editors should be geared towards improving the reporting rather than commenting on the design. It is possible, however, that the peer reviewers and editors may raise concerns over the design. These suggestions, although potentially valuable, are not typically within the purview of a journal and usually cannot be implemented by the investigators when the trial is already underway. When appropriate, we may ask the authors to discuss the issues and limitations raised by reviewers’ concerns. In addition, the readers of the final results paper can consider these potential limitations because the comments made by reviewers for Trials are open to the public and published alongside the article within the Open Peer Review reports. Finally, the editors may invite a commentary on the design of a trial or a set of trials.

Although our goal is to facilitate public availability of trial protocols so that readers can judge the trials’ scientific rigor, we reject some protocol submissions because of a lack of clarify in reporting, poor adherence to SPIRIT guidelines without justification, or serious concerns over the design that preclude replication of the trial by other researchers.

Can we improve the process?

The turnaround time for providing the authors with the first decision on a protocol submission could be shortened if (1) the authors optimised the quality of reporting and submitted a manuscript that followed the journal’s instructions and requirements, (2) the editors and peer reviewers were familiar with the journal’s position on the issues stated above, (3) more of the trials community contributed to the journal as editors and peer reviewers, and (4) peer reviewers continued by providing constructive comments intended to improve the quality of reporting. Ultimately, Trials aims to increase value and reduce waste in the reporting of clinical trials [7]. The principal recommendations by Chan et al. for reducing inaccessible research (see box below) illustrate the value that investigators can derive from fully accessible and clear reporting of clinical trial protocols [7], as well as the role that Trials plays in facilitating this.
Box

Recommendations from Chan et al. for reducing inaccessible research in The Lancet’s Increasing Value, Reducing Waste series [7]

1. Institutions and funders should adopt performance metrics that recognise full dissemination of research and reuse of original datasets by external researchers

2. Investigators, funders, sponsors, regulators, research ethics committees, and journals should systematically develop and adopt standards for the content of study protocols and full study reports, and for data sharing practices

3. Funders, sponsors, regulators, research ethics committees, journals, and legislators should endorse and enforce study registration policies, wide availability of full study information, and sharing of participant-level data for all health research

We encourage researchers to continue sending the protocols for their randomised controlled trials to Trials for consideration for publication. We also encourage researchers who are committed to full and transparent reporting to join Trials as editors or peer reviewers. We will strive to make the process for publishing trial protocols more efficient and useful to trial investigators and readers.

Declarations

Acknowledgements

Not applicable.

Funding

Not applicable.

Availability of data and material

Not applicable.

Authors’ contributions

TL led the writing of this editorial, and all other authors (IB, RASS, EC, EF, JMG and DGA) contributed equally with comments and feedback. All authors read and approved the final manuscript.

Competing interests

DGA and JMG are editors-in-chief of Trials, and TL, IB, RASS and EC are senior editors of Trials. EF is an employee of BioMed Central.

Consent for publication

Not applicable.

Ethics approval and consent to participate

Not applicable.

Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Authors’ Affiliations

(1)
Department of Epidemiology, Johns Hopkins Bloomberg School of Public Health
(2)
University Paris Descartes, Inserm Methods Team, CRESS-UMR 1153
(3)
Centre d’épidémiologie Clinique, Hôpital Hôtel-Dieu
(4)
Centre for Clinical Brain Sciences, University of Edinburgh
(5)
Statistics and Operations Research Department, Barcelona Tech
(6)
BioMed Central
(7)
Ottawa Hospital Research Institute
(8)
Centre for Statistics in Medicine, University of Oxford, Botnar Research Centre

References

  1. Summerskill W, Collingridge D, Frankish H. Protocols, probity, and publication. Lancet. 2009;373(9668):992.View ArticlePubMedGoogle Scholar
  2. Lassere M, Johnson K. The power of the protocol. Lancet. 2002;360(9346):1620–2.View ArticlePubMedGoogle Scholar
  3. Jones G, Abbasi K. Trial protocols at the BMJ. BMJ. 2004;329(7479):1360.View ArticlePubMedPubMed CentralGoogle Scholar
  4. International Conference on Harmonisation of Technical Requirements for Pharmaceuticals for Human Use (ICH) E6 Good Clinical Practice: Consolidated Guidance, dated April 1996. http://www.fda.gov/downloads/Drugs/.../Guidances/ucm073122.pdf. Accessed 19 Dec 2016.
  5. Altman DG, Furberg CD, Grimshaw JM, Rothwell PM. Lead editorial: Trials - using the opportunities of electronic publishing to improve the reporting of randomised trials. Trials. 2006;7:6.View ArticlePubMedPubMed CentralGoogle Scholar
  6. Krleža-Jerić K, Chan AW, Dickersin K, Sim I, Grimshaw J, Gluud C. Principles for international registration of protocol information and results from human trials of health related interventions: Ottawa statement (part 1). BMJ. 2005;330(7497):956–8. A published erratum appears in BMJ. 2005;330(7502):1258.Google Scholar
  7. Chan AW, Song F, Vickers A, Jefferson T, Dickersin K, Gøtzsche PC, et al. Increasing value and reducing waste: addressing inaccessible research. Lancet. 2014;383(9913):257–66.View ArticlePubMedPubMed CentralGoogle Scholar
  8. Zarin DA, Tse T, Williams RJ, Carr S. Trial reporting in ClinicalTrials.gov—the final rule. N Engl J Med. 2016;375(20):1998–2004.View ArticlePubMedGoogle Scholar
  9. Hudson KL, Lauer MS, Collins FS. Toward a new era of trust and transparency in clinical trials. JAMA. 2016;316(13):1353–4.View ArticlePubMedPubMed CentralGoogle Scholar
  10. Chan AW, Hróbjartsson A, Haahr MT, Gøtzsche PC, Altman DG. Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. JAMA. 2004;291(20):2457–65.View ArticlePubMedGoogle Scholar
  11. Al-Marzouki S, Roberts I, Evans S, Marshall T. Selective reporting in clinical trials: analysis of trial protocols accepted by The Lancet. Lancet. 2008;372(9634):201.View ArticlePubMedGoogle Scholar
  12. Pildal J, Chan AW, Hróbjartsson A, Forfang E, Altman DG, Gøtzsche PC. Comparison of descriptions of allocation concealment in trial protocols and the published reports: cohort study. BMJ. 2005;330(7499):1049.View ArticlePubMedPubMed CentralGoogle Scholar
  13. Mhaskar R, Djulbegovic B, Magazin A, Soares HP, Kumar A. Published methodological quality of randomized controlled trials does not reflect the actual quality assessed in protocols. J Clin Epidemiol. 2012;65(6):602–9.View ArticlePubMedPubMed CentralGoogle Scholar
  14. Hróbjartsson A, Pildal J, Chan AW, Haahr MT, Altman DG, Gøtzsche PC. Reporting on blinding in trial protocols and corresponding publications was often inadequate but rarely contradictory. J Clin Epidemiol. 2009;62(9):967–73.View ArticlePubMedGoogle Scholar
  15. Scharf O, Colevas AD. Adverse event reporting in publications compared with sponsor database for cancer clinical trials. J Clin Oncol. 2006;24(24):3933–8.View ArticlePubMedGoogle Scholar
  16. Chan AW, Hróbjartsson A, Jørgensen KJ, Gøtzsche PC, Altman DG. Discrepancies in sample size calculations and data analyses reported in randomised trials: comparison of publications with protocols. BMJ. 2008;337:a2299. doi:10.1136/bmj.a2299.View ArticlePubMedPubMed CentralGoogle Scholar
  17. Dwan K, Altman DG, Cresswell L, Blundell M, Gamble CL, Williamson PR. Comparison of protocols and registry entries to published reports for randomised controlled trials. Cochrane Database Syst Rev. 2011;1:MR000031.Google Scholar
  18. Blümle A, Meerpohl JJ, Rücker G, Antes G, Schumacher M, von Elm E. Reporting of eligibility criteria of randomised trials: cohort study comparing trial protocols with subsequent articles. BMJ. 2011;342:d1828.View ArticlePubMedPubMed CentralGoogle Scholar
  19. Berendt L, Callréus T, Petersen LG, Bach KF, Poulsen HE, Dalhoff K. From protocol to published report: a study of consistency in the reporting of academic drug trials. Trials. 2016;17:100.View ArticlePubMedPubMed CentralGoogle Scholar
  20. Raghav KP, Mahajan S, Yao JC, Hobbs BP, Berry DA, Pentz RD, et al. From protocols to publications: a study in selective reporting of outcomes in randomized trials in oncology. J Clin Oncol. 2015;33(31):3583–90.View ArticlePubMedPubMed CentralGoogle Scholar
  21. Horton R. Pardonable revisions and protocol reviews. Lancet. 1997;349(9044):6.View ArticlePubMedGoogle Scholar
  22. Siegel JP. Editorial review of protocols for clinical trials. N Engl J Med. 1990;323(19):1355.PubMedGoogle Scholar
  23. Foss A, Westcott M. Major journals should peer review trials at protocol stage. BMJ. 1997;314(7095):1691–2.View ArticlePubMedPubMed CentralGoogle Scholar
  24. PubMed. National Center for Biotechnology Information, U.S. National Library of Medicine, USA. https://www.ncbi.nlm.nih.gov/pubmed. Accessed 30 Nov 2017 with search criteria (“study protocol”[Title] AND “trial”[Title]) and (“study protocol”[Title] AND “trial”[Title]) AND “Trials”[Journal].
  25. Submission guidelines. https://trialsjournal.biomedcentral.com/submission-guidelines. Accessed 28 Sept 2016.
  26. Goodman SN. Update on position and design papers. Clin Trials. 2004;1:415–6.View ArticleGoogle Scholar
  27. Multicenter Uveitis Steroid Treatment Trial Research Group. The multicenter uveitis steroid treatment trial: rationale, design, and baseline characteristics. Am J Ophthalmol. 2010;149(4):550–61. e10.View ArticleGoogle Scholar
  28. Holbrook JT, Kempen JH, Prusakowski NA, Altaweel MM, Jabs DA, Multicenter Uveitis Steroid Treatment (MUST) Trial Research Group. Challenges in the design and implementation of the Multicenter Uveitis Steroid Treatment (MUST) Trial – lessons for comparative effectiveness trials. Clin Trials. 2011;8(6):736–43.View ArticlePubMedPubMed CentralGoogle Scholar
  29. Chan AW, Tetzlaff JM, Altman DG, Laupacis A, Gøtzsche PC, Krleža-Jerić K, et al. SPIRIT 2013 statement: defining standard protocol items for clinical trials. Ann Intern Med. 2013;158(3):200–7.View ArticlePubMedPubMed CentralGoogle Scholar
  30. Coordinating Center Models Project Research Group. Coordinating Center Models Project: a study of coordinating centers in multicenter clinical trials IV. Terminology. Bethesda: National Heart, Lung, and Blood Institute; 1979. Appendix, p. 9.Google Scholar
  31. Chan AW, Tetzlaff JM, Gøtzsche PC, Altman DG, Mann H, Berlin JA, et al. SPIRIT 2013 explanation and elaboration: guidance for protocols of clinical trials. BMJ. 2013;346:e7586.View ArticlePubMedPubMed CentralGoogle Scholar
  32. International Conference on Harmonisation of Technical Requirements for Registration of Pharmaceuticals for Human Use. ICH Harmonised Tripartite Guideline: Statistical Principles for Clinical Trials E9. Step 4 version, dated 5 Feb 1998. http://www.ich.org/fileadmin/Public_Web_Site/ICH_Products/Guidelines/Efficacy/E9/Step4/E9_Guideline.pdf. Accessed 21 Nov 2016.
  33. Krishan A, Stocken D, Lewis S, Juszczak E, Dore C, Williamson P, et al. Development of guidance for statistical analysis plans (SAPs) for clinical trials [abstract]. Trials. 2015;16 Suppl 2:O41.View ArticlePubMed CentralGoogle Scholar

Copyright

© The Author(s). 2017