This article has Open Peer Review reports available.
The citation of relevant systematic reviews and randomised trials in published reports of trial protocols
© The Author(s). 2016
Received: 3 May 2016
Accepted: 17 November 2016
Published: 7 December 2016
It is important that planned randomised trials are justified and placed in the context of the available evidence. The SPIRIT guidelines for reporting clinical trial protocols recommend that a recent and relevant systematic review should be included. The aim of this study was to assess the use of the existing evidence in order to justify trial conduct.
Protocols of randomised trials published over a 1-month period (December 2015) indexed in PubMed were obtained. Data on trial characteristics relating to location, design, funding, conflict of interest and type of evidence included for trial justification was extracted in duplicate and independently by two investigators. The frequency of citation of previous research including relevant systematic reviews and randomised trials was assessed.
Overall, 101 protocols for RCTs were identified. Most proposed trials were parallel-group (n = 74; 73.3%). Reference to an earlier systematic review with additional randomised trials was found in 9.9% (n = 10) of protocols and without additional trials in 30.7% (n = 31), while reference was made to randomised trials in isolation in 21.8% (n = 22). Explicit justification for the proposed randomised trial on the basis of being the first to address the research question was made in 17.8% (n = 18) of protocols. A randomised controlled trial was not cited in 10.9% (95% CI: 5.6, 18.7) (n = 11), while in 8.9% (95% CI: 4.2, 16.2) (n = 9) of the protocols a systematic review was cited but did not inform trial design.
A relatively high percentage of protocols of randomised trials involves prior citation of randomised trials, systematic reviews or both. However, improvements are required to ensure that it is explicit that clinical trials are justified and shaped by contemporary best evidence.
The onus on clear research reporting and indeed optimal yield from randomised trials has been highlighted increasingly in recent years with much research now accepted as being suboptimal with ensuing financial and systemic waste . Among the more pressing shortcomings are failure to consider questions of relevance to clinicians and patients, inappropriate design and methods, publication bias, and biased and incomplete reporting .
Accepted prerequisites for randomised trials are the existence of genuine uncertainty concerning the relative merits of competing interventions and appropriate design to permit meaningful answers . An appreciation of the existence of previous research is therefore necessary to avoid unnecessary duplication, and due consideration of the evidence base is important in informing appropriate design and methodology. Notwithstanding this, replication of previous studies may be justified in confirming previous results or in an effort to assess the generalizability of novel findings .
It is accepted that relevant research, including systematic reviews where they exist and randomised trials, should be cited in the introduction section of reports of randomised controlled trials (RCTs) . There are also growing concerns in relation to publication bias and selective outcome reporting in biomedical journals [5, 6]. Consequently, in recent years there has been an increasing drive to publish research protocols both to prevent unwanted duplication and to mitigate the risk of reporting bias . The Standard Protocol Items: Recommendations for Interventional Trials (SPIRIT) guidelines  aim to improve the reporting and quality of clinical trial protocols and incorporate a checklist of recommended items to include in the trial protocol . Within the SPIRIT guidelines, citation of prior research in the introduction section and justification of randomised trials on the basis of gaps in the underlying evidence base are advocated . The SPIRIT checklist strongly recommends placing: “The trial in the context of available evidence, it is strongly recommended that an up-to-date systematic review of relevant studies be summarised and cited in the protocol.”
Previous research has addressed the issue of recognition of prior studies in stimulating or informing further research [8–10]. Robinson and Goodman  in an analysis of RCTs contributing to meta-analyses identified that less than one quarter of relevant trials had been cited. Similarly, Clarke and Hopewell  in a survey of five leading medical journals reported sparse referencing of systematic reviews either in the introduction or discussion sections. Consequently, randomised trial reports may routinely fail to place research findings in appropriate context potentially hampering the end user’s ability to reach balanced, informed decisions about important healthcare interventions. Failure to consider the available and latest evidence may jeopardize the justification for new randomised research on ethical grounds.
To our knowledge no studies exist assessing the justification for randomised trials based on the available evidence at the protocol stage and in accordance with SPIRIT. The aim of this meta-epidemiological study was to assess the extent to which published protocols of randomised trials adhere to SPIRIT and include an updated systematic review to inform the trial design in the background or rationale sections.
Search strategy for protocol selection
“protocol[ti] AND random*”
Filter for “randomised trials” and time range
1. randomised controlled trial [pt] OR controlled randomised trial [pt] OR randomised [tiab] OR randomised [tiab] OR placebo [tiab] OR randomly [tiab] OR trial [tiab]
2. protocol [tiab]
3. systematic review OR meta-analysis [ti] OR meta analysis [ti] OR review [ti] OR Review [pt] OR Meta-Analysis [pt] OR Comment [pt] OR Letter [pt] OR Editorial [pt] OR News [pt]
4. #1 AND #2 NOT #3
Document describing the objectives, design, methodology, statistical considerations, and organisation of a clinical trial including justification and rationale for the trial.
Cochrane definition of a trial as randomised: “the individuals (or other units) followed in the trial were assigned prospectively to one of two (or more) alternative forms of health care using random allocation.”
randomised trial is the first to address the question
updated systematic review was used to inform trial design
systematic review and new trials (published since the systematic review was published) were used to inform trial design
previous systematic review was discussed but was not used in trial design
references to other randomised trials
no references to other randomised trials or claim to be the first trial
Data were independently extracted by two investigators and entered on pre-piloted standardized forms for the eligible protocols. Initial calibration was performed between the two researchers on ten articles. Disagreements were resolved by discussion or, if necessary, with adjudication by a third reviewer and a consensus was reached for all protocols. Descriptive statistics were undertaken and data were tabulated with respect to selected protocol characteristics with the use of STATA® version 14.1 software (Stata Corporation, College Station, TX, USA).
Distribution of demographic variables within the protocols assessed (n = 101)
Characteristics of assessed protocols
Annals of Translational Medicine
BMC umbrella journals
Clinical and Translational Allergy
Contemporary Clinical Trials
Danish Medical Journal
Implementation Science: IS
JMIR Research Protocols
Journal of Advanced Nursing
Journal of Diabetes Science and Technology
The Journal of Cardiovascular Nursing
Biological or vaccine
Surgery or procedure
No. centres (according to authors’ affiliations)
Continent of authorship
Asia and other
No. sites (based on trial conduct)
Other (or mixed-type designs)
Type of funding
Conflict of interest
Data upon request
Data stored in central repository
No information provided
Distribution of protocol characteristics in terms of citation of relevant research (n = 101)
Total (n = 101)
Claims that randomised trial is the first to address the question
Contains an updated systematic review used to inform trial design
Contains systematic review and new trials (published since the systematic review was published) that were used to inform trial design
Previous systematic review discussed but not used in trial design
Contains references to other randomised trials
Does not contain references to other randomised trials or claim to be the first trial
From the 101 included protocols 89 of those were published in journals endorsing the SPIRIT guidelines.
The CONSORT statement has stipulated that findings from a randomised trial should be placed in the context of the “totality of the available evidence” . Similarly, when designing a new study it is important that the setting and methodology is informed by previous research. This cross-sectional analysis is the first to assess the citation of high-level research, including both systematic reviews and randomised controlled trials, in reports of published protocols of planned randomised trials. Overall, 41% of the protocols cited a systematic review or a randomised trial that was used to inform trial design.
Previous analyses of reports of randomised trials have indicated that systematic reviews are infrequently cited in reports of randomised trials with sequential audits of trials published in leading medical journals alluding to failure to refer to systematic reviews in 24% of reports in 1997, 10% in 2001, 33% in 2005, 46% in 2009 and 39% in 2012 [8–10]. This finding has arisen despite the pervasion of systematic reviews with in excess of 5000 full Cochrane reviews now published in the Cochrane Database of Systematic Reviews  and several thousand more systematic reviews published on an annual basis . Our study indicates that trial justification based on systematic reviews and randomised trials at the protocol level is slightly higher compared to published trials and this finding is encouraging, despite the potential imprecision in obtaining these higher values reflected by the wide confidence intervals. Reasons for this are unclear at this stage but it could be speculated that the utility of systematic reviews has improved over time, that a greater onus on trial justification exists at the protocol stage with funding agencies placing an emphasis on evidence of a recent systematic review to justify further clinical research. However, this is the first study to assess this and further research may be required to confirm this pattern within research protocols.
The SPIRIT guidelines were developed in response to inadequate description of trial details such as outcomes and interventions, but also due to unclear description of trial methodology . These shortcomings may prompt protocol amendments, might hamper trial conduct and can ultimately lead to inadequate reporting . Moreover, it is important that previous research is considered at the design stages to avoid unnecessary duplication, to inform trial design in order to maximize the yield from expensive and lengthy randomised studies . Where genuine uncertainty concerning the effectiveness of an intervention is lacking, undertaking such a trial is also considered unethical. It is therefore recommended both within the SPIRIT and CONSORT statements that relevant randomised trials and systematic reviews are identified. It is accepted, however, that compliance with established reporting and conduct guidelines is suboptimal throughout the biomedical literature . While meta-epidemiological studies focusing on compliance with SPIRIT have not yet been reported, the present investigation does report some encouraging results.
A potential limitation of the present study is the possibility that there may not have been relevant trials or systematic reviews to cite at the time of the design of the new trial. It is therefore possible that the prevalence of failure to consider relevant research may be overstated slightly. However, previous studies have attempted to identify similar studies  but have found broadly similar rates of isolated reports. Furthermore, the present study was limited to a restricted time period and few sources. The search strategy looked specifically for protocols in the title and therefore it is possible that a number of trial protocols might have been overlooked. However, the journals included are likely to represent best practice in terms of protocol design and reporting as most of them endorse SPIRIT guidelines with the majority of trial protocols likely remaining unpublished at this juncture. It is therefore likely that the prevalence of failure to identify relevant related research found in the present study represents a best-case scenario. Finally, a limited number of pharmacological studies were identified over the study period; further research focusing on the protocol reporting characteristics for these studies, in particular, would therefore be welcome.
A relatively high percentage of protocols of randomised trials involves prior citation of either randomised trials, systematic reviews or both. Overall, 41% of protocols involved citation of a systematic review or a randomised trial that was used to inform trial design.
The authors have not received funding to undertake this research.
Availability of data and materials
Available with submission.
SH conceived the study, conducted the search, performed data extraction, and edited the manuscript for intellectual content. NP selected the abstracts, performed data extraction, created data tabulations, and edited the manuscript for intellectual content. DK performed data extraction, created data tabulations, and edited the manuscript for intellectual content. PF extracted data, prepared the first manuscript draft, and edited the manuscript for intellectual content. All authors read and approved the final manuscript.
The authors declare that they have no competing interests.
Consent for publication
Ethics approval and consent to participate
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
- Glasziou P, Altman DG, Bossuyt P, Boutron I, Clarke M, Julious S, et al. Reducing waste from incomplete or unusable reports of biomedical research. Lancet. 2014;383:267–76.View ArticlePubMedGoogle Scholar
- Chalmers I, Glasziou P. Avoidable waste in the production and reporting of research evidence. Lancet. 2009;374:86–9.View ArticlePubMedGoogle Scholar
- Schulz KF, Altman DG, Moher D, CONSORT Group. CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. PLoS Med. 2010;7:e1000251.View ArticlePubMedPubMed CentralGoogle Scholar
- Ioannidis JPA. How to make more published research true. PLoS Med. 2014;11:e1001747.View ArticlePubMedPubMed CentralGoogle Scholar
- Dwan K, Altman DG, Arnaiz JA, Bloom J, Chan A-W, Cronin E, et al. Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLoS One. 2008;3:e3081.View ArticlePubMedPubMed CentralGoogle Scholar
- All Trials. 2014. http://www.alltrials.net. Accessed 3 May 2016.
- Chan A-W, Tetzlaff JM, Altman DG, Dickersin K, Moher D. SPIRIT 2013: new guidance for content of clinical trial protocols. Lancet. 2013;381:91–2.View ArticlePubMedGoogle Scholar
- Clarke M, Chalmers I. Discussion sections in reports of controlled trials published in general medical journals: islands in search of continents? JAMA. 1998;280:280–2.View ArticlePubMedGoogle Scholar
- Clarke L, Clarke M, Clarke T. How useful are Cochrane reviews in identifying research needs? J Health Serv Res Policy. 2007;12:101–3.View ArticlePubMedGoogle Scholar
- Clarke M, Hopewell S. Many reports of randomised trials still don’t begin or end with a systematic review of the relevant evidence. J Bahrain Med Soc. 2013;24:145–7.Google Scholar
- Robinson KA, Goodman SN. A systematic examination of the citation of prior research in reports of randomized, controlled trials. Ann Intern Med. 2011;154:50–5.View ArticlePubMedGoogle Scholar
- Clarke M, Hopewell S, Chalmers I. Clinical trials should begin and end with systematic reviews of relevant evidence: 12 years and waiting. Lancet. 2010;376:20–1.View ArticlePubMedGoogle Scholar
- Clarke M, Li Y. The current state of knowledge, alongside recommendations for future research and practice. J Evid-Based Med. 2012;5:47.View ArticlePubMedGoogle Scholar
- Bastian H, Glasziou P, Chalmers I. Seventy-five trials and eleven systematic reviews a day: how will we ever keep up? PLoS Med. 2010;7:e1000326.View ArticlePubMedPubMed CentralGoogle Scholar
- Getz KA, Zuckerman R, Cropp AB, Hindle AL, Krauss R, Kaitin KI. Measuring the incidence, causes and repercussions of protocol amendments. Drug Inf J. 2011;45:265–75.Google Scholar
- Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman DG. Epidemiology and reporting characteristics of systematic reviews. PLoS Med. 2007;4:e78.View ArticlePubMedPubMed CentralGoogle Scholar