Open Access
Open Peer Review

This article has Open Peer Review reports available.

How does Open Peer Review work?

The citation of relevant systematic reviews and randomised trials in published reports of trial protocols

  • Nikolaos Pandis1Email author,
  • Padhraig S. Fleming2,
  • Despina Koletsi3 and
  • Sally Hopewell4
Trials201617:581

https://doi.org/10.1186/s13063-016-1713-6

Received: 3 May 2016

Accepted: 17 November 2016

Published: 7 December 2016

Abstract

Background

It is important that planned randomised trials are justified and placed in the context of the available evidence. The SPIRIT guidelines for reporting clinical trial protocols recommend that a recent and relevant systematic review should be included. The aim of this study was to assess the use of the existing evidence in order to justify trial conduct.

Methods

Protocols of randomised trials published over a 1-month period (December 2015) indexed in PubMed were obtained. Data on trial characteristics relating to location, design, funding, conflict of interest and type of evidence included for trial justification was extracted in duplicate and independently by two investigators. The frequency of citation of previous research including relevant systematic reviews and randomised trials was assessed.

Results

Overall, 101 protocols for RCTs were identified. Most proposed trials were parallel-group (n = 74; 73.3%). Reference to an earlier systematic review with additional randomised trials was found in 9.9% (n = 10) of protocols and without additional trials in 30.7% (n = 31), while reference was made to randomised trials in isolation in 21.8% (n = 22). Explicit justification for the proposed randomised trial on the basis of being the first to address the research question was made in 17.8% (n = 18) of protocols. A randomised controlled trial was not cited in 10.9% (95% CI: 5.6, 18.7) (n = 11), while in 8.9% (95% CI: 4.2, 16.2) (n = 9) of the protocols a systematic review was cited but did not inform trial design.

Conclusions

A relatively high percentage of protocols of randomised trials involves prior citation of randomised trials, systematic reviews or both. However, improvements are required to ensure that it is explicit that clinical trials are justified and shaped by contemporary best evidence.

Keywords

Randomised trials Systematic reviews Citation Protocol SPIRIT statement

Background

The onus on clear research reporting and indeed optimal yield from randomised trials has been highlighted increasingly in recent years with much research now accepted as being suboptimal with ensuing financial and systemic waste [1]. Among the more pressing shortcomings are failure to consider questions of relevance to clinicians and patients, inappropriate design and methods, publication bias, and biased and incomplete reporting [2].

Accepted prerequisites for randomised trials are the existence of genuine uncertainty concerning the relative merits of competing interventions and appropriate design to permit meaningful answers [3]. An appreciation of the existence of previous research is therefore necessary to avoid unnecessary duplication, and due consideration of the evidence base is important in informing appropriate design and methodology. Notwithstanding this, replication of previous studies may be justified in confirming previous results or in an effort to assess the generalizability of novel findings [4].

It is accepted that relevant research, including systematic reviews where they exist and randomised trials, should be cited in the introduction section of reports of randomised controlled trials (RCTs) [3]. There are also growing concerns in relation to publication bias and selective outcome reporting in biomedical journals [5, 6]. Consequently, in recent years there has been an increasing drive to publish research protocols both to prevent unwanted duplication and to mitigate the risk of reporting bias [7]. The Standard Protocol Items: Recommendations for Interventional Trials (SPIRIT) guidelines [7] aim to improve the reporting and quality of clinical trial protocols and incorporate a checklist of recommended items to include in the trial protocol [7]. Within the SPIRIT guidelines, citation of prior research in the introduction section and justification of randomised trials on the basis of gaps in the underlying evidence base are advocated [7]. The SPIRIT checklist strongly recommends placing: “The trial in the context of available evidence, it is strongly recommended that an up-to-date systematic review of relevant studies be summarised and cited in the protocol.”

Previous research has addressed the issue of recognition of prior studies in stimulating or informing further research [810]. Robinson and Goodman [11] in an analysis of RCTs contributing to meta-analyses identified that less than one quarter of relevant trials had been cited. Similarly, Clarke and Hopewell [10] in a survey of five leading medical journals reported sparse referencing of systematic reviews either in the introduction or discussion sections. Consequently, randomised trial reports may routinely fail to place research findings in appropriate context potentially hampering the end user’s ability to reach balanced, informed decisions about important healthcare interventions. Failure to consider the available and latest evidence may jeopardize the justification for new randomised research on ethical grounds.

To our knowledge no studies exist assessing the justification for randomised trials based on the available evidence at the protocol stage and in accordance with SPIRIT. The aim of this meta-epidemiological study was to assess the extent to which published protocols of randomised trials adhere to SPIRIT and include an updated systematic review to inform the trial design in the background or rationale sections.

Methods

Sample

All published protocols of planned randomised controlled trials (RCTs) were identified over a 1-month period (December 2015) indexed in PubMed based on a defined search strategy (Table 1).
Table 1

Search strategy for protocol selection

“protocol[ti] AND random*”

Filter for “randomised trials” and time range

1. randomised controlled trial [pt] OR controlled randomised trial [pt] OR randomised [tiab] OR randomised [tiab] OR placebo [tiab] OR randomly [tiab] OR trial [tiab]

2. protocol [tiab]

3. systematic review OR meta-analysis [ti] OR meta analysis [ti] OR review [ti] OR Review [pt] OR Meta-Analysis [pt] OR Comment [pt] OR Letter [pt] OR Editorial [pt] OR News [pt]

4. #1 AND #2 NOT #3

Data extraction

Titles and abstracts were screened by one author (NP) in order to select all eligible publications. The selected citations were entered in Endnote reference management software and the corresponding full texts were retrieved for further evaluation. The criteria for defining a report as a “randomised controlled trial protocol” were as follows:
  • Document describing the objectives, design, methodology, statistical considerations, and organisation of a clinical trial including justification and rationale for the trial.

  • Cochrane definition of a trial as randomised: “the individuals (or other units) followed in the trial were assigned prospectively to one of two (or more) alternative forms of health care using random allocation.”

The following information was extracted in duplicate by two authors (NP, PSF) from each eligible protocol: title, first author name, number of authors, country and geographic region (i.e. Europe, Americas, Asia, Other) of first author, details of registration, funding, number of trial sites, trial design, funding type, conflict of interest, and data sharing. Following the SPIRIT recommendation within the introduction section the research context used to inform or justify the trial was assessed in one of six ways [12]:
  • randomised trial is the first to address the question

  • updated systematic review was used to inform trial design

  • systematic review and new trials (published since the systematic review was published) were used to inform trial design

  • previous systematic review was discussed but was not used in trial design

  • references to other randomised trials

  • no references to other randomised trials or claim to be the first trial

Data analysis

Data were independently extracted by two investigators and entered on pre-piloted standardized forms for the eligible protocols. Initial calibration was performed between the two researchers on ten articles. Disagreements were resolved by discussion or, if necessary, with adjudication by a third reviewer and a consensus was reached for all protocols. Descriptive statistics were undertaken and data were tabulated with respect to selected protocol characteristics with the use of STATA® version 14.1 software (Stata Corporation, College Station, TX, USA).

Results

The PubMed search identified 411 records, 310 of which were excluded as they did not satisfy the inclusion criteria resulting in 101 protocols suitable for full data extraction (Fig. 1). The Additional file 1 includes the raw data extracted. A variety of medical conditions were considered in the selected protocols (Table 2). The majority of studies were undertaken in multiple centres (84.2%). In terms of geographic region, the highest percentage of studies was carried out in Europe (54.5%; Table 2). Most proposed trials were parallel-group (n = 74; 73.3%), while 9.9% (n = 10) were cluster designs. The vast majority of studies reported funding (94.1%) with non-industry funding in isolation in 82.2%. Conflict of interest statements were made in the vast majority of protocols (n = 96; 95.0%), with the majority having no conflicts to declare (n = 77; 76.2%). No commitment to data sharing was made in the majority of protocols (n = 93, 92.0%), 7% (n = 7) reported that data will be available upon request and only one (1%) protocol indicated the data repository.
Fig. 1

RCT protocol selection flow diagram

Table 2

Distribution of demographic variables within the protocols assessed (n = 101)

Characteristics of assessed protocols

Total

N (101)

%

Journal

 Annals of Translational Medicine

1

1.0

 BMC umbrella journals

23

22.7

 BMJ Open

15

14.8

 Clinical and Translational Allergy

1

1.0

 Contemporary Clinical Trials

4

4.0

 Danish Medical Journal

1

1.0

 Implementation Science: IS

1

1.0

 JAMA Surgery

1

1.0

 JMIR Research Protocols

2

2.0

 Journal of Advanced Nursing

3

3.0

 Journal of Diabetes Science and Technology

1

1.0

 Nutrients

1

1.0

 Reproductive Health

1

1.0

 Springer Plus

1

1.0

 The Journal of Cardiovascular Nursing

1

1.0

 Trials

44

43.5

Subject

 Behavioural

49

48.5

 Biological or vaccine

1

1.0

 Dietary supplement

6

5.9

 Drugs

18

17.8

 Surgery or procedure

14

13.9

 Other

13

12.9

No. centres (according to authors’ affiliations)

 Multi-centre

85

84.2

 Single-centre

16

15.8

Continent of authorship

 America

19

18.8

 Europe

55

54.5

 Asia and other

27

26.7

No. sites (based on trial conduct)

 Multiple

52

51.5

 Single

49

48.5

Trial design

 Parallel

74

73.3

 Cluster

10

9.9

 Non-inferiority

5

5.0

 Superiority

2

2.0

 Other (or mixed-type designs)

10

9.8

Type of funding

 Non-industry

83

82.2

 Part industry

5

4.0

 Industry

7

6.9

 None/unknown

6

5.9

Conflict of interest

 No

77

76.2

 Yes

19

18.8

 Not described

4

4.0

 Unclear

1

1.0

Data sharing

 Data upon request

7

7.0

 Data stored in central repository

1

1.0

 No information provided

93

92.0

Total

101

100.0

In terms of citation of relevant research (Table 3), reference to a previous systematic review with additional randomised trials was found in 9.9% (95% CI: 4.9, 17.5) (n = 10) of protocols and without additional trials was alluded to in 30.7% (95% CI: 21.9, 40.7) (n = 31). A randomised controlled trial was not cited in 10.9% (95% CI: 5.6, 18.7) (n = 11), while in 8.9% (95% CI: 4.2, 16.2) (n = 9) of the protocols a systematic review was cited but did not inform trial design. Of the cited systematic reviews (n = 41), used to inform trial design, 41.5% (n = 17) were Cochrane reviews.
Table 3

Distribution of protocol characteristics in terms of citation of relevant research (n = 101)

Protocol characteristics

Total (n = 101)

 

N

%

95% CI

Claims that randomised trial is the first to address the question

18

17.8

10.9, 26.7

Contains an updated systematic review used to inform trial design

31

30.7

21.9, 40.7

Contains systematic review and new trials (published since the systematic review was published) that were used to inform trial design

10

9.9

4.9, 17.5

Previous systematic review discussed but not used in trial design

9

8.9

4.2, 16.2

Contains references to other randomised trials

22

21.8

14.2, 31.1

Does not contain references to other randomised trials or claim to be the first trial

11

10.9

5.6, 18.7

From the 101 included protocols 89 of those were published in journals endorsing the SPIRIT guidelines.

Discussion

The CONSORT statement has stipulated that findings from a randomised trial should be placed in the context of the “totality of the available evidence” [3]. Similarly, when designing a new study it is important that the setting and methodology is informed by previous research. This cross-sectional analysis is the first to assess the citation of high-level research, including both systematic reviews and randomised controlled trials, in reports of published protocols of planned randomised trials. Overall, 41% of the protocols cited a systematic review or a randomised trial that was used to inform trial design.

Previous analyses of reports of randomised trials have indicated that systematic reviews are infrequently cited in reports of randomised trials with sequential audits of trials published in leading medical journals alluding to failure to refer to systematic reviews in 24% of reports in 1997, 10% in 2001, 33% in 2005, 46% in 2009 and 39% in 2012 [810]. This finding has arisen despite the pervasion of systematic reviews with in excess of 5000 full Cochrane reviews now published in the Cochrane Database of Systematic Reviews [13] and several thousand more systematic reviews published on an annual basis [14]. Our study indicates that trial justification based on systematic reviews and randomised trials at the protocol level is slightly higher compared to published trials and this finding is encouraging, despite the potential imprecision in obtaining these higher values reflected by the wide confidence intervals. Reasons for this are unclear at this stage but it could be speculated that the utility of systematic reviews has improved over time, that a greater onus on trial justification exists at the protocol stage with funding agencies placing an emphasis on evidence of a recent systematic review to justify further clinical research. However, this is the first study to assess this and further research may be required to confirm this pattern within research protocols.

The SPIRIT guidelines were developed in response to inadequate description of trial details such as outcomes and interventions, but also due to unclear description of trial methodology [7]. These shortcomings may prompt protocol amendments, might hamper trial conduct and can ultimately lead to inadequate reporting [15]. Moreover, it is important that previous research is considered at the design stages to avoid unnecessary duplication, to inform trial design in order to maximize the yield from expensive and lengthy randomised studies [1]. Where genuine uncertainty concerning the effectiveness of an intervention is lacking, undertaking such a trial is also considered unethical. It is therefore recommended both within the SPIRIT and CONSORT statements that relevant randomised trials and systematic reviews are identified. It is accepted, however, that compliance with established reporting and conduct guidelines is suboptimal throughout the biomedical literature [16]. While meta-epidemiological studies focusing on compliance with SPIRIT have not yet been reported, the present investigation does report some encouraging results.

A potential limitation of the present study is the possibility that there may not have been relevant trials or systematic reviews to cite at the time of the design of the new trial. It is therefore possible that the prevalence of failure to consider relevant research may be overstated slightly. However, previous studies have attempted to identify similar studies [11] but have found broadly similar rates of isolated reports. Furthermore, the present study was limited to a restricted time period and few sources. The search strategy looked specifically for protocols in the title and therefore it is possible that a number of trial protocols might have been overlooked. However, the journals included are likely to represent best practice in terms of protocol design and reporting as most of them endorse SPIRIT guidelines with the majority of trial protocols likely remaining unpublished at this juncture. It is therefore likely that the prevalence of failure to identify relevant related research found in the present study represents a best-case scenario. Finally, a limited number of pharmacological studies were identified over the study period; further research focusing on the protocol reporting characteristics for these studies, in particular, would therefore be welcome.

Conclusions

A relatively high percentage of protocols of randomised trials involves prior citation of either randomised trials, systematic reviews or both. Overall, 41% of protocols involved citation of a systematic review or a randomised trial that was used to inform trial design.

Declarations

Acknowledgements

Not applicable.

Funding

The authors have not received funding to undertake this research.

Availability of data and materials

Available with submission.

Authors’ contributions

SH conceived the study, conducted the search, performed data extraction, and edited the manuscript for intellectual content. NP selected the abstracts, performed data extraction, created data tabulations, and edited the manuscript for intellectual content. DK performed data extraction, created data tabulations, and edited the manuscript for intellectual content. PF extracted data, prepared the first manuscript draft, and edited the manuscript for intellectual content. All authors read and approved the final manuscript.

Competing interests

The authors declare that they have no competing interests.

Consent for publication

Not applicable.

Ethics approval and consent to participate

Not applicable.

Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Authors’ Affiliations

(1)
Department of Orthodontics and Dentofacial Orthopedics, School of Dental Medicine, Medical Faculty, University of Bern
(2)
Barts and The London School of Medicine and Dentistry, Queen Mary University of London
(3)
London School of Hygiene and Tropical Medicine, University of London
(4)
Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, Oxford Clinical Trials Research Unit, University of Oxford

References

  1. Glasziou P, Altman DG, Bossuyt P, Boutron I, Clarke M, Julious S, et al. Reducing waste from incomplete or unusable reports of biomedical research. Lancet. 2014;383:267–76.View ArticlePubMedGoogle Scholar
  2. Chalmers I, Glasziou P. Avoidable waste in the production and reporting of research evidence. Lancet. 2009;374:86–9.View ArticlePubMedGoogle Scholar
  3. Schulz KF, Altman DG, Moher D, CONSORT Group. CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. PLoS Med. 2010;7:e1000251.View ArticlePubMedPubMed CentralGoogle Scholar
  4. Ioannidis JPA. How to make more published research true. PLoS Med. 2014;11:e1001747.View ArticlePubMedPubMed CentralGoogle Scholar
  5. Dwan K, Altman DG, Arnaiz JA, Bloom J, Chan A-W, Cronin E, et al. Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLoS One. 2008;3:e3081.View ArticlePubMedPubMed CentralGoogle Scholar
  6. All Trials. 2014. http://www.alltrials.net. Accessed 3 May 2016.
  7. Chan A-W, Tetzlaff JM, Altman DG, Dickersin K, Moher D. SPIRIT 2013: new guidance for content of clinical trial protocols. Lancet. 2013;381:91–2.View ArticlePubMedGoogle Scholar
  8. Clarke M, Chalmers I. Discussion sections in reports of controlled trials published in general medical journals: islands in search of continents? JAMA. 1998;280:280–2.View ArticlePubMedGoogle Scholar
  9. Clarke L, Clarke M, Clarke T. How useful are Cochrane reviews in identifying research needs? J Health Serv Res Policy. 2007;12:101–3.View ArticlePubMedGoogle Scholar
  10. Clarke M, Hopewell S. Many reports of randomised trials still don’t begin or end with a systematic review of the relevant evidence. J Bahrain Med Soc. 2013;24:145–7.Google Scholar
  11. Robinson KA, Goodman SN. A systematic examination of the citation of prior research in reports of randomized, controlled trials. Ann Intern Med. 2011;154:50–5.View ArticlePubMedGoogle Scholar
  12. Clarke M, Hopewell S, Chalmers I. Clinical trials should begin and end with systematic reviews of relevant evidence: 12 years and waiting. Lancet. 2010;376:20–1.View ArticlePubMedGoogle Scholar
  13. Clarke M, Li Y. The current state of knowledge, alongside recommendations for future research and practice. J Evid-Based Med. 2012;5:47.View ArticlePubMedGoogle Scholar
  14. Bastian H, Glasziou P, Chalmers I. Seventy-five trials and eleven systematic reviews a day: how will we ever keep up? PLoS Med. 2010;7:e1000326.View ArticlePubMedPubMed CentralGoogle Scholar
  15. Getz KA, Zuckerman R, Cropp AB, Hindle AL, Krauss R, Kaitin KI. Measuring the incidence, causes and repercussions of protocol amendments. Drug Inf J. 2011;45:265–75.Google Scholar
  16. Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman DG. Epidemiology and reporting characteristics of systematic reviews. PLoS Med. 2007;4:e78.View ArticlePubMedPubMed CentralGoogle Scholar

Copyright

© The Author(s). 2016

Advertisement