Skip to main content

Research transparency promotion by surgical journals publishing randomised controlled trials: a survey

Abstract

Objective

To describe surgical journals’ position statements on data-sharing policies (primary objective) and to describe key features of their research transparency promotion.

Methods

Only “SURGICAL” journals with an impact factor higher than 2 (Web of Science) were eligible for the study. They were included, if there were explicit instructions for clinical trial publication in the official instructions for authors (OIA) or if they had published randomised controlled trial (RCT) between 1 January 2016 and 31 December 2018. The primary outcome was the existence of a data-sharing policy included in the instructions for authors. Data-sharing policies were grouped into 3 categories, inclusion of data-sharing policy mandatory, optional, or not available. Details on research transparency promotion were also collected, namely the existence of a “prospective registration of clinical trials requirement policy”, a conflict of interests (COIs) disclosure requirement, and a specific reference to reporting guidelines, such as CONSORT for RCT.

Results

Among the 87 surgical journals identified, 82 were included in the study: 67 (82%) had explicit instructions for RCT and the remaining 15 (18%) had published at least one RCT. The median impact factor was 2.98 [IQR = 2.48–3.77], and in 2016 and 2017, the journals published a median of 11.5 RCT [IQR = 5–20.75].

The OIA of four journals (5%) stated that the inclusion of a data-sharing statement was mandatory, optional in 45% (n = 37), and not included in 50% (n = 41).

No association was found between journal characteristics and the existence of data-sharing policies (mandatory or optional). A “prospective registration of clinical trials requirement” was associated with International Committee of Medical Journal Editors (ICMJE) allusion or affiliation and higher impact factors. Journals with specific RCT instructions in their OIA and journals referenced on the ICMJE website more frequently mandated the use of CONSORT guidelines.

Conclusion

Research transparency promotion is still limited in surgical journals. Standardisation of journal requirements according to ICMJE guidelines could be a first step forward for research transparency promotion in surgery.

Peer Review reports

Background

Surgical journals have a key role in ensuring transparency, openness, and reproducibility [1] to increase value and reduce waste in the research they publish [2]. Editorial standards promoting transparency are expected when it comes to randomised controlled trials (RCT) because their importance is paramount in drafting guidelines that can impact medical practice worldwide. Surgical interventions are invasive and in some surgical trials, participants may expose themselves to heightened risk with uncertain benefits. This results in an implicit social contract imposing an ethical obligation whereupon the results lead to the greatest possible benefit to society [3].

The latest breakthrough was the adoption of a policy that encourages RCT data sharing and requires a data-sharing statement to be included in the reports of published clinical trials by the International Committee of Medical Journal Editors (ICMJE) [4]. According to the ICJME recommendations, the data-sharing statement must indicate if individual anonymised data will be shared. The statement must clearly specify the start and end date that the data will be available, with whom the data may be shared, the modality of access, and other documents that would be available.

Other aspects of research transparency promotion have been previously promoted, such as registration of the trial [5], adoption of the CONSORT statement [6], and declaration of conflict of interest (COI) [7]. However, transparent practices in the surgical community could be suboptimal, as suggested by the underreporting of COI [8].

If we hypothesise that editors can be the first motivators for research transparency promotion, reviewing current editors’ practices and policies in relation to the transparency of the research is pertinent, before evaluating the evolution of those practices in papers published in surgical journals.

The aim of this study is to describe the surgical journal position statement on data-sharing policies (primary objective) and to describe the key features of their research transparency promotion.

Methods

The protocol of the survey of surgical journals was registered in the Open Science Framework (OSF) on February 25, 2019 [9].

Eligibility criteria and journal selection

Two reviewers (NL and AG) used Web of Science to select journals classified in the “SURGICAL” category with a 2017 impact factor higher than 2. Surgical journals were included if there were explicit instructions for clinical trial publication in the instructions for authors, or if they published at least one RCT between 1 January 2016 and 31 December 2018. Two authors (NL and AG) independently extracted the data. Disagreements were resolved by consensus or in consultation with a third reviewer (DB). The list of journals was extracted in December 2018, and the official instructions for authors (OIA) were downloaded on 13–14 January 2019.

Our primary outcome was the existence of a data-sharing policy in the instructions for authors. The following classification was used to describe the types of policies for data sharing: “Inclusion of data-sharing policy mandatory”, “inclusion of data-sharing policy optional” and “no data-sharing policy available”.

Additional details on research transparency promotion were also collected, namely, the existence of a “prospective registration of clinical trials requirement” policy, a “COIs” disclosure requirement, and a specific reference to reporting guidelines such as CONSORT for RCT.

Other variables were also extracted, in order to detail journals features as well as potential predictors for journals’ adherence to research transparency rules. The 2017 journal impact factors were extracted from the Web of Science data base. The number of RCT published in 2016 and 2017 (1/1/16 to 12/31/17) was extracted from PubMed. As described in the protocol, it was originally planned to extract the number of RCT published in 2016, 2017, and 2018, but this was not possible because at the time of data extraction, not all RCT published in 2018 were fully indexed in PubMed. ICMJE “affiliation” was defined as journals referenced at time of data extraction as “Journals stating that they follow ICMJE Recommendations” at: http://www.icmje.org/journals-following-the-icmje-recommendations. Publishing model was classified according to the details described in the instructions to authors referring to the method used to fund publications: either by the authors (open access), reader (subscription to the journal or pay per view), and optional (can be the author, equivalent to open access or the reader).

Statistical analyses

Analyses of all included journals were performed using R statistical software (http://www.r-project.org/). Continuous data were presented using median and interquartile range (IQR) and compared with the Mann-Whitney U test. Categorical data were presented as a percentage and compared with a chi-squared test or a two-sided Fisher’s exact test when Chi-squared test application conditions were not met. Univariate exploratory analyses were performed to explore the associations between journal features and the various transparency policies. For exploratory analyses, publishers were grouped into 3 classes according to the number of journal titles included in this study: “high” with over 20 titles, “middle” with 10 to 20 titles, and “low” with less than 10 titles. The number of RCTs published and the 2017 journal impact factor were separated into quartiles using the quant.cut () function, with following parameters, including the lowest variable and excluded the right. For exploratory analyses, the threshold for statistical significance was set at P < 0.05/36 (P = 0.0014) with a Bonferroni correction due to multiple comparisons (n = 36). Multivariable analysis was not performed due to sparse data (i.e. too small sample size in some groups).

Results

Of the 87 surgical journals identified, 82 were included in the analysis: 67 (82%) had explicit instructions for RCT and the remaining 15 (18%) had published at least one RCT between 2016 and 2018 (Fig. 1). The characteristics of these journals are detailed in Table 1. The median impact factor was 2.98 [IQR = 2.48–3.77]. In 2016 and 2017, the journals published a median of 12 RCT [IQR = 5–21]. The publishing model was “optional” in most cases (89%), and America was the principal geographical area of journal editorial committees (56%).

Fig. 1
figure1

Journal selection process. RCT, randomised controlled trial

Table 1 Journal characteristics and policies

Adherence to research transparency rules are detailed in Table 2. Data-sharing statement instructions were present in the OIAs in 41 journals (50%). The OIA for four journals (5%) stated that the inclusion of a data-sharing statement was mandatory, optional in 45% (n = 37), and not included in 50% (n = 41). COI disclosure was mandatory in 77 journals (94%). A reference to CONSORT guidelines was made in 24 journals (29%). Prospective registration of clinical trials was mandatory in 53 cases (65%). Table 3 details the relationship between data-sharing policies and journal impact factors and the number of RCT published in 2016 and 2017 grouped by quartiles. The association of other research transparency promotion item relationships with journal impact factor and the number of RCT published are illustrated in the Fig. 2. Only the association between journal impact factor and trial registration mandatory rule was statistically significant (P = 0.003). Other exploratory analysis of journal features and the different transparency policies are presented in Table 4. As for data-sharing statements, no association was found between journal characteristics and the existence of data-sharing policies (mandatory or optional). A “prospective registration of clinical trials requirement” was associated with ICMJE allusion (P < 0.001), ICMJE affiliation (P < 0.001), and higher impact factors (P < 0.001). Journals with specific RCT instructions in their OIA (P = 0.04) and journals referenced on the ICMJE website (P = 0.03) more frequently mandated the use of CONSORT guidelines, but those results were not significant using our P value threshold (P = 0.0014). No other pertinent association was found.

Table 2 Adherence to research transparency rules
Table 3 Relationship between data-sharing policies and journal impact factors and the number of RCT during the study period grouped by quartiles
Fig. 2
figure2

Percent of journal with mandatory rules concerning a trial registration, b COI disclosure, and c CONSORT guidelines according to the number of RCT quartiles published during the study period and to the impact factor quartiles. COI, conflict of interest; RCT, randomised controlled trial. Significant P value (P = 0.03) is highlight with “*”

Table 4 Exploratory analysis, association between journal features, and transparency policies with univariate analysis

Discussion

We noted low rates of implementation of data-sharing policies, i.e. 50% of the journals had no explicit policy included in their instructions for authors. When explicit, these policies were mostly optional contrary to the ICMJE recommendation that make a data-sharing statement mandatory for RCT. Of course, the study was performed during a time of change, and one could argue that the ICMJE position on data sharing was fairly recent (data extraction started 6 months after the ICMJE statement) and that a number of journals may not have sufficient time to implement it when our survey was conducted. However, this policy was announced in 2017 [4], and 35 (43%) journals are listed on the ICMJE website. Interestingly, the implementation of older policies was also suboptimal, even for making a specific reference to reporting guidelines such as CONSORT for RCT, which date from 1996 [6]. Except for COI disclosure, those recommendations were mostly non-binding. These disappointing results are not new. In 2014, Chapman et al. [10] warned about sub-optimal transparency policies in 10 leading surgical journals.

We considered a journal’s policies presented on its website as a surrogate marker of implementation of these policies. However, it is possible that editors of journals with a policy do not implement them in an optimal manner [11] or, conversely, that a journal with no specific policy documented on the website requires authors to comply with some of the features we explored. Of note, previous research has shown that journal requirements can have a significant impact on changing researcher practices [12], an obvious next step will be to explore the transparency features of the published RCT in these journals.

Of concern, we found no association of research transparency items with impact factors nor with the number of RCT published except for prospective trial registration among the surveyed surgical journals. This is of concern since impact factor is commonly used as a surrogate to assess the quality of a given journal and sometimes of an individual paper [13, 14].

Some limitations of our study have to be highlighted. Firstly, we want to stress that the goal of the study was to evaluate transparency research promotion by surgical journals through their official instruction for authors. Consequently, no conclusion about a relation between our results and the quality of published papers in terms of transparency can be drawn. Future research about journals or publishers’ policies’ impact on published papers will be important to assess the potential impact of such policies on reporting. Secondly, the official registration of our study protocol on OSF occurred after the initiation of the study, even though it was written before the review had begun and was and not modified since (author’s statement). The complete descriptive aspect of our main outcome limits the potential bias in this situation. Finally, we have chosen arbitrarily to combine mandatory and optional data-sharing policies based on the verbatim analysis we made during extraction. Data-sharing statement requirements were rarely mandatory but, in most cases, only optional. For exploratory analysis (journal features associated to research transparency promotion items), we combined these two categories in acknowledgment of the fact that journals suggesting data sharing is an important step in changing practice.

Conclusion

Data-sharing policies appear to be sub-optimally adopted and promoted by surgical journals. We suggest that indicators of quality such as prospective audits of policies and published papers must be used to assess journals, instead of journal impact factors. We encourage surgical journals to improve their research transparency promotion. Standardisation of journal requirements according to ICMJE guidelines could be the first step forward for research transparency promotion in surgery.

Availability of data and materials

Study protocol is already available on Open Science Framework. Data extracted and statistical codes are available on Open Science Framework.

Abbreviations

RCT:

Randomised controlled trial

ICMJE:

International Committee of Medical Journal Editors

COI:

Conflict of interest

OIA:

Official Instructions for Authors

IQR:

Interquartile interval range

References

  1. 1.

    Nosek BA, Alter G, Banks GC, Borsboom D, Bowman SD, Breckler SJ, et al. Promoting an open research culture. Science. 2015;348:1422–5.

    CAS  Article  Google Scholar 

  2. 2.

    Macleod MR, Michie S, Roberts I, Dirnagl U, Chalmers I, Ioannidis JPA, et al. Biomedical research: increasing value, reducing waste. Lancet. 2014;383:101–4.

    Article  Google Scholar 

  3. 3.

    Bauchner H, Golub RM, Fontanarosa PB. Data sharing: an ethical and scientific imperative. JAMA. 2016;315:1238.

    Article  Google Scholar 

  4. 4.

    Taichman DB, Sahni P, Pinborg A, Peiperl L, Laine C, James A, et al. Data sharing statements for clinical trials - a requirement of the International Committee of Medical Journal Editors. N Engl J Med. 2017;376:2277–9.

    Article  Google Scholar 

  5. 5.

    De Angelis C, Drazen JM, Frizelle FA, Haug C, Hoey J, Horton R, et al. Clinical trial registration: a statement from the International Committee of Medical Journal Editors. N Engl J Med. 2004;351:1250–1.

    Article  Google Scholar 

  6. 6.

    Limb C, White A, Fielding A, Lunt A, Borrelli MR, Alsafi Z, et al. Compliance of randomized controlled trials published in general surgical journals with the CONSORT 2010 statement. Ann Surg. 2019;269:e25–7.

    Article  Google Scholar 

  7. 7.

    International Committee of Medical Journal Editors. Conflict of interest. Ann Intern Med. 1993;118:646–7.

    Article  Google Scholar 

  8. 8.

    Ziai K, Pigazzi A, Smith BR, Nouri-Nikbakht R, Nepomuceno H, Carmichael JC, et al. Association of compensation from the surgical and medical device industry to physicians and self-declared conflict of interest. JAMA Surg. 2018;153(11):997–1002.

    Article  Google Scholar 

  9. 9.

    Bergeat D, Nicolas L, Anis G, Florian N. Research transparency promotion by surgical journals: OSF; 2019. Available from: osf.io/gkc7f.

  10. 10.

    Chapman SJ, Shelton B, Mahmood H, Fitzgerald JEF, Harrison E, Bhangu A. Promoting transparency in clinical research: systematic review of disclosure and data-sharing policies in surgical journals. Int J Surg. 2014;12:S52.

    Article  Google Scholar 

  11. 11.

    Naudet F, Sakarovitch C, Janiaud P, Cristea I, Fanelli D, Moher D, et al. Data sharing and reanalysis of randomized controlled trials in leading biomedical journals with a full data sharing policy: survey of studies published in The BMJ and PLOS Medicine. BMJ. 2018;360:k400.

    Article  Google Scholar 

  12. 12.

    Giofrè D, Cumming G, Fresc L, Boedker I, Tressoldi P. The influence of journal submission guidelines on authors’ reporting of statistics and use of open research practices. PLoS One. 2017;12:e0175583.

    Article  Google Scholar 

  13. 13.

    Ioannidis JPA, Thombs BD. A user’s guide to inflated and manipulated impact factors. Eur J Clin Investig. 2019;49:e13151.

    Google Scholar 

  14. 14.

    San Francisco Declaration on Research Assessment. Available from: https://sfdora.org/read/. Accessed 26 June 2019.

Download references

Acknowledgements

None.

Funding

This study had no financial support. Florian Naudet’s works on reproducibility are part of the ReiTheR (reproducibility in therapeutic research) project. This project is funded by the French National Research Agency (ANR, reference number ANR-17-CE-36-0010-01). The sponsor had no role concerning preparation, review, or approval of the manuscript.

Author information

Affiliations

Authors

Contributions

Study concept and design: DB, FN. Acquisition of data: NL, AG, DB. Analysis and interpretation of data: DB, NL, FN. Drafting of the manuscript: DB, NL, FN. Critical revision of the manuscript for important intellectual content: FN, KB, LS, AG. Statistical analysis: DB, FN. The authors read and approved the final manuscript.

Corresponding author

Correspondence to D. Bergeat.

Ethics declarations

Ethics approval and consent to participate

For this type of study, formal consent was not required.

Consent for publication

Not applicable.

Competing interests

All authors have completed the ICMJE uniform disclosure form at http://www.icmje.org/coi_disclosure.pdf (available by request from the corresponding author) and declare that (1) no authors has support from any company for the submitted work; (2) none has relationships (travel/accommodations expenses covered/reimbursed) who might have an interest in the work submitted in the previous 3 years; (3) none have relationship with any company that might have an interest in the work submitted; (4) no author’s spouse, partner, or children have any financial relationships that could be relevant to the submitted work; and (5) none of the authors have any non-financial interests that could be relevant to the submitted work.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Lombard, N., Gasmi, A., Sulpice, L. et al. Research transparency promotion by surgical journals publishing randomised controlled trials: a survey. Trials 21, 824 (2020). https://doi.org/10.1186/s13063-020-04756-7

Download citation

Keywords

  • Data sharing
  • Conflict of interest
  • CONSORT statement
  • Randomised controlled trial