Publication of clinical trials on medicinal products: follow-up on trials authorized in Hungary
Trials volume 23, Article number: 330 (2022)
Clinical research should provide reliable evidence to clinicians, health policy makers, and researchers. The reliability of evidence will be assured once study planning, conducting, and reporting of results are transparent. The present research investigates publication rates, time until publication, and characteristics of clinical trials on medicinal products associated with timely publication of results, measures of scientific impact, authorship, and open access publication.
Clinical trials authorized in Hungary in 2012 were followed until publication and/or June 2020. Corresponding scientific publications were searched via clinical trial registries, PubMed (MEDLINE), and Google.
Overall, 330 clinical trials were authorized in 2012 of which 232 trials were completed for more than 1 year in June 2020. The proportion of industry initiation was high (97%).
Time to publication was 21 (22) months [median (IQR)]. Time to publication was significantly shorter when trials involved both European and non-European countries (26 vs 69 months [median]; hazard ratio = 0.38, 95% CI 0.22–0.66, p< 0.001), and were registered in both EU CTR and clinicaltrials.gov (27 vs 88 months; hazard ratio = 0.24, 95% CI 0.11–0.54; p< 0.001) based on survival analyses.
A significant amount (24.1%) of unpublished clinical trial results were accessible in a trial register. The majority of available publications were published “open access” (70.93%). A minority of identified publications had a Hungarian author (21.5%).
We encourage academic researchers to plan, register and conduct trials on medicinal products. Registries should be considered as an important source of information of clinical trial results. Publications with domestic co-authors contribute to the research output of a country. Measurable domestic scientific impact of trials on medicinal products needs further improvement.
Clinical research should provide reliable evidence to clinicians, health policy makers, and researchers . This is achieved once results are made publicly available . On a national level, published research means that the resources expended are not waisted and results become part of the international medical knowledge. Published research with a domestic co-author contributes to the assessment of the scientific performance of a country.
As of 2014, registration in the European Union Clinical Trials Register (EU CTR) administered by the European Medicines Agency (EMA) is mandatory for all trial of any medicinal product conducted within a member state of the European Union (EU). The 2012 European Commission guideline 2012/c302/03 requires sponsors to ensure all trials registered on EU CTR disclose their results to the EMA within 12 months of trial completion; phase I trials are exempt unless they are part of a pediatric investigation plan . Voluntary initiatives  and recommendations  have begun to emphasize the importance of registration of clinical trials and subsequent reporting of results. The EU CTR also attempts to increase awareness on mandatory posting of results  and has recently launched a page with a tutorial to facilitate the posting of results on the EU CTR webpage .
Beyond mandatory posting in the EU CTR, many European researchers additionally register their studies on ClinicalTrials.gov, the largest trial register worldwide. It remains unclear if these “multiple registrations” are an intent to improve the transparency and whether they are associated with a more likely publication of trial results.
Results posted in registries have limited impact and awareness in the scientific community. Study results in registries appear in a standardized format which facilitates data provision [8, 9]; however, these results do not undergo rigorous evaluation as do full scientific publications during the peer review process. Besides, publications and scientometrics are currently—despite international initiatives to change this —an integral part of research evaluation and play a crucial role in decision making for national research policies, funding, promotions, and the careers of scientists . Results posted in registries do not contribute to the total research output of either the participating researchers or the participating country.
The aim of this methodological cohort study was to investigate the visibility of authorized medical research conducted in a given country. The study equally analyzed how this visibility affects the research output of that country. We investigated publication rates, time until publication, and the relationship between results posted in trial registers and published as full scientific publications. Further, we aimed to identify trial characteristics associated with timely publication of trial results, measures of scientific impact, authorship, and open access publication in a representative sample of clinical trials authorized in Hungary.
We used the advanced search function of the EU Clinical Trials Register (www.clinicaltrialsregister.eu) to identify clinical trials registered from January 1, 2012, to December 31, 2012, with Hungary as a participating research center.
Inclusion and exclusion criteria
Clinical trials were eligible for our study if (a) Hungary was a site of the clinical trial, (b) trials were registered in EU Clinical Trials Register by the responsible Hungarian authority (National Institute of Pharmacy, Hungary) in 2012, and (c) no restrictions were applied to the trial phase, trial status or participant characteristics (e.g., age, gender, disease group).
Identification of trials in clinicaltrials.gov and data extraction from registries
We extracted the following pre-defined study characteristics from EU CTR: full title of the trial, authorization date, trial start and completion, information on participating countries, sponsor, funder, trial scope, trial design, blinding, sample size, study phase, therapeutic area and presence of a data monitoring committee (DMC).
We tried to identify included trials in the register clinicaltrials.gov by searching the EU CTR identifier or by the use of specific PICO terms.
We determined whether study results were available in the study registries EU Clinical Trial Register and ClinicalTrials.gov. In this current paper, we aim to distinguish results available in the registries (“results in registries”) from results published as full scientific publications (“publication”).
Identification of corresponding scientific publications
Full scientific publications were defined as papers published in any scientific journal and reporting study results on pre-defined outcomes. We excluded methods papers, published protocols and publications which reported results of a secondary analysis.
The first screening was performed in February 2019 and then 16 months later in June 2020. Publications were identified in a step-by-step process for each trial. First, we checked whether publications were already added to the register. In a second step, we searched for publications in the PubMed database with the following identification data: (a) the trial register number, (b) the investigators’ names, and (c) keywords describing the intervention or the condition (PICO elements). The third step involved a Google search with the same search terms.
All identified publications belonging to the registered study were checked for their content (study design, population characteristics, dates of recruitment, intervention, comparator). Publications which clearly described the results of the originally planned and registered study were included.
Data extraction from scientific publications
We extracted the following data from the identified publications: the presence of author(s) with a Hungarian affiliation; the number of Hungarian authors or whether Hungarian participation in the study was mentioned in a way other than author affiliation; the journal’s name, and date of publication. In cases when there were different forms of publishing (e.g., published electronically ahead of print), we recorded the first date when the full text of the final manuscript was accessible.
To estimate the time to publication, we counted the total months elapsed between trial end date available in the EU Clinical Trials Register and the publication date. We expressed publication rate as the percentage of clinical trials with a full scientific publication divided by all clinical trials. The number of all clinical trials was calculated separately for each month after trial completion.
We also wanted to determine the public’s access to the results of published scientific papers. We therefore investigated whether scientific publications were published openly or with closed access. No distinctions were made between publications published in an open access journal and publications published in a hybrid journal by using the open access option.
Results were summarized as frequencies and proportions for binary data, and as medians and interquartile ranges for continuous data. We considered three analysis sets: a dataset based on all trials authorized in 2012 in Hungary, a dataset based on the trials completed for longer than one year in June 2020, and a dataset based on corresponding publications. Data was analyzed by descriptive statistics and cross-tabulation. Time to publication was estimated with the nonparametric Kaplan-Meier estimator and the logrank test (by Mantel-Cox) was used to estimate the potential effects of investigated factors on time to publication. Hazard ratios and confidence intervals were calculated on the basis of a Cox regression model. We used the statistical program SPSS version 26 (SPSS INC., Chicago, IL, USA) for our analyses.
Included clinical trials
A total of 614 clinical trials were identified in our search. Trials were excluded if the year 2012 did not correspond to the Hungarian registration date, but to the registration date of another participating country of multinational trials. A total of 330 Hungarian national or international clinical trials were eligible for inclusion into our methodological cohort (see Additional file 1). Eight years after trial authorization (in June 2020), 232 were “completed” trials for at least 1 year. Baseline characteristics of these trials are presented in Table 1.
Most of the trials were international, initiated, and funded by the industry. The majority assessed both efficacy and safety of a therapeutic intervention. Of the investigated clinical trials, 91.8% were registered not only in EU CTR, but both in the EU CTR and the clinicaltrials.gov database.
Publication rate over time is shown in Fig. 1.
Twelve months after completion, 19.8% of clinical trials were published as a full scientific paper. While five years after trial completion 19.4% of studies were still not available as a full publication. The time between the end of the clinical trial and the publication of the full scientific paper was 21 (22) months [median (IQR)].
At the time of our search, 74.1% of completed clinical trials had an available corresponding scientific publication. Overall, 70.7% of trial results were available as both full scientific publications and were posted in registries; 3.4 % were publications without posted results in registries, 24.1 % were in registries without an available full scientific publication, and 1.7 % were completed for longer than a year without available results (see Additional file 2).
Factors influencing time to publication
Time to publication was significantly shorter for trials conducted both inside and outside of Europe compared to trials only in Europe (26 vs 69 months [median]; Additional file 3; log-rank p< 0.001); and in case of trials registered in both EU CTR and clinicaltrials.gov compared to trials registered in EU CTR only (27 vs 88 months [median]; Additional file 4; p< 0.001). Time to publication was not influenced by either being a national or an international trial (37 vs 27 [median]; Additional file 5; p=0.45), an RCT or non-RCT (27 vs 32 [median]; Additional file 6; p=0.56), or by the presence or lack of a DMC (25 vs 33 [median]; Additional file 7; p=0.28) (Table 2).
Scientific results were published earlier, if published in a Q1 as compared to a Q2–Q3 journal (Fig. 2; log-rank p=0.001; HR [95% CI]: 2.14 [1.32–3.48], p=0.002).
Measures of scientific impact
Hungarian authorship and participation
Hungary was mentioned in 48.0% of the scientific publications (either as an author’s affiliation or as a study site listed in the text). Publications had at least one author with a Hungarian affiliation in 21.5% of cases (16.3% with one Hungarian author, 5.2% with two or more Hungarian authors).
Trials conducted within Europe (RR 2.184 [1.104–4.321]) and trials initiated by the academy instead of the industry (RR 3.706 [1.953–7.032]) significantly increased the probability of a scientific publication with a Hungarian author. Other investigated factors, such as studies registered in EU CTR only (0.769 [0.125–0.707), the lack of a DMC (1.071 [0.594–1.933]), non-RCT studies (1.216 [0.588–2.511]), and national trial studies (1.565 [0.308–7.957), had no effect on Hungarian authorship.
Impact factor and Scimago ranking of the journal
No differences were observed between the impact factor of publications with a Hungarian author compared to publications without (IF16.02 [16.59] vs. 19.47 [23.89]; mean [SD]; p=0.389). The impact factor tended to be higher for international compared to national trials (18.98 [22.63] vs. 4.58 [4.48]; p=0.171), for trials conducted in and outside of Europe compared to trials within Europe only (19.28 [22.77] vs.12.47 [19.09]; p=0.264), and for trials registered in both the clinicaltrials.gov and the EU CTR compared to trials registered in EU CTR only (19.08 [22.80] vs.8.92 [8.21]; p=0.445).
Trials with a data monitoring committee (DMC; i.e., a group of experts external to the study monitoring safety during study conduct) were more likely to publish results in a Q1 journal (1.131 [1.008–1.270]), while studies initiated by the industry were less likely (0.875 [0.826–0.926]). Other investigated factors, such as registration in two or only one registry (1.055 [0.735–1.516]), international versus national trial (2.663 [0.537–13.204]), trials conducted outside of versus within Europe (1.249 [0.893–1.748]), or study design (1.020 [0.871–1.195]) did not affect the Scimago journal ranking of the subsequent publication.
Study results available for all, as part of an open access publication
A total of 70.93% of scientific publications were open access, making the results accessible to the public. None of the investigated factors increased the probability for open access publication (see Additional file 8).
Influence of the industry on transparency and scientific impact of a trial
The number of authorized clinical trials initiated by the academy was extremely low (2.7% of all authorized trials). Clinical trials initiated by the industry or the academy are shown and compared in Table 3.
Industry-initiated trials with accessible information regarding funding were all funded by the industry. Investigator-initiated clinical trials were also partly funded by the industry. Information on funding was unavailable for one third of investigator-initiated trials.
A DMC was available to a much larger extent in trials initiated by the industry.
Results of investigator-initiated trials were significantly less likely to be posted in a clinical trial register (73.1% vs. 11.1%) and also slightly less likely to be published as a full scientific publication (62.8% vs. 55.6%).
Both the rate of publications with at least one Hungarian author (33.3% vs. 12.2%), and the rate of publications mentioning Hungarian participation in any form (44.4% vs. 27.5) was higher among investigator-initiated as compared to industry-initiated trials.
Summary of findings
Our study provides empirical evidence about the publication tendencies of authorized clinical trials in Hungary and their impact on the scientific reputation of the authorizing country.
Overall, 97.3% of authorized and EU CTR-registered clinical trials were initiated by the industry. About 20% of clinical trials were published within one year after trial completion. Trials conducted only within Europe and registered only in the EU CTR register were published significantly later.
Universality is a fundamental principle of science ; open access publications have therefore the largest impact on the scientific community. In this study, 70.93% of publications were found to be openly accessible to the public. However, we were not able to identify trial characteristics which might influence the access to scientific publications.
Only 21.5% of publications included Hungarian co-authors. Hungarian authors were most often present in publications of clinical trials conducted in Europe and trials initiated by the academy.
Trials registered both in the EU Clinical Trials Register and clinicaltrials.gov database were more likely to result in full publications. Almost one quarter of the results of the investigated clinical trials were available in registries, but not as a full scientific publication. Only a minority of clinical trials were available as full scientific papers and without results posted in registries.
Strength and limitations of this methodological research study
Our study analyzed a representative sample of trials authorized in Hungary which allows the results to be generalized to the country. All trials were registered in study registries, thus basic study information was available for all the trials included in our study. All data extractors were trained and the main outcome data were double-checked and double-extracted.
Our study has limitations. The cohort was composed of trials that were authorized and included in the EU Clinical Trials Register by the national authority. Due to the low number of investigator-initiated trials, it is difficult to draw firm conclusions on their publication tendencies. Nevertheless, our results may indicate a publication trend also for investigator-initiated trials.
The EU Clinical Trials Register defined trials as completed when a trial „has been completed in accordance with the full requirements of the protocol”, which might be interpreted differently by researchers and may have impacted our results. We searched for scientific publications in 2020 and trials completed within one year before the search date were excluded.
Comparison with other studies
To the best of our knowledge, this is the first study that investigates to what extent research authorized and conducted in a given country becomes visible and affects t domestic scientific performance.
Authorship issues were previously discussed by several papers, analyzing gender distribution of authorship [12, 13]; association between sponsorship and authorship [14, 15]; under-representation of researchers from specific regions in papers published from research done in these regions [16, 17]; and difficulties and possibilities in determining authorship in multicenter clinical trials [18, 19]. These studies were mostly based on publication data sets. We demonstrate a new approach to prospectively follow trials authorized in a given country until publication and investigate authorship in this matter.
We found one large cohort study that investigated compliance with result reporting in the EU Clinical Trials Register up to December 2016 . Trials with a commercial sponsor more likely posted results in the EU CTR than studies with a non-commercial sponsor (68.1% v 11.0%) . This is in line with the results of the present study: industry-initiated trials posted significantly more results in the EU CTR (73.1%) than investigator-initiated trials (11.1%). Investigator-initiated trials are in need of standardized procedures , and the proper implementation of the following additional requirements: periodic quality control assessments during trial implementation , improved reporting about funding , improvement of reporting , mandatory trial registration and trial result posting.
Implications of findings for practice, policy and future research
All interventional clinical trials on medicinal products authorized in the European Union—without any distinction by type of sponsor—should be registered in the EU Clinical Trials Register as a first step. Trial registration in clinicaltrials.gov can further increase the visibility of registered European trials.
The present research revealed that a surprisingly low number of clinical trials initiated by the academy get registered in the EU Clinical Trial Register in Hungary. Academic clinical trials have an important place on the map of clinical research. These studies focus on specific questions that arise during clinical care and are extremely important in everyday medical practice. These include but are not limited to facilitating the optimization of a therapy, or the discovery of potential new clinical areas where a therapeutic intervention can be used. Increased transparency of results of academic clinical trials is essential for evidence-based medical decision-making and optimal patient management.
Posting trial results in study registries might be the first step to allow study results to become openly available for the public; however, results should be additionally published as soon as possible as a scientific publication after trial completion. Systematic reviewers and guideline developers are advised to search clinical trial registers in addition to electronic databases to identify study results, which have not been published as full text publications at the time of the search.
The participation of Hungarian researchers in industry-initiated studies on medicinal products has only partial measurable scientific benefits, since Hungarian researchers appear as authors in only a fraction of scientific publication derived from these trials. Several publications did not even include the list of countries of trial participants. The Lancet journals strongly support the inclusion of domestic authors in papers reporting studies from those countries; we also would like to “encourage authors to include researchers who originally collected the data, where possible, and to share expertise in analysis and other skills, so that the research capacity of the country from which the data were obtained is strengthened” , i.e., to enable local researchers to fulfill the criteria for authorship developed by the International Committee for Medical Journal Editors.
The scientific performance of universities and countries is evaluated and ranked—despite valuable initiatives for change—based on research productivity (i.e., the number of scientific publications), research impact, and research excellence (i.e., the number of scientific papers in high-impact journals). Slightly over a fifth of authorized Hungarian trials result in scientific publications with a Hungarian co-authorship. We can conclude that the authorized, mainly industry-initiated clinical trials on medicinal products currently result in limited measurable scientific benefits to the participating researchers and their countries.
We call researchers of investigator-initiated clinical trials to register their trials in an openly available clinical trial register. Trial registers have to be considered as an important source of information of clinical trial results, as they may contain results from unpublished trials or trials published with closed access. Domestic scientific impact of trials on medicinal products has to be further improved. An increase in the number and role of investigator-initiated trials might help to achieve this goal.
Availability of data and materials
The data supporting the conclusions of this article is included within the article (and its additional files).
Ioannidis JP. Why most clinical research is not useful. PLoS Med. 2016;13(6):e1002049.
DeVito NJ, Goldacre B. Catalogue of bias: publication bias. BMJ Evidence Based Med. 2019;24(2):53–4.
Commission Guideline — Guidance on posting and publication of result‐related information on clinical trials in relation to the implementation of Article 57(2) of Regulation (EC) No 726/2004 and Article 41(2) of Regulation (EC) No 1901/2006 (2012/C 302/03). Available at: https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:52012XC1006(01)&from=EN. Accessed 08 Apr 2022.
Brown T. It’s time for alltrials registered and reported. Cochrane Database Syst Rev. 2013;4:Ed000057.
Taichman DB, Sahni P, Pinborg A, Peiperl L, Laine C, James A, et al. Data sharing statements for clinical trials: a requirement of the International Committee of Medical Journal Editors. PLoS Med. 2017;14(6):e1002315.
Joint Letter by the European Commission, EMA and HMA. Letter to stakeholders regarding the requirements to provideresults for authorised clinical trials in EudraCT: RefAres; 2019. 3933941 -20/06/2019
Bornmann L, Leydesdorff L. Scientometrics in a changing research landscape: bibliometrics has become an integral part of research quality evaluation and has been changing the practice of research. EMBO Rep. 2014;15(12):1228–32.
Goldacre B, DeVito NJ, Heneghan C, Irving F, Bacon S, Fleminger J, et al. Compliance with requirement to report results on the EU Clinical Trials Register: cohort study and web resource. BMJ (Clinical research ed). 2018;362:k3218.
Turner L, Shamseer L, Altman DG, Schulz KF, Moher D. Does use of the CONSORT Statement impact the completeness of reporting of randomised controlled trials published in medical journals? A Cochrane review. Syst Rev. 2012;1:60.
San Francisco Declaration on Research Assessment. Available at: https://sfdora.org/read/ (accessed on 01 Febr 2021).
Schiltz M. Science Without Publication Paywalls: cOAlition S for the Realisation of Full and Immediate Open Access. PLoS Med. 2018;15(9):e1002663.
Xu GM, Zavalkoff S, de Wildt SN, Duffett M. Gender and Authorship in Pediatric Critical Care Randomized Control Trials. In: Pediatric critical care medicine : a journal of the Society of Critical Care Medicine and the World Federation of Pediatric Intensive and Critical Care Societies; 2020.
Bernardi K, Lyons NB, Huang L, Holihan JL, Olavarria OA, Martin AC, et al. Gender Disparity in Authorship of Peer-Reviewed Medical Publications. Am J Med Sci. 2020;360(5):511–6.
Raman S, Moraes FY, Mendez LC, Taunk NK, Suh JH, Souhami L, et al. The relationship of study and authorship characteristics on trial sponsorship and self-reported conflicts of interest among neuro-oncology clinical trials. J Neuro-Oncol. 2018;139(1):195–203.
Tauber M, Paul C. Authorship selection in industry-sponsored publications of dermatology clinical trials. Br J Dermatol. 2017;176(6):1669–71.
Mbaye R, Gebeyehu R, Hossmann S, Mbarga N, Bih-Neh E, Eteki L, et al. Who is telling the story? A systematic review of authorship for infectious disease research conducted in Africa, 1980-2016. BMJ Glob Health. 2019;4(5):e001855.
Kelaher M, Ng L, Knight K, Rahadi A. Equity in global health research in the new millennium: trends in first-authorship for randomized controlled trials among low- and middle-income country researchers 1990-2013. Int J Epidemiol. 2016;45(6):2174–83.
Whellan DJ, Kraus WE, Kitzman DW, Rooney B, Keteyian SJ, Piña IL, et al. Authorship in a multicenter clinical trial: The Heart Failure-A Controlled Trial Investigating Outcomes of Exercise Training (HF-ACTION) Authorship and Publication (HAP) scoring system results. Am Heart J. 2015;169(4):457–63.e6.
Dulhunty JM, Boots RJ, Paratz JD, Lipman J. Determining authorship in multicenter trials: a systematic review. Acta Anaesthesiol Scand. 2011;55(9):1037–43.
Mudaranthakam DP, Phadnis MA, Krebill R, Clark L, Wick JA, Thompson J, et al. Improving the efficiency of clinical trials by standardizing processes for Investigator Initiated Trials. Contemp Clin Trials Commun. 2020;18:100579.
Figer BH, Sapra KP, Gogtay NJ, Thatte UM. A comparative study to evaluate quality of data documentation between investigator-initiated and pharmaceutical industry-sponsored studies. Perspect Clin Res. 2020;11(1):13–7.
Madeira C, Santos F, Kubiak C, Demotes J, Monteiro EC. Transparency and accuracy in funding investigator-initiated clinical trials: a systematic search in clinical trials databases. BMJ Open. 2019;9(5):e023394.
Landewé RB, Smolen JS, Weinblatt ME, Emery P, Dougados M, Fleischmann R, et al. Can we improve the performance and reporting of investigator-initiated clinical trials? Rheumatoid arthritis as an example. Ann Rheum Dis. 2014;73(10):1755–60.
The Lancet P. Diversity and inclusion: from priority setting to publication. Lancet Psychiatry. 2018;5(11):855.
We would like to acknowledge the valuable help of the student participants, Bettina Kovács and Dávid Varga with data extraction from the registries. We would like to thank to Matthias Briel for his valuable comments on the manuscript draft.
Open access funding provided by University of Pécs. The preparation of this work was supported by the János Bolyai Research Scholarship of the Hungarian Academy of Sciences for SL (BO/00498/17/5). The funder had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Ethics approval and consent to participate
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Flowchart of clinical trial selection
Information transfer process from trial authorization until the publication of trial results
Publication rates over time in clinical trials involving only European and both European and non-European countries
Publication rates over time in clinical trials registered in both EU CTR and clinicaltrials.gov
Publication rates over time in national and international clinical trials
Publication rates over time according to trial design
Publication rates over time in clinical trials with and without a data Monitoring Committee
Probability of open access publication
About this article
Cite this article
Sándor-Bajusz, K.A., Kraut, A., Baasan, O. et al. Publication of clinical trials on medicinal products: follow-up on trials authorized in Hungary. Trials 23, 330 (2022). https://doi.org/10.1186/s13063-022-06268-y
- Trial registration
- Publication rates
- Time to publication
- Research impact
- Evidence-based medicine