Quality, scope and reporting standards of randomised controlled trials in Irish Health Research: an observational study
Trials volume 21, Article number: 494 (2020)
Despite efforts to improve the accuracy and transparency of the design, conduct, and reporting of randomised controlled trials (RCTs), deficiencies remain. Such deficiencies contribute to significant, avoidable waste of health research investment and impede reproducibility. This study aimed to synthesise and critically analyse changes over time in the conduct and reporting of internationally published evidence on patient and/or population health-oriented RCTs conducted in one country.
This observational study drew on systematic review methods. We searched six databases for published RCTs (database inception to December 2018) where ≥ 80% of participants were recruited in the Republic of Ireland. RCTs of interventions targeted at patients, providers and/or policy makers intended to improve health, healthcare or health research were included. For each study, screening, data extraction and methodological quality appraisal were conducted by one member of the author team.
From 17,560 titles and abstracts, 752 unique RCTs were published in 745 papers between 1968 and 2018, with a steady year-on-year increase since 1968. The number of participants was in the range of 2–8628. The majority were parallel design (86%) and classified as treatment evaluation. Of the 418 RCTs published since the introduction of mandatory clinical trial registration by the International Committee of Medical Journal Editors in 2005, 32% (n = 134) provided a trial registration number. This increased to 47% when taking studies published between 2013 and 2018 (n = 232). Since the 1996 publication of the CONSORT statement, 16% of included RCTs made specific reference to a standardised reporting guideline and this increased to 31% for more recent studies published between 2013 and 2018. Overall, 7% (n = 53) of studies referred to a published study protocol, increasing to 20% for studies published between 2013 and 2018.
Evidence from this single-country study of RCTs published in the international literature suggests that both the number overall, the number registered and the number referencing reporting guidelines have increased steadily over time. Despite widespread endorsement of reporting standards, reporting of RCTs remains suboptimal in domains such as compliance with the CONSORT statement and prospective trial registration. Researchers, funders and journal editors, nationally and internationally, should continue to focus on improving reporting and examining avoidable waste of health research investment.
Randomised controlled trials (RCTs), when appropriately designed, conducted and reported, represent the gold standard in evaluating the effectiveness of healthcare interventions and have traditionally been considered the backbone of evidence-based medicine [1, 2]. The findings of RCTs should be published, rigorously conducted and clearly reported to inform decision making.
The number of RCTs conducted and reported has continually increased over time and it is difficult to quantify the exact volume of this research activity . The extent to which published RCTs reflect the efficacy of interventions, however, depends on the completeness and accuracy of published results. Two important initiatives have emphasised the need to increase the accuracy and transparency with respect to the performance and reporting. In 2005, the International Committee of Medical Journal Editors (ICMJE) introduced mandatory trial registration guidelines and member journals require prospective registration of RCTs before patient enrolment as a condition of publication . The Consolidated Standards of Reporting in Trials (CONSORT) reporting guideline was developed and published in 1996 to improve the accuracy and transparency of reports of RCTs . Both initiatives aim to increase the adequacy of reporting of important aspects of how trials are designed, analysed and interpreted to overcome issues such as outcome switching or publication bias. Despite these and other efforts to improve the accuracy and transparency of the design, conduct, analysis and interpretation of RCTs, deficiencies and lack of adequate reporting continue . A large systematic review updated in 2012 showed that, despite improvements in the completeness of reporting for 22 of 25 CONSORT checklist items, there are still major reporting deficiencies in some areas . Subsequent reviews and research have confirmed that reporting remains suboptimal in areas such as sample size calculations and consistency between pre-specified and reported outcomes [6,7,8,9]. Adherence to the CONSORT statement for Abstracts is also variable and incomplete . Adherence to the ICMJE’s prospective registration policy also remains suboptimal, with high-impact journals frequently publishing unregistered trials and trials registered after potential observation of primary outcome data . Such deficiencies contribute to significant and avoidable waste of health research investment and impede reproducibility .
It has been suggested that research-active organisations and companies should routinely audit their activity with regard to the registration and publication of RCTs [13, 14]. Over the last 20 years, there has been significant investment in improving health research in Ireland, highlighting a commitment to developing research capacity [15, 16]. There is also a shift away from funding biomedical research (broadly focused on the investigation of the biologic process and the causes of disease through careful experimentation, observation, laboratory work, analysis and testing) towards population health (research with the goal of improving the health of the population through investigation of social, cultural, environmental, occupational and economic factors or through the identification of effective interventions) and health services research (research with the goal of improving the efficiency and effectiveness of health professionals and the healthcare system) by some funding bodies such as the Health Research Board of Ireland .
Current research indicates that substantial amounts of publicly funded RCTs are not published or available to inform future research and practice . In this context, it is important to evaluate locally generated and published research and use this to improve standards and to contextualise the evidence that informs local healthcare and health policy. Th aim of the present study was to synthesise and critically analyse the changes in the reporting of published evidence from patient or population health-oriented RCTs conducted in one country, the Republic of Ireland, over time.
This observational study drew on systematic review methods to identify, appraise and synthesise RCTs. As we were not attempting to determine the effectiveness of any one particular intervention, a full systematic review was not deemed necessary. We used a systematic review approach to conducting the literature search but literature screening, data extraction and quality appraisal were not conducted in duplicate, as would be standard practice in a systematic review.
Data sources and search strategy
A literature search was performed including PubMed, Embase, Scopus, CINAHL, PsychINFO and the Cochrane Central Register of Controlled Trials, from database inception to December 2018. Within MEDLINE, we used the methodology filter for RCTs, ‘Randomized Controlled Trial’ (Publication Type), which has a sensitivity for retrieving RCTs of approximately 94% [19, 20]. Combinations of key words and MeSH terms were used to search the other databases. The search strategy was developed with the support of a Health Sciences Librarian with expertise in systematic review searching (Additional file 1). No language restrictions were applied.
Full-text studies were included if they met the following inclusion criteria:
Population: all studies conducted on human subjects of all age groups in an Irish healthcare setting (hospital and community) or where ≥ 80% of participants were recruited in the Republic of Ireland. International studies where ≥ 80% of participants were recruited in Ireland were also included.
Intervention: Any intervention targeted at patients, providers and/or policy makers intended to improve health, healthcare or health research (including biomedical research and population health and health services research).
Comparators: no limitation on comparators.
Primary outcome: any health outcome focused on patient/population health.
Study design: described by authors as a RCT of any design (e.g. parallel, cluster, factorial, cross-over, stepped wedge), with two or more groups. However, author-described RCTs which did not state explicitly that random allocation was employed were managed by consensus and studies that stated information to the contrary were excluded, e.g. authors stating the paper as a RCT but describing recruitment as consecutive. Clinical trial phases II and above were also included, so long as they were randomised.
Initially identified study titles were divided between authors and assessed in relation to the inclusion criteria by reading titles and abstracts. Potentially eligible studies were read fully and their suitability for inclusion determined individually by author team members. Full-text papers were obtained from a combination of institutional library access, online profiles (such as Researchgate) and by contacting authors. Multicentre, international studies where there was one or more Irish centre but < 80% of participants recruited in Ireland were recorded, but are not included in the analysis presented here.
Included full texts were divided between authors and reviewers extracted data from the included studies using a pre-established data extraction form. This was not done in duplicate. The reviewers extracted the information on study characteristics, participant demographics, study documentation and methodological quality from each study, including:
study characteristics: author, year of publication, study title, journal, study design as reported by the authors, objectives, inclusion criteria, clinical domain or condition as reported by the authors, clinical setting, composition of intervention/control groups, study dates and duration, number of outcomes, results for primary outcome (positive or negative, significant or non-significant), sample size, funding source.
documentation of reporting quality: registration number, protocol publication (yes or no), standardised reporting guideline explicitly used (yes or no).
methodological quality: Cochrane domains (see below).
The Health Research Classification System (HRCS) , developed by The UK Clinical Research Collaboration (UKCRC), was used to categorise research activity. The specific interventions were further classified by the primary disease focus (coded using International Classification of Diseases, 10th revision [ICD-10]), the primary procedures (using the Australian Classification of Health Interventions [ACHI]), the primary behavioural intervention (provider or patients) or the primary service delivery intervention.
One reviewer individually assessed the risk of bias of each RCT using the Cochrane Collaboration’s risk of bias tool including the standard domains of sequence generation, allocation concealment, blinding of participants, blinding of outcome assessors, incomplete outcome data, selective reporting and other bias.
A narrative synthesis of included studies is presented. Descriptive statistics are used to summarise trends over time in terms of the quantity and reporting quality of included RCTs across all years and a subset of the most recently published studies. A pragmatic cut-off of studies published between 2013 and 2018 was used as the most recently published.
We screened 17,220 titles and abstracts and identified 5205 studies for full-text review. Of these, 752 unique RCTs from 745 papers (some papers reported greater than one RCT) published between 1968 and 2018 were included (Fig. 1). The number of studies published per year has steadily been increasing since 1968 (Fig. 2). The majority of papers were published in international journals, with 2% publishing in an Irish specific journal (e.g. Irish Medical Journal).
Characteristics of included studies
Study design, setting and population
As shown in Table 1, the majority of studies were of parallel design (n = 643, 85%) and were most frequently conducted in the hospital, inpatient setting (n = 314, 42%). Less than 10% (n = 73) were conducted in primary care. The majority of studies were conducted with adult populations (n = 549, 73%), with older people (≥ 65 years, n = 43) and pregnant women (n = 47) being the specified population in 6% of studies each. The number of participants included in each RCT was in the range of 2–8628 (median 60, mean 142).
Research activity and interventions
An analysis by research activity (using HRCS categories) showed the majority of research was in the domain of treatment evaluation, with the top three areas of activity focused on pharmaceuticals (n = 556, 47%), physical activity (n = 88, 12%) and psychological and behavioural interventions (n = 75, 10%) (Table 2). The main comparators (Table 1) were an alternative intervention (n = 298, 40%), usual care (n = 270, 36%) or placebo (n = 160, 21%).
Over half (n = 401, 53%) of all studies focused on specific conditions with diseases of the circulatory system being the most frequently studied (n = 52, 13%). The conduct of procedures (e.g. hip surgery) were the primary focus in 34% (n = 253) of studies with non-invasive, cognitive and other interventions being most common in that category (n = 415, 45%). The top five conditions and procedures are presented in Table 3. The remaining studies were classified as behavioural intervention (provider or patients) or the primary service delivery interventions.
Of the 752 RCTs, 412 (55%) did not report their sources of funding. Government bodies were cited by 14% (n = 103), industry funding was cited by 12% (n = 89), with the remaining studies funded by charities, universities and other funders. Taking studies published between 2013 and 2018 (n = 232) as a subset of the most recently published, 35% (n = 81) did not report their sources of funding, government bodies were cited by 28% (n = 65), industry funding was cited by 5% (n = 12), with the remaining studies funded by charities, universities and other funders.
Of the 418 RCTs published since the ICMJE introduced mandatory clinical trial registration in 2005, 32% (n = 134 published between 2005 and 2018) provided a trial registration number although year-on-year increases were observed (Fig. 2). Taking studies published between 2013 and 2018 (n = 232) as a subset of the most recently published, this figure rose to 47% (n = 109) providing a trial registration number. Overall, 7% (n = 53) of studies referred to a published study protocol, with 20% (n = 47) of studies published between 2013 and 2018 referring to a published study protocol. Sixteen percent of RCTs published between 1996 (the CONSORT statement original publication year) and 2018 made specific reference to a standardised reporting guideline within the publication. Year-on-year increases were observed here also, with 31% (n = 72) of studies published between 2013 and 2018 making specific reference to a standardised reporting guideline (Fig. 2). Only 304 (40%) RCTs mentioned a sample size calculation, with 273 achieving the required sample size. Year-on-year increases were observed here also with 62% (n = 143) of studies published between 2013 and 2018 referring to a sample size calculation. Overall, 79% (n = 594) of studies reported having obtained approval by an ethics committee.
As summarised in Fig. 3, the risk of bias was low in the majority of studies for attrition bias (n = 594, 79%), reporting bias (n = 594, 79%) and detection bias (n = 414, 55%). Risk of bias was high or unclear in the majority of studies for performance bias (n = 392, 52%), allocation concealment (n = 467, 62%) and sequence generation (n = 376, 50%).
This study identified 752 unique RCTs published internationally across 745 papers between 1968 and 2018. Evidence from this single-country, observational study suggests that both the number and the reporting quality of RCT articles have increased steadily over time. However, reporting of RCTs remains suboptimal, with only 32% of included RCTs published since 2005 providing a trial registration and 16% of included RCTs published since 1996 making specific reference to a standardised reporting guideline; therefore, areas for improvement in a number of domains were identified.
Prospective registration of RCTs is critical for promoting transparency and integrity in research by discouraging undeclared protocol deviations or outcome switching which increase the likelihood that reported effects have arisen through bias, chance or are exaggerated. The RCTs identified in this study were particularly poor in adhering to the ICJME’s prospective registration policy, with only 32% of the RCTs published since 2005, the year ICMJE introduced mandatory clinical trial registration, providing a trial registration number and < 10% of studies overall referring to a published study protocol. This suggests that authors are either not registering trials or not reporting trial registration numbers, and journal editors are either not checking trial registries for appropriate registration or have identified reasons for non-registration and chose to publish it without transparently providing any documentation of this decision-making process. This is not an issue specific to research in Ireland. Adherence to the ICMJE’s prospective registration policy has been identified as suboptimal internationally, with high-impact journals frequently publishing unregistered trials and trials registered after potential observation of primary outcome data [11, 22]. The RCTs identified in this study were also poor in referring to and adhering to several items the CONSORT statement. The COMPare (Centre for Evidence-Based Medicine Outcome Monitoring Project) study has recently highlighted, based on five high-impact journals endorsing CONSORT, that journals exhibit extensive breaches of the CONSORT guidance, particularly with respect to outcome reporting . Our results confirm and extend this finding, given that we did not restrict by journal and the 752 RCTs were published across a range of journals and health research domains.
A particular area of poor reporting was study funding. The majority of studies included in this study did not report the source of funding within their manuscripts, although reporting in this area has improved over time. Most journals in which these papers were published, particularly for newer publications, require the reporting of the funding source. This reflects both a suboptimal compliance by authors with this requirement and poor implementation by journals internationally. The reporting of funding sources is necessary to allow adequate appraisal of primary research by individuals, those involved in evidence synthesis and decision makers. Funding sources may influence the reporting of research findings and the interpretation of results. Industry-funded trials are more likely to report favourable efficacy results and conclusions than those funded by other sources [23, 24]. Where RCTs do report funding, considerable variability in the reporting of funding source, amount and roles of funders has been identified and a standardised approach to reporting of funding information has been proposed . However, given the poor compliance to reporting guidelines identified here and in the international literature [5, 26], it is unlikely that such standards would be adhered to. Use of reporting guidelines is largely determined by a combination of factors such as individual factors (e.g. having multiple reasons for use of reporting guidelines), the professional culture in which people work, environmental factors (e.g. policies of journals) and practical factors (e.g. having time to use reporting guidelines) . Therefore, multifaceted interventions that target these factors are required to increase compliance .
We would echo the COMPare project in their call for greater journal transparency in how they comply with CONSORT in terms of outcome reporting, with greater consistency in use of trial registries and changes to CONSORT’s mechanisms for enforcement . It is also important to raise awareness in the peer-review process for the need to compare information recorded on a trial register to that reported in the manuscript as peer reviewers may not see this as their role, especially as editors may not clearly request the reviewer to assess this . This may be time-consuming, so, alternatively, authors should be required to submit a copy of their registration as part of the journal submission process. Increased focus on improving peer review may be a mechanism to improve reporting quality. A number of journals have educational and training material available to improve peer review; however, training in the process is not mandatory. Given the increasing funding allocated each year to evaluate and develop medical treatments, understanding the patterns of publication of research findings among publicly funded research is important for public benefit, to identify areas of good and bad practice, reduce research waste and identify ways to better utilise public funding. Like journal editors, funders such as the Health Research Board (HRB) of Ireland and Science Foundation Ireland may have a role in enforcing greater transparency in complying with reporting guidelines and mandatory trial registrations.
We conducted an extensive search of the published literature across six electronic databases. However, this study has some limitations including the inherent difficulty in identifying RCTs from a specific geographical location. We encountered two main obstacles in this process. First, most databases do not easily facilitate the identification of the country of origin of the study or study population. We used combinations of key words and controlled vocabulary in the search string. We did not include search terms based on author affiliation as some databases limit this field to the first author affiliation only and we found variability in how authors report their affiliations (for example, it was not always clear if the affiliation was their current position or where the research was conducted). Second, in screening published papers, we found a large number of studies that gave no information at all as to where the study was conducted so we therefore excluded those studies. Third, we did not conduct a grey literature search or include forwards or backwards citation checking as part of our search, as this was not a full systematic review. Taken together, these limitations might have resulted in some omissions, underestimating the rate of publication. We would recommend that future publications are more specific about reporting where studies are conducted.
While this observational study drew on systematic review methods to identify, appraise and synthesis RCTs, it was not a full systematic review. Given the size and scope of the literature, the team did not conduct all screening and data extraction in duplicate which also might have induced some omissions of included studies and data errors. A full assessment of adherence to all the CONSORT statement items was not undertaken.
Evidence from this single-country study suggests that both the number and the quality of RCT articles published internationally have increased steadily over time. Despite widespread endorsement of reporting standards, reporting of RCTs remains suboptimal in terms of adhering to several items the CONSORT statement and prospective trial registration, in keeping with international literature. Future research and journal editors should focus on improving reporting and examining avoidable waste of health research investment.
Availability of data and materials
Data have been uploaded to Zenodo https://doi.org/10.5281/zenodo.3754967.
Schulz K, Altman D, Moher D. CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials. BMJ. 2010;340:c332.
Frieden TR. Evidence for health decision making — beyond randomized, controlled trials. N Engl J Med. 2017;377(5):465–75.
Bastian H, Glasziou P, Chalmers I. Seventy-five trials and eleven systematic reviews a day: how will we ever keep up? PLoS Med. 2010;7(9):e1000326.
De Angelis C, Drazen JM, Frizelle FA, Haug C, Hoey J, Horton R, et al. Clinical trial registration: a statement from the International Committee of Medical Journal Editors. N Engl J Med. 2004;351(12):1250–1.
Turner L, Shamseer L, Altman DG, Weeks L, Peters J, Kober T, et al. Consolidated standards of reporting trials (CONSORT) and the completeness of reporting of randomised controlled trials (RCTs) published in medical journals. Cochrane Database Syst Rev. 2012;11:Mr000030.
Zhai X, Wang Y, Mu Q, Chen X, Huang Q, Wang Q, et al. Methodological reporting quality of randomized controlled trials in 3 leading diabetes journals from 2011 to 2013 following CONSORT statement: a system review. Medicine. 2015;94(27):e1083.
Rutterford C, Taljaard M, Dixon S, Copas A, Eldridge S. Reporting and methodological quality of sample size calculations in cluster randomized trials could be improved: a review. J Clin Epidemiol. 2015;68(6):716–23.
Stevens A, Shamseer L, Weinstein E, Yazdi F, Turner L, Thielman J, et al. Relation of completeness of reporting of health research to journals’ endorsement of reporting guidelines: systematic review. BMJ. 2014;348:g3804.
Goldacre B, Drysdale H, Dale A, Milosevic I, Slade E, Hartley P, et al. COMPare: a prospective cohort study correcting and monitoring 58 misreported trials in real time. Trials. 2019;20(1):118.
Hays M, Andrews M, Wilson R, Callender D, O'Malley PG, Douglas K. Reporting quality of randomised controlled trial abstracts among high-impact general medical journals: a review and analysis. BMJ Open. 2016;6(7):e011082.
Gopal AD, Wallach JD, Aminawung JA, Gonsalves G, Dal-Re R, Miller JE, et al. Adherence to the International Committee of Medical Journal Editors’ (ICMJE) prospective registration policy and implications for outcome integrity: a cross-sectional analysis of trials published in high-impact specialty society journals. Trials. 2018;19(1):448.
Glasziou P, Altman DG, Bossuyt P, Boutron I, Clarke M, Julious S, et al. Reducing waste from incomplete or unusable reports of biomedical research. Lancet (London, England). 2014;383(9913):267–76.
Powell-Smith A, Goldacre B. The TrialsTracker: automated ongoing monitoring of failure to share clinical trial results by all major companies and research institutions. F1000Res. 2016;5:2629.
Tompson AC, Petit-Zeman S, Goldacre B, Heneghan CJ. Getting our house in order: an audit of the registration and publication of clinical trials supported by the National Institute for Health Research Oxford Biomedical Research Centre and the Musculoskeletal Biomedical Research Unit. BMJ Open. 2016;6(3):e009285.
Department of Health and Children. Action plan for health Research 2009–13. Dublin: Department of Health and Children; 2009.
Department of Jobs Enterprise and Innovation. Report of the research prioritisation steering group. Dublin: Department of Jobs, Enterprise and Innovation; 2011.
Health Research Board. HRB Strategic Business Plan 2010–2014: the future of Irish health research. Dublin: Health Research Board; 2009.
Ross JS, Tse T, Zarin DA, Xu H, Zhou L, Krumholz HM. Publication of NIH funded trials registered in ClinicalTrials.gov: cross sectional analysis. BMJ. 2012;344:d7292.
Glanville JM, Lefebvre C, Miles JNV, Camosso-Stefinovic J. How to identify randomized controlled trials in MEDLINE: ten years on. J Med Libr Assoc. 2006;94(2):130–6.
McKibbon KA, Wilczynski NL, Haynes RB, for the Hedges T. Retrieving randomized controlled trials from medline: a comparison of 38 published search filters. Health Info Libr J. 2009;26(3):187–202.
UKCRC Health Research Classification System. https://hrcsonline.net/research-activities/. Accessed 1 June 2020.
Zarin DA, Tse T, Williams RJ, Rajakannan T. Update on trial registration 11 years after the ICMJE policy was established. N Engl J Med. 2017;376(4):383–91.
Bhandari M, Busse JW, Jackowski D, Montori VM, Schünemann H, Sprague S, et al. Association between industry funding and statistically significant pro-industry findings in medical and surgical randomized trials. CMAJ. 2004;170(4):477–80.
Lundh A, Lexchin J, Mintzes B, Schroll JB, Bero L. Industry sponsorship and research outcome. Cochrane Database Syst Rev. 2017;2:Mr000033.
Hakoum MB, Jouni N, Abou-Jaoude EA, Hasbani DJ, Abou-Jaoude EA, Lopes LC, et al. Characteristics of funding of clinical trials: cross-sectional survey and proposed guidance. BMJ Open. 2017;7(10):e015997.
Jin Y, Sanger N, Shams I, Luo C, Shahid H, Li G, et al. Does the medical literature remain inadequately described despite having reporting guidelines for 21 years? - A systematic review of reviews: an update. J Multidiscip Healthc. 2018;11:495–510.
Fuller T, Pearson M, Peters J, Anderson R. What affects authors’ and editors’ use of reporting guidelines? Findings from an online survey and qualitative interviews. PLoS One. 2015;10(4):e0121585.
Chauvin A, Ravaud P, Baron G, Barnes C, Boutron I. The most important tasks for peer reviewers evaluating a randomized controlled trial are not congruent with the tasks most often requested by journal editors. BMC Med. 2015;13:158.
No specific funding was obtained for this work; however, BC, FB, NM, FM, EW, AB, TD, SMS and TF were supported by the HRB Centre for Primary Care Research (grant HRC/2014/01), EM by Public and Patient Involvement (PPI) Ignite, DD by the HRB-Trials Methodology Research Network, and AM by the HRB Primary Care Clinical Trials Network Ireland. The funders had no role in the study design, data collection, data analysis, data interpretation, writing of the report or in the decision to submit the paper.
Ethics approval and consent to participate
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Clyne, B., Boland, F., Murphy, N. et al. Quality, scope and reporting standards of randomised controlled trials in Irish Health Research: an observational study. Trials 21, 494 (2020). https://doi.org/10.1186/s13063-020-04396-x