Missing clinical trial data: the evidence gap in the safety of potential COVID-19 drugs

Background: Several drugs are being repurposed for the treatment of the coronavirus disease 2019 (COVID-19) pandemic based on in vitro or early clinical ndings. As these drugs are being used in varied regimens and dosages, it is important to enable synthesis of existing safety data from clinical trials. However, availability of safety information is limited by a lack of timely reporting of clinical trial results on public registries or through academic publication. We aimed to analyse the evidence gap in safety data by quantifying the number of missing clinical trial results for drugs potentially being repurposed for COVID-19 by conducting a rapid review of results posting on ClinicalTrials.gov and in academic publications. Methods: ClinicalTrials.gov was searched for 19 drugs that have been identied as potential treatments for COVID-19. Relevant clinical trials for any prior indication were listed by identier (NCT number) and checked for results and for timely result reporting (within 395 days of the primary completion date). Additionally, PubMed and Google Scholar were searched to identify publications of results not listed on the registry. A second, blinded search of 10% of trials was conducted to assess reviewer concordance. Results: Of 3754 completed trials, 1516 (40.4%) did not post results on ClinicalTrials.gov or in the academic literature. 1172 (31.2%) completed trials had tabular results on ClinicalTrials.gov. A further 1066 (28.4%) completed trials had results from the literature search, but did not report results on ClinicalTrials.gov. Key drugs missing clinical trial results include hydroxychloroquine (37.0% completed trials unreported), favipiravir (77.8%) and lopinavir (40.5%). Conclusions: There is an important evidence gap for the safety of drugs being repurposed for COVID-19. This uncertainty could cause a large burden of additional morbidity and mortality during the pandemic. We recommend caution in experimental drug use for non-severe disease and urge clinical trial sponsors to report missing results retrospectively.


Background
Coronavirus disease 2019 (COVID-19) is a pandemic infection caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2). Its global spread has been rapid and unprecedented, at the time of writing 24.5 million con rmed cases have been reported with 832,378 deaths across 188 countries (1).
Currently, treatment options for COVID-19 are limited. However, several drugs developed for other indications have shown promising results against SARS-CoV-2 in vitro, in animal models, or in compassionate use trials (2,3). Many of these drugs are now being experimentally repurposed for COVID-19 or are undergoing clinical trials in humans (4). Such candidates include nitazoxanide, remdesivir, favipiravir, lopinavir, darunavir, hydroxychloroquine, chloroquine, and ivermectin amongst others (5).
Investigations into the e cacy of these experimental treatments for COVID-19 are ongoing. Meanwhile, in response to positive media coverage, some speculative rather than evidence-based, governments are stockpiling vast supplies of these treatments in anticipation of their licensing for COVID-19. Furthermore, national regulatory institutions, meant to safeguard against the unsafe use of drugs, are under increasing pressure to relax approval standards to accelerate market-entry for COVID-19 treatments. For example, on April 27th 2020 the U.S. Food and Drug Administration (FDA) approved the antimalarial drug hydroxychloroquine for emergency treatment of COVID-19 with unknown optimal dosage and duration of treatment (6). However, the e cacy of hydroxychloroquine is still under question (7). Furthermore, it is cardiotoxic at the higher doses which may be indicated for COVID-19, causing QT prolongation leading to ventricular tachycardia and death (8). Care must be taken not to lose the rigorous safety standards usually stipulated for pharmaceuticals, even during a pandemic, to avoid unnecessary morbidity and mortality worldwide.
As many of these drugs have been used widely in other indications for years, there should be substantive information on safety, tolerability and pharmacokinetics available in the public domain, including public trial registries. Yet, whilst the International Committee of Medical Journal Editors (ICMJE) policy requires prospective registration of interventional studies on a WHO primary registry or on ClinicalTrials.gov, the ICMJE does not currently require researchers to report summary results on these registries before academic publication (9). However, according to the FDA Amendment Act 2007, the responsible party for some applicable clinical trials that are registered on ClinicalTrials.gov must report results to a public register within twelve months of the primary completion date, or in some cases risk a ne of $11,569 for every day results are delayed (10,11). Failure to share clinical trial results publicly can additionally have far-reaching consequences for health and public expenditure, as illustrated by the widespread stockpiling and prescription of oseltamivir (Tami u) during the swine u outbreak in 2009, despite a lack of evidence on safety and e cacy (12). When the clinical study reports for this therapeutic were nally made publicly accessible, the bene ts of the product turned out to be exaggerated and misrepresented in the journal publications compared to the underlying data. Following the same H1N1 swine u outbreak a novel vaccine (Pandemrix) was rapidly rolled out, but it only emerged seven years later that the manufacturer had failed to disclose internal pharmacovigilance data showing narcolepsy to be a rare side-effect of recipients of the vaccine (13). Clinical trial transparency is therefore vital in order to maximise and unify the sharing of safety information and data on e cacy, during this pandemic and beyond.
Public clinical trial registries are an important tool for transparent collaborative research. On these registries, safety and e cacy data can be uploaded freely, shortly after completion of the study, and protocol and data collection methods are still quality assessed (14,15). In contrast, academic publication may stall for signi cant periods of time and can be costly and selective, with time-intensive writing and review processes. Furthermore, clinical trial registry data is available free of charge and can be pooled without concern for silent outcome switching or publication bias (16-18). Indeed, inclusion of unpublished study results from clinical trial registries in meta-analysis may provide important additional information on adverse events and more precise risk estimates than looking at journal publications alone (19). Moreover, academic publications often fail to disclose all information on adverse events occurring during clinical trials, and a systematic review comparing the journal publication with unpublished documents has suggested in 75% of the studies reviewed, the number of named adverse events was lower in published medical literature (20). This is especially important during the COVID-19 pandemic, as interest in potential treatments for COVID-19 generates even greater incentive than normal for the publication of studies with positive results only. Without rapid sharing of datasets for drugs that may be repurposed for COVID-19, secondary analyses of safety data will be arduous and often incomplete (14,21). This may slow down the biomedical innovation process and could lead to preventable side effects occurring in vulnerable patients if safety information remains missing. As the pharmaceutical pipeline is accelerated to address the COVID-19 pandemic, enhancing clinical trial transparency is more important than ever.
In this rapid review of ClinicalTrials.gov, we aimed to determine the scale of unpublished clinical trial results which may hinder safety reviews of repurposed drugs for COVID-19. We reviewed the number of completed or terminated trials that have not reported results for an extensive, list of medications being repurposed for COVID-19, looking at all previous indications for these drugs. Speci cally, we searched for any trial results that have not been made available to the public, with no results published on either the ClinicalTrial.gov registry or in the academic literature.
The U.S. clinical trials registry (ClinicalTrials.gov), was searched for all trials that listed these drugs as an intervention. Results of the search were downloaded on 4th April 2020 (25). Numbers of trials with and without results on the registry were recorded. For trials without results, trial status was determined (completed, ongoing, suspended, terminated or withdrawn). Trials listing 'primary completion date' in the future were counted as ongoing, if no primary completion date was available then the study completion date was used. Listed trial status was used to identify terminated, suspended and withdrawn trials.
For all trials without results on ClinicalTrials.gov, a three-step process was followed between 4th -27th April 2020 to determine whether results were reported elsewhere through academic publication (Fig. 1).
1. Publications automatically indexed by clinical trial identi er (NCT number) on ClinicalTrials.gov were screened and included based on criteria below. If multiple publications were listed, the earliest dated publication was selected. 2. If results were not available on the registry, the NCT number was used to search and screen academic publications in PubMed. 3. If the PubMed search did not retrieve an academic publication, an additional search was conducted in Google Scholar using the following search terms in succession: clinical trial identi er; listed title; intervention name with primary investigator's name. For each search, the rst twenty results were screened.
[  Flow chart depicting the methodology used to search for and identify relevant publications for each of the trials listed on CT.gov If a publication did not include the clinical trial identi er, it was cross-referenced with the primary investigator, study design, intervention and outcomes listed on ClinicalTrials.gov to assess relevance. We excluded publications that had fewer than 500 words, as well as conference abstracts, posters, presentations, and non-English texts.
Publications of results were recorded by PMID, DOI, and publication date. Trials with results on ClinicalTrials.gov combined with those with a journal publication gave a total number of trials with results where results were found in the public domain. This allowed approximation of registered trials without results.
Additionally, overdue trials were calculated as any completed trial with no result on the registry and a primary completion date before 18th April 2019, 395 days prior to nal analysis (1 year + 30-day grace period). This is the standard outlined in the FDAAA 2007 and used as a reference throughout this study despite not all included trials being covered by the law (10). This is also consistent with international ethical standards for timely results dissemination set by the World Health Organisation (26).
A second review was conducted by a different researcher on 10% of trials for each drug to check concordance between reviewers. The protocol during the second review remained unchanged and researchers were blinded to the results of the rst review. A random number generator was used to select trials for second review. Concordance was calculated by simple percent agreement on a results judgement; above 80% was deemed acceptable from consulting experts and in line with recommendations in the literature (27). Results published between the dates of rst and second review (29th April -9th May 2020) were not counted in the concordance.

Results
Nineteen drugs were screened, encompassing a total of 4553 clinical trials registered on ClinicalTrials.gov (Table 1). We excluded 799, with 661 ongoing (primary completion date in the future) and 138 with a trial status of suspended or withdrawn. Figure 2 shows the number of trials found on ClinicalTrials.gov, those excluded from this analysis, and the nal results status of all included trials. All recorded percentages in text are in relation to the 3754 completed trials, seen in Table 2.
[FIGURE 2] Figure 2 Flow diagram displaying the numbers of registered trials identi ed on CT.gov and the proportion of these which ultimately have no results available. The number of trials which had results available from various sources or had been withdrawn/suspended or trails for which results were not yet due are also shown. In the blinded second review of 341 (10%) trials, the same results status was found 83.6% of the time meeting our 80-100% threshold for acceptable concordance in searches.  Bar chart displaying the proportion of trial results available across all potential COVID-19 interventionscategorised into those registered fully on CT.gov, those with results available elsewhere and those with no results available.

Discussion
Of the completed clinical trials for existing drugs that may be repurposed for COVID-19, 40.4% did not report results on either ClinicalTrials.gov or through academic publication (  (28)(29)(30). These drugs are currently being used in COVID-19 patients and clinical trials across the globe, sometimes in novel regimens and doses (7,31,32). Clinicians currently have few treatment options available, but with greater transparency and proactiveness from trial sponsors regarding the posting of results, there would be less risk of unforeseen adverse outcomes, especially in the treatment of mildmoderate COVID-19 as in the PIONEER trial (31).
It is recognised that there are sources of safety data outside of clinical trial registries, for example pharmacovigilance registers, however these databases are not always free to access and publicly available. In contrast with clinical trial registries, there is often no obligation for all adverse events to be reported by trial sponsors. Public health decision-makers, guideline developers, clinicians, and patients therefore rely on clinical trial registries, systematic reviews, and meta-analyses to inform treatment decisions. Evidence gaps and publication bias therefore have the potential to in uence clinical practice and drug usage worldwide, particularly in a treatment landscape as changeable as the COVID-19 pandemic. If clinical decisions are based on incomplete evidence, this can result in avoidable morbidity and mortality, especially if unsafe drugs or ineffective treatments are given on a large scale. Sponsors and researchers alike carry an ethical responsibility towards clinical trial participants, who consent to participate in research in order to contribute to scienti c understanding and improved clinical practice, to make results publicly available (33).
Our study reveals an important evidence gap regarding existing pharmaceuticals potentially being repurposed for COVID-19. However, the proportion of studies with results available in the academic literature is an approximation and there are several limitations to our study. Firstly, our trial population was limited only to those registered on ClinicalTrials.gov. While ClinicalTrials.gov is the largest registry in the world, with over 340,000 registrations as of writing and an order of magnitude greater than the next largest journal, additional trials on these therapies may have been registered elsewhere. However, it is unlikely that these would be expected to report at a different rate than those registered to ClinicalTrials.gov. Secondly, our strategy for locating publications included only those listed on ClinicalTrials.gov and identi ed through searches on PubMed and Google Scholar, which are open-access resources that should cover a majority of published clinical research. While including proprietary databases like Scopus or Ovid may have located some additional publications, we do not believe this would have substantially impacted our overall results (34). Thirdly, we are aware that trials that were not registered at all or published in non-English language journals without inclusion of the NCT number would not have been captured by our methodology. Finally, searcher heterogeneity and di culty identifying results publication in the academic literature limits accuracy in any manual publication search, however, our search strategy was standardised and produced a high level of agreement between assessors (83.6%) in a check of a 10% random sample. However, any discordance between reviewers only reveals the inherent di culties in nding results for the drugs in question, especially if the trial identi cation number was not included in line with CONSORT standards (35).
Our ndings add to the existing evidence of the dearth of clinical trial reporting on public registries. This analysis investigated clinical trials of existing drugs currently being considered for use for COVID-19.
However, given the diversity of drug classes included in this report, ndings are likely to be representative of many pharmaceuticals. This presents a major problem for researchers attempting to summarise safety and e cacy by pooling trial data (36). Since academic publications often summarise key ndings only, secondary research efforts are impinged by the incomplete publishing of all trial outcomes. ClinicalTrials.gov, in contrast, provides a forum to share complete safety and e cacy data reports, as well as facilitating consistent data reporting in a timely manner (14,15). Prior research has shown that results reported to ClincialTrials.gov were often more complete, especially for safety data, when compared to matched journal publications (16-18). However, the availability of this data depends on researchers registering trials and uploading results in a timely manner, within twelve months of the primary completion date.
The International Committee of Medical Journal Editors (ICMJE) and the editorial o ces of medical journals could play an important role in improving the lack of timely results posting on clinical trial registries by demanding submission of a link to summary results on public registries before academic publication, although this may mean that the publication bias of positive, 'publishable' results may trickle down to reporting on public registries as well. Furthermore, public funders and institutional publication funds could demand that trial sponsors post their results before allocating funding for academic (openaccess) publication. These funding bodies could also deny individual sponsors funding if they have in the past violated clinical trial reporting rules (37). At the very least, journals should conform to the CONSORT statement in ensuring that registry identi cation numbers are clearly indicated in the abstract, full-text, and meta-data of published clinical trials in order promote discoverability and record linkage between registries and publications (35). Finally, clinical trial sponsors, such as universities, hospitals, public research institutions, and pharmaceutical companies, should themselves work towards improving their institutional clinical trial reporting performance by making use of available resources that provide detailed step-by-step instructions as to how to go about this task (11). Especially during the COVID-19 pandemic, it is of great importance that trials sponsors release summary results on these registries retrospectively to inform decision making around existing treatments being re-purposed for COVID-19.

Conclusions
Overall, our ndings reveal a signi cant evidence gap for the safety of drugs being repurposed for COVID-19. This uncertainty could cause a large burden of extra morbidity in the global pandemic. We recommend caution in experimental drug use for non-severe disease and urge trial sponsors to report missing results retrospectively. Medicine during the COVID-19 pandemic cannot be evidence-based if a large proportion of the evidence is missing. Availability of data and materials

List Of Abbreviations
The datasets generated and/or analysed during the current study are available on the U.S. National Library of Medicine (https://clinicaltrials.gov/ct2/home).

Competing interests
Three of the co-authors on this paper are part of Universities Allied for Essential Medicines U.K. However, views expressed in this paper are not necessarily that of Universities Allied for Essential Medicines Europe.

Funding
This research received no speci c grant from any funding agency in the public, commercial or not-forpro t sectors.
Authors' contributions FR and TP conceptualised the study, devised the methodology, and coordinated the research team. SK recruited the research team. All authors were involved in data collection and analysis, and contributed to the nal manuscript.