Identification of high-risk cardiovascular medical devices
One author (MJS) identified novel, high-risk cardiovascular medical devices from the publicly accessible FDA PMA database (www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfpma/pma.cfm) between January 1, 2005, and January 1, 2020, excluding automated external defibrillators (AEDs), studies that had missing data, and summaries that leveraged a meta-analysis for the pivotal study (Fig. 1). We excluded AEDs because FDA published a final order on January 29, 2015, stating that AED clinical study information can be leveraged from both published studies and clinical data previously submitted under the 510(k) process instead of requiring the conduct of a pivotal trial to support FDA approval [17]. Otherwise, devices were selected if they met both of the following parameters: “Cardiovascular” under advisory committee and “Originals Only” under supplement type. All devices were characterized by the following using publicly available information on the FDA website: FDA review type (priority/standard), implantable designation (yes/no), life-sustaining designation (yes/no), and combination product (yes/no). Sponsor company management (public/private) was also determined by Google searching the sponsor company name along with “publicly traded,” “stock price,” “IPO,” or “privately held.”
Characterization of pivotal clinical trials
For each device, one author (MJS) then identified the pivotal clinical studies that supported device approval from the “Summary of Safety and Effectiveness” documents. Pivotal clinical studies supporting approvals were categorized as pre-FDAAA if the clinical trial primary completion date was before December 26, 2007 (the date the policy took effect), in a manner described previously [9]; all other studies were categorized as post-FDAAA. Also, we categorized pivotal trials by specific design characteristics: use of randomization (yes/no), use of blinded allocation (yes/no), primary efficacy endpoint (surrogate marker/clinical outcome or scale), and study center and patient enrollment numbers. These metrics have been widely reported as assessing the validity and quality of clinical trials [9, 13, 18,19,20].
Study characteristics and data were abstracted from the FDA summaries by one author (MJS); a 10% subsample (17 devices) was randomly selected for validation by a second author (JLJ) at the beginning of data extraction. There were no disagreements in the 7 product/study characteristics extracted from FDA summaries and other sources (e.g., FDA review pathway, life-sustaining designation, implantable designation, combination product, company management, and use of randomization and blinding): percent agreement = 100% (119/119).
Following this independent validation, JLJ verified all device extractions for which MJS had been unable to identify clinical trial registration, result reporting, or publication, or for which MJS determined there was discordance in results reported or trial interpretation. There were a total of 132 transparency measure extractions for these devices, on which MJS and JLJ agreed on 97 and disagreed on 35 (percent agreement of 73.5%). Among these 35 disagreements, 3 related to trial registration, 5 to result reporting, 18 to publication, 8 to result concordance, and 1 to interpretation concordance. All disagreements were resolved via consensus among all authors. This process demonstrated that the initial search was overly reliant on the ClinicalTrials.gov hyperlinks, which were not always accurately reported and do not reliably identify trial publications [21]. Afterwards, the search strategy was revised and repeated to identify publications using and comparing clinical trial titles, product names, methods, number of study centers, enrollment numbers, primary efficacy endpoints, primary results, and study sponsors.
Identification of trials on ClinicalTrials.gov and published in the peer-reviewed literature
For each pivotal trial identified from FDA documents, we conducted a comprehensive search of ClinicalTrials.gov and PubMed’s listing of MEDLINE-indexed journals to identify any corresponding trial registration or publication, respectively. One author (MJS) conducted the initial search; a second author (JLJ) reviewed all pivotal trials for which a clinical trial registration or publication was not identified after the initial search by MJS; differences (n = 3 and 18, respectively) were reconciled by consensus among all authors. All document and website searches were performed during July 2020. Our search strategy included using and comparing clinical trial titles, product names, methods, number of study centers, enrollment numbers, primary efficacy endpoints, primary results, and study sponsors. While more recent FDA PMAs include ClinicalTrials.gov registration hyperlinks and ClinicalTrials.gov manually and automatically indexes corresponding publications of results to their registration by National Clinical Trial (NCT) number, these identification numbers did not reliably identify pivotal trial registrations and publications for older PMAs, consistent with prior reviews [21]. Among publications identified in PubMed, abstracts and conference reports were excluded. Publications reporting multiple studies, such as reviews and meta-analyses, were also excluded unless the results of each study were analyzed and discussed individually at the level of detail as one would expect from a full-length publication.
Comparison to corresponding publications
First, for each pivotal trial for which a publication was identified, we compared the primary effectiveness endpoint specified in the FDA documents with the effectiveness endpoint specified as primary in the publication. If there was more than one primary effectiveness endpoint reported in the FDA documents, we verified the one that matched the primary endpoint specified in the publication. If none of the specified primary endpoints matched, we categorized the primary effectiveness outcomes reported as discordant. If one matched, we determined whether the primary effectiveness endpoint result reported in the FDA documents was the same as the result reported in the publication. The outcomes reported were categorized as concordant if they shared all five defined elements of an endpoint (i.e., domain, measure, metric, method of aggregation, and timepoint) and were an exact numerical match or if there was a relative difference of less than 5% when compared to the FDA PMA, a conservative estimate intending to identify clinically meaningful differences while recognizing that there might be changes in analytical approaches over time [22]. Otherwise, the outcomes reported were categorized as discordant, as well as if the documented primary endpoint in the FDA materials was included in the publication but reported as a secondary outcome and if the primary endpoint in the FDA documentation pre-specified with the FDA was switched by the sponsor in their FDA documentation from what was pre-specified in a protocol or a registry entry. Second, for each pivotal trial for which a publication in the peer-reviewed literature was identified, we compared the overall study interpretation between the two sources. The overall interpretation was categorized as positive, equivocal, or negative based on the FDA officer’s language in the “Effectiveness Conclusions” and “Overall Conclusions” subsections of the “Summary of Safety and Effectiveness” document and the author’s language in the conclusion of the publication; the FDA and publication interpretation were categorized as concordant or discordant. One author (MJS) conducted the outcome comparisons; a second author (JLJ) verified the outcome comparisons classified as discordant by MJS and differences were reconciled by consensus among all authors.
Statistical analysis
We determined the rate of ClinicalTrials.gov registration, ClinicalTrials.gov result reporting, and PubMed publication for all identified pivotal trials, overall and stratified by device and design characteristics. We then determined the overall rate of concordant primary outcome reporting between the FDA PMA summaries and corresponding publications, as well as the overall rate of concordance between the FDA PMA reviewer’s interpretations and the trial publication’s interpretations. Summary statistics were calculated for each comparison, presented as numbers, percentages, means, standard deviations, and ranges, as appropriate. Chi-square and two-tailed Fisher exact tests were used to compare rates pre- and post-FDAAA of registration, result reporting, publication, concordant outcome reporting, and concordant interpretation, as appropriate. All statistical tests were two-tailed and used the Bonferroni method to correct our alpha value to account for multiple comparisons of five transparency measures across the sample of clinical trials for these devices: (1) registration, (2) result reporting, (3) publication, (4) concordant outcome reporting, and (5) concordant interpretation. Statistical significance was set at p ≤ 0.01. Analyses were performed using Microsoft Excel (version 16.35) and SPSS (version 27).
Research guidelines and ethics
This study was prepared in accordance with the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guideline for cross-sectional studies [23]. The study did not require institutional review board approval or patient informed consent because it was based on publicly available information and involved no patient records.