Table 1 Characteristics of included studies
Study | Study description | Number of sites (sample size) | Metrics reported by each study | |
|---|---|---|---|---|
Included as site performance metric | Excluded as not site performance metrica | |||
Studies proposing performance metrics | ||||
Bose 2012 [14] | Paper discussing trial management through central monitoring | Not applicable | • Site location potential index based on an assessment of the number of patients at an individual site with the disease of interest • Trial compliance index based on a number of suggested factors including the number of late visits, failure to achieve recruitment target, number of dosing errors, etc. | • Drug adversity measurement (B) • Drug potential index (B) |
Djali 2010 [15] | Paper discussing a data-driven quality management system | Not applicable | • Enrolment number per siteb • Recruitment period per site • Number of AEs per site • Number of protocol deviations and violations per site | • Number of discontinuations per site (A) • Deaths per site (D) |
Elsa 2011 [16] | Methodology of developing ‘key risk indicators’ for monitoring of a large international clinical trial | Not applicable | • Rate of SAE reporting per site: centres assigned a dichotomous score depending on whether they showed extreme deviation from comparable sites (arbitrarily defined as half the observed median rate across sites) • Short visit duration: centres assigned a dichotomous score depending on whether they showed extreme deviation from comparable sites (arbitrarily defined as half the observed median rate across sites) | • Measures of compliance with study treatment (A) • Blood results/other continuous variables examined for unusual patterns (A) |
Glass 2007 [17] | Study analysing data retrospectively from 262 clinical trials to determine variables associated with successful trial delivery | Not applicable | • Actual number participants randomised per site • Number successfully completing the study’s protocol per site • Time between when an individual site randomises its first participant and the time the first site in that study enrols its first patient | |
Hanna, 2013 [11] | Development of a list of quality indicators for trial performance based on the consensus of experts | Not applicable | • SAE reporting measured by the number of SAEs reported/number of SAEs identified in trial database or trial follow-up documents • Transfer of CRF to CTU measured by the number of completed CRF received by CTU within 30 days/number of completed CRF received by the CTU in 3 months | |
Jou, 2013 [18] | Aim of the main study: treatment-naïve, hepatitis C patients randomised to two peginterferon regimens. Primary outcome virologic response. A retrospective analysis was performed of individual site performance using trial data | 118 (3070) | • Rates of screen failure defined as the percentage of participants screened who failed screening • Completion and discontinuation of treatment, defined as the percentage of participants who completed treatment/ percentage of participants who discontinued treatment • Completion / discontinuation of follow-up, defined as the percentage who completed follow-up/ percentage who discontinued follow-up | • Treatment adherence (B) |
Khatawkar 2014 [19] | Retrospective analysis of data queries using clinical trial data | Not applicable | • Data query (DQ) rate per page • DQ rate per page by phase of study | • DQ rate per page by country (B) • DQ rate per page by therapeutic area (B) |
Lee 2012 [20] | Paper describing the output of a Delphi survey to establish an ‘evaluation framework’ for clinical trial data | Not applicable | • Rapid enrolment, defined as time taken to reach target enrolment • Timely data entry, defined as time taken for data entry after completion of informed consent • Timely manual query management, defined as time taken for response to manual query request from data centre • Timely database lock, defined as time taken for database lock after the last visit of last participant per site • Data discrepancy management metric encompassing number of manual queries per CRF for missing data; number of manual queries per CRF for out -of-range data; number of manual queries per CRF for logical consistency • Protocol compliance metric encompassing: rate of ‘dropout’ of total participants; rate of false ‘dropout’ of total dropouts; rate of late detection of ‘dropout’ • Enrolment success defined as % eligible per study | • Weeks after go-live, i.e. after the point of protocol amendment (A) |
Rojavin, 2005 [21] | Paper describing and discussing one proposed metric | Not applicable | • Recruitment Index (RI) = (LPFV − FPFV) x S/P where LPFV = date of the last participant first visit FPFV = date of the first participant first visit S = number of participating sites P = number of participants who successfully completed the study | |
Rosendorf, 1993 [22] | Trials of treatment for HIV. No further details. An evaluation tool was proposed to monitor individual site performance within a multicentre randomised trial. | 59 (ns) | Intensity adjusted score (IAS) = IAS = IS0 + don x IS1 + doff x IS2 where: IS0 = score assigned for enrolling a new participant during the 6 month evaluation period don = number of days the participant was on the study medication during the evaluation period doff = number of days the participant was off the study medication IS1 = intensity score for the days in which the participant is receiving study medication IS2 = intensity score for the days in which the participant is off all study medication ISA is calculated for each participant and then summing scores across all participants, once during the evaluation period • Funding adjusted score = IAS divided by the amount awarded for total direct costs during the given time period • Summary quartiles = total number of new and continuing participants on study | |
Sweetman, 2011 [23] | Retrospective analysis of publications of 80 clinical trials on protocol violations reporting | Not applicable | Occurrence of protocol violations, defined as total number of protocol violations divided by the number of enrolled participants | |
Thom, 2011 [12]a | Report of a centre performance assessment tool used within a clinical trial network to assess individual site performance | Not applicable | • Protocol adherence, defined as average rate of protocol violations per enrolled participant • Data quality, defined as average rate of edit checks per participant • Data timeliness, defined as the percentage of forms entered late • Time of starting after the first centre start date • Sum of protocol adherence, data quality, data timeliness and timeliness of study start-up to give overall rank • Timeliness of study start-up | • Recruitment, defined as average percentage of participants contributed over all studies conducted (B) • Retention, defined as average percentage of participants with complete follow-up data (B) • Recruitment/retention, defined as sum of recruitment + retention to give overall rank (B) • Adherence/quality (A) • Quality of laboratory samples collected (A) |
Tudur Smith, 2014 [24] | Paper describing monitoring methods using a ‘risk proportionate approach’ used by an individual clinical trials unit | Not applicable | • Consent form completion, defined as consent forms returned within 7 days of completion by sites. • Recruitment process, defined as frequency of eligible participants who do not provide consent. • Missing primary outcome data, defined as cumulative percentage of participants with missing primary outcome data at each site • SAEs, defined as cumulative percentage of participants with at least one SAE across the trial as a whole and at each site /measure of time, e.g. 1 month • Sum of all SAEs/sum of all follow-up for the trial • Sum of all follow-up at site x overall SAE rate for the trial • Visit dates, defined as time between actual date of visit versus expected date of visit | • Case report form completion, defined as timely submission (A) |
Wilson, 2014 [25] | Theoretical paper describing methods of monitoring the conduct of trials | Not applicable | • Quality metric encompassing: average number of major audit findings per audited site; percentage per site of unreported, confirmed SAEs; number of significant protocol deviations per site • Frequency of protocol violations for eligibility criteria and randomisation per site • Rates of withdrawal by site | • Proportion of the enrolled population comprising the non-randomised parallel cohorts (measured by percentage agreement and kappa statistic) (C) • Radiologic inter-observer agreement (C) |
Studies using performance metrics | ||||
Berthon-Jones 2015 [2] | Aim of main study: treatment-naïve HIV patients randomised to 2 different types of ART. Primary outcome plasma HIV-RNA, change from baseline to week 48. Performance across 5 geographical regions was assessed using performance metrics | 36 (322) | • Time from protocol release to ethics/regulatory submission • Time from protocol release to ethics/regulatory approval • Time from protocol release to first participant randomised (FPR) • Time from protocol release to last participant randomised (LPR) • Time from site opened to first participant randomised (FPR) • Time from first participant randomised (FPR) to last participant randomised (LPR) • Actual versus estimated recruitment • Time from participant visit to electronic data capture (EDC) initiation • Time from EDC initiation to completion • Number of missing values per participant • Number of data queries per participant • Number of SAEs reported per participant • Time from SAE occurrence to initial report • Time from initial SAE report to final report • Number of samples collected versus number required by protocol | • Number of missed visits per region (B) • Quality of laboratory sample/s collected (A) • Number of plasma samples collected versus protocol-mandated samples to be collected (C) • Number of buffy-coat samples collected versus protocol-mandated samples to be collected (C) |
Katz, 2015 [26] | Aim of main studies: osteoarthritis (2 trials), lower back pain (1 trial) randomised to fulranumab infusion or placebo. Primary outcomes unspecified. Within these three clinical trials a method of monitoring individual site performance was applied | 40–88 (91–157) | • Time to data query response | • Compliance with study drug (D) |
Kim, 2011 [27]a | Aim of main study: patients with acute cerebral haemorrhage randomised to early intensive antihypertensive or standard regimen. Primary outcome death or disability at 3 months. A site performance monitoring tool was incorporated for monitoring individual site performance during the trial | 100 (1280) | • Participant recruitment per site • CRF data collection timeliness + completeness • Protocol violations per site • SAE reporting per site | • Participant study progress (A) • Site data monitoring visit findings (A) • Data clarification request processing (A) • Regulatory document collection and tracking (A) |
Rifkind, 1983 [28] | Aim of the main study: men with primary type 2 hyper-lipoproteinaemia randomised to bile acid sequestrant or placebo. Primary outcome CHD death and/or nonfatal myocardial infarction. Within this study measures of individual site recruitment performance were monitored. | 12 (3550) | • Proportion of initial contacts proceeding to first protocol visit by recruitment source • Proportion of first protocol visits proceeding to study entry by recruitment source | |
Saunders, 2015 [29]a | Aim of the main study: critical care patients randomised to probiotic or placebo. Primary outcome ventilator associated pneumonia. Within this study the team focused on screening performance in individual centres | 14 (285) | • Non-screening weeks = proportion of weeks during which participants were not screened for trial eligibility | • |
Sun, 2008 [30] | Aim of the main study: patients with major depression randomised to aprepitant or placebo. Primary outcome change in Hamilton Depression Scale. Within this study measures of individual site performance were captured | Not reported | • Administration excellence, defined as site administration performance and interaction with central study team rated 1, 2 or 3 • Data quality, defined as data completeness and correctness at initial submission rated 1, 2, or 3 • Proportion of participants with protocol violation, defined as: proportion of participants in each site who do not meet eligibility criteria; have medication compliance < 75%, or take prohibited concomitant medication or wrong study medication; or other serious violation • Level of visit non-compliance, defined as mean absolute difference of the days between visits and the protocol-specified days between visits for participants in a specific centre | • Level of medication non-compliance, defined as the mean percentage of days participants from each centre taking less than the prescribed number of doses of study-assigned medication (B) |
Wear, 2010 [31]a | Aim of the main study: patients with multiple myeloma, multiple clinical trials. No further details. Performance metrics utilised during the study | Not reported | • First patient dosed (FPD), defined as time from receipt of final protocol to the first participant treated • Enrolment commitment (EC), defined as commitment from the study site to provide a predicted number of participants who will receive at least 1 dose of study drug (e.g. number of participants randomised and completing first part of intervention • Baseline enrolment timeline (BET), defined as target time period to obtain EC | |