Our findings show that the evidence available to support the ten most-used trial retention strategies by CTUs in the UK is weak or lacking entirely but that the cost of using them can be very large.
The most routinely used strategy outlined by Kearney et al. [6] is “newsletters”, this strategy was found to be one of the cheaper retention methods particularly emailing newsletters (€546.59 (£470.07) to €1094.17 (£940.96)). Another cheap strategy across all the trials would be “telephone reminders for questionnaire response” costing between €34.58 (£29.74) and €568.62 (£489.01). We are able to say that these retention strategies would be cheap, but more evidence is needed to show that they are also effective at retaining trial participants.
The second most routinely used retention strategy outlined by Kearney et al., [6] is “a timeline of participant visits for sites”. The site reminder schedule alone would be cheap, costing between €79.18 (£68.09) and €112.23 (£96.52). Integrating a participant reminder schedule would significantly increase the cost (€304.74 (£262.08)–€14,803.70 (£12,730.79)). Similarly “data collection scheduled with routine care” (€900 (£774.00)–€32,503.25 (£27,951.92)), “routine site visits by CTU staff” (€777.67 (£668.80)–€14,753.48 (£12,688)) and “investigator meetings face-to-face” (€777.67 (£668.80)–€14,753.48 (£12,688)), would also be expensive to implement yet none of these have compelling evidence demonstrating that they are effective at retaining trial participants [9]. They may be very effective. The point is that we cannot say with any certainty whether they work or not, and therefore substantial amounts of money and other resources are potentially being invested into strategies that lead to no improvement in retention. We recognise that participant retention may not be the only reason for some of these strategies to be implemented in a trial. For example “routine visits by CTU staff” may also be used for monitoring purposes and the costs may be justified for purposes other than retaining participants. However, it is worth remembering that all these strategies were specifically identified by CTUs as strategies they use to support retention and if a strategy is to be used to improve retention, the jury is out as to whether the majority of the top ten strategies are effective.
Even some of the less costly strategies have limited evidence showing effectiveness and much of the existing evidence is from single studies often with low GRADE ratings [9]. “Telephone reminders for questionnaire response” (€34.58 (£29.74)–€568.62 (£489.01)), “newsletters (emailed)” (€546.59 (£470.07)–€1,094.17 (£940.96)), “targeted recruitment of sites/GPs” (€30 (£25.80)–€1,620 (£1393.20)) and “a timeline of participant visits for sites”—site reminder alone (€79.18 (£68.09)–€112.23 (£96.52)) would be less costly compared to the other strategies but not all have evidence in support of them. A cheap but ineffective strategy is still not something worth using, especially if it takes resources away from other potentially more useful strategies.
Lack of trial process evidence is a sadly enduring problem and despite the paucity of evidence, trial teams need to try and ensure retention is high. Based on our work, a reasonable approach might be to use strategies that are inexpensive when compared to the overall cost of conducting the trial. Building in an evaluation, perhaps a Study Within a Trial (SWAT) [17] would improve decision-certainly for the next trial. For more expensive strategies, we’d suggest only using them in the context of an evaluation. An estimate of the clinical trial cost per participant in the UK’s National Institute for Health Research Health Technology Assessment (NIHR HTA) Programme is £2,987 GBP [18]. Some of that cost is due to retention problems, often because trial teams increasing the recruitment target to compensate for later retention loss. Having a collection of evidence-informed retention strategies just might bring down the average cost per participant, and perhaps reduce some recruitment problems too.
Strengths and limitations
We acknowledge that a limitation of this study is that we have had to make assumptions to calculate our cost estimates and these may not be truly representative, or the assumptions made may not be accurate depending on how trials are run, especially those outside of Ireland and the UK. Making assumptions was unavoidable due to the lack of information regarding the strategies available from the study by Kearney et al. [6]. We also acknowledge that others may make different assumptions, that there is not a standardised approach to implementing these strategies and that the assumptions affect the costs. However, to help to address this limitation, we have made the costing spreadsheet available as an additional file, which means readers can modify it to suit their own trial and input their own information on how the strategy will be implemented.
One of the strengths of this study is that regardless of the costs, it highlights the lack of evidence for routinely-used trial retention strategies. Even if our estimates are very wrong, no strategy costs nothing and if there is weak or no evidence in support of it, we should pause and consider what we want to do. If trialists go ahead and use the strategy, we think at least some of them should use SWATs or other research designs to investigate the impact of the strategy on retention. The combination of routine use of a strategy to support retention and a lack of evidence that the strategy actually improves retention is a recipe for research waste.
Recommendations for future research
This paper highlights the need for further research into the effects of trial retention strategies. The cost of some of the strategies that are currently routinely used is significant, and so is the lack of evidence to support their use. We recommend the wider use of SWATs to evaluate the effects of retention strategies used in clinical trials to avoid persistent and widespread research waste. Replication of evaluations will add to the existing evidence to support/not support the use of these strategies.
We also think it would be useful for trial teams to include clear descriptions of their retention strategies and the associated costs in trial publications. The challenges of how to implement a retention strategy based on current descriptions became very clear in this work. Communicating the costs of retention strategies can be helpful to other trial teams to estimate the budgets required for implementing similar strategies. A better idea of costs will allow for better ‘cost-per-participant-retained’ calculations, which in turn will give trial teams another way to compare retention strategies when making choices about which to use in their trial.