How much is the lack of retention evidence costing trial teams in Ireland and the UK?

Abstract

Background

Evidence to support the use of many retention strategies in clinical trials is lacking. Despite this, trial teams still need to have some form of retention strategy in their trials to try and avoid high attrition rates. This study aimed to estimate how much this lack of retention evidence might be costing trials in Ireland and the UK.

Methods

We selected the top ten most routinely used retention strategies by Clinical Trial Units in the UK and made assumptions as to how each of these strategies was most likely to be implemented and the costs involved in doing this. We applied our costing model to a hypothetical trial scenario in both Ireland and the UK as well as to three published trial protocols. We developed the costing model and calculated the costs in Microsoft Excel.

Results

Retention strategies were often poorly specified, meaning we had to make assumptions about implementation and in some cases about the strategy itself. Based on our assumptions, some retention strategies can be extremely expensive; some of the costliest strategies included “data collection scheduled with routine care” (€900–€32,503.25), “a timeline of participant visits for sites”—with integrated participant reminder (€304.74–€14,803.70), and “routine site visits by CTU staff” and “investigator meetings face to face”, both costing (€777.67–€14,753.48). Others such as “telephone reminders for questionnaire response” (€34.58–€568.62), “a timeline of participant visits for sites”—site reminder alone (€79.18–€112.23), and “targeted recruitment of sites/GPs” (€30–€1620) were less costly compared to the other strategies.

Discussion

The resources invested in the use of some retention strategies may outweigh known or imagined benefits on retention. Where benefits are currently unknown, evaluation should be a priority.

Conclusion

More evaluation of the effectiveness and cost of trial retention strategies is needed to avoid widespread use of strategies that are both expensive and ineffective.

Background

Randomised trials can be no better than the data they collect. If retention has been poor, in other words, data are missing for many participants, then the usefulness of the trial starts to come into question. Patchy datasets can lead to a trial becoming underpowered for its primary outcome and very underpowered for its secondaries, which were anyway probably not part of power calculations. Participants who do not provide data may differ to those who remain [1,2,3], making interpretation harder. It might be possible to overturn trial conclusions by simply imagining that the results from missing participants had gone against those conclusions [4]. In short, potential users of the trial results now have doubts and doubt undermines trials.

Poor retention is a major cause of research waste as it can delay the implementation (or removal) of healthcare interventions [5], as missing primary outcome data can reduce the power of the study to detect significant findings [6]. Poor retention may also increase trials costs [5, 7]. A recent prioritisation exercise (PRioRiTy 2) identified 20 priority unanswered research questions for research in trial retention, including questions around what motivates participants to stay involved, how to provide information and how what is done at recruitment might influence retention [8].

Although there are plenty of unanswered research questions in trial retention, trial teams nevertheless have to use some form of retention strategy in their trials. In the UK, Clinical Trials Units (CTUs) registered with the UK Clinical Research Network use many approaches and Kearney et al. asked them what these strategies were. Thirty-three (70%) CTUs responded outlining commonly used strategies [6]. While evidence is available for some strategies, none have compelling evidence of benefit [9].

The current study aimed to estimate how much this lack of retention evidence might be costing trials in the UK and Ireland. When referring to retention strategy we mean an activity that is done with the explicit aim of improving trial data completeness and/or quality. Trialists have many options for how to do this but we focus on estimating the costs of the ten retention strategies used most often by UK CTUs as identified in the study by Kearney and colleagues [6].

Methods

The top ten most routinely used retention strategies used by UK CTUs are shown in Table 1, along with the percentage of CTUs that use this strategy routinely [6] and the evidence of benefit provided by the most recent Cochrane Systematic Review [9]. The top ten strategies were identified in a survey of UK CTUs [6], where respondents outlined the strategies they use to improve retention. However, the responses provided to Kearney and colleagues did not contain detail as to how trial teams implement these strategies. We extracted as much information as possible from the paper and Dr Kearney provided additional information where available but we were nevertheless left to make many assumptions as to how retention strategies were implemented, particularly for the strategy “a timeline of participant visits”, where the very nature of the strategy itself remained unclear.

Table 1 Top 10 most used missing data strategies by CTUs in the UK [6]

To estimate the cost of the retention strategies, we made assumptions as to how each of these strategies would be implemented and the costs involved. We called this our costing model for each strategy. We contacted experienced clinical trial professionals such as trial managers, clinical research nurses, and professionals working in clinical research facilities in the UK and Ireland for information to inform our costing models. We identified these individuals through workplace inquiries and personal knowledge of suitable personnel to answer specific costing queries.

We applied our costing models to each of the ten strategies in two ways. First, we created a hypothetical trial scenario (Table 2). We made assumptions about how the retention strategies would most likely be conducted and then calculated and applied the costs of running each of the ten retention strategies in both Ireland and the UK. The costing models for some strategies differed slightly between Ireland and the UK due to differences in the responsibilities held by staff members involved in running clinical trials and differences in how some retention strategies are conducted in each country (See Additional file 1 for full details).

Table 2 Hypothetical trial characteristics

Secondly, we chose three trial protocols published in the journal Trials between 2016 and 2020. We chose these three for convenience reasons: they fit into our costing model well, represented trials of various sizes, and the trials had different characteristics. We used these to estimate the cost associated with each retention strategy in a “real-life” randomised controlled trial. We applied our costing model for the previously identified top ten retention strategies to each of the three randomised controlled trials regardless of whether the trial protocol stated that these retention methods were used or not.

An example costing model for both a hypothetical trial and published trial protocol [14], with costs and assumptions, is shown in Table 3. A full description of the costing models and the assumptions we made to create them, together with details of the three trials selected from the Trials journal are outlined in Additional file 1.

Table 3 Assumptions and costings for “Newsletters”—hypothetical trial Ireland and the MAMI trial [14]

In addition to the ten most routinely used retention strategies, we further elaborated and included additional actions that are likely to be carried out by trial units when trying to retain participants that were not specifically outlined by Kearney et al. [6]. These assumptions reflect actions that are likely to be conducted by trial teams during the implementation of the listed strategies. For example, under the strategy “inclusion of pre-paid envelopes (questionnaires)” along with sending out pre-paid envelopes to enhance questionnaire response we also assumed a reminder schedule would be sent out to 30% of participants who initially did not return the questionnaire. A full list of the additional assumptions can be found in Additional file 1.

Where there was evidence of effectiveness of the retention strategy we calculated the cost per participant retained. For example for the ease of calculation, we calculated the cost per participant retained by using pre-paid return envelopes based on a 4% benefit [9]. In the Irish hypothetical trial scenario, 4% of 500 participants is 20. The total cost of “inclusion of pre-paid envelopes (questionnaires)” was €1690. Therefore the cost per participant retained was €84.50.

All our calculations were done within Microsoft Excel. Additional file 2 contains our Excel spreadsheet with our cost calculations. For ease of comparison, all our costs are presented in Table 4 in both EUR and GBP based on exchange rates of 1 GBP = 1.16279 EUR, 1 EUR = 0.860001 GBP, 1 USD = 0.843802 EUR and 1 USD = 0.725647 GBP, taken from Xe Currency Converter - LIVE Foreign Exchange Rates on the 7th of September 2021.

Table 4 Total cost and cost per participant for each retention strategy for all trials

Results

We discussed the top ten most routinely used trial retention strategies in the UK and made assumptions as to how each of these strategies would be implemented and the costs involved. We applied this costing model to our hypothetical trial scenario, (1-year participant follow-up, 10 sites, 500 participants, 3 trial visits, 1 questionnaire sent out) in Ireland and the UK, and found that the strategies ranged from very cheap to very expensive to implement. The cheapest strategy to implement in both countries would be “a timeline of participant visits for sites”—site reminder alone costing €79.18 (£68.09) and €111.95 (£96.28), respectively, and the most expensive strategy would be “data collection scheduled with routine care” costing €20,250 (£17,415.02) in Ireland and €15,697.67 (£13,500) in the UK.

For the “real life” trials, “a timeline of participant visits for sites”—site reminder alone would be the cheapest strategy in the CINNAMON trial (€112.23 (£96.52)). For the MAMI and MOON trials, “targeted recruitment of sites/GPs” would be the cheapest strategy in these two single-centre trials costing €30 (£25.80) and €35.35 (£30.40), respectively. The most expensive strategy in the MAMI trial would be either “investigator meetings face to face” or “routine site visits by CTU staff” both costing €1600 (£1376). For the MOON and CINNAMON trials, the costliest strategy for the trial teams to use would be “data collection scheduled with routine care” costing €4834.88 (£4158) and €32,503.25 (£27,951.92), respectively.

The cost of retention strategies differed across the trials due to differences in the trial size (number of sites and participants), staff roles and responsibilities in different countries and the number of follow-up visits. For the smaller single-site trials, such as the MAMI trial (60 participants, 1 site, 1 trial visit) it naturally costs less to implement retention strategies compared to the larger trials such as the CINNAMON trial (428 participants, 5 sites and 9 trial visits).

Newsletters were identified as the most routinely used retention strategy by CTUs in the UK [6]. The cost of sending newsletters would be one of the cheaper retention strategy options ranging from €755.17 (£649.45) to €2040 (£1754.40) for manually posting the newsletters, and €546.59 (£470.07) to €1094.17 (£940.96) to electronically send newsletters. However, the most up-to-date evidence suggests there is no retention benefit when using this strategy “RD = − 0% (95% CI − 4% to 3%); GRADE: very low” [9]. This costing model has shown that “data collection scheduled with routine care” is one of the most expensive strategies across all trials (€900 (£774)–€32,503.25 (£27,951.92)).

The retention strategy with the best available evidence is “inclusion of pre-paid envelopes" for questionnaire return. The cost of this retention strategy ranges from €170.98 (£147.04) to €1,690 (£1453.40), one of the cheapest retention strategies trial teams can use. The Cochrane review found that return postage which included “preaddressed second class stamped envelope”, “high priority stamp to the mailing” and “personalised postal follow-up” may lead to a 4% retention benefit “RD= 4% (95% CI − 0% to 9%)”. However, the evidence was based on three low-quality studies (n=1543), and the single study of pre-paid return envelopes itself did not find a beneficial effect on retention [9].

Discussion

Our findings show that the evidence available to support the ten most-used trial retention strategies by CTUs in the UK is weak or lacking entirely but that the cost of using them can be very large.

The most routinely used strategy outlined by Kearney et al. [6] is “newsletters”, this strategy was found to be one of the cheaper retention methods particularly emailing newsletters (€546.59 (£470.07) to €1094.17 (£940.96)). Another cheap strategy across all the trials would be “telephone reminders for questionnaire response” costing between €34.58 (£29.74) and €568.62 (£489.01). We are able to say that these retention strategies would be cheap, but more evidence is needed to show that they are also effective at retaining trial participants.

The second most routinely used retention strategy outlined by Kearney et al., [6] is “a timeline of participant visits for sites”. The site reminder schedule alone would be cheap, costing between €79.18 (£68.09) and €112.23 (£96.52). Integrating a participant reminder schedule would significantly increase the cost (€304.74 (£262.08)–€14,803.70 (£12,730.79)). Similarly “data collection scheduled with routine care” (€900 (£774.00)–€32,503.25 (£27,951.92)), “routine site visits by CTU staff” (€777.67 (£668.80)–€14,753.48 (£12,688)) and “investigator meetings face-to-face” (€777.67 (£668.80)–€14,753.48 (£12,688)), would also be expensive to implement yet none of these have compelling evidence demonstrating that they are effective at retaining trial participants [9]. They may be very effective. The point is that we cannot say with any certainty whether they work or not, and therefore substantial amounts of money and other resources are potentially being invested into strategies that lead to no improvement in retention. We recognise that participant retention may not be the only reason for some of these strategies to be implemented in a trial. For example “routine visits by CTU staff” may also be used for monitoring purposes and the costs may be justified for purposes other than retaining participants. However, it is worth remembering that all these strategies were specifically identified by CTUs as strategies they use to support retention and if a strategy is to be used to improve retention, the jury is out as to whether the majority of the top ten strategies are effective.

Even some of the less costly strategies have limited evidence showing effectiveness and much of the existing evidence is from single studies often with low GRADE ratings [9]. “Telephone reminders for questionnaire response” (€34.58 (£29.74)–€568.62 (£489.01)), “newsletters (emailed)” (€546.59 (£470.07)–€1,094.17 (£940.96)), “targeted recruitment of sites/GPs” (€30 (£25.80)–€1,620 (£1393.20)) and “a timeline of participant visits for sites”—site reminder alone (€79.18 (£68.09)–€112.23 (£96.52)) would be less costly compared to the other strategies but not all have evidence in support of them. A cheap but ineffective strategy is still not something worth using, especially if it takes resources away from other potentially more useful strategies.

Lack of trial process evidence is a sadly enduring problem and despite the paucity of evidence, trial teams need to try and ensure retention is high. Based on our work, a reasonable approach might be to use strategies that are inexpensive when compared to the overall cost of conducting the trial. Building in an evaluation, perhaps a Study Within a Trial (SWAT) [17] would improve decision-certainly for the next trial. For more expensive strategies, we’d suggest only using them in the context of an evaluation. An estimate of the clinical trial cost per participant in the UK’s National Institute for Health Research Health Technology Assessment (NIHR HTA) Programme is £2,987 GBP [18]. Some of that cost is due to retention problems, often because trial teams increasing the recruitment target to compensate for later retention loss. Having a collection of evidence-informed retention strategies just might bring down the average cost per participant, and perhaps reduce some recruitment problems too.

Strengths and limitations

We acknowledge that a limitation of this study is that we have had to make assumptions to calculate our cost estimates and these may not be truly representative, or the assumptions made may not be accurate depending on how trials are run, especially those outside of Ireland and the UK. Making assumptions was unavoidable due to the lack of information regarding the strategies available from the study by Kearney et al. [6]. We also acknowledge that others may make different assumptions, that there is not a standardised approach to implementing these strategies and that the assumptions affect the costs. However, to help to address this limitation, we have made the costing spreadsheet available as an additional file, which means readers can modify it to suit their own trial and input their own information on how the strategy will be implemented.

One of the strengths of this study is that regardless of the costs, it highlights the lack of evidence for routinely-used trial retention strategies. Even if our estimates are very wrong, no strategy costs nothing and if there is weak or no evidence in support of it, we should pause and consider what we want to do. If trialists go ahead and use the strategy, we think at least some of them should use SWATs or other research designs to investigate the impact of the strategy on retention. The combination of routine use of a strategy to support retention and a lack of evidence that the strategy actually improves retention is a recipe for research waste.

Recommendations for future research

This paper highlights the need for further research into the effects of trial retention strategies. The cost of some of the strategies that are currently routinely used is significant, and so is the lack of evidence to support their use. We recommend the wider use of SWATs to evaluate the effects of retention strategies used in clinical trials to avoid persistent and widespread research waste. Replication of evaluations will add to the existing evidence to support/not support the use of these strategies.

We also think it would be useful for trial teams to include clear descriptions of their retention strategies and the associated costs in trial publications. The challenges of how to implement a retention strategy based on current descriptions became very clear in this work. Communicating the costs of retention strategies can be helpful to other trial teams to estimate the budgets required for implementing similar strategies. A better idea of costs will allow for better ‘cost-per-participant-retained’ calculations, which in turn will give trial teams another way to compare retention strategies when making choices about which to use in their trial.

Conclusions

Without evidence regarding the effectiveness of trial retention strategies, trial teams will continue to put substantial amounts of money into strategies that potentially have no beneficial impact on participant retention. More evaluation of the effectiveness and cost of trial retention strategies is needed to avoid the widespread use of strategies that are both expensive and ineffective.