Data were grouped into four broad themes:
-
1.
Understanding and acceptance of the need for SWATs
-
2.
Wider context of trial development and funding
-
3.
Specific stages of trial development, implementation and delivery
-
4.
Organisational aspects of CTUs (using the McKinsey 7S model)
Understanding and acceptance of the need for SWATs
Recognition of the need for SWATs was well established amongst CTU leads, senior trialists and trial managers interviewed here. As one senior trialist commented:
… for at least 50 years people have struggled with the idea of making trials as efficient as they can be…and every time you looked at the literature, people were still struggling … and they still are…clearly things are not improving in the way we think they should.
However, this recognition was not seen to extend to the wider trial community, where Chief Investigators (CIs) or other trial management group members were frequently cited as barriers to employing SWAT methodology. The types of concerns encountered tended to focus on resisting further complexity, increasing workload, a belief in a particular methodological approach (lack of equipoise) or fear of reducing recruitment in one arm:
I did raise it with one CI … around payment for follow up data … they said ’we don’t want a randomised payment or no payment because we believe it works’ … they wanted to give it to everybody because they don’t want to risk reducing the recruitment in one arm…
I did put in a section [on the application] saying we are going to do an embedded trial, but all the other applicants thought this was a distraction and wanted me to take it out, which I did
Good relationships (normally between CTU and CI and occasionally between trial manager and CI) and shared values around methodology research were cited numerous times as important facilitators:
If you’ve got a long standing relationship, that’s fine, but if they are a new CI you might not want to raise it necessarily.
Opinion was divided on the role of academic clinicians in promoting or supporting embedded methodology research and SWATs:
I think clinician CIs, because they are less experienced can go along with whatever you say, whereas academics, who maybe feel they could get the grant in their own right, are more likely to want to do what they want…
Another potential facilitator of SWATs was the extent to which CTUs could make inclusion of a SWAT conditional for accepting a trial or application, and the continued promotion of this type of work through dissemination, networking and support.
Some interviewees questioned the need for or suitability of SWATs. Exceptions included examples of improved trial procedure that they did not feel needed to be tested:
Within the CTU we’re always looking at how we can improve our systems and processes of running trials, we just don’t do it by, sort of, formally randomising to do method A or method B … We know it’s working because we know it happens quicker…
Some trial settings were seen as inappropriate for a SWAT:
A lot of the time we’re recruiting patients the night before they go in for major surgery … so it’s difficult to imagine what we could embed methodologically given the sort of patient pathway we work with most of the time.
Finally, there were mixed comments from funder representatives, with one contending that SWATs are too narrow in focus: the approach misses key determinants of successful trials. Issues are not how to recruit, but management style and leadership:
My impression of the literature … is that it’s adopted a rather too narrow study design paradigm … in terms of fixed interventions and …randomised controlled trials. I think there’s often modest differences between individual interventions and the trials that are successful are not because they’ve adopted a particular approach … it’s because they’ve been just really well organised
Other funder views aligned more with those of CTU leads, acknowledging the need to improve the evidence base on recruitment and retention strategies.
Wider trial development and funding context
Substantial differences existed in perceptions about availability of funding for methodology work. For those engaged in delivering research, difficulty in accessing funds was the most significant barrier. None of the current funding streams were seen as open to funding stand-alone SWAT applications:
So we think it’s a good thing, and also one of our other objectives is to do more methodological work and there aren’t many funding streams for methodological work. In fact I can’t really find any, so…
Views from representatives of funding agencies were mixed, but on the whole reflected a more positive view, citing examples where the opportunity for ‘methodological bolt-ons’ existed:
The [methodological bolt-ons] were put on there in the sense that if people are funded to do a trial using some new method … is there some additional research that might be done to either evaluate or test the validity of the approach that is being done? So, we haven’t any more thoughts on that …
Some funders reported shifts in policy towards more applied work:
It was decided to look at whether there should be changes to the structure and membership of the [Panel] to try and make sure we got more of an applied bent … it was an attempt to increase [applied methodology research] but of course we’re entirely reliant on the applications that come to us
Outside these programmes, there was some indication from funders that applications for SWATS as part of a full trial funding application could be considered ‘in scope’:
But, you know, something of modest scale that represents an internal evaluation of recruitment methods, in order to inform the conduct of the later phase of the trial, seems entirely reasonable.
However, from the perspective of trialists, there was not always clarity about access to funds:
I guess knowing that the funder actually wants to see it and is prepared to pay for it and upfront explicitly say, because otherwise you’ll just think, oh well they’ll say, why have you put this extra thing in, it just makes it more complicated and more expensive?
From a funder perspective that meant clearer communication about funding for SWATs:
Interviewer: There seems to be a perception that it’s [SWATs] not something [funder] would fund …
Respondent: Yes … I think it’s an incorrect perception, but a widely held perception
Several other contextual factors were identified by researchers as barriers to the adoption of SWATs. Timing pressures in the trial funding cycle were commonly mentioned as the main factor working against the added complexity of developing and introducing SWATS:
At the moment the way they’ve got the expression of interest to the full in seven weeks, either over Christmas, over summer or over Easter is just not giving you the opportunity to start being creative about method study.
However, several interviewees noted that efforts to co-ordinate ideas and develop ‘off-the-shelf’ interventions could act as a facilitator in this setting.
A second barrier of considerable concern relates to costs and competition. Many perceive current funding strategy as placing great emphasis on cost saving, value for money and efficiency. Ironically, in this context CIs and CTUs tend to strip design back and are keen to ensure no extraneous costs in funding proposals, acting as a barrier to SWATs. CTU leads in particular expressed concerns about whether additional costs for SWATs could have an impact on their competitiveness:
I think like with any trials unit we are always being pushed back on our costs, and our costs, I would argue, are actually very competitive. There’s always that pressure to do all that you want to do for less because of the fear that it will get knocked back by the funding board if it’s seen to be too expensive.
Specific stages of trial development, implementation and delivery
There were many comments about the detailed process of implementing a SWAT. A number of key stages of trial development, implementation and delivery were noted, but a considerable proportion related to activities that could be considered as earlier stages. Moving from support for the principle to idea generation for a specific SWAT was one area of difficulty. Concerns related partly to timing (as noted above) but also the broader question of what constitutes a worthwhile intervention to test:
Yes. And I think the other element to all of that is the timing of it all. We’ve had a lot of discussions about how much of a time window we have between people approaching us and then a funding deadline, so it’s then on top of all of what you have to do anyway, building in this. It’s whether you’ve got something ready to go and you think, oh, this trial would be perfect, or whether it’s developing something from scratch.
There was particular support for some form of agenda-setting for research priorities and co-ordinating efforts around specific questions to have maximum impact, possibly forming collaborations with small groups of CTUs:
Yes, one of the key things I always talk about with this Trial Forge idea which is sort of distilling down now to essentially one line, three words, it’s ‘coordination and collaboration’. The first bit is that coordination, it’s what should we do? Again that removes a barrier if you can say, well, here are some things to do, and in fact if we take the IQuaD example, not only: here are some things, but here is the thing to do. It removes a barrier, you just make a decision, do I want to take part or not? You haven’t had to dream the thing up.
Host trial characteristics, stage of trial development and the level of service provided by the CTU were seen as critical for determining suitability for a SWAT. The complexities associated with running a SWAT were felt to be reduced where CTUs provide a full trial service (as opposed to data management only) and where they are involved in the design and development of the trial from an early stage. Other potential facilitators included checklists or guidance for identifying suitable host trials, detailing what needs to be taken into account when negotiating access for the SWAT:
In X CTU you buy the whole service. They are very, very reluctant to take on a trial if they are not running it completely, so there is the potential to bolt stuff on. …in Y CTU there is a much wider range of support …[they] would run things completely but also there were investigators who would just buy data management … so if they’re doing trial management themselves and you suggest doing an embedded methodological study that will affect the working life of their trial manager, I think you’re going to struggle …
External review as part of funding or ethics applications was seen as problematic, with a SWAT seen as having the potential to cause confusion, detract from the main trial or, at worst, generate objections that undermine the host trial or delay its implementation. Providing directions from funders to peer reviewers on how SWATs should be assessed was seen as useful:
Funders and the referee, it’s not just funders, it’s referees get…and they might, sort of, focus on making hugely critical comments on an embedded trial that they profoundly disagree with or something and that might damage the whole…there’s a fear it might damage the whole proposal.
Greater uncertainty was expressed about how SWATs would be reviewed within ethics, with a general sense that a SWAT could cause complications. This was felt particularly where SWAT methodology might diverge from standard practice. Examples included cases where there is no pre-notification of participants in a SWAT or where a power calculation for a SWAT is less appropriate.
Potential facilitators included engaging with the ethics committees to discuss whether guidance could be given on how this type of work should be treated:
Some [research ethics committees ] still specialise in drugs, some will specialise in radiotherapy, some around consent for children…why not have a few ethics committees think about embedded methodology, or methodology in general?
The major barriers perceived by researchers in relation to trial process and conduct were focused on the early phases of the research process (concept, development, funding and review). However, a number of concerns relating to trial implementation were also identified. Trial setup is a time-critical period in the life of the trial. Reservations were expressed about the practicality of running a SWAT if it added to the burden of setup. Various solutions were suggested, including delaying the start of the SWAT or using a design (such as cluster randomisation) that simplified the recruitment work on site, although there were concerns that these types of actions could in themselves cause complications:
…it’s to do with the centre of gravity, they don’t want the centre of gravity of the project to shift from what they are actually really interested in. So if … you have a substantial chunk of your money and effort is refocused on a methodological piece of work, you are starting to move the centre of gravity away from what they [funders/main trial] are really interested in and that won’t fly.
There was also concern about monitoring and implementation fidelity of the embedded trial, particularly on large multi-site trials where the CTU or trial team might be remote from day-to-day recruitment practice. It was felt that this would have to be considered when selecting host trials.
A final concern raised was over ownership of the embedded trial research and data generated and the need to establish this early in the process.
Organisational aspects of CTUs
CTU lead interviewees were asked to consider barriers and facilitators to embedded methodology research within their own organisations and networks. Responses were analysed using the McKinsey 7S model and related to five aspects: strategy, skills, structure, style and systems.
Strategy: plan or course of action leading to the allocation of an organisation’s finite resources to reach identified goals
The potential of SWATs to contribute to CTU strategy was identified in a number of key areas such as increasing unit capacity, staff skills and methodology work. However, the extent to which SWATs formed part of CTU strategy varied considerably. Examples ranged from explicit and enacted strategy in one CTU to different stages of consideration or implementation:
Well it’s across the board, we try and do it on every trial…
Okay, well we’ve been thinking about it for a long time because we’re obviously aware that other people do it, and some people get a lot of papers out of it, and also it’s a potential for trial managers or trial staff to get their own papers to lead on a paper and be a first author. So we think it’s a good thing, and also one of our other objectives is to do more methodological work
This included in some cases testing the idea with external collaborators or as part of a broad strategy to expand, increase skills and capacity, and forge links with other university departments:
So as I said with this…this was the first kind of formal raising of it [SWATs] at this meeting [with a group of collaborators] and basically it got very positive feedback and I think it will get positive feedback from the others [collaborator group meetings] as well, and I think part of the reason for that is that we have… but if you look on our website you will see that we are going to build up a very strong track record of methodology
There was also widespread recognition of how the CTU network could be used to facilitate greater strategic commitment to SWATs:
I mean, perhaps it should be…we should try and persuade one or two others just to start doing this [SWATs] and then perhaps it would then, you know, percolate around more.
However, given the operating pressures within CTUs, it was unclear whether such additional work would be undertaken without further strategic drivers, primarily around access to funding and perceived importance. There was awareness of new initiatives such as Trial Forge [10] and the SWAT repository (go.qub.ac.uk/SWAT-SWAR); however, few reported engagement, and the links between these aspects of the trials infrastructure are only just being made.
Skills: distinctive capabilities of key personnel and the organisation as a whole
As might be expected, skill profiles across CTUs are broadly similar, with core staff including statisticians, methodologists and data managers. Running SWATs was felt to be well within the existing CTU skill set, both in terms of individual and organisational skills:
We have all the team we need really, trial managers, data managers, the usual randomisation services, databases, statisticians. Then we liaise with health economists
Two aspects of CTU staffing were seen as intrinsic to successful SWAT implementation and delivery, both focused on the role of trial managers. Developing trial manager skills was largely described in terms of on-the-job training, with SWATs recognised as offering important opportunities for staff to experience all aspects of trial management and improve career options both through skill development and lead authorship on SWAT publications:
My main driving force is to help my trial managers to get publication for their CVs, that’s one of the main reasons for doing embedded trials, as well as it’s nice to generate the evidence, but it’s also nice to get them through promotions committees which are easier to do if they’ve got publications on their CVs.
The second set of comments in relation to staff concerned the location of trial managers. There are much better prospects for implementing SWATs where trial managers are based in a CTU or closely linked department, and several CTUs insist on this as a condition of taking on a study:
…it will always be really important for us to have an academic co-applicant on external studies so that we are in a position as a CTU to be able to influence the overall shape and design of the study…
CTUs operating a more remote model (e.g. with no co-investigator on the trial or where they provide a single service, such as data management) saw this as a considerable barrier.
Structure: who is in charge, how decisions are made, authority relationships
CTU leads all described broadly similar decision-making structures within their organisations, with a board or senior management team assessing each clinical trial proposal. However, for SWATs, decision-making power does not necessarily sit with the CTU. This can affect their ability to make direct decisions about implementing SWATs in a particular trial where the CTU is dependent on a receptive CI:
One of the problems when you’re just doing it purely service, is this issue around how much control you have, and CIs are very different. Some are very happy to have all the input of a CTU and in some ways actually allow the CTU to effectively lead the study. Others it’s the opposite
It can also have an impact on broader decisions about CTU strategy:
I know some other units … have their own cost centre and are, like, independent organisations in the universities, but we’re not. So we’re in a bigger department and that has its up and downsides, I mean, the upside is that we have access to other specialties, other people more easily and we can spread the risk sometimes of if we have a short fall in work, sometimes people can be moved to projects within the department. I suppose the downside is we have less control of our own destiny.
Style: characterisation of how key managers behave in order to achieve the organisation’s goals
There was strong recognition amongst interviewees of the importance of senior management support if SWATs are to be more widely adopted. This ranged from creating a culture of expectation that all trials will seek to adopt SWATs, to seeking and promoting specific opportunities to work collaboratively:
If you look at XXXX [CTU]… the director of that unit will support trial managers who are interested in running a piece of [methodological] research to do so and to get a publication out of it… my take on it is that a) it’s supporting the methodological literature, but b) it’s also supporting the individual trial manager and it’s part of the culture. If you’re in a different unit where [the leadership] see the trial manager’s role as someone who essentially implements the trial, it might be much harder.
Systems: procedures and routine processes, including how information moves around the organisation
The nature of trials work means that CTUs have high levels of expertise in developing and implementing systems and offer great scope for routinizing SWATs. However, the lack of SWATs means that, in practice, these systems rarely exist. Additionally, because most SWATs have been conducted in an ad hoc way to date, there is little shared learning on what processes seem to work best.
Looking at existing systems and identifying where SWAT elements could be included was felt to be a straightforward process for CTUs, but it was also felt to be a minor consideration in relation to the other barriers faced, such as funding, consensus around what to test and resources to develop ideas and materials.