Maximising the relevance of randomised trials to primary care: a qualitative study.

Background Pragmatic trials have been suggested as a way to improve the relevance of clinical trial results to practice. PRECIS-2 is a trial design tool which considers how pragmatic a trial is across a number of domains. It is not known whether a pragmatic approach to all PRECIS-2 domains leads to results being more relevant to primary care. The aim of this study was to investigate the views of people with inuence on primary care practice towards the design of randomised trials, pragmatic approaches to trial design, and the PRECIS-2 domains. Methods We carried out semi-structured interviews with people who inuence practice in primary care in the UK. A thematic analysis was undertaken using the framework approach. Results We conducted individual or small group interviews involving 17 individuals in total. We found a wide range of views. These highlighted that an exclusively pragmatic approach may not always make the results of trials more applicable to primary care. For example, it may be better to have less exibility in the way interventions are delivered in randomised trials than in practice. In addition, an appropriate balance needs to be struck when thinking about levels of resourcing and the intensity of steps needed to improve adherence in a trial. Across other aspects of a trial’s design, for example the population and trial setting, a pragmatic approach was viewed as more appropriate. Conclusions To maximize the relevance of research directed at primary care, trials should be conducted with the same populations and settings that are found in primary care. Across other aspects of trials it is not always necessary to match the conditions found in practice.

who in uence practice in primary care in the UK. A thematic analysis was undertaken using the framework approach. Results We conducted individual or small group interviews involving 17 individuals in total. We found a wide range of views. These highlighted that an exclusively pragmatic approach may not always make the results of trials more applicable to primary care. For example, it may be better to have less exibility in the way interventions are delivered in randomised trials than in practice. In addition, an appropriate balance needs to be struck when thinking about levels of resourcing and the intensity of steps needed to improve adherence in a trial. Across other aspects of a trial's design, for example the population and trial setting, a pragmatic approach was viewed as more appropriate. Conclusions To maximize the relevance of research directed at primary care, trials should be conducted with the same populations and settings that are found in primary care. Across other aspects of trials it is not always necessary to match the conditions found in practice.

Background
Randomised trials are seen by many as being the best design for providing evidence about the effectiveness of different interventions. However, they do not always produce evidence that is relevant to primary care because they are frequently conducted under conditions that are different from those found in primary care (1). In addition, there may be challenges to the implementation of new interventions in primary care, such as restrictions on resourcing (2), and a need for complex interventions involving multiple interacting elements (3).
Pragmatic trials have been suggested as a solution to the problem of evidence not being relevant to clinicians, policymakers and patients. Pragmatic trials are often thought of as randomised trials that test interventions under the conditions found in routine care: aside from aspects of routine care modi ed by the intervention itself, other aspects of care should be as they usually would be (4)(5)(6)(7)(8)(9). Research funders including the National Institute of Health Research (NIHR) in the UK, the National Institute of Health (NIH) and Patient Centred Outcome Research Institute (PCORI) in the USA aim to fund pragmatic trials. There is growing interest in pragmatic trials from the pharmaceutical industry with the GetReal collaboration aiming to show how real world evidence, including pragmatic trials, could be used in pharmaceutical research and development (10).
When designing pragmatic trials, compromises have to be made as routine care conditions cannot always be completely replicated within the trial. PRECIS-2 (11) ( gure 1) is a trial design tool which has been developed to help make decisions about a trial's design, by highlighting how pragmatic a trial is across nine different domains: eligibility, recruitment, setting, organisation, exibility of delivery, exibility of adherence, follow-up, primary outcome, and primary analysis. Once the tool has been used, investigators may choose to modify the design of their trial or reasons for the design may become more apparent. PRECIS-2 can be applied to individually randomised trials, cluster randomised trials (12) and also to systematic reviews (13).
To date, little research speci c to primary care has been conducted to help those designing pragmatic trials take design decisions that maximise the relevance of their results to practice. Previous research highlights challenges in de ning complex interventions in pragmatic trials, deciding what steps should be taken to ensure compliance, and deciding the level of exibility given to those delivering the intervention (1,14,15).
The overarching aim of this work was to help guide the design of randomised trials of interventions to be delivered in primary care. Speci cally, in this study we investigated how evidence from trials is used and which aspects of trials are considered by those using evidence from randomised trials to determine practice and policy in primary care. We also explored views about pragmatic trials, pragmatic approaches to the nine PRECIS-2 domains, and the usefulness of PRECIS-2.

Methods
We conducted individual and small group interviews to investigate the views of people involved in in uencing primary care practice based on the results of randomised trials, on how best to design trials that are relevant to practice and on PRECIS-2. We sampled from groups who had a professional interest in the wider dissemination of research ndings into primary care practice.
Through research team discussion and consultation with academics in primary care, we identi ed eight categories of people who used the results of primary care trials to in uence practice, and aimed to include people from each category in our sample. The different categories were journal editors, primary care educators, guideline developers, research charities, research funders, clinical commissioning group leads, and quality improvement organisations. We identi ed individuals from each category either from appropriate websites or through personal networks of the research team and approached via email. We invited participants until we had at least one person from each category. We invited more than one person in each category at a time to maximise recruitment, if more than one person from a category responded to our invitation we interviewed all of them. The sample size was limited as the study targeted an elite group of stakeholders tasked with the job of synthesising and rolling out research evidence.
The interviews were carried out by GF (Gordon Forbes), either face to face or via video link (skype). Face to face interviews were carried out the participants place of work or other location of their choosing. The interviews lasted approximately 1 hour. At the time of the interviews GF was a male, research methods fellow who had received training in qualitative research methods. For two interviews GF was joined by an academic GP who co-led the interview. All participants were unknown to the interviewers prior to the interviews taking place.
GF developed the topic guide for the interviews through discussions with the research team. This guide was re ned throughout the research process, for example some vignettes of trials used in early interviews were dropped from later interviews because discussion of these vignettes left little time for discussion of other important topics. The nal topic guide is reported in box 1.
Box 1: Topic guide · Interviewer Introduction -GF introduced himself and the study. He introduced himself as a researcher interested in pragmatic trials, describing the research as a study into how best to design primary care trials so that they are relevant to practice. PRECIS is mentioned in passing but not the details of how the tool works.
· Participant(s) introduction · How participants use evidence from randomised trials · What aspects of trials do participants think are important when deciding whether research is relevant to practice · PRECIS-2 is introduced by the interviewer with the participant shown an example PRECIS-2 wheel. The concepts of pragmatic and explanatory trials are also explained.
· How important is it that research is similar to routine practice for each of the PRECIS domains for the results of the trial to be relevant to practice? · Closing. Interviewee(s) is asked if there is anything else they would like to say or comment on and given the opportunity to ask any questions they have about the study.
Prior to the interviews participants provided consent to participate in the study. Interviews were recorded and transcribed by an independent transcription service. One interview was transcribed by GF. Transcripts and ndings of the study were not returned to participants for comment.
A thematic analysis was carried out following the Framework method (17). GF read all of the interview transcripts and notes. SE (Sandra Eldridge), KL (Kirsty Loudon) and MC (Megan Clinch) each read a different subset so that all the interview transcripts were read by two different people. Analysis was conducted using NVIVO. A thematic framework was developed by GF and reviewed with SE, KL and MC. Indexing and Charting was carried out by GF and reviewed by SE, KL and MC. Interpretation of the results was carried out by all authors of the study.

Results
Twenty four individuals or groups were invited to participate. Seventeen individuals took part in total, nine via individual interviews -and the rest via group-interviews (table 1). Interviews were conducted between October 2014 and February 2015. Three of the individual interviewees did not give permission for their interviews to be recorded. For two interviews the recording equipment failed so in total seven interviews (two group and ve individual) were recorded and transcribed. We used contemporaneous eld notes from the remaining ve interviews. Four main themes emerged from our interviews: How evidence is used; aspects of trials considered when assessing evidence; views on PRECIS-2 domains; and perceptions of pragmatic and explanatory trials.

How evidence is used
Interviewees who applied evidence from randomised trials in a clinical setting reported using evidence differently from those who produced evidence synthesis or guidelines. The latter followed a formal process starting with a systematic way of identifying evidence, critical appraisal and then production of a summary of the evidence as a review or guideline. Challenges in applying evidence came from combining the results of heterogeneous evidence and deciding whether evidence is applicable to the question of interest. In overcoming these challenges judgement has to be incorporated into the formal processes. The greatest area of judgement was around whether evidence is applicable to the question of interest.
"... the guideline development group members look at the evidence tables. And make a, what we call, a considered judgement on evidence from the trials, taking into account other aspects, whether it's generalizable, whether it's applicable." (Interview C, Guideline Developer) For those applying evidence in a clinical setting, sources of evidence included guidelines or evidence summaries as well as less formal sources such colleagues or experts in a particular eld. Due to time pressure and a need for accountability regarding decisions about patient care, the results from single studies were not commonly used to in uence practice. We found a mistrust of early evidence due to the potential of trials to show greater bene t of new interventions than would be found in practice. To overcome this challenge some individuals employed a deliberate strategy of waiting before implementing new evidence. A further challenge in applying evidence was the limited resources to implement novel interventions.
" "…let's just take the new oral anticoagulant agents … There's been some early meta-analyses, but everybody feels the early trials were always more optimistic… (Interview D, Primary care support and quality improvement) And then the question is, do you wish to be an early adopter, are the advantages so great that you want to take a risk or would you like to do it later." (Interview D, Primary care support and quality improvement)

Perceptions of pragmatic trials
Interviewees perceptions of pragmatic trials included enthusiasm, mistrust alongside limited knowledge of the term and misconceptions about its meaning. Interviewees from public research funders and journals were most enthusiastic about pragmatic trials, showing existing familiarity with the concept and positively expressing that those are the sort of trials they would be most interested in.
"We are more interested in funding pragmatic trials than explanatory and would like to see trials which were as pragmatic as possible..." (Interview H, Research funder) Interviewees from the research charity, involved in primary care education or from CCGs, and some interviewees involved in guideline development had little pre-existing knowledge of the term "pragmatic trials", expressing misconceptions as to what the concept meant or it being a term they were not familiar with prior to the interview.
Amongst those who used evidence from guidelines or evidence synthesis, pragmatic trials were welcomed as they simpli ed the judgement around whether evidence is applicable to clinical settings. There was concern, however, that they could be signi cantly different from trials in the existing evidence base leading to heterogeneity which could make meta-analysis more di cult.
"Pragmatic trials may have to be considered separately in meta-analysis due to heterogeneity with other trials" (Interview J, Health technology assessor) "I can just imagine that people would be really thrilled to see a pragmatic trial, of diabetes or something, [set here] or something and that people could then use it... It would be so much better for recommendations." (Interview C, Guideline Developer) Pragmatic trials also carried negative connotations, sometimes being viewed as inferior to more traditional approaches to randomised trials. There was also concern that the degree of pragmatism of trials could be used to manipulate the systematic review process.
"During discussions of evidence it can be di cult to consider the relative pragmatism of evidencesometimes it is used by people to try and exclude evidence that doesn't agree with their point." (Interview I, Research funder) "Pragmatic can be a dirty word when describing trials, people like to shoot at them…" (Interview K, Journal publishing systematic reviews).
Pragmatic trials were con ated with complex interventions and with alternative trial designs such as stepped-wedge or cluster randomised. Some interviewees also drew a distinction between pragmatic trials and randomised controlled trials.
"...to include these kind of trials, you know… the guideline developers would have to be really careful to explain the difference between this and RCT [randomised controlled trial]." (Interview C, Guideline Developer) Views on the PRECIS-2 domains ( g 1) The population in the trials was by far the most discussed aspect of a trial's design, with interviewees preferring trial populations to have few exclusions, including patients with comorbidities and older patients. Population is handled by PRECIS-2 over three domains: eligibility, recruitment and setting.
"Well I think, rst of all, they want to know that people, that patients like mine, are entered." (Interview D, Primary care support and quality improvement) "So again, if you do it in your tertiary centres, then it becomes almost inapplicable in primary care, because I don't have those resources. I don't see all of those patients at that stage in that illness. I see them either way before or way after they've seen the tertiary care people. So yes I think setting is very important." (Interview E, Primary care educator) Aside from its part in determining the population in trials, Recruitment was an aspect of trial design not explicitly considered by most interviewees when assessing evidence. Amongst those who did consider the impact of recruitment on the relevance of trial results, there was concern that very intensive recruitment could bring being people into trials who would not usually present for treatment in routine practice.
"…we wouldn't assess recruitment routinely, but it's about whether recruitment is applicable to the question that we're trying to address." (Interview D, Guideline Developer) "Sometimes recruitment can be too intensive and bring people into the trial who would not usually present for a condition" (Interview I, Chair of funding panel) The organisation domain focuses on the level of expertise and resources made available to deliver an intervention compared with what would be available in practice. Here, we identi ed a tension between a pragmatic and an explanatory approach. Those applying research in practice were more concerned about resourcing issues and preferred trials to test interventions that could be implemented with the limited resources available in primary care. Those from the research charity or guideline developers believed there was also a place for evidence from trials of interventions requiring resources over and above those currently available. They felt that sometimes research showed that a resource intensive intervention was effective could lead to those resources becoming available in routine care.
"The only thing in primary care that would be a limiting factor is the resources aren't this sort of over owing bucket" (Interview E, Primary care educator) "And sometimes groups will make the gold standard recommendation and that will push forward what resources are brought in" (Interview C, Guideline Developer) For exibility of delivery interviewees producing evidence synthesis or guidelines favoured reduced exibility as this allows more understanding to be gained as to what is causing any effect, makes it easier to include a trial in a meta analysis, and also reduces bias from other treatments being initiated. Reduced exibility was also preferred as it allowed greater understanding of what the intervention being delivered in the trial actually is. For the clinicians interviewed there was an appreciation and expectation that trials would have less exibility in the way interventions were delivered.
" After ten years of doing this I would prefer to see strict control [in how interventions are delivered] but if there are variation they need to be described properly so in attempting to make sense of this you can see what has happened." (Interview B, Guideline Developer) "We appreciate that you've got to stick to strict guidelines when you're doing the research, otherwise it doesn't become very accurate at the end of it. So as long as it's not hugely different, we appreciate there is a little bit of leeway in real life, but we wouldn't expect that in a clinical trial." (Interview E, Primary care educator) Poor adherence to interventions in trials was a concern as this can reduce the potential effect of a successful intervention. For some, best practice with regard to exibility of adherence was for adherence issues to be identi ed prior to the trial and in the trial itself no extra steps to be taken to improve adherence. Others, notably the interviewees from the research charity, suggested knowing an intervention can be effective can lead to measures to help adherence being developed.
"Researchers shouldn't include intense follow up to ensure adherence. Steps should be taken to collect as much primary end point data as possible but this is separate to ensuring that people adhere to the intervention." (Interview I, Chair of funding panel) "The more exibility there, the more you're going to actually mask any real effect because of the amount of variation... ... nd out why people aren't adhering and what can we do to try and help people adhere to that particular exercise programme, for example. So, you know, I feel that the exibility [of adherence] should be pushed tighter."

(Interview G, Research charity)
For follow up we identi ed a balance to be struck between collecting data that is useful for research and follow up in uencing participant behaviour or increasing the burden on those taking part.
" ...in terms of having quite a lot follow-up, you can get some really useful answers and actually you probably want to do that. But it's thinking about the way in which you follow up and so that you're not actually in uencing their behaviour and their any sort of clinical outcome through taking those measurements."

(Interview G, Research charity)
Where primary outcome and primary analysis was discussed all Interviewees favoured a more pragmatic approach with patient centred primary outcomes and intention to treat analysis. There was concern expressed about primary outcomes being measured at too early a time point.

Comments on applying PRECIS-2
The guideline developers, research charity and research funders considered PRECIS-2 a useful tool, with the guideline developers saying that it covered many of the areas of judgement they were required to make and those involved in research funding commenting that being able to justify design decisions across the PRECIS-2 domains would strengthen funding applications.
"we never used this instrument, it looks very helpful, we were often left to make the judgement... to what degree was it moving towards the pragmatic, and to what degree was it explanatory. (Interview B, Guideline Developer) "if they thought about all of them in advance and they've got a good reason why it's explanatory on this and it's pragmatic on this one, then I think that's, that would make a strong application coming through." (Interview G, Research Charity) Alternative uses for PRECIS-2 were also suggested including use as a teaching aid, trial reporting, and aiding judgements around applicability. It was noted that PRECIS-2 is subjective so care would have to be taken for reported PRECIS-2 scores to be justi ed.
"…actually in the paper they should put [PRECIS-2 wheels] in… you just want a quick summary. That might be helpful" (Interview F, Primary Care Commissioner)

Issues raised by participants but not covered by PRECIS-2 domains
Outside of PRECIS-2 domains, interviewees raised issues around internal validity, in particular blinding where possible, and size of the trial -with larger trials preferred. Internal validity, sometimes referred to by interviewees as "quality", was typically assessed before generalisability when developing guidelines or assessing research for funding, typically using risk of bias tools. Issues around internal validity were raised both by those involved in evidence synthesis and those applying evidence to practice.
"So if the quality is poor, that will get marked before people even think whether it's generalizable or not." (Interview C, Guideline Developer) Of the factors unrelated to the design of trials, reporting of the trial was the most important issue outside PRECIS-2 domains, to be raised. Poor reporting was seen as an obstacle to using evidence in practice whereas good reporting was seen as something which could enhance the generalisability of evidence.
Areas of reporting that were seen as most important included details of what the intervention was, how the intervention was implemented in the trial and discussion of the generalizability of results. If usual care is used as a comparator it was seen as important to report in detail what usual care consisted of.
"I think we've found that people have not been able to use evidence before because they haven't been clear on what the usual care has been." (Interview C, Guideline Developer) Other issues that were raised included patient acceptability of the intervention, whether research was carried out in collaboration with practice and whether the intervention addressed an important clinical problem either effecting a large number of people or a particular problem for a minority that has been hard to reach.

Summary
Whilst broadly supporting the principle of pragmatic trials this study identi es a number issues that those conducting and funding trials of interventions to be delivered in primary care should take into account.
The term "pragmatic trial" is not universally recognised and sometimes misunderstood. Whilst pragmatic trials were welcomed by some of our interviewees others showed less familiarity with the concept or expressed suspicion towards trials labelled as pragmatic.
Across the PRECIS-2 domains eligibility, setting, primary outcome and primary analysis the universal response from our interviewees was that more pragmatic trial designs would make results most useful, in particular including the same population as would present in practice, having patient centred outcomes and conducting intention to treat analysis. For the domains recruitment, exibility of adherence, organisation, follow up, a balance needs to be struck between testing the intervention under more restricted conditions and a more pragmatic approach. For recruitment and follow-up it was acknowledged that for some trials to be successful they cannot mirror routine care and a slightly less pragmatic approach could be necessary. However, extreme departures from routine care for follow up or recruitment were discouraged. For organisation and exibility of adherence there was a tension between recognising the limitations that an intervention will encounter in every day practice and providing results that can lead to change, either in the resourcing available for an intervention or to patient adherence to the intervention.
There was a contrast in responses between the clinicians interviewed and some of our other interviewees, most notably those from the research charity. The clinicians generally favoured a more pragmatic approach, taking into account the constraints of the system in which they worked. Interviewees from the research charity and involved in guideline development saw a place for designs that are less pragmatic in terms of organisation or exibility of adherence as these trials may provide evidence that leads to system level change or further interventions to improve adherence to treatment. For exibility of delivery a less pragmatic approach was favoured. Being able to clearly identify what the intervention delivered in a trial was more important to our participants than trying to reproduce in the trial the amount of exibility that would exist in practice when delivering interventions.
Our ndings relating to how evidence is used by clinicians are not new and have been explored in more detail by others (18). It is worth noting, however, that clinicians routinely accessed evidence from clinical trials via guidelines, highlighting the importance of trials being conducted in a way that is amenable to the guideline development process. One of the key challenges identi ed for those developing guidelines is assessing whether evidence is applicable. In addition, good reporting, particularly details of the intervention, can enhance the applicability of a trials results. Steps taken to maximise internal validity are also important, for example blinding, as this is often assessed before the applicability of trials is considered.

Strengths and limitations
This study obtained the views of a wide range of people involved in applying evidence from randomised trials in primary care towards the design of pragmatic trials. Whilst the study included people with a wide variety of backgrounds, the sample was small so we may not have covered all views on trial design. Furthermore, the researchers conducting the study presented themselves as researchers involved in carrying out pragmatic trials, so more critical views of pragmatic trials may not have been encountered. The focus of this study was on the design of randomised trials and as a consequence our sample did not include primary care clinicians as a separate category. This may have limited the range of views we found on how evidence is used in primary care.
Discussing PRECIS-2 in interviews presented challenges as it was only possible to convey a relatively super cial level of understanding of the tool. On the other hand the use of PRECIS-2 enabled a detailed discussion of the speci cs of pragmatic trial design, without relying on the interviewees understanding of what it means for a trial to be pragmatic.

Comparison with existing literature
This study is the rst to examine pragmatic trials from the point of view of those funding and disseminating evidence for primary. Too much exibility in the way interventions are delivered has been identi ed as posing problems in pragmatic trials by Sonnad (19) for reasons similar to those identi ed in this study, by Jansen (20) who was concerned about the challenges too much exibility may present to those delivering the intervention, and by Kalkman (21) who identi ed safety concerns when implementing a new intervention without strict guidelines. The tension between delity to the intervention and its delivery and exibility so that the intervention can be implemented widely is also described in implementation research (22). The trade-off between testing interventions within the resource constraints found in practice, and conducting trials that can lead to better resources in practice, has not received much attention previously in the pragmatic trial literature. The need for enhanced descriptions of interventions in pragmatic trials is highlighted in the consort extension for pragmatic trials (7).

Conclusions
Funders and trialists investigating interventions that will be applied in primary care should fund and conduct randomised trials that are pragmatic in terms of the population included in the trial, the setting and trial outcomes. Particular care should be taken in the areas of trial design highlighted here by our interviewees (recruitment, organisation, exibility of adherence, exibility of delivery, and follow up) where a completely pragmatic approach may not be best. The lack of universal understanding of the term pragmatic trials shows a need for promoting better understanding of pragmatic trials and a need for those conducting pragmatic trials to be explicit about how their trial is, and is not, pragmatic. The PRECIS-2 tool can assist with de ning how pragmatic a trial is and could be used to help people understand what it means for a trial to be pragmatic. Good reporting of trials is important to ensuring their applicability, particularly details of the intervention delivered.