Overall, the study fell short of its targets. At 29%, recruitment was below the target of 54% of referrals to the service. Even after an extension of the recruitment period from 18 months to 23 months, only 76 people were recruited - 24% short of the target of 100 recruits. At follow-up, 58% of TER scores were obtained (target 80%) and 58% of CSRI (target 80%). Reasons for the shortfall were largely unanticipated and uncontrollable changes to the service in which recruitment took place. A major influence on recruitment at any research site is the support given by managers and staff. Research is viewed as a non-essential activity, and essential issues need to be satisfactory before research can be given adequate support . Over the 32 months of this project, major and unexpected challenges were experienced: changes to the treatments provided; a freeze on staff recruitment; withdrawal of services in some localities; freeze on receipt of referrals to the service; long waiting times for service users to get into group treatment; withdrawal of clinicians’ time to conduct the PCI; and departure of the site co-investigator. The research was conducted in a turbulent service and this no doubt had an impact on recruitment and retention.
Balanced against recruitment problems is evidence that most service users who were in a position to report their experiences found the PCI acceptable (67%; target 80%) and they rated the PCI’s usefulness as 8 out of 10. Opinions revealed that the PCI benefited engagement both through specific processes, helping people to clarify important personal goals that therapy could help them attain, and general processes, such as familiarizing people with the venue and personnel providing therapy services and building confidence for starting therapy. The characteristics of the interviewer were highly relevant to perceived beneficial effects, with interviewees expressing appreciation for a respectful and sensitive style. Those who did not complete the interview may have had less favorable views.
Perhaps the most persuasive argument for suggesting the need for a full RCT in the face of recruitment difficulties is the evidence of a significant positive impact of the PCI on treatment attendance (Cohen’s d = 0.44), clarity of therapy goals (Cohen’s d = 1.86), and treatment engagement (Cohen’s d = 1.62), although treatment engagement was rated by unblinded therapists, which may account for this effect. At a cost of only £145 per session, a full-scale evaluation of the PCI as pre-therapy preparation seems well worth pursuing. The substantial effect sizes also indicate that the sample size required to test the effectiveness of the PCI interview will not be impractically large. The question is whether we have learned sufficient about the implementation of procedures to be confident of the viability of a full-scale trial.
The first lesson relates to the risk inherent in relying on a single site. A multisite trial would offer a degree of protection against unanticipated and uncontrollable organizational changes that can thwart even the best designed of trials. However, the design of a multisite trial would need to take into account variations in practices between sites.
Furthermore, site selection is of critical importance. Typically, sites are selected on objective criteria, such as the number of eligible patients. Although these are clearly important, other, more subjective criteria should also be taken into consideration, such as clinicians’ enthusiasm for trial participation, support from clinical directors and service managers, and the leadership qualities of the site co-investigators . Gauging the stability of the service also requires attention.
Another important lesson relates to the timing of the initial approach to potential participants. We made this during the assessment phase, when service users were giving and receiving large amounts of information. Overload at this time led many people to turn down the invitation to meet with the researcher on the grounds that they did not want to volunteer for the extra assessments and input associated with the research. The initial contact to inform service users about the research should be made before or after the most intensive assessment period. Most of those who agreed to meet with the researcher to find out more about the project consented to participate.
We also learned lessons about the delivery of the PCI interview. The timing of delivery of the PCI was important, in that adverse conditions such as current physical and mental health problems meant that people were not receptive to the PCI interview. The PCI interview does ask about health and medical matters that may be of current concern to interviewees, but it may be that certain problems prevent people engaging from the start so that they do not get to these questions. After describing the purpose of the PCI interview, it might be useful to ask a preliminary question about any issues that may prevent the interviewee from engaging in the procedure. The addition of written materials to accompany the interview is likely to be of benefit, and a booklet summarizing the procedure in general and recording the individual’s own responses should be produced.
In an effectiveness trial, it will be important to rule out alternative explanations for any observed effect. Clearly, care needs to be taken that blinded researchers collect follow-up data. Blinding the therapists who provide the usual treatment to whether or not the participant has undertaken the PCI interview would rule out biases in staff ratings of participant engagement in therapy. Reliance on dedicated PCI interviewers may not reflect how services would operate in actual clinical practice, but this might be a useful procedure for controlling the quality of the PCI interviews.
Given these uncertainties, further preparatory work is required to estimate the likely rates of recruitment and retention of participants in a multisite trial, and to properly test the acceptability and viability of procedures such as the delivery of the PCI by clinicians and the blinding of researchers. This is in accordance with the Medical Research Council guidance on the evaluation of complex interventions. The Medical Research Council recommends thorough piloting and feasibility work to be confident that the intervention can be delivered as intended, safe assumptions about effect sizes and variability can be made, and rates of recruitment and retention can be gauged prior to designing a main evaluation study . Full-scale evaluations are expensive and preparatory work through a series of studies may be required to progressively refine the design before embarking on a full-scale evaluation. Conducting a pilot study of the PCI evaluation is the next step.