Skip to main content

Advertisement

A review of current practice in the design and assessment of internal pilots in UK NIHR clinical trials

Article metrics

Abstract

Background

Internal pilots provide useful information which can help to optimise the running of the main trial. Although some recommendations exist in the literature for the design of internal pilots, little is known about current practice in terms of the specification and also the assessment of progression criteria. The aim of the review is to provide an overview of current practice.

Methods

A cohort of clinical trials with an internal pilot, funded by the National Institute for Health Research (NIHR), Health Technology Assessment programme (HTA), extracted in 2017 was reviewed. Data were extracted from: project descriptions; summary of changes from the first stage; feedback about the full application; monitoring notes; progress report history and protocols, for information about the design and assessment of internal pilots.

Results

Fifty-seven studies were reviewed. An internal pilot was first proposed in the early stages of the trial in the majority of cases. Target number for recruitment, rate of randomisation, retention/primary outcome ascertainment rate, rate of treatment adherence and consent rate were included as progression criteria. All but one study was permitted to continue to the main trial; however, 25% did not strictly meet the progression criteria. Changes were made to the design of the main trial for 25% of studies, mainly in terms of conduct of recruitment.

Conclusions

This review provides insight into the process of designing and assessing internal pilots. Progression criteria are sometimes not met; however, committees involved in the reviewing process will recommend continuation to the main trial, usually accompanied by a second review or close monitoring. Recommendations are made to optimise the process.

Background

Running a clinical trial can be expensive and complicated. Anticipation of hurdles at the design stage will optimise trial conduct. One method is to run a pilot trial, mirroring the larger-scale trial but with fewer participants and fewer resources. Pilot trials can impart unique insight into the day-to-day issues in running a trial and can provide a justification for progressing to, or abandoning, the full-scale trial. Different types of pilot trial exist [1,2,3]. An external pilot (also referred to as an external feasibility study) stands alone from the main trial and can be used to test the feasibility of elements within the trial including recruitment. Data from the external pilot are not used in the main trial analysis. An internal pilot is a phase within the main trial where data from the experience over a pre-specified period are used to determine the design of the rest of the trial, amended or not from the original proposal. Data from the internal pilot are used within the final analysis. The internal pilot should have clear pre-specified progression criteria for moving on to the main trial.

To optimise the benefit of including an internal pilot the progression criteria must be chosen carefully. Recommendations are made in the literature for developing these criteria including: basing recruitment targets on rates per centre per unit time, as opposed to an overall target number at a set time point, to account for unpredictability in site openings; inclusion of intervention adherence and cross-over; event rates; and review of missing data in terms of primary and secondary outcomes. Inclusion of both the funders and Trial Steering Committee (TSC) in the internal pilot review process is recommended [4].

In terms of choosing the pilot sites, it is advised that these must be representative of the sites to be included in the main trial [4, 5]. The protocol should detail the timing of the internal pilot analysis and outline the progression criteria [2, 4, 5].

Little is known, however, about internal pilot trials in practice [4] and few empirical studies are undertaken yet they are widely recommended by funding bodies. The aim of this study was to provide an insight into current practice regarding internal pilots in clinical trials, specifically the design and review process, by assessing a cohort of studies funded by the National Institute for Health Research (NIHR) Health Technology Assessment (HTA) programme.

Methods

The cohort of clinical trials involving an internal pilot, funded by the NIHR HTA programme, was extracted in March 2017. Trials were identified and data extracted via a search of the NIHR Management Information System by the NIHR Evaluation, Trials and Studies Coordinating Centre (NETSCC). All trials from May 2012 that were funded and active or in the publication process with the NIHR Journals Library were included to the point of date of extraction.

After approval from the respective trial chief investigators (CIs) was obtained by NETSCC, the following information was collated: project descriptions; a summary of changes from the first stage; feedback about the full application; monitoring notes and progress report history. Protocols were accessed from the online NIHR Journals Library [6]. Development of a classification for continuation to full trial was carried out by the team and confirmation and checks were discussed with the NETSCC Research on Research (RoR) team. The classification consisted of the following categories: criteria met, pilot passed; criteria not met, pilot passed; unknown whether criteria met, pilot passed; pilot did not pass.

Data were extracted on: choice of progression criteria; the review process; the decision on continuation and any changes made to the design based on the pilot.

Data extracted were quality checked via double-data extraction, carried out for three trials (5%). The second reviewer was chosen on the basis of having had some involvement in the trial, i.e. CI or statistician, in order to assess whether the information available depicted the trial well.

Descriptive statistics were calculated using SAS Version 9.4.

Results

Seventy-four trials were identified as eligible. Seventeen were removed from the cohort (four CIs declined, four organisation contracts offices did not reply, two CIs had not been able to consult their TSC for approval and seven did not respond to the email); a total of 57 trials comprise the cohort. For the 17 trials excluded, four had their pilots extended and the remaining 13 had passed.

The available information for each trial is summarised in Table 1. Sixteen trials had missing project descriptions, summaries of changes from the first stage and feedback about the full application, due to the trials pre-dating the development of the NETSCC Management Information System (April 2012).

Table 1 Details of the design of the internal pilots

The majority of internal pilots were first proposed in the initial stages (Table 1).

Thirty-six trials (63%) specified the progression criteria in the latest available version of the protocol and of the 22 with available previous versions, 19 (86%) specified the progression criteria in all available previous versions of the protocol.

Fifty-four trials (96%) included the target number for recruitment in their progression criteria, 16 (29%) the retention/primary outcome ascertainment rate, 13 (23%) the rate of treatment adherence, 11 (20%) the rate of randomisation, 10 (18%) the number of sites actively recruiting, 8 (14%) the consent rate, 5 (9%) the number of sites opened, 3 (5%) the proportion of eligible participants. One trial did not specify the criteria.

In the majority of cases (91%) the internal pilot followed the design agreed with the funder. Five (9%) pilots did not follow their original design: two internal pilots were extended due to poor recruitment, one of which also amended the progression criteria to remove the first criterion of the number of centres recruiting their first participant and revised the criterion of a target number to be in terms of average patients recruited per site per month; three were delayed – one due to recruitment being put on hold, one due to a delay in recruiting the trial manager and one due to early delays with no further information given (Table 2).

Table 2 Details of the assessment of the internal pilots

Alongside the funder, the TSC was most commonly reported to be involved in the reviewing of the progression criteria, reviewing 94% of trials. No discussion of, or rationale for which committees were chosen to review the results, or in what order, was given for any study.

Fifty-six trials (98%) continued to the full trial, although 14 did not meet the pre-specified progression criteria (see Additional file 1: Table S1); 5 (36%) of which underwent a second review prior to continuation where new criteria were agreed by the trialists and the funder, 3 (21%) put a recovery plan in place and 3 (21%) continued with close monitoring. For nine trials it was unclear whether the criteria were met. For one trial the decision on continuation was on hold at the time of data extraction due to delays in the pilot period.

Forty-four trials (79%) gave a planned number of sites to participate in the internal pilot, 16 (36%) of which specifically named the sites in the protocol.

Twenty-eight (74%) trials adhered to the plan regarding the number of sites. Ten (26%) trials did not adhere to the plan: six opened more sites due to poor recruitment and four opened fewer sites than planned. There was no information on adherence to pilot site choice for six of the trials.

Changes were made to the design of the main trial as a result of the internal pilot in 14 (25%) trials; see Table 3.

Table 3 Details of the outcomes of internal pilots

Discussion

There is increasing use of an internal pilot design to optimise main trial conduct. This review examined the design of internal pilots and how progression criteria were chosen and used to inform progression to the main study. Of 57 trials from an HTA cohort the majority were first proposed in the early stages whether of funder- or researcher-led trials. Although most trials specify the progression criteria in the protocol, a considerable proportion of trials do not (37%), despite recommendations in the literature. It was clear from the included trials that progress to the main trial includes a number of steps and can at times be complex. The decision-making process from the internal pilot to the main trial included a multidisciplinary approach, whereby the funders, trialists, TSC, Independent Data and Safety Monitoring Committee (IDSMC), Trial Management Group (TMG) and sponsor contributed to the progression of a trial. If the reporting of the progression criteria were more transparent in the protocol, this could help to eliminate any uncertainties during the internal pilot stage.

The pilot and feasibility trials extension to the Consolidated Standards of Reporting Trials (CONSORT) 2010 Statement states that objectives of the pilot study should be stipulated in the trial report [7]. The objectives were clear for the trials in this review but none provided a specific rationale for choosing the progression criteria; giving the rationale within the protocol may aid decision-making if certain criteria are not met.

Previous work has examined how feasibility trials (i.e. standalone randomised pilot studies) or ‘external pilots’ funded by the NIHR Research for Patient Benefit (RfPB) programme progressed to main studies. In a cohort of 89 trials which had closed by May 2016, 20 (30%) were judged ‘not feasible’, of which 6 (7%) were intending to apply for funding for the main trial [8]. However, in the current review all but one internal pilot, which was still under review at the time of analysis, continued to the main trial, even when progression criteria had not been met. This is reassuring as a standalone feasibility study is deliberately performed when the level of success for the main trial is much more uncertain than when an internal pilot is undertaken. However, our findings also suggest that a nuanced approach is used to assess the viability of trial when reviewing an internal pilot which can include a second review, ongoing monitoring and changes to the design of the main trial in terms of recruitment and outcomes. Although some of the trials underwent further reviews, none specified this in the protocol; consideration should be given to include plans for this in the protocol when pre-specifying the progression criteria in the event that criteria are not met. Sometimes changes to the trial design are invoked by the results of the internal pilot; in such cases these changes should be documented clearly within the trial report [6].

The majority (77%) of CIs agreed to share their monitoring data and only 4 (5%) CIs explicitly declined; this is important for a move towards more transparent reporting of the decision-making process. Given that the pilots in excluded trials either passed the progression criteria or had their pilot phases extended there was no bias in terms of non-response due to non-progression to the main trial.

One limitation of the review is that the data extraction came from various, sometimes incomplete, sources. An extension of the work would be to contact the respective CIs to ask for a review of the data extracted for their trial to confirm validity or to add where data may have been missing. This was not carried out for this review as firstly, it would have been too time-consuming and, secondly, it was assumed that the information provided was sufficient to give an overall summary which was confirmed by the double-data extraction. These findings may not necessarily be extrapolated to other funding bodies; thus, this work could be extended to include trials from other NIHR programmes as well as other funding bodies to compare progression criteria and progress across funders.

Seventeen trials had a qualitative element which either ran alongside the internal pilot or was incorporated within it. Objectives of the qualitative study were mainly focussed around understanding the barriers and facilitators to recruitment and the consent process, including patients’ motives for giving or refusing consent. There were qualitative elements in some of the other included trials; however, these were either scheduled to take place pre or post internal pilot. Further work can be done to assess the impact of these qualitative elements on the decision to proceed to the main trial.

Conclusions

This review has provided an overview of current practice in planning and carrying out internal pilots and gives a behind-the-scenes insight into the review process with the hopes of moving towards transparent reporting in the design and analysis of internal pilots. Protocols should state progression criteria and give consideration to reporting the rationale of each criterion. Consideration should also be given to add plans for second reviews of the progression criteria to the protocol and any changes made to the trial design as a result of the internal pilot.

Availability of data and materials

The datasets generated and/or analysed during the current study are available from the corresponding author on reasonable request.

Abbreviations

CI:

Chief investigator

HTA:

Health Technology Assessment

IDSMC:

Independent Data and Safety Monitoring Committee

NETSCC:

NIHR Evaluation, Trials and Studies Coordinating Centre

NIHR:

National Institute for Health Research

RfPB:

Research for Patient Benefit

RoR:

Research on Research

TMG:

Trial Management Group

TSC:

Trial Steering Committee

References

  1. 1.

    Blatch-Jones AJ, Pek W, Kirkpatrick E, Ashton-Key M. Role of feasibility and pilot studies in randomised controlled trials: a cross-sectional study. BMJ Open. 2018;8:e022233.

  2. 2.

    Lancaster GA, Dodd S, Williamson PR. Design and analysis of pilot studies: recommendations for good practice. J Eval Clin Pract. 2004;10:307–12.

  3. 3.

    NIHR.:Feasibility and Pilot Studies. 2018. https://www.nihr.ac.uk/funding-and-support/documents/funding-for-research-studies/research-programmes/PGfAR/Feasibility%20and%20Pilot%20studies.pdf. Accessed on 21 Dec 2018.

  4. 4.

    Avery KN, Williamson PR, Gamble C, O’Connell Francischetto E, Metcalfe C, Davidson P, Williams H, Blazeby JM. members of the Internal Pilot Trials Workshop supported by the Hubs for Trials Methodology Research. Informing efficient randomised controlled trials: exploration of challenges in developing progression criteria for internal pilot studies. BMJ Open. 2017;7:e013537.

  5. 5.

    Hampson LV, Williamson PR, Wilby MJ, Jaki T. A framework for prospectively defining progression rules for internal pilot studies monitoring recruitment. Stat Methods Med Res. 2018;27:3612–27.

  6. 6.

    NIHR.:NIHR Journals Library. https://www.journalslibrary.nihr.ac.uk/. Accessed on 21 Dec 2018.

  7. 7.

    Eldridge SM, Chan CL, Campbell MJ, Bond CM, Hopewell S, Thabane L, Lancaster GA, PAFS Consensus Group. CONSORT 2010 Statement: extension to randomised pilot and feasibility trials. Pilot Feasibility Stud. 2016;2:64.

  8. 8.

    Morgan B, Hejdenberg J, Hinrichs-Krapels S, Armstrong D. Do feasibility trials contribute to, or avoid, waste in research. PLoS One. 2018;13(4):e0195951.

Download references

Acknowledgements

We are grateful to the NIHR for providing the data and to Amanda Blatch-Jones and Fay Chinnery for responding to our queries.

Funding

The project was funded by the Medical Research Council Network of Hubs for Trials Methodology Research MR/L004933/2 Project N85. The funder did not play any role in the design of the study, collection of data and analysis; nor in the interpretation of data and writing of the manuscript. JB is part funded by the NIHR Biomedical Research Centre at the University Hospitals Bristol NHS Foundation Trust and the University of Bristol.

Author information

PRW proposed the research project. PRW, CG and JB designed the questionnaire. ARH extracted the data, summarised the results and wrote the first draft of the manuscript. PRW, CG and JB reviewed and made changes to the manuscript. All authors read and approved the final manuscript.

Correspondence to Anna Rosala-Hallas.

Ethics declarations

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Additional file

Additional file 1:

Table S1. Further details on progression criteria not met. Table S2. List of included trials. (DOCX 27 kb)

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Keywords

  • Internal pilot
  • Clinical trials
  • Methodological research

Comments

By submitting a comment you agree to abide by our Terms and Community Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate. Please note that comments may be removed without notice if they are flagged by another user or do not comply with our community guidelines.