Skip to main content
Download PDF

Remote, real-time expert elicitation to determine the prior probability distribution for Bayesian sample size determination in international randomised controlled trials: Bronchiolitis in Infants Placebo Versus Epinephrine and Dexamethasone (BIPED) study

Trials volume 23, Article number: 279 (2022) Cite this article

Abstract

Background

Bayesian methods are increasing in popularity in clinical research. The design of Bayesian clinical trials requires a prior distribution, which can be elicited from experts. In diseases with international differences in management, the elicitation exercise should recruit internationally, making a face-to-face elicitation session expensive and more logistically challenging. Thus, we used a remote, real-time elicitation exercise to construct prior distributions. These elicited distributions were then used to determine the sample size of the Bronchiolitis in Infants with Placebo Versus Epinephrine and Dexamethasone (BIPED) study, an international randomised controlled trial in the Pediatric Emergency Research Network (PERN). The BIPED study aims to determine whether the combination of epinephrine and dexamethasone, compared to placebo, is effective in reducing hospital admission for infants presenting with bronchiolitis to the emergency department.

Methods

We developed a Web-based tool to support the elicitation of the probability of hospitalisation for infants with bronchiolitis. Experts participated in online workshops to specify their individual prior distributions, which were aggregated using the equal-weighted linear pooling method. Experts were then invited to provide their comments on the aggregated distribution. The average length criterion determined the BIPED sample size.

Results

Fifteen paediatric emergency medicine clinicians from Canada, the USA, Australia and New Zealand participated in three workshops to provide their elicited prior distributions. The mean elicited probability of admission for infants with bronchiolitis was slightly lower for those receiving epinephrine and dexamethasone compared to supportive care in the aggregate distribution. There were substantial differences in the individual beliefs but limited differences between North America and Australasia. From this aggregate distribution, a sample size of 410 patients per arm results in an average 95% credible interval length of less than 9% and a relative predictive power of 90%.

Conclusion

Remote, real-time expert elicitation is a feasible, useful and practical tool to determine a prior distribution for international randomised controlled trials. Bayesian methods can then determine the trial sample size using these elicited prior distributions. The ease and low cost of remote expert elicitation mean that this approach is suitable for future international randomised controlled trials.

Trial registration

ClinicalTrials.gov NCT03567473

Peer Review reports

Background

Bayesian statistical methods use Bayes’ theorem to combine data with previous evidence, characterised in a prior distribution, to make inferences about the parameters in a statistical model [1]. Bayesian methods are increasingly popular in clinical research as concern about frequentist methods has increased [2]. Bayesian methods also formally incorporate external evidence into the trial conclusions, rather than making definitive conclusions based on a single trial [3]. Finally, they provide a more natural interpretation of uncertainty [4] and easily support frequent monitoring and adaptive designs [5].

To use Bayesian methods, the proposed trial sample size must be determined. Bayesian methods for sample size determination (SSD) have several advantages over frequentist methods. First, Bayesian SSD methods incorporate the statistical uncertainty that is inherent in the estimates of key quantities [6]. This contrasts to frequentist methods where the required sample size is highly sensitive to the fixed values that must be specified for several key quantities, such as size and the target difference [7, 8]. Secondly, frequentist SSD methods do not consider clinicians’ current beliefs about a treatment, meaning that trial results that contradict strong beliefs are often not convincing enough to change clinical practice [9]. Finally, sample sizes calculated using frequentist methods are often hard to achieve or unfeasible in rare diseases [10]. In this setting, Bayesian SSD methods can reduce the required sample size by combining trial data with other information, such as expert knowledge or earlier studies, to provide a similar level of scientific certainty [11].

To utilise Bayesian SSD methods, a “prior distribution” must be defined to represent the currently available evidence about all model parameters [12]. This prior distribution can be defined using historical data [13], expert knowledge or a combination of the two [8]. To use expert knowledge, it must be converted into a quantitative expression. This is commonly achieved through a structured “elicitation process” [14] in which experts are assisted in converting their knowledge into a distribution through a process that is viewed as formal data acquisition process based on validated methodologies [15].

Expert elicitation in clinical trials is becoming more frequent with 42 studies related to clinical trial design and analysis included in a recent review of 460 studies discussing Bayesian prior elicitation [16]. Elicitation has been used in randomised controlled trials (RCTs) that compare treatments for trauma resuscitation [17], bacterial corneal ulcers [18] and in rare diseases [19]. However, these studies required experts to meet in person, which can be difficult to arrange, extremely expensive, especially in international studies, and has been restricted due to the COVID-19 pandemic. Alternative approaches to in-person meetings are asynchronous surveys and exercises [20,21,22] or remote, real-time panels [23]. Asynchronous elicitation exercises often have low response and engagement rates and only allow for limited assistance during the elicitation session [22]. Furthermore, experts are less able to discuss and calibrate their beliefs [21], which is key to many elicitation frameworks [24, 25].

As RCTs aim to gather robust empirical evidence that could change clinical practice and health outcomes, the prior for the parameters in an international RCT should robustly represent the beliefs of experts in all health systems where the results would be implemented. This representation is particularly important in diseases where there are regional (international) differences in clinical practice and presentation patterns. Therefore, to avoid the restrictions of in-person exercises and the limitations of asynchronous elicitation, we implemented an efficient, remote, real-time elicitation process to generate representative priors to support Bayesian SSD.

Bronchiolitis, a viral infection of the small and medium airways, is the most common reason infants younger than 1 year of age are admitted to hospital in the developed world. Bronchiolitis also has strong regional differences in clinical practice [26]. Current management recommended by national guidelines is predominantly the provision of parenteral fluids for hydration and oxygen for hypoxemia, called “supportive care” [27,28,29,30,31,32]. Despite a lack of high-quality evidence, the use of additional pharmacotherapy such as nebulised epinephrine, albuterol, hypertonic saline or oral corticosteroids varies by region, with an odds of use of any of these of 11.5 in Canada and 6.8 in the USA, compared to Australia and New Zealand [26]. While using pharmacotherapy alongside supportive care is not supported by most guidelines, exploratory evidence suggests that the combination of inhaled epinephrine and oral corticosteroids has the potential to reduce hospital admission by a third in infants presenting to emergency departments (EDs) with bronchiolitis [33].

The Bronchiolitis in Infants with Placebo versus Epinephrine and Dexamethasone (BIPED) study is an international RCT comparing inhaled epinephrine and oral dexamethasone (a corticosteroid) to placebo for infants presenting to EDs with bronchiolitis for the primary outcome of reducing admission into hospital, taking place in Canada, New Zealand and Australia. Given the regional differences in bronchiolitis management and the geographical spread of BIPED sites, we describe our remote, real-time elicitation exercise to provide a well-justified, representative prior for the SSD and analysis of the BIPED study and the resulting Bayesian SSD.

Methods

The BIPED study

The BIPED study is a phase III, multi-centre, randomised, double-blind, placebo-controlled trial within the Pediatric Emergency Research Network (PERN) [34] to determine whether the combination of inhaled epinephrine and oral dexamethasone (EpiDex) reduces hospitalisation within the 7 days following an initial presentation to an ED with bronchiolitis. The BIPED study enrols participants across 12 international sites in the global network of networks PERN: six sites in Canada (part of the Pediatric Emergency Research Canada Network) and 3 in New Zealand and 3 in Australia (members of the PREDICT network). The study will enrol infants aged between 60 days and 1 year who present to the ED with an episode of wheezing or crackles, alongside signs of an upper respiratory tract infection during the peak season for respiratory syncytial virus (RSV). The active treatment, to be compared with a placebo control, is two treatments of epinephrine (either via nebulisation (3 mg) or via metered-dose inhaler and spacer (625 mcg)) given 30 min apart in the ED and two doses of once daily oral dexamethasone (0.6 mg/kg per dose, up to a maximum of 10 mg). Participants will be randomised in a 1:1 ratio to either the placebo or the EpiDex combination therapy. The BIPED study aims to provide the requested additional evidence [35, 36] after a previous study unexpectedly found that EpiDex reduced hospitalisation within 7 days of an ED visit by one-third [33].

Research ethics approval

The BIPED study was approved by Health Canada and the local research ethics committee at each study site prior to enrollment. The remote elicitation exercise was approved by the Hospital of Sick Children research ethics committee. Implied consent was used for the remote elicitation exercise, meaning that by partaking in the elicitation exercise, the experts agreed that their data could be used for research.

Designing the remote elicitation exercise

Key parameters and clinical setting

The primary outcome in the BIPED study is admission to hospital within 7 days following initial presentation to ED with bronchiolitis, which can be modelled using a binomial distribution. The parameters of interest in the BIPED study are the probability of hospital admission within 7 days for each arm, placebo and EpiDex, denoted π1 and π2, respectively. As beta distributions commonly model beliefs about probabilities [37], we assume that each expert’s prior can be expressed as a beta distribution.

To enable the elicitation, we developed a clinical case study (Supplementary Material) of an infant with bronchiolitis, who would meet the inclusion/exclusion criteria of the BIPED study, and was likely equivocal with respect to admission into hospital (i.e., EpiDex could potentially improve infant prognosis if prior beliefs supported benefit). Experts were asked to determine the number of patients, out of 100, with characteristics like this patient who would be admitted to hospital within 7 days under the two different treatment options. Thus, we elicited the two probabilities as proportions among a population of similar patients [38] and felt that 100 patients were sufficient to capture uncertainty in this proportion rather than sampling variability. We elicited the probabilities of hospital admission, rather than a treatment effect, as this more closely reflects everyday clinical decision-making and thus the experts were more able to provide information on these probabilities. However, in doing so, we assume independence between the two admission probabilities, which may be incorrect.

The elicitation exercise aimed to determine prior distributions for the BIPED Bayesian SSD and analysis. However, as placebo is not offered in routine care, we elicited the probability of admission under supportive care. In the BIPED study, placebo will be offered with supportive care (to avoid bias) and, thus, we assumed that the prior for the placebo in our Bayesian SSD was represented by the prior for supportive care.

Developing an online elicitation tool

Our remote elicitation exercise used an adapted version of the Sheffield Elicitation Framework (SHELF) methodology [17, 25], which aligns closely with the IDEA protocol [39]. Online tools have supported the use of the SHELF framework [40, 41] and we adapted these tools for our elicitation. We built a Web-based interactive elicitation tool using R software and the shiny package [42, 43] (https://phebelan.shinyapps.io/Elicitation/). In this tool, experts were asked to provide the lower and upper plausible values that subjectively described their beliefs about the number of infants with bronchiolitis who would be hospitalised within 7 days. We assumed that the lower and upper plausible values represented the limits of the 95% central credible interval in the beta distribution. Experts then provided their “Best” estimate for the number of hospitalisations. This best estimate was assumed to be the point of highest probability density in the prior and, thus, represented its mode. Within the online tool, we restricted the value for the mode to be within the plausible interval. By eliciting the upper and lower plausible values, followed by the mode, we aimed to prevent experts from anchoring to their initial most likely value and thereby underestimating uncertainty [24]. The online tool provided experts with a real-time individual beta distribution plot, with parameters estimated using R function, BetaExpert [44], and a quantitative summary of their beliefs so they could adjust if the fitted beta distribution did not represent their beliefs (Supplementary Material).

While the online tool supported the elicitation process, the Research Electronic Data Capture (REDCap) application, a Web-based application designed to support secure data capture for research studies [45, 46], collected the elicited distributions from each expert. Once we developed the online elicitation tool and REDCap database, we piloted our workshop three times internally (AP, SD, TK, MO) to ensure clarity of expression, understanding and acceptability of the tool. These workshops were piloted remotely to ensure seamless delivery and the efficient use of experts’ time.

Selecting the experts

We targeted experts from Canada, the USA, Australia and New Zealand to determine representative aggregate priors across the regions in the BIPED study, avoiding selection bias. Participants were eligible for the elicitation workshop if they (i) were identified as experts in bronchiolitis and its treatment and (ii) had experience in paediatric emergency medicine (PEM). Participants were excluded if they had extensive prior involvement with the BIPED study, i.e. serving as a site principal investigator. Potential participants were invited to contribute by email. We aimed to recruit between 10 and 20 experts to ensure a breadth of experience in terms of geography and speciality [14, 47].

Determining an aggregate prior distribution

In elicitation, determining an aggregate prior distribution is viewed as a consensus formation process, in which the pooled prior distributions should fairly represent all individuals’ beliefs [48]. In our elicitation study, each expert i = 1, …, N generates a prior distribution for each trial arm j = 1, 2,

$$ {p}_{i,j}\mid {\boldsymbol{x}}_{i,j}\sim Beta\left({\alpha}_{i,j},{\beta}_{i,j}\right), $$

where \( {\boldsymbol{x}}_{i,j}=\left({x}_{i,j}^1,{x}_{i,j}^2,{x}_{i,j}^3\right) \) is the lower plausible value, mode and upper plausible value, respectively, from the expert elicitation process. These individual-level distributions are combined using the equal-weighted linear pooling method as it can reduce biases introduced by overoptimism and overconfidence [49]. The equal-weighted linear pooling is also easier to implement than other aggregation methods [50]. However, it can lead to less informative prior distributions [50, 51]. We chose to prioritise the ease of use as the experts were not available for long sessions; the lack of informativeness would be overcome by combining the prior with RCT data.

Thus, the aggregate distributions are equal to

$$ {\pi}_j\sim \frac{1}{N}\ \sum \limits_{i=1}^N{p}_{i,j}\mid {\boldsymbol{x}}_{i,j} $$

and we assume that they represent group’s beliefs on the admission rate of infants with bronchiolitis under supportive care and EpiDex, respectively, for j = 1, 2. We generate separate pooled distributions for each region and for each workshop to explore differences.

The remote, real-time elicitation workshop

Pre-workshop materials

One week prior to the workshop, all participants were sent a study dossier to read before attending the workshop. This dossier aimed to introduce the concept of an elicitation exercise and the current literature on treatments for bronchiolitis [24]. Our study dossier included a published elicitation study similar to our study [17] and four manuscripts presenting the use of epinephrine and/or dexamethasone as a treatment for bronchiolitis [33, 52,53,54]. The previous elicitation study was included to introduce the concept of Bayesian probability distributions and elicitation, while the other studies were included to complement the experts’ knowledge with the current literature.

Remote, real-time expert elicitation workshop

We conducted three remote, real-time elicitation workshops using Zoom, a cloud-based video conferencing platform [55], and a standardised script (Supplementary Material) [49]. Three facilitators from the BIPED study team with statistical and medical content specific-expertise attended each workshop (JL, SD, AP, TK, AH). The workshop began with an introduction to Bayesian statistics, probability and the BIPED study. To familiarise experts with the elicitation procedure, an example using our online elicitation tool was then shown (Supplementary Material). Experts then used the online elicitation tool to provide their personal beliefs about the chance of hospitalisation for the patient identified in the case study.

The elicitation exercise was structured over two rounds with a group discussion between the two rounds [24, 25]. In the first round, experts provided their individual prior distribution for the probability of hospitalisation with supportive care and EpiDex. The facilitator (JL) then generated a deidentified boxplot (Fig. S1) to display all the individual-level priors and support the group discussion. The group discussion allowed the experts to adjust and calibrate their responses but did not aim to reach a consensus [24]. The group discussion began with the facilitator interpreting the individual boxplots before the experts were encouraged to share their beliefs and discuss their thoughts around the observed variations in beliefs across experts. When the group discussion no longer resulted in an exchange of information, the facilitator managed the discussion to help promote critical thinking [39]. Following the group discussion, experts were asked to use the online elicitation tool to characterise their beliefs and these results then generated the individual prior distributions to be pooled.

Following the remote, real-time elicitation workshop

Following the completion of all three workshops, the experts were sent the pooled distributions for the probability of hospital admission with supportive care and EpiDex. The experts were also sent the workshop-specific pooled distribution for each workshop and their own individual distributions for comparison and were invited to provide comments on the aggregate distribution.

Bayesian sample size determination

To determine the sample size in the BIPED study, we used the average length criterion (ALC) for Bayesian SSD [56]. This method selects the smallest sample size for which the average length of a specified posterior credible interval is below a given threshold. The ALC uses a preposterior analysis where the length of the posterior credible interval is estimated across the prior-predictive distribution of the potential studies [56]. To achieve this, we simulated the probability of hospitalisation within 7 days under the two treatments based on the priors from the expert elicitation exercises using a binomial likelihood. These simulated data were combined with our aggregated prior to determine the posterior for the two probabilities of hospitalisation, using Markov chain Monte Carlo (MCMC) methods [57]. We then calculated the 95% high-density posterior credible interval for the difference in the probability of admission across the two treatments, placebo and EpiDex. We estimated the average posterior credible interval length for sample sizes between 400 and 630 using 1500 simulations from the prior-predictive distribution and 5000 simulations from the posterior. We selected the sample size for which the ALC is below 0.09.

In the BIPED study, we will declare that EpiDex is superior to placebo if the posterior probability that the probability of hospitalisation under EpiDex is greater than the probability of hospitalisation under placebo exceeds 0.99;

$$ P\left({\pi}_1<{\pi}_2\right)>0.99. $$

To determine whether our study is likely to reach a conclusion, we compute the relative predictive power of the study, defined as the probability of declaring EpiDex is superior based on simulated studies from the prior predicative distribution, standardised by the prior probability that EpiDex is superior to placebo. These calculations were based on 8000 simulated trials with 5000 simulations from the posterior. All Bayesian analyses were performed using JAGS through R [43, 58].

Results

Elicitation workshop

Baseline characteristics

We invited 25 PEM clinicians from Canada, the USA, Australia and New Zealand to participate in our three remote elicitation workshops. Fifteen of these experts agreed to participate in the study: 9 from North America (NA) and 6 from Australia and New Zealand (ANZ). The three workshops contained 5 (2 NA; 1 ANZ), 4 (4 NA) and 6 (3 NA; 3 ANZ) participants, respectively. Table 1 displays the baseline characteristics for these 15 experts. Experts from NA had more experience treating bronchiolitis with epinephrine and dexamethasone, separately and combined. However, most experts do not currently use either treatment in their routine practice.

Table 1 Baseline characteristics of experts, by region of practice
Full size table

Prior distributions

Figure 1 displays the individual prior distributions for the two probabilities of hospitalisation, with supportive care on the left and EpiDex on the right. The individual responses from both rounds were highly varied, both in terms of the central tendency of the distributions across individuals and the width of the plausible interval within individuals, although most experts believe that the probability of admission for infants with bronchiolitis is slightly lower for those receiving EpiDex compared to supportive care. Figure 2 displays the pooled prior distributions from all experts for each round in the elicitation workshop. In both rounds, the pooled prior distributions show a slight reduction in the probability of admission for infants with bronchiolitis who are treated with EpiDex. However, experts were less certain about the size of this reduction in the second round, demonstrating that the group discussion led the experts to be more conservative. These distributions are multi-modal, which is a result of the equal-weights linear pooling method. Thus, they do not represent a consensus distribution for a single rational impartial observer but rather acknowledge that there are differences in beliefs across the experts [50]. No experts provided critiques on these priors by email.

Fig. 1
figure 1

Individual-level elicited prior distributions for hospitalisation probability under (a) supportive care (left) or (b) treatment with the combination of epinephrine and dexamethasone (EpiDex, right). Each line depicts the distribution scored by an individual participant (n = 15). Distributions for first elicitation round on top; second round at bottom

Full size image
Fig. 2
figure 2

Pooled elicited prior distributions for hospitalisation probability under (a) supportive care (supportive, solid black line) or (b) treatment with the combination of epinephrine and dexamethasone (EpiDex, dashed red line). Distributions for first elicitation round top; second round bottom

Full size image

Pooled prior distributions demonstrate that similar beliefs about the probability of hospitalisation are held in NA and ANZ (Fig. S2) but that the aggregate distributions were different for each workshop (Fig. S3). However, this is likely due to the diversity of our experts and the limited number of individuals in each workshop.

Bayesian sample size determination

We computed the average length of the 95% high-density posterior credible interval for the difference in admission probability between the two arms (Fig. 3). From these results, we specify a sample size of 410 participants per arm for the BIPED study ensuring the average 95% credible interval is shorter than 9%, compared to 610 participants per arm with a uniform prior. Adjusting for an expected 5% loss to follow up, the total sample size of the BIPED study is 432 per arm. The average 95% credible interval would be less than 8% if the BIPED study recruits 610 participants per arm. With 410 participants per arm, the study has a relative predictive power of 90%.

Fig. 3
figure 3

The width of the average 95% posterior credible interval length for “admission probability difference” between placebo and the combination of epinephrine and dexamethasone EpiDex plotted across the BIPED clinical trial sample sizes increasing between 400 and 630 in increments of 5 (solid black line). Average length criterion (ALC) thresholds of 0.09 and 0.08 are plotted as dashed black lines (see text)

Full size image

Discussion

We implemented a remote, real-time elicitation exercise that provides a practical and convenient method for expert elicitation using an online tool. Expert belief, elicited using this framework, then formed the basis of a Bayesian SSD and analysis for an international RCT, where the prior distribution should represent the diverse beliefs across the regions enrolling patients. The remote nature of our elicitation allowed us to practically obtain diverse opinions by running a synchronous online exercise with a reasonably large number of diverse experts. We quickly enrolled 15 experts from 4 countries (Canada, USA, New Zealand and Australia), on a limited budget, under COVID-19-related travel restrictions and determined a pooled prior distribution that represents the diversity of perspectives in an international trial. As our elicitation exercise involved a relatively short time commitment, we had high response rates, resolving issues seen with asynchronous elicitation [22]. Finally, we were able to hold multiple elicitation workshops assisted by a facilitator to further broaden the range of experts who could attend.

Another advantage of our elicitation framework, compared to asynchronous elicitation methods, is that we were able to have real-time facilitation and a group discussion [21]. This allowed us the interaction between experts and the identification of issues within the workshops. As there were differences between the distributions between the two rounds, the group discussion was critical in calibrating the experts’ beliefs. The experts raised external factors that would influence the decision to admit an infant with bronchiolitis, such as hospital resources and family circumstances, and shared their thoughts and clarifications related to the design of the elicitation exercise and their understanding of the elicitation task. We were also able to respond to any technical issues and ensure that all enrolled experts were able to provide responses.

Our biggest challenge was scheduling the workshops as we needed to accommodate large differences in time zones between the countries and the shift patterns of practising PEM clinicians working in the ED. We decided to run multiple workshops so more experts could participate and aimed to include experts from each region in each workshop and ensure there were enough participants to allow a fruitful group discussion. While we were largely successful, we found that scheduling of these workshops was a significant challenge and recommend inviting a higher number of experts than required as some schedules may be incompatible, especially across multiple time zones. Furthermore, scheduling multiple workshops prevented us from using expert consensus to determine the final prior distribution. Instead, we used the equal-weights linear pooling method, which has been found to provide less informative prior distributions [50].

A strength and limitation of our remote, real-time elicitation exercise is the time taken for the workshop. Each workshop was scheduled for 90 min, and the experts were invited to read five manuscripts before attending the workshop as preparation — estimated to take another 90 min. This minimal time commitment, compared to day-long meetings and travel, allowed us to recruit a range of experts to our study and was key to enrolling practising PEM physicians. However, the set 90 min meeting-slot did limit the time available for presenting the theory behind elicitation and for calibration exercises, which could have impacted the quality of our elicited prior distribution [38] and prevented us from using alternative pooling methods [59]. We also did not consider multivariate elicitation [60], which would have allowed to consider dependence between the two admission probabilities.

Conclusions

To enable application of Bayesian methods for SSD, we implemented a remote, real-time elicitation exercise that offers a comprehensive, practical, affordable approach to obtaining prior distributions for a Bayesian analysis of an international RCT. This prior distribution was then used to determine the sample size for the proposed Bayesian analysis. Thus, this remote, real-time elicitation exercise can promote the use of Bayesian methods in randomised controlled trials.

Availability of data and materials

Our consent procedures do not allow for data sharing due to the small number of participants in the study and the ease of identifying these individuals. The pooled distributional forms are available from the corresponding author on reasonable request.

Abbreviations

ANZ:

Australia and New Zealand

BIPED:

Bronchiolitis in Infants Placebo Versus Dexamethasone Study

REDCap:

Research Electronic Data CAPture

EpiDex:

Combination therapy of nebulised epinephrine and dexamethasone

ED:

Emergency department

NA:

North America

PEM:

Paediatric emergency medicine

References

  1. van de Schoot R, Depaoli S, King R, Kramer B, Märtens K, Tadesse M, et al. Bayesian statistics and modelling. Nat Rev Methods Primers. 2021;1(1). https://doi.org/10.1038/s43586-020-00001-2.

  2. Berry D. Bayesian clinical trials. Nat Rev Drug Discov. 2006;5(1):27–36.

    Article  CAS  PubMed  Google Scholar 

  3. Spiegelhalter D, Freedman L, Parmar M. Bayesian approaches to randomized trials. J R Stat Soc A (Statistics in Society). 1994;157(3):357–87.

    Article  Google Scholar 

  4. Jack Lee J, Chu C. Bayesian clinical trials in action. Stat Med. 2012;31(25):2955–72.

    Article  PubMed  Google Scholar 

  5. Berry S, Carlin B, Lee J, Muller P. Bayesian adaptive methods for clinical trials. Boca Raton: CRC Press; 2010.

    Book  Google Scholar 

  6. Adcock C. Sample size determination: a review. J R Stat Soc D (The Statistician). 46(2):261–283, 2997.

  7. Cook J, Julious S, Sones W, Hampson L, Hewitt C, et al. DELTA2 guidance on choosing the target difference and undertaking and reporting the sample size calculation for a randomised controlled trial. BMJ. 2018;363:k3750.

    Article  PubMed  PubMed Central  Google Scholar 

  8. Joseph L, Du Berger R, Bélisle P. Bayesian and mixed Bayesian/likelihood criteria for sample size determination. Stat Med. 1997;16(7):769–81.

    Article  CAS  PubMed  Google Scholar 

  9. Chaloner K, Rhame F. Quantifying and documenting prior beliefs in clinical trials. Stat Med. 2001;20(4):581–600.

    Article  CAS  PubMed  Google Scholar 

  10. Hampson L, Whitehead J, Eleftheriou D, Tudur-Smith C, Jones R, Jayne D, et al. Elicitation of expert prior opinion: application to the MYPAN trial in childhood polyarteritis nodosa. PLoS One. 2015;10(3):e0120981.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  11. Choy S, O'Leary R, Mengersen K. Elicitation by design in ecology: using expert opinion to inform priors for Bayesian statistical model. Ecology. 2009;90(1):265–77.

    Article  PubMed  Google Scholar 

  12. Cao J, Lee J, Alber S. Comparison of Bayesian sample size criteria: ACC, ALC, and WOC. J Stat Plan Inference. 2009;139(12):4111–22.

    Article  PubMed  PubMed Central  Google Scholar 

  13. De Santis F. Using historical data for Bayesian sample size determination. J R Stat Soc A (Statistics in Society). 2007;170(1):95–113.

    Article  Google Scholar 

  14. Dias L, Morton A, Quigley J. Elicitation: the science and art of structuring judgement. Cham: Springer; 2018.

    Book  Google Scholar 

  15. Estévez R, Mardones F, Álamos F, Arriagada G, Carey J, Correa C, et al. Eliciting expert judgements to estimate risk and protective factors for Piscirickettsiosis in Chilean salmon farming. Aquaculture. 2019;507:402–10.

    Article  Google Scholar 

  16. Azzolina D, Berchialla P, Gregori D, Baldi I. Prior elicitation for use in clinical trial design and analysis: a literature review. Int J Environ Res Public Health. 2021;18(4):1833.

    Article  PubMed  PubMed Central  Google Scholar 

  17. Jansen J, Wang H, Holcomb J, Harvin J, Richman J, Avritscher E, et al. Elicitation of prior probability distributions for a proposed Bayesian randomized clinical trial of whole blood for trauma resuscitation. Transfusion. 2020;60(3):498–506.

    Article  PubMed  PubMed Central  Google Scholar 

  18. See C, Srinivasan M, Saravanan S, Oldenburg C, Esterberg E, Ray K, et al. Prior elicitation and Bayesian analysis of the steroids for corneal ulcers trial. Ophthalmic Epidemiol. 2012;19(6):407–13.

    Article  PubMed  Google Scholar 

  19. Ramanan A, Hampson L, Lythgoe H, Jones A, Hardwick B, Hind H, et al. Defining consensus opinion to develop randomised controlled trials in rare diseases using Bayesian design: an example of a proposed trial of adalimumab versus pamidronate for children with CNO/CRMO. PLoS One. 2019;14(6):e0215739.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  20. Browne E, Rathinam S, Kanakath A, Thundikandy R, Babu M, Lietman T, et al. A Bayesian analysis of a randomized clinical trial comparing antimetabolite therapies for non-infectious uveitis. Ophthalmic Epidemiol. 2017;24(1):63–70.

    Article  PubMed  Google Scholar 

  21. McBride M, Garnett S, Szabo J, Burbidge A, Butchart S, Christidis L, et al. Structured elicitation of expert judgments for threatened species assessment: a case study. Methods Ecol Evol. 2012;3(5):906–20.

    Article  Google Scholar 

  22. Grigore B, Peters J, Hyde C, Stein K. EXPLICIT: a feasibility study of remote expert elicitation in health technology assessment. BMC Med Inform Decis Making. 2017;17(1):1–10.

    Article  Google Scholar 

  23. Aspinall W, Cooke R, Havelaar A, Hoffmann S, Hald T. Evaluation of a performance-based expert elicitation: WHO global attribution of foodborne diseases. PLoS One. 2016;11(3):e0149817.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  24. Gosling J. SHELF: the Sheffield elicitation framework, in Elicitation, vol. 8. Cham: Springer; 2021. p. 61–93.

    Google Scholar 

  25. A. O'Hagan and J. Oakley, SHELF: the Sheffield elicitation framework. Sheffield, URL: http://tonyohagan.co.uk/shelf. Accessed 7 Jul 2021.

  26. Schuh S, Babl F, Dalziel S, Freedman S, Macias C, Stephens D, et al. Practice variation in acute bronchiolitis: a pediatric emergency research networks study. Pediatrics. 2017;140(6):e20170842.

    Article  PubMed  Google Scholar 

  27. Babl F, Borland M, Ngo P, Acworth J, Krieser D, Pandit S, et al. Paediatric Research in Emergency Departments International Collaborative (PREDICT): first steps towards the development of an Australian and New Zealand research network. Emerg Med Australas. 2006;18(2):143–7.

    PubMed  Google Scholar 

  28. Ricci V, Nunes V, Murphy M, Cunningham S. Bronchiolitis in children: summary of NICE guidance. BMJ. 2015;350:h2305.

    Article  PubMed  Google Scholar 

  29. Ralston S, Lieberthal A, Meissner H, Alverson B, Baley J, Gadomski A, et al. Clinical practice guideline: the diagnosis, management, and prevention of bronchiolitis. Pediatrics. 2014;134(5):e1474–502.

    Article  PubMed  Google Scholar 

  30. Friedman J, Rieder, J M. Walton and Canadian Paediatric Society, Acute Care Committee, Bronchiolitis: recommendations for diagnosis, monitoring and management of children one to 24 months of age. Paediatr Child Health. 2014;19(9):485–91.

    Article  PubMed  PubMed Central  Google Scholar 

  31. Baumer J. SIGN guideline on bronchiolitis in infants. Arch Dis Child Educ Pract. 2007;92(5):ep149–51.

    Google Scholar 

  32. Quinonez R, Schroeder A. Safely doing less and the new AAP bronchiolitis guideline. Pediatrics. 2015;135(5):793–5.

    Article  PubMed  Google Scholar 

  33. Plint A, Johnson D, Patel H, Wiebe N, Correll R, Brant R, et al. Epinephrine and dexamethasone in children with bronchiolitis. N Engl J Med. 2009;360(20):2079–89.

    Article  CAS  PubMed  Google Scholar 

  34. Klassen T, Dalziel S, Babl F, Benito J, Bressan S, Chamberlain J, et al. The Pediatric Emergency Research Network (PERN): a decade of global research cooperation in paediatric emergency care. Emerg Med Australas. 2021. https://doi.org/10.1111/1742-6723.13801.

  35. Hartling L, Fernandes R, Bialy L, Milne A, Johnson D, Plint A, et al. Steroids and bronchodilators for acute bronchiolitis in the first two years of life: systematic review and meta-analysis. BMJ. 2011;342:d1714.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  36. Wainwright C. Acute viral bronchiolitis in children-a very common condition with few therapeutic options. Paediatr Respir Rev. 2010;11(1):39–45.

    Article  PubMed  Google Scholar 

  37. Bolstad W, Curran J. Introduction to Bayesian statistics. Hoboken: Wiley; 2016.

    Google Scholar 

  38. Garthwaite P, Kadane J, O'Hagan A. Statistical methods for eliciting probability distributions. J Am Stat Assoc. 2005;100(470):680–701.

    Article  CAS  Google Scholar 

  39. Hemming V, Burgman M, Hanea A, McBride M, Wintle B. A practical guide to structured expert elicitation using the IDEA protocol. Methods Ecol Evol. 2018;9(1):169–80.

    Article  Google Scholar 

  40. Morris D, Oakley J, Crowe J. A web-based tool for eliciting probability distributions from experts. Environ Model Softw. 2014;52:1–4.

    Article  Google Scholar 

  41. Mason A, Gomes M, Grieve R, Ulug P, Powell J, Carpenter J. Development of a practical approach to expert elicitation for randomised controlled trials with missing health outcomes: application to the IMPROVE trial. Clin Trials. 2017;14(4):357–67.

    Article  PubMed  PubMed Central  Google Scholar 

  42. W. Chang, J. Cheng, J. Allaire, S. C., B. Schloerke, Y. Xie, J. Allen, J. McPherson, A. Dipert and B. Barbara Borges, shiny: Web application framework for R. R package version 1.6.0., https://CRAN.R-project.org/package=shiny, Vienna, Austria, 2021.

  43. R Core Team. R: a language and environment for statistical computing. Vienna: R Foundation for Statistical Computing; 2020.

    Google Scholar 

  44. B. Devleesschauwer, P. Torgerson, J. Charlier, B. Levecke, N. Praet, S. Roelandt, S. Smit, P. Dorny, D. Berkvens and N. Speybroeck, prevalence: tools for prevalence assessment studies., R package version 0.4.0, Vols. http://cran.r-project.org/package=prevalence, 2014.

    Google Scholar 

  45. Harris P, Taylor R, Minor B, Elliott V, Fernandez M, O'Neal L, et al. The REDCap consortium: building an international community of software platform partners. J Biomed Inform. 2019;95:103208.

    Article  PubMed  PubMed Central  Google Scholar 

  46. Harris P, Taylor R, Thielke R, Payne J, Gonzalez N, Conde J. Research electronic data capture (REDCap)—a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009;42(2):377–81.

    Article  PubMed  Google Scholar 

  47. European Food Safety Authority. Guidance on expert knowledge elicitation in food and feed safety risk assessment. EFSA J. 2014;12(6):3734.

    Google Scholar 

  48. Dietrich F, List C. Probabilistic opinion pooling. In: The Oxford handbook of probability and philosophy. Oxford: Oxford University Press; 2016. p. 1–27.

    Google Scholar 

  49. Johnson S, Tomlinson G, Hawker G, Granton J, Feldman B. Methods to elicit beliefs for Bayesian priors: a systematic review. J Clin Epidemiol. 2010;63(4):355–69.

    Article  PubMed  Google Scholar 

  50. Williams C, Wilson K, Wilson N. A comparison of prior elicitation aggregation using the classical method and SHELF. J R Stat Soc A (Statistics in Society). 2021;184:920–40.

    Article  Google Scholar 

  51. Clemen R. Comment on Cooke’s classical method. Reliab Eng Syst Saf. 2008;93(5):760–5.

    Article  Google Scholar 

  52. Corneli H, Zorc J, Mahajan P, Shaw K, Holubkov R, Reeves S, et al. A multicenter, randomized, controlled trial of dexamethasone for bronchiolitis. N Engl J Med. 2007;357(4):331–9.

    Article  CAS  PubMed  Google Scholar 

  53. Florin T, Plint A, Zorc J. Viral bronchiolitis. Lancet. 2017;389(10065):211–24.

    Article  PubMed  Google Scholar 

  54. Kirolos A, Manti S, Blacow R, Tse G, Wilson T, Lister M, et al. A systematic review of clinical practice guidelines for the diagnosis and management of bronchiolitis. J Infect Dis. 2020;222(7):S672–9.

    Article  PubMed  Google Scholar 

  55. Zoom Video Communications, Zoom Cloud Meetings, URL: https://zoom.us/. Accessed 9 Aug 2021, 2021.

  56. Joseph L, M'Lan C, Wolfson D. Bayesian sample size determination for binomial proportions. Bayesian Anal. 2008;3(2):269–96.

    Article  Google Scholar 

  57. Gilks W, Richardson S, Spiegelhalter D. Markov chain Monte Carlo in practice. Boca Raton, FL: CRC press; 1995.

    Book  Google Scholar 

  58. Plummer M. JAGS: a program for analysis of Bayesian graphical models using Gibbs sampling. In: Proceedings of the 3rd international workshop on distributed statistical computing, vol. 124, no. 125.10; 2003. p. 1–10.

    Google Scholar 

  59. Cooke R, Mendel M, Thijs W. Calibration and information in expert resolution; a classical approach. Automatica. 1988;24(1):87–93.

    Article  Google Scholar 

  60. Werner C, Hanea A, Morales-Nápoles O. Eliciting multivariate uncertainty from experts: considerations and approaches along the expert judgement process, in Elicitation. Cham: Springer; 2018. p. 171–210.

    Google Scholar 

Download references

Acknowledgements

We thank all members of the Pediatric Emergency Research Canada (PERC)-KidsCAN Innovative Pediatric Clinical Trials BIPED Study Group and the PERC-KidsCAN Innovative Pediatric Clinical Trials Methods Core for assistance in developing the protocol and statistical analysis plan for the BIPED trial. We acknowledge the PERC-KidsCAN Innovative Pediatric Clinical Trials team who provided valuable input on the study design and documents. We thank the PERC network of health care professionals, the KidsCAN Trials Network and the Pediatric Emergency Research Network (PERN) for their contribution and support to this project and pediatric clinical research in Canada and globally. Finally, we thank all participants in the remote, real-time elicitation workshops, Trevor Kuang, Antonia Stang, Robert Porter, James Chamberlain, Vikram Sabhaney, Jo Cole, and those who wished to remain anonymous.

Funding

This work is funded through an Innovative Clinical Trials Multi-year Grant from the Canadian Institutes of Health Research (funding reference number MYG-151207; 2017–2020), as part of the Strategy for Patient-Oriented Research and in partnership with the Alberta Children’s Hospital Research Institute (Calgary, Alberta), Centre Hospitalier Universitaire Sainte-Justine (Montreal, Quebec), Children’s Hospital Research Institute of Manitoba (Winnipeg, Manitoba), CHEO Research Institute (Ottawa, Ontario), Hospital for Sick Children Research Institute (Toronto, Ontario), Stollery Children’s Hospital (Edmonton, Alberta), Research Manitoba (Winnipeg, Manitoba), University of Western Ontario (London, Ontario) and the Women and Children’s Health Research Institute (Edmonton, Alberta). SRDs time was supported by Cure Kids New Zealand. ACP is supported by a Tier I University of Ottawa Research Chair. TPK is supported by the Canada Research Chair in Clinical Trials. AH is supported by the Canada Research Chair in Statistical Trial Design.

Author information

Authors and Affiliations

  1. Child Health Evaluative Sciences, The Hospital for Sick Children, 686 Bay Street, Toronto, ON, M5G 0A4, Canada

    Jingxian Lan, Martin Offringa & Anna Heath

  2. Division of Emergency Medicine, Children’s Hospital of Eastern Ontario, Ottawa, Canada

    Amy C. Plint

  3. Departments of Pediatrics and Emergency Medicine, University of Ottawa, Ottawa, Canada

    Amy C. Plint

  4. Children’s Hospital of Eastern Ontario Research Institute, Ottawa, Ontario, Canada

    Amy C. Plint

  5. Departments of Surgery and Paediatrics: Child and Youth Health, University of Auckland, Auckland, New Zealand

    Stuart R. Dalziel

  6. Children’s Emergency Department, Starship Children’s Hospital, Auckland, New Zealand

    Stuart R. Dalziel

  7. University of Manitoba, Winnipeg, Manitoba, Canada

    Terry P. Klassen

  8. Children’s Hospital Research Institute of Manitoba, Winnipeg, Manitoba, Canada

    Terry P. Klassen

  9. Institute of Health Policy, Management and Evaluation, University of Toronto, Toronto, Ontario, Canada

    Martin Offringa

  10. Division of Neonatology, The Hospital for Sick Children, University of Toronto, Toronto, Ontario, Canada

    Martin Offringa

  11. Division of Biostatistics, Dalla Lana School of Public Health, University of Toronto, Toronto, Ontario, Canada

    Anna Heath

  12. Department of Statistical Science, University College London, London, UK

    Anna Heath

Authors
  1. Jingxian Lan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Amy C. Plint
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Stuart R. Dalziel
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Terry P. Klassen
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Martin Offringa
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Anna Heath
    View author publications

    You can also search for this author in PubMed Google Scholar

Consortia

on behalf of the Pediatric Emergency Research Canada (PERC) KIDSCAN/PREDICT BIPED Study Group

Contributions

JL and AH designed the elicitation workshop. SRD, TPK, MO and ACP piloted the elicitation workshop and drafted the clinical case. JL, SRD, TPK, ACP and AH facilitated the elicitation workshop. JL analysed the data. JL and AH drafted the article. SRD, TPK, MO and ACP offered substantive revisions. All authors conceived the elicitation study and read, edited and approved the final manuscript.

Corresponding author

Correspondence to Anna Heath.

Ethics declarations

Ethics approval and consent to participate

We have obtained primary ethics approval from the Hospital for Sick Children Research Ethics Board with reference number: 1000075048. The elicitation study used implied consent in line with our ethics approval. No patients were involved.

Consent for publication

Not applicable

Competing interests

The authors declare that they have no competing interests. The authors alone are responsible for the writing and content of this article.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1:

Supplementary material

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lan, J., Plint, A.C., Dalziel, S.R. et al. Remote, real-time expert elicitation to determine the prior probability distribution for Bayesian sample size determination in international randomised controlled trials: Bronchiolitis in Infants Placebo Versus Epinephrine and Dexamethasone (BIPED) study. Trials 23, 279 (2022). https://doi.org/10.1186/s13063-022-06240-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s13063-022-06240-w

Keywords

Trials

ISSN: 1745-6215