Sample size determination for a binary response in a superiority clinical trial using a hybrid classical and Bayesian procedure

Abstract

Background

When designing studies that have a binary outcome as the primary endpoint, the hypothesized proportion of patients in each population experiencing the endpoint of interest (i.e., π ₁,π ₂) plays an important role in sample size and power calculations. Point estimates for π ₁ and π ₂ are often calculated using historical data. However, the uncertainty in these estimates is rarely addressed.

Methods

This paper presents a hybrid classical and Bayesian procedure that formally integrates prior information on the distributions of π ₁ and π ₂ into the study’s power calculation. Conditional expected power (CEP), which averages the traditional power curve using the prior distributions of π ₁ and π ₂ as the averaging weight conditional on the presence of a positive treatment effect (i.e., π ₂>π ₁), is used, and the sample size is found that equates the pre-specified frequentist power (1−β) and the conditional expected power of the trial.

Results

Notional scenarios are evaluated to compare the probability of achieving a target value of power with a trial design based on traditional power and a design based on CEP. We show that if there is uncertainty in the study parameters and a distribution of plausible values for π ₁ and π ₂, the performance of the CEP design is more consistent and robust than traditional designs based on point estimates for the study parameters. Traditional sample size calculations based on point estimates for the hypothesized study parameters tend to underestimate the required sample size needed to account for the uncertainty in the parameters. The greatest marginal benefit of the proposed method is achieved when the uncertainty in the parameters is not large.

Conclusions

Through this procedure, we are able to formally integrate prior information on the uncertainty and variability of the study parameters into the design of the study while maintaining a frequentist framework for the final analysis. Solving for the sample size that is necessary to achieve a high level of CEP given the available prior information helps protect against misspecification of hypothesized treatment effect and provides a substantiated estimate that forms the basis for discussion about the study’s feasibility during the design phase.

Background

When designing a study that has a binary outcome as the primary endpoint, the hypothesized proportion of patients in each population experiencing the endpoint of interest (i.e., π ₁,π ₂) plays an important role in sample size determination. In a two-arm study comparing two independent proportions, |π ₂−π ₁| represents the true hypothesized difference between groups, sometimes also known as the minimal relevant difference [1]. While the treatment effect may also be parameterized equivalently using an odds ratio or relative risk, when appropriate, the most frequently used sample size formula expresses the treatment effect using the difference between groups [2, 3]. In the case of proportions, the variance of the difference depends on the individual hypothesized values for the population parameters π ₁ and π ₂ under the alternative hypothesis. Thus, the sample size required to detect a particular difference of interest is affected by both the magnitude of the difference and the individual hypothesized values.

Traditional sample size formulas incorporate beliefs about π ₁ and π ₂ through single point estimates [1]. However, there is often uncertainty in these hypothesized proportions and, thus, a distribution of plausible values that should be considered when determining sample size. Misspecification of these hypothesized proportions in the sample size calculation may lead to an underpowered study, or one that has a low probability of detecting a smaller and potentially clinically relevant difference when such a difference exists [4]. Alternatively, if there is strong evidence in favor of a large difference, a study may be overpowered to detect a small hypothesized difference. Thus, a method for determining sample size that formally uses prior information on the distribution of study design parameters can mitigate the risk that the power calculation will be overly optimistic or overly conservative.

Similar difficulty surrounding the choice of study parameters for a continuous endpoint with known variance [5] and for a continuous endpoint with unknown variance [6] has been discussed previously. We have presented methods that formally incorporate the distribution of prior information on both the treatment effect and the variability of the endpoint into sample size determination. In this paper, we extend these methods to a binary endpoint by using a “hybrid classical and Bayesian” [7] technique based on conditional expected power (CEP) [8] to account for the uncertainty in study parameters π ₁ and π ₂ when determining the sample size of a superiority clinical trial. Unlike traditional power, which is calculated assuming the truth of a point alternative hypothesis (π ₂−π ₁=Δ _A ) for given values of π ₁ and π ₂, CEP conditions on the truth of a composite alternative of superiority (e.g., π ₂−π ₁>0 or π ₂>π ₁). CEP formally incorporates available prior information on both π ₁ and π ₂ into the power calculations by averaging the traditional power curve using the product of the prior distribution of π ₁ and the conditional prior distribution of π ₂,p(π ₂ | π ₂>π ₁), as the averaging weight. Based on the available prior information, the sample size that yields the desired level of CEP can be used when estimating the required sample size of the study.

While there has been much research in the area of Bayesian sample size determination [9–12], the hybrid classical and Bayesian method presented here aligns more with the ideas found in traditional frequentist sample size determination. Unlike traditional frequentist methods, however, we do not assume that the true parameters under the alternative hypothesis are known. This assumption rarely holds; typically, parameter values are estimated from early phase or pilot studies, studies of the intervention in different populations, or studies of similar agents in the current population [13, 14]. Thus, there is uncertainty surrounding the estimation of these population parameters and natural prior distributions of plausible values of these parameters that should be incorporated into the assessment of a trial’s power. Our method incorporates knowledge on the magnitude and uncertainty in the parameters into the traditional frequentist notion of power through explicit prior distributions on these unknown parameters to give CEP. As discussed in the “Methods” Section, CEP is not only well behaved, but it allows us to maintain a definition of power that intuitively converges to the traditional definition. Bayesian methodology is used only during the study design to allow prior information, through the prior distributions, to inform a choice for the sample size. Traditional type I and type II error rates, which have been accepted in practice, are maintained, and inferences are based on the likelihood of the data. The probability of achieving a target value of power using this method is compared to the performance of a traditional design. It is our hope that this formal method for incorporating prior knowledge into the study design will form the basis of thoughtful discussion about the feasibility of the study in order to reduce the number of poorly designed, underpowered studies that are conducted.

Methods

CEP for dichotomous outcome

Suppose that the study endpoint is dichotomous so that the probability (risk) of experiencing the event of interest in group 2 (the experimental treatment group), π ₂, is compared to that in group 1 (the control group), π ₁. The responses (i.e., the number of successes) in each group follow a binomial distribution. Assume that after n observations in each independent group or N=2n total observations, the two-sample Z-test of proportions is performed to test the null hypothesis H ₀:π ₂=π ₁ (i.e., π ₂−π ₁=Δ=0) versus the two-sided alternative hypothesis H ₁:π ₂≠π ₁ (i.e., π ₂−π ₁=Δ≠0), where π ₂>π ₁ indicates benefit of the experimental treatment over the control. The test is based on the test statistic T=p ₂−p ₁, or the difference in the proportion of successes in each sample. Under H ₀:π ₂=π ₁=π,T · ∼ N(0,σ ₀) in large samples, where σ ₀ is the standard deviation of the normal distribution. Assuming equal sample sizes n in each group gives $\sigma _{0} = \sqrt {2 \pi (1-\pi)/n }$, where π=(π ₁+π ₂)/2. In this setting, H ₀ is rejected at the α-level of significance if $|T| \geq z_{{}_{1-\alpha /2}} \, \hat {\sigma }_{0}$, where $\phantom {\dot {i}\!}z_{{}_{1-\alpha /2}}$ is the critical value for lower tail area 1−α/2 of the standard normal distribution and π is estimated by p=(p ₁+p ₂)/2 in $\hat {\sigma }_{0}$. A positive conclusion, D ₁, occurs if $Z = T/\hat {\sigma }_{0} \geq z_{{}_{1-\alpha /2}}$.

Under $H_{1}: \pi _{2}-\pi _{1} = \Delta _{A}, T \overset {\cdot }{\sim } N(\Delta _{A}, \sigma _{1})$, where $\sigma _{1} = \sqrt {(\pi _{2} (1-\pi _{2}) + \pi _{1} (1-\pi _{1}))/n}$. Thus, the traditional power of this test to detect the hypothesized difference corresponding to values of π ₁ and π ₂ under H ₁ is

$$ \begin{aligned} {}P(D_{1} \,|\, \pi_{1}, \pi_{2}) = \Phi\left[ \frac{\sqrt{N} \, |\pi_{2}-\pi_{1}| - 2 z_{{}_{1-\alpha/2}} \sqrt{\pi(1-\pi)}}{\sqrt{2 \pi_{2} (1-\pi_{2}) + 2 \pi_{1} (1-\pi_{1})}} \right] \!= 1-\beta, \end{aligned} $$

(1)

where Φ[ ·] is the standard normal cumulative distribution function. Since the traditional power curve is discontinuous at π ₂=π ₁ for a two-sided test, we assume a successful outcome or π ₂>π ₁ when calculating power; thus, |π ₂−π ₁|=π ₂−π ₁ in (1). One may plot the power function for fixed N and π ₁ over values of π ₂ or equivalently over values of π ₂−π ₁ to give the traditional power curve. Figure 1 shows the traditional power surfaces for N=48 and for N=80 with hypothesized values of π ₂=0.7 and π ₁=0.3. Power curves for fixed π ₂=0.7 and variable π ₁ and for fixed π ₁=0.3 and variable π ₂ are highlighted. Sample size is chosen to give high traditional power (e.g., 0.80≤1−β≤0.90) to detect an effect at least as large as the hypothesized difference for π ₂ and π ₁ by solving (1) for N [2]:

$$ \begin{aligned} {}N = \left[\frac{2 z_{{}_{1-\alpha/2}} \sqrt{\pi(1-\pi)} + z_{{}_{1-\beta}} \sqrt{2 \pi_{2} (1-\pi_{2}) + 2 \pi_{1} (1-\pi_{1})}}{\pi_{2}-\pi_{1}} \right]^{2}. \end{aligned} $$

(2)

Fig. 1

Traditional power surfaces when hypothesized values of π ₁=0.3 and π ₂=0.7 for N=48 and N=80

The traditional power curve does not account for the uncertainty associated with the unknown population parameters π ₂ and π ₁ and does not indicate if the planned sample size is adequate given this uncertainty. Average or expected power (EP) was developed as a way to use the distribution of prior beliefs about the unknown parameters to provide an overall predictive probability of a positive conclusion [8, 9, 15–24]. EP, also known as assurance [20], probability of study success [23], or Bayesian predictive power [24], averages the traditional power curve using the prior distributions for the unknown parameters to weight the average without restricting the prior distributions to assume treatment superiority. In the case of a binomial response, assuming π ₁ and π ₂ are independent yields a special case of the general multivariate formulation which allows the joint distribution p(π ₁,π ₂) to be factored into the product of the two prior distributions p(π ₁) and p(π ₂). Thus, the traditional power curve P(D ₁ | π ₂,π ₁) is averaged using the product of the prior distributions for π ₂ and π ₁,p(π ₂) and p(π ₁), respectively, as the averaging weight [8], which gives the following formulation for EP:

$$P(D_{1}) = \int\limits_{\pi_{1}} \int\limits_{\pi_{2}} \, P(D_{1} \,|\, \pi_{1}, \pi_{2}) \, p(\pi_{2}) \, p(\pi_{1}) \, d\pi_{2} \, d\pi_{1}. $$

Expected power conditional on the experimental treatment’s superiority, π ₂>π ₁, is known as conditional expected power (CEP) [8]. Unlike EP, CEP is found by using the conditional prior distribution for π ₂,p(π ₂ | π ₂>π ₁), in the averaging weight. Since this conditional prior is now dependent on π ₁ and equals zero when π ₂≤π ₁, to ensure integration to 1 when P(π ₁>π ₂)>0, the conditional prior is scaled by the normalization factor P(π ₂>π ₁)⁻¹, or the inverse probability of the experimental treatment’s superiority. This gives the following formulation for CEP:

$$ \begin{aligned} {}P(D_{1} | \pi_{2} > \pi_{1}) &\,=\, \int\limits_{\pi_{1}} \int\limits_{\pi_{2}>\pi_{1}} \!P(D_{1} | \pi_{1}, \pi_{2}) p(\pi_{2} | \pi_{2}>\pi_{1}) p(\pi_{1}) \, d\pi_{2} \, d\pi_{1}\\[-3pt] &=\! \frac{1}{P(\pi_{2}\!\!>\!\!\pi_{1}\!)}\!\! \int\limits_{\pi_{1}=0}^{1} \int\limits_{\pi_{2}=\pi_{1}}^{1}\! \! \!\!P(D_{1} | \pi_{1}\!, \pi_{\!2}) p(\pi_{\!2}) p(\pi_{\!1}) d\pi_{\!2} d\pi_{1} \!, \end{aligned} $$

(3)

where

$$\begin{array}{*{20}l} &P(\pi_{2}>\pi_{1})=\int \limits_{\pi_{1}=0}^{1} \int \limits_{\pi_{2}=\pi_{1}}^{1} p\left(\pi_{1}\right) p\left(\pi_{2} \right) d\pi_{2} d\pi_{1}. \end{array} $$

(4)

The unconditional prior distributions p(π ₁) and p(π ₂) are defined such that π ₁∉ [0,1]⇒p(π ₁)=0 and π ₂∉ [ 0,1]⇒p(π ₂)=0 (e.g., beta or uniform(0,1) distributions).

Combining (1) and (3) gives the following equation for CEP:

$$ \begin{aligned} &{}P(D_{1} |\pi_{2} \!>\! \pi_{1}) \,=\, \frac{1}{P(\pi_{2}\!>\!\pi_{1})}\! \int\limits_{\pi_{1}} \int\limits_{\pi_{2}>\pi_{1}} \, \\[-2pt] &{}\times\!\Phi\!\left[\! \frac{\sqrt{N} \, (\pi_{2}\,-\,\pi_{1}) \,-\, 2 z_{{}_{1-\alpha/2}} \sqrt{\pi(1\,-\,\pi)}}{\sqrt{2 \pi_{2} (1\,-\,\pi_{2}) \,+\, 2 \pi_{1} (1\,-\,\pi_{1})}} \!\right] \!p(\pi_{\!2}) p(\pi_{\!1}) d\pi_{2} d\pi_{\!1}\!. \end{aligned} $$

(5)

Note, any appropriate sample size and power formulas may be used to evaluate CEP in (5). For example, continuity-corrected versions of (2) or the arcsine approximation [25, 26] may alternatively be utilized instead of (2) to determine sample size, while related power formulas may be used instead of (1) for CEP calculations.

To evaluate CEP under a proposed design, find N in (2) for the hypothesized values of π ₁ and π ₂, significance level α, and traditional power level 1−β. Numerical integration may then be used to evaluate CEP (5) for the assumed prior distributions p(π ₁) and p(π ₂). If CEP for the proposed design is less than 1−β, the study is expected to be underpowered under the treatment superiority assumption, and if the CEP is greater than 1−β, the study is expected to be overpowered. To ensure that the study is expected to be appropriately powered under the treatment superiority assumption, an iterative search procedure can be used to find the value of the sample size N in (5) that gives CEP equal to the threshold of traditional power 1−β. The value of N that achieves this desired level is denoted N ^∗. As in traditional power, we would like the probability of detecting a difference when a positive difference exists to be high (i.e., 0.80≤1−β≤0.90). Pseudo-code 1 outlines the steps for this process.

If the prior distributions put all their mass at a single positive point, essentially becoming a traditional point alternative hypothesis, EP and CEP reduce to the traditional formulation of power. However, for prior distributions where P(π ₁>π ₂)>0, CEP will be greater than EP, with CEP approaching 1 and EP approaching P(π ₂>π ₁) as N→∞:

$$\begin{aligned} &{}{\text{If }\pi_{2}\!<\!\pi_{1}\!,}\!{\; {\lim}_{{N}\to \infty}\!\Phi\!\! \left[\! \frac{\sqrt{N} \!(\pi_{2}\,-\,\pi_{1}) \,-\, 2 z_{{}_{1-\alpha/2}} \sqrt{\!\pi(1\,-\,\pi)}}{\!\sqrt{2 \pi_{2} (\!1\,-\,\pi_{\!2}) \,+\, 2 \pi_{\!1} (\!1\,-\,\pi_{1}\!)}} \!\right]}{\,=\,\!{\lim}_{z\to -\infty} \!\Phi\!\left[z\right]}{}{=\!0}{} \\ &{}{\text{If}\,\pi_{2}>\!\!\pi_{1}\!,}\!{\; {\lim}_{{N}\to \infty}\!\Phi\!\! \left[ \!\frac{\sqrt{N} (\pi_{2}\,-\,\pi_{1})\! -\! 2 z_{{}_{1-\alpha/2}} \sqrt{\!\pi\!(\!1\,-\,\pi\!)}}{\sqrt{\!2 \pi_{2} (\!1\,-\,\pi_{\!2}) \,+\, 2 \pi_{\!1} \!(\!1\,-\,\pi_{\!1}\!)}} \right]}{={\lim}_{z\to \infty} \Phi\left[z\right]}{}{=1}{} \end{aligned} $$

$$\begin{aligned} &{}{\implies} \!\!\! \!{\lim}_{N \to \infty} \!P(D_{1}\!)\! =\!\! {\lim}_{\!N \!\to\! \infty}\int\limits_{\pi_{1}} \int\limits_{\!\pi_{2}<\pi_{1}} \!\! \!\!\Phi\!\!\left[\! \frac{\!\sqrt{\!N} \! (\pi_{2}\,-\,\pi_{1}) \,-\, 2 z_{{}_{1-\alpha/2}} \!\sqrt{\!\pi(1\,-\,\pi\!)}}{\sqrt{2 \pi_{2} (1\,-\,\pi_{2})\! +\! 2 \pi_{1} \!(1\,-\,\pi_{1})}} \!\right] \!p(\!\pi_{2}\!) p(\pi_{1}\!) \, \!d\pi_{2} \, \!d\pi_{1}\\ &\qquad{+\! {\lim}_{N \to \infty} \int\limits_{\pi_{1}} \int\limits_{\pi_{2} \!>\! \pi_{1}} \!\! \!\Phi\!\!\left[\! \frac{\!\sqrt{N} \! (\pi_{2}\,-\,\pi_{1}\!) \,-\, 2 z_{{}_{1-\alpha/2}} \sqrt{\!\pi\!(1\,-\,\pi\!)}}{\sqrt{2 \pi_{2} (\!1\,-\,\pi_{\!2}\!) \,+\, 2 \pi_{\!1} (1\,-\,\pi_{\!1})}} \right]\! p(\pi_{2}) \, p(\pi_{1}) \, d\pi_{2} \, d\pi_{1}} &\\ &\qquad{= P(\pi_{2}>\pi_{1})} & \end{aligned} $$

When there is no doubt of a beneficial effect (i.e., P(π ₂>π ₁)=1), CEP equals EP.

Previous work in this area almost exclusively uses expected power P(D ₁) to account for uncertainty in study design parameters [8, 9, 15–24], and finds the sample size that gives the desired level of P(D ₁). Our preference for using CEP as opposed to EP to inform the design of a study is twofold. EP gives the predictive probability of a positive conclusion, regardless of the truth of the alternative hypothesis. CEP, however, is conceptually analogous to traditional power in that it is conditional on the truth of the benefit of the experimental treatment, which provides a more familiar framework for setting the desired level of CEP for a study. Secondly, if P(π ₁>π ₂)>0, then EP will not approach 1 as the sample size goes to infinity because ${\lim }_{N\to \infty } P(D_{1})=1-P(\pi _{1}>\pi _{2})$. CEP, however, is conditioned on π ₂>π ₁, so it approaches 1 as the sample size increases since ${\lim }_{N\to \infty } P(D_{1} \,|\, \pi _{2} > \pi _{1}) = \frac {1-P(\pi _{1}>\pi _{2})}{P(\pi _{2}>\pi _{1})}=1$. Thus, CEP is also more mathematically analogous to traditional power in that the probability of correctly reaching a positive conclusion is assured as the sample size goes to infinity.

Prior distributions

The prior distributions p(π ₁) and p(π ₂) reflect the current knowledge about the response rate in each treatment group before the trial is conducted. In the design phase of a clinical trial, a review of the literature is often performed. This collection of prior evidence forms a natural foundation for specifying the prior distributions. Historical data are commonly pooled using traditional meta-analysis techniques to calculate an overall point estimate [27, 28]; however, a Bayesian random-effects meta-analysis [29–31] may be more appropriate when the goal is to hypothesize a prior distribution. The priors can also incorporate the pre-trial consensus of experts in the field [9] or Phase II trial data [22]. Translating and combining prior evidence and opinions to form a prior distribution is often hailed as the most challenging part of using a Bayesian framework [7], and several works [32–35] describe techniques for eliciting a prior distribution.

A beta distribution, which is defined on the interval [ 0,1], can be used to describe initial beliefs about the parameters π ₁ and π ₂. If π _j ∼Beta(a,b), then

$$ p(\pi_{j}) = \frac{\Gamma(a+b)}{\Gamma(a) \, \Gamma(b)} \pi_{j}^{a-1} \, (1-\pi_{j})^{b-1} $$

where shape parameters a>0 and b>0. The mean, variance, and mode of the prior distribution are given by: μ=a/(a+b),τ ²=ab/((a+b)²(a+b+1)), and m=(a−1)/(a+b−2) for a,b>1, respectively. For fixed μ, larger values of a and b decrease τ ². One may choose the shape parameters a and b by fixing the mean and variance of the distribution at fixed values μ and τ ², which yields a=μ ²(1−μ)/τ ²−μ and b=a(1−μ)/μ. For skewed distributions, one may wish to describe central tendency using the mode m rather than the mean. Under a traditional design, the difference in modes, m ₂−m ₁, is a natural estimate for the hypothesized difference in proportions. When fixing m and τ ², the corresponding value of b may be found by solving the general cubic equation Ab ³+Bb ²+Cb+D=0, with coefficients

$$\begin{aligned} &{}A=-\frac{m^{3}}{(m-1)^{3}}+\frac{3m^{2}}{(m-1)^{2}}-\frac{3m}{(m-1)}+1\\ &{}B=\frac{6m^{3}-3m^{2}}{(m-1)^{3}}+\frac{-11m^{2}+6m}{(m-1)^{2}}+\frac{4m+\frac{m}{\tau^{2}}-3}{(m-1)}+1\\ &{}C=\frac{-12m^{3}+12m^{2}-3m}{(m-1)^{3}}+\frac{8m^{2}-10m+3}{(m-1)^{2}}-\frac{4m-2+\frac{1-2m}{\tau^{2}}}{(m-1)}\\ &{}D=\frac{8m^{3}-12m^{2}+6m-1}{(m-1)^{3}}+\frac{4m^{2}-4m+1}{(m-1)^{2}}. \end{aligned} $$

The corresponding value of a is given by $a=\frac {2m-mb-1}{m-1}$. (Table 2 in the Appendix reports the values of a and b for given m and τ ².) Notice that for a given variance τ ², the value of a when the mode =m equals the value of b when the mode =1−m. Thus, when m=0.5,a=b.

A uniform prior distribution may also be assumed for π _j with limits within the interval [ 0,1]. The uniform prior has lower bound a and upper bound b, or π _j ∼U(a,b), and is constant over the range [ a,b]. The prior is centered at μ=(a+b)/2 with variance τ ²=(b−a)²/12. The non-informative prior distribution that assumes no values of π _j are more probable than any others is U(0,1)≡Beta(1,1). One may also restrict the range of the uniform distribution to focus on smaller ranges for π ₁ and π ₂. Rather than setting the lower and upper bounds of the uniform, one may set the mean μ<1 and variance $\tau ^{2} < \frac {\min (\mu ^{2}, (1-\mu)^{2})}{3}$ of the prior distribution, which gives lower bound $a = \mu - \sqrt {3 \, \tau ^{2}}$ and upper bound $b = \mu + \sqrt {3 \, \tau ^{2}}$. Again, under a traditional design, the difference in means μ ₂−μ ₁ is a natural estimate for the hypothesized difference in proportions when presented with uniform prior evidence. (Table 3 in the Appendix reports the values of a and b for given μ and τ ².) Notice that restrictions exist for the variances assumed for certain means to maintain bounds between [ 0,1].

Results

The procedures described in the “Methods” Section were applied to a set of notional scenarios to compare traditionally designed studies to those designed using CEP. The integration step of Pseudo-code 1 was approximated using Riemann sums with step size 0.0001.

An example scenario assumed beta-distributed priors for π ₁ and π ₂, such that π ₁∼Beta(6.62,14.11) and π ₂∼Beta(14.11,6.62). For this scenario, a traditionally designed study would select a sample size of N=48 based on (2) to achieve 80% power and a two-sided type I error of 5%, with hypothesized values of π ₁=mode(Beta(6.62,14.11))=0.3 and π ₂=mode(Beta(14.11,6,62))=0.7. However, based on the assumed prior distributions, a study with a sample size of 48 could achieve less than 80% power when π ₁≠0.3 or π ₂≠0.7. In fact, based on (5), the study with sample size N=48 would give CEP=67.8%. Figure 2 a displays the joint distribution of π ₁ and π ₂, conditional on π ₂>π ₁, and highlights the region where power would be less than 80% under a traditional design when the sample size is N=48. For this scenario, the study with sample size N=48 would achieve power less than the target value in more than 56% of instances when π ₂>π ₁.

Fig. 2

Conditional joint prior density p(π ₁)p(π ₂|π ₂>π ₁) for π ₁∼Beta(6.62,14.11) and π ₂∼Beta(14.11,6.62). a Highlighting region where power <80% under a traditional design. b Highlighting region where power <80% under a CEP design

For the same scenario, a CEP-designed study would select a sample size of N ^∗=80 based on Pseudo-code 1 to achieve 80% CEP with a two-sided type I error of 5%. Figure 2 b displays the joint distribution of π ₁ and π ₂, conditional on π ₂>π ₁, and highlights the region where power would be less than 80% under a CEP design when the sample size is N ^∗=80. For this scenario, the study with sample size N ^∗=80 would achieve power less than the target value in approximately 33% of instances when π ₂>π ₁. Note that the intersection of the two regions corresponds to values of π ₁ and π ₂ that give power from (1) equal to 80% with sample size N=80.

The probability of achieving power at least equal to the target value, conditional on the experimental treatment’s superiority (π ₂>π ₁), is here termed the performance of the design. While CEP provides a point estimate of power under the treatment superiority assumption, performance indicates how robust the design is. The performance of the design is given by:

$$ \begin{aligned} &{}\text{Performance}=\! \frac{1}{P(\!\pi_{2}\!\!>\!\!\pi_{1}\!)} \!\int\limits_{\pi_{1}} \int\limits_{\!\pi_{2}>\pi_{1}} \!\!\! \textsf{\!F}\!\left(N\!,\!\pi_{\!1}\!,\pi_{\!2}\!,z_{{}_{1-\alpha/2}}\!\right) \!p(\pi_{\!2}) \, p(\pi_{\!1}) d\pi_{2} d\pi_{1}\!, \end{aligned} $$

(6)

where

$$\begin{aligned} &{}\textsf{F}\left(N\!,\!\pi_{1\!},\pi_{2}\!,z_{{}_{\!1-\alpha/2}}\right)\,=\, \begin{cases} \!1 & \text{\!\!\!if }\!\Phi\!\left[\! \!\frac{\sqrt{N} \, \!(\pi_{2}\,-\,\pi_{1}) \,-\, 2 z_{{}_{1-\alpha/2}} \sqrt{\!\pi(1\!\,-\,\pi)}}{\!\sqrt{\!2 \pi_{2} (1\,-\,\pi_{2}) \,+\, 2 \pi_{1} (1\,-\,\pi_{1}\!)}}\! \right]\! \!\geq\!\! 1\,-\,\beta\!,\\ \!0 & \text{\!\!\!otherwise} \end{cases} \end{aligned} $$

Thus, the traditionally designed study from the example scenario produced a performance of (100−56)%=44%, while the CEP design, which explicitly accounts for uncertainty, produced a more robust performance of (100−33)%=67%. However, this increase in performance required an increase in sample size from N=48 to N ^∗=80. The increase in performance divided by the increase in sample size is here termed the marginal benefit for the scenario due to CEP. The marginal benefit for the example scenario due to CEP is given by (67−44)%/(80−48)=0.71%. If there is no uncertainty in the design parameters, then there would be no marginal benefit due to CEP, since the probability of achieving less than the target power would be assumed 0 for a traditionally designed study and the CEP-designed study would give N ^∗=N. On the other hand, if the uncertainty in the design parameters is very large, the marginal benefit may approach 0, since the CEP-designed study could give N ^∗>>N with limited increase in performance. This is important to consider, since a very small marginal benefit could make it impractical to achieve a desired value for CEP or a desired threshold of performance.

Since the performance and marginal benefit result from the prior distributions of π ₁ and π ₂, several notional scenarios were evaluated to explore the relationship between prior distributions, CEP, and performance. Tables 4, 5 and 6 in the Appendix display the results of several scenarios that assumed Beta-distributed priors for π ₁ and π ₂. The mode and variance of p(π _j ),j=1,2, are denoted m _j and $\tau ^{2}_{j}$, respectively. The procedure for generating the results from Table 4 in the Appendix, for which $\tau ^{2}_{1}=\tau ^{2}_{2}$, is given below:

• 1. The modes, m ₁ and m ₂, and variances, $\tau ^{2}_{1}=\tau ^{2}_{2}$, were used to hypothesize a beta prior distribution for π ₁ and π ₂, respectively.

• 2. For each pair of prior distributions (p(π ₁),p(π ₂)) considered:

  • (a) Traditional sample size is found using (2) by setting the hypothesized values of π ₁ and π ₂ equal to the mode of each prior, m ₁ and m ₂, respectively. Two-sided type I error α=0.05 and traditional power 1−β=0.80 are assumed. Traditional sample size is denoted $\hat {N}$. If $\hat {N}$ is odd, the sample size is increased by 1 to provide equal sample size for both groups.

  • (b) The CEP of the traditional design is found using (5), with $N=\hat {N}$, two-sided α=0.05, and 1−β=0.80.

  • (c) The performance of the traditional design is found using (6), with $N=\hat {N}$, two-sided α=0.05, and 1−β=0.80.

  • (d) The smallest sample size for which CEP evaluates to ≥1−β is found using Pseudo-Code 1 and is denoted N ^∗. If N ^∗ is odd, the sample size is increased by 1 to provide equal sample size for both groups.

  • (e) The probability of a positive treatment effect, P(π ₂>π ₁), is found using (4) with Riemann sum integral approximations.

  • (f) The conditional expected difference, E(π ₂−π ₁|π ₂>π ₁), is found using Riemann sum integral approximations of

    $$\begin{aligned} {}E(\pi_{2} \!- \!\pi_{1}|\pi_{2} \!\!>\!\! \pi_{1})\,=\,\frac{1}{P(\pi_{2}\!\!>\!\!\pi_{1})}\!\! \int \limits_{\pi_{1}=0}^{1} \int \limits_{\pi_{2}=\pi_{1}}^{1} \!\!(\pi_{2}\,-\,\pi_{1}) p\!\left(\pi_{1}\right) \!p\!\left(\pi_{2} \right) \!d\pi_{2} d\pi_{1}. \end{aligned} $$

  • (g) The performance of the CEP design is found using (6), with N=N ^∗, two-sided α=0.05, and 1−β=0.80.

  • (h) The marginal benefit due to CEP for the scenario is found by dividing the difference between the CEP design performance and the traditional design performance by the difference between the CEP sample size and the traditional sample size, $N^{*}-\hat {N}$.

Table 4 in the Appendix shows that when m ₂−m ₁>1/3, the performance of the traditional design decreases as $\tau _{1}^{2}=\tau _{2}^{2}$ increases. This is explained by the fact that the conditional expected difference is less than the hypothesized difference that was used in the traditional design sample size calculation. This occurs for m ₂−m ₁>1/3 since both prior distributions are approaching U(0,1) as $\tau _{1}^{2}=\tau _{2}^{2}$ increases, and E(π ₂−π ₁|π ₂>π ₁)=1/3 for π ₁,π ₂∼U(0,1). Thus, when m ₂−m ₁<1/3, the performance of the traditional design increases as $\tau _{1}^{2}=\tau _{2}^{2}$ increases since the hypothesized difference is less than the limit of the conditional expected difference. When m ₂−m ₁ is smaller than E(π ₂−π ₁|π ₂>π ₁), CEP will be high for a traditional design with hypothesized difference m ₂−m ₁, since it is designed to detect a difference smaller than the expected difference.

The procedure was also applied to scenarios where $\tau _{1}^{2} = 0.001$ and $\tau _{2}^{2} > 0.001$ (Table 5 in the Appendix) and scenarios where $\tau _{1}^{2} = 0.08$ and $\tau _{2}^{2} < 0.08$ (Table 6 in the Appendix), corresponding to small and large uncertainty, respectively, in the proportion experiencing the outcome in the control group. Table 5 in the Appendix shows that the performance of the traditional design is similar to the performance seen in Table 4 in the Appendix. However, when $\tau _{1}^{2}$ is fixed at 0.001,E(π ₂−π ₁|π ₂>π ₁) begins near m ₂−m ₁ and approaches (1−m ₁)/2 as $\tau _{2}^{2}$ increases because p(π ₂|π ₂>π ₁) is approaching U(m ₁,1). Thus, when m ₂−m ₁>(1−m ₁)/2, the performance of the traditional design decreases as $\tau _{2}^{2}$ increases, and when m ₂−m ₁<(1−m ₁)/2, the performance of the traditional design increases as $\tau _{2}^{2}$ increases.

When $\tau _{1}^{2}$ is fixed at 0.08,E(π ₂−π ₁|π ₂>π ₁) approaches 1/3 from m ₂/2. If E(π ₂−π ₁|π ₂>π ₁) is increasing towards 1/3 as $\tau _{2}^{2}$ increases, then the performance of the traditional design will increase. If E(π ₂−π ₁|π ₂>π ₁) decreases towards 1/3 as $\tau _{2}^{2}$ increases, then the performance of the traditional design will decrease. If m ₂/2>1/3, then the performance of the traditional design will decrease as $\tau _{2}^{2}$ increases. This happens because, as $\tau _{2}^{2}$ increases, the hypothesized difference is decreasing from m ₂/2 to 1/3. The behavior of the traditional design is summarized in Table 1.

Table 1 Analysis of traditional design performance

Excursions with uniform priors were performed. Table 7 in the Appendix shows that the performance of a traditional design under a uniform prior is similar to the performance observed in Table 4 in the Appendix. However, fewer trends are visible because the parameters of the uniform distribution are more restricted than the parameters of the beta distribution.

As expected, the performance of the CEP design changes minimally as $\tau _{1}^{2}=\tau _{2}^{2}$ increases, since N ^∗ is chosen to explicitly account for changes in $\tau _{1}^{2}=\tau _{2}^{2}$. Note, N ^∗ is directly tied to E(π ₂−π ₁|π ₂>π ₁): N ^∗ increases as the conditional expected difference decreases, and N ^∗ decreases as the conditional expected difference increases. This occurs because increasing the variability can increase the conditional expected difference if the resulting conditional priors give more relative weight to larger differences and less relative weight to smaller differences compared to the unconditional priors. This is more likely to occur when m ₁ is large, since increasing the variability when m ₁ is large will make smaller values of π ₁ more likely due to the condition that π ₂>π ₁. Similarly, when m ₂ is small, larger values of π ₂ are more likely under the assumption that π ₂>π ₁.

The marginal benefit due to CEP is greatest for small values of $\tau _{1}^{2}=\tau _{2}^{2}$. This is so because the relative difference between $\hat {N}$ and N ^∗ is smallest when the uncertainty is low (i.e., when the traditional assumptions closely approximate the CEP assumptions). However, the marginal benefit due to CEP decreases minimally or remains constant as the uncertainty increases because the difference in performance is always less than 1, while the difference in sample size, $N^{*}-\hat {N}$, can be greater than 200 in some cases. Furthermore, as $\tau _{1}^{2}=\tau _{2}^{2}$ increases, the performance of the traditional design can improve even though $\hat {N}$ remains constant, while N ^∗ may have to increase to maintain the performance of the CEP design.

When $\tau _{1}^{2}$ is fixed at 0.001, the performance of the CEP design remains stable at approximately 0.7. However, the marginal benefit is greater with fixed, low uncertainty in π ₁ compared with the changing uncertainty in Table 4 in the Appendix. The sample size required to achieve CEP of 1−β with fixed $\tau _{1}^{2}$ is reduced compared to scenarios with changing $\tau _{1}^{2}$. This is because uncertainty in the control group is small, which indicates that reducing the uncertainty in the control parameter can increase the benefit of CEP to the study.

When $\tau _{1}^{2}$ is fixed at 0.08, the performance of the CEP design remains stable at approximately 0.71. However, the marginal benefit is very small because N ^∗ is always greater than that in Table 4 or Table 5 in the Appendix due to the larger uncertainty in π ₁. Again, this demonstrates that it is beneficial to minimize the uncertainty in π ₁ to increase the marginal benefit.

Note that for small differences in m ₂−m ₁ and any large variance, the CEP design can reduce the sample size from the value determined from a traditional design. The reason is that increased uncertainty under the treatment superiority assumption increases the likelihood of differences greater than m ₂−m ₁.

Discussion

Many underpowered clinical trials are conducted with limited justification for the chosen study parameters used to determine the required sample size [36, 37] with scientific, economic, and ethical implications [36, 38]. While sample size calculations based on traditional power assume no uncertainty in the study parameters, the hybrid classical and Bayesian procedure presented here formally accounts for the uncertainty in the study parameters by incorporating the prior distributions for π ₁ and π ₂ into the calculation of conditional expected power (CEP). This method allows available evidence on both the magnitude and the variability surrounding the parameters to play a formal role in determining study power and sample size.

In this paper, we explored several notional scenarios to compare the performance of the CEP design to that of a design based on traditional power. We show that if there is uncertainty in the study parameters and a distribution of plausible values for π ₁ and π ₂, the performance of the CEP design is more consistent and robust than that of traditional designs based on point estimates for the study parameters. Traditional sample size calculations based on point estimates for the hypothesized study parameters tend to underestimate the required sample size needed to account for the uncertainty in the parameters.

The scenarios demonstrate that reducing uncertainty in the control parameter π ₁ can lead to greater benefit from the CEP-designed study, because the relative difference between $\hat {N}$ and N ^∗ is smallest when uncertainty is low. Therefore, it is worthwhile to use historical information to reduce the variability in the control group proportion rather than focusing only on the prior for the experimental treatment group. Nonetheless, when there is significant overlap between the prior distributions and a small hypothesized difference m ₂−m ₁, traditional study designs can be overpowered under the treatment superiority assumption compared to the CEP design, and the CEP design would result in a smaller sample size. This happens because increased uncertainty under the treatment superiority assumption increases the relative likelihood of differences greater than m ₂−m ₁.

In the scenarios we evaluated, the performance of the traditional design was highly dependent on the prior distributions but exhibited predictable behavior. The CEP design, however, consistently generated performance near 70% across all scenarios. This indicates that power greater than the target 1−β would not be uncommon for a CEP design. This begs the question of whether or not 1−β is an appropriate target for CEP, since it could apparently lead to overpowered studies. To avoid this issue, one may use a lower target for CEP or instead design the study using a target value of performance and use our iterative N ^∗ search to find the design that achieves acceptable performance.

Additionally, when comparing the method based on CEP to similar methods based on expected power, the sample size from a CEP design will always be less than or equal to the sample size required to achieve equivalent EP. While pure Bayesian methods of sample size determination that compute prior effective sample size to count the information contained in the prior towards the current study will generally yield a smaller sample size than traditional frequentist methods [10], the method presented here does not assume that prior information will be incorporated into the final analysis.

Conclusions

The hybrid classical and Bayesian procedure presented here integrates available prior information about the study design parameters into the calculation of study sample size for a binary endpoint. This method allows prior information on both the magnitude and uncertainty surrounding the parameters π ₁ and π ₂ to inform the design of the current study through the use of conditional expected power. When there is a distribution of plausible study parameters, the design based on conditional expected power tends to outperform the traditional design. Note that if the determined sample size N ^∗ is greater than what can be feasibly recruited in the proposed trial, this may indicate excessive uncertainty about the study parameters and should encourage serious discussion concerning the advisability of the study. Thus, we do not recommend that N ^∗ be blindly used as the final study sample size, but we hope that this method encourages a careful synthesis of the prior information and motivates thoughtful discussion about the feasibility of the study in order to reduce the number of poorly designed, underpowered studies that are conducted.

Appendix

Table 2 presents the values of shape paramaters [a, b] for given m and τ ² for the beta distribution. Table 3 reports the values of minimum and maximum parameters [ a,b] for given μ and τ ² for the uniform distribution.

Table 2 Shape parameters [ a,b] of beta distribution given mode m and variance τ ²

Table 3 Minimum and maximum parameters [ a,b] of uniform distribution given mean μ and variance τ ²

Table 4 Sample scenarios assuming beta priors p(π ₁) and p(π ₂) where $\tau _{1}^{2}=\tau _{2}^{2}$. Hypothesized values of π ₁ and π ₂ set equal to m ₁ and m ₂, respectively, under the traditional design. Two-sided α=0.05 and 1−β=0.80 assumed

Table 5 Sample scenarios assuming beta priors p(π ₁) and p(π ₂) where $\tau _{1}^{2}=0.001$. Hypothesized values of π ₁ and π ₂ set equal to m ₁ and m ₂, respectively, under the traditional design. Two-sided α=0.05 and 1−β=0.80 assumed

Table 6 Sample scenarios assuming beta priors p(π ₁) and p(π ₂) where $\tau _{1}^{2}=0.08$. Hypothesized values of π ₁ and π ₂ set equal to m ₁ and m ₂, respectively, under the traditional design. Two-sided α=0.05 and 1−β=0.80 assumed

Table 7 Sample scenarios assuming uniform priors p(π ₁) and p(π ₂) where $\tau _{1}^{2}=\tau _{2}^{2}$. Hypothesized values of π ₁ and π ₂ set equal to μ ₁ and μ ₂, respectively, under the traditional design. Two-sided α=0.05 and 1−β=0.80 assumed