Table 2 Summary of simulation results
Measure | Continuous outcomes | Binary outcomes |
|---|---|---|
Inability to perform analysis | Rarely occurs for the estimation of the unadjusted mean difference or for conducting a t-test, even for trials including as few as 100 participants. | Estimation of odds ratio and calculation of the chi-squared test are frequently prohibited at small sample sizes due to zero events in at least one arm, unless incidence of intermediate response is high. |
Bias | Not materially impacted by sample size, nor by the size of the treatment effect on outcome. Bias increases with strength of unmeasured confounding between intermediate and outcome, although not substantial for typical treatment effects on the intermediate response variable, unless there is an interaction between this effect and confounding variables. Bias is reduced when the incidence of the intermediate response variable increases. | Substantial impact on bias for small trial sizes, unless incidence of intermediate response is high. Impacted by the size of treatment effect on the outcome. At larger sample sizes, bias is impacted by the magnitude of confounding between intermediate and outcome but is not substantial. |
Coverage | Close to the nominal level, unless there is strong confounding between the intermediate and outcome variables, in which case coverage may be related to treatment effects on the intermediate, particularly for larger sample sizes. | Coverage is generally too high for smaller sample sizes due to missing data (model SE exceeds empirical SE of the computable estimates), unless incidence of intermediate response variable is high. |
Type 1 error | Notably affected when there is an interaction between treatment effect on the intermediate response variable and confounding variables, for large sample size. Affected when confounding between intermediate response variable and confounding is strong, particularly for larger treatment effects on the intermediate. | Chi-squared tests perform poorly at smaller trial sizes but are close to a nominal level for larger trial sizes. Fisher’s test performs consistently poorly. |