Table 1 A suggested strategy for measuring and addressing heterogeneity when planning a future study to update a network meta-analysis

1. Measure and characterize heterogeneity. Latest methodological developments encourage meta-analysts to estimate the heterogeneity variance which measures the variability of the true treatment effect and compare it to its expected value. Within the context of planning new studies, existing heterogeneity is large when even a new study with several thousands of participants fails to produce effect sizes that are precise enough to enable decision-making. Consequently, characterizing heterogeneity as large or small depends on the context. We recommend that when the heterogeneity variance is below the expected, sample size calculations are carried out; if the estimated sample size is unrealistically large, then heterogeneity is clearly too large to proceed with conditional trial design.

2. Understand heterogeneity. Identify potential effect modifiers and explore the changes of the treatment effect in various subgroup analyses using study-level and patient-level characteristics (such as risk of bias, trial setting). The impact of continuous characteristics (such as trial duration or sample size) can be explored using meta-regression models. Interpret with caution any findings from patient-level covariates as they can be subject to ecological bias.

3. Reduce heterogeneity. If some variables are associated with heterogeneity, then consider the treatment effects about a particular group of patients and trial settings. For example, investigators might choose to restrict the analysis in low risk of bias or recent studies or estimate the treatment effect for a very large study (from a meta-regression on sample size). Any post hoc decisions about the analyzed dataset need to be clearly documented to avoid selection bias.

4. What to do in the presence of large, unexplained heterogeneity. If the heterogeneity remains substantial even after efforts to pinpoint its source and reduce it, then investigators have the following options: (1) collect individual-patient data to better investigate the role of patient characteristics in modifying the treatment effect and go to step 2 above; (2) develop assumptions about what is possibly causing the heterogeneity and plan a new exploratory study to investigate these assumptions. For example, if investigators believe that treatment dose is associated with differential effects (and this information is scarcely reported in the included studies) a new multi-arm study can be planned to investigate this assumption.