Volume 16 Supplement 2

3rd International Clinical Trials Methodology Conference

Open Access

A new large-scale meta-epidemiological study on bias in randomized trials using routinely collected risk-of-bias assessments by cochrane reviewers: results from the robes study

  • Jelena Savovic1,
  • Rebecca Turner2,
  • David Mawdsley1,
  • Julian Higgins1 and
  • Jonathan Sterne1
Trials201516(Suppl 2):P168


Published: 16 November 2015


Empirical evidence suggests that certain aspects of trial design may lead to biased intervention effect estimates. We examined the influence of risk-of-bias judgements from Cochrane reviews for sequence generation, allocation concealment, blinding and incomplete data on intervention effect estimates in a large collection of meta-analyses (MAs).


We selected MAs with dichotomous outcomes and > 4 included trials from intervention reviews with fully completed risk-of-bias tool, published in issue 4, 2011 of the Cochrane Library. We classified outcome measures as mortality, other objective or subjective, and estimated the effect of risk-of-bias domain judgements on average bias (ratios of odds ratios [ROR] with 95% credible intervals [Cr-I]) using Bayesian hierarchical models.


Among 2815 trials in 256 meta-analyses, intervention effect estimates were on average exaggerated in trials with high or unclear risk-of-bias (versus low) for random sequence generation (ROR 0.91 [95% Cr-I 0.86, 0.98]), for allocation concealment (ROR 0.92 [95% Cr-I 0.86-0.98]) and for blinding (ROR 0.87 [95% Cr-I 0.80, 0.93]). Unlike our previous study, we did not observe consistently different bias or between-trial heterogeneity in bias in MAs with subjective outcomes compared to mortality. Results from analyses of the influences of incomplete data were inconclusive.


Possible inconsistency in criteria for risk-of-bias judgments applied by individual reviewers is a likely limitation of routinely collected bias assessments.


Inadequate randomization or lack of blinding may lead to exaggeration of intervention effect estimates in trials, but it is unclear if this effect differs by outcome type.

Authors’ Affiliations

University of Bristol
MRC Biostatistics Unit


© Savovic et al. 2015

This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.