Cochrane Reviews are systematic reviews of research in human health care and health policy, and are internationally recognized as a high quality source of evidence for decision making. They bring together research evidence on the effects of healthcare interventions or the accuracy of diagnostic tests into the world’s largest, standardized collection of systematic reviews, with more than 5,000 full reviews and protocols for another 2,000 published to date. Outcome selection in systematic reviews needs to be relevant to patients, clinicians and policy-makers if the findings of a review are to influence practice and future research. Inconsistent choice of outcome measures in clinical trials means that many meta-analyses are unable to include data from all the relevant studies. For example, the five most accessed Cochrane Reviews in 2009, together with the top cited review in that year, all described inconsistencies in the outcomes reported in eligible trials, which hampered the ability of the reviewers to resolve the uncertainties about health care that they set out to tackle . An additional problem that can affect the quality of evidence within a systematic review is that of missing outcome data from all eligible trials. Missing outcome data in trials can occur for a number of reasons. Participant data can be missing as a result of attrition or as a result of the selective non-reporting of an outcome in a study. Recent research has shown that outcome reporting bias (ORB), that is, results-based selection for publication of a subset of the original outcome variables, is also a major problem in randomized trials . The Outcome Reporting Bias in Trials (ORBIT) study of the impact of ORB in randomized trials on the results of Cochrane Reviews found that ORB is an ‘under-recognised problem that affects the conclusions in a substantial proportion of Cochrane reviews’ . This study found that more than half (157/283 (55%)) of the reviews did not include full data for the review’s primary outcome of interest from all eligible trials. The median amount of review primary outcome data missing for any reason was 10%, whereas 50% or more of the potential data were missing in 70 (25%) reviews. In this same study, an assessment of the impact of ORB was undertaken for reviews with a single meta-analysis of the primary review outcome where there was missing outcome data for a whole study that was known or suspected of being measured but not reported. Of the 25 reviews that met these criteria, a sensitivity analysis revealed that the conclusions to eight of these reviews were not robust to ORB—that is, the treatment effect estimate changed from a significant result favoring treatment to a non-significant result. For all the 25 reviews assessed, the median percentage change in the treatment effect estimates after the adjustment based on the sensitivity analysis was 39% (interquartile range 18% to 67%) .
One way to reduce the amount of missing outcome data by ensuring that eligible trials contribute usable information is the definition and implementation of an agreed minimum set of standardized outcomes, to be measured and reported in all trials for a particular disease or condition, referred to as a ‘Core Outcome Set’ (COS) [4, 5]. Furthermore, some Cochrane Review Groups (CRGs) now include a Summary of Findings (SoF) table in their reviews . The SoF table presents the results of the review for up to seven outcomes that are important to patients and aims to improve the understanding and retrieval of key findings.
In this paper, we present results for missing data in single review primary outcomes from the Cochrane Reviews included in the ORBIT study. We also report the results of a survey of the Co-ordinating Editors of CRGs, undertaken to obtain their views on the benefits and challenges of core outcome sets and how they might improve the quality of the reviews in their CRG. We were also interested in establishing whether the amount of missing data from a systematic review was influenced by whether or not a CRG had a centralized policy on outcome selection. The Co-ordinating Editors have responsibility for overseeing the work of their CRG and deciding on the publication of each of the Cochrane Reviews from within their CRG.