Table 5 Themes in journals’ criticisms of COMPare

Misrepresentation of COMPare’s methods

COMPare’s method is a “simple check for an exact word match between outcomes entered in a registry and those reported in a manuscript, but that oversimplifies a highly nuanced process” (Annals to BMJ).

This is untrue. COMPare did not seek literal word matches: each pre-specified outcome was manually checked and re-checked, as per previous research on outcome misreporting, using CONSORT as gold standard.

“The initial trial registry data… serve as COMPare’s ‘gold standard’” (Annals Editors critique, 01/03/16).

This is untrue. As explained in our publicly accessible protocol, COMPare used the registry entry only as a last resort where there was no pre-commencement protocol publicly available, as CONSORT 6b requires that changes after commencement be noted in the trial report. Notably, no Annals trial had a publicly accessible pre-commencement protocol.

Stating that COMPare correspondence and raw data sheets contained insufficient information

“In addition, some of the information in your letters is vague, containing only numbers and not specific outcomes, making it difficult to understand the specific issues or reply to them. Moreover, the last 2 paragraphs of the letters you have submitted, concerning CONSORT and the COMPare project, are identical” (JAMA emails, 09/12/15).

All correction letters linked to the COMPare online repository where all underlying raw data sheets were shared in full, specifying in detail each pre-specified primary and secondary outcome, whether and how each pre-specified outcome was reported, each additional non-pre-specified outcome reported, and whether each non-pre-specified outcome added was correctly declared as non-pre-specified. This JAMA letter was received halfway through the COMPare study period. To address the reasons given for letter rejection, despite word length limits imposed by JAMA for correspondence, all subsequent JAMA letters had no repetition and extensive detail within the text on specific misreported outcomes. However, none of these subsequent letters was published and we received no further replies.

Warning readers against COMPare’s assessments

“Until the COMPare Project’s methodology is modified to provide a more accurate, complete and nuanced evaluation of published trial reports, we caution readers and the research community against considering COMPare’s assessments as an accurate reflection of the quality of the conduct or reporting of clinical trials” (Annals Editors critique, 01/03/16), (Trial 25, Annals, 14/12/15), (Trial 44, Annals, 15/12/16), (Trial 45, Annals, 15/12/15), (Trial 68, Annals, 30/12/15).

All Annals’ critiques on matters of fact were incorrect; Annals rejected replies demonstrating this to readers. Following this comment posted by Annals under all COMPare correspondence, no trialists engaged with any of our evidence of their failure to correctly report pre-specified outcomes. We regarded Annals’ advising authors not to engage with reasonable professional criticism of their methods and results as a breach of ICMJE guidance on correspondence.

Claim that COMPare coding incorrect on specific outcomes

NEJM gave journalists a detailed review of COMPare’s assessment of one trial, which NEJM stated had identified six errors in COMPare’s assessment. This was reviewed, and NEJM were wrong on all six counts; full details are presented in the table above and in the correspondence appendix (NEJM first comments on trial 22). Another NEJM review of a COMPare letter was also factually wrong on all three issues it raised (NEJM second comments on trial 22 (2)).

The editors were wrong on all nine issues raised. The document they sent exemplified misunderstandings around the importance of reporting all pre-specified time points for each pre-specified outcome.