Skip to main content

Importance of swift event adjudication of endpoints for adequate reporting to data and safety monitoring boards in clinical trials—lessons from CULPRIT-SHOCK

Peer Review reports

Conduction of randomized controlled clinical trials is cost- and time-consuming, especially if study enrolment is performed at multiple sites in multiple countries with different medical traditions, medical record systems, and language barriers. Typically, three distinct boards are created to coordinate these trials. First of all, the principal investigator and associated advisors form the Steering Committee, which is responsible for executive tasks such as trial design and oversight. Secondly, a Data Safety Monitoring Board (DSMB) is needed to review the event and outcome data. This board consists of independent members and is responsible for ongoing evaluation of participant safety, overall conduction as well as the progress of the trial and might even recommend to modify or terminate the trial in case of respective concerns. Thirdly, the Clinical Event Adjudication Committee (CEAC) reviews the anonymized participant data and adjudicates events and outcome data. While the DSMB has direct access to the random code, the CEAC remains blinded throughout the trial to guarantee unbiased adjudications. Whereas the latter is often considered a necessity for the final reporting and thus robustness of the conclusions it is also quite common that substantial delays occur between investigator reporting and final adjudication of events. There are practical and financial interests to have fewer CEAC gatherings. While this delay is commonly accepted by the steering committee as it occurs simultaneously with other activities before the data lock of a trial, this delay may have important consequences for the interpretation during interim analyses, i.e. for the DSMB. Cardiovascular outcomes trials often include interventions which could benefit patients but inherently also might incur harm. In recent years there have been several examples of clinical trials being stopped prematurely due to either benefit or risk, even with an unfavorable risk-benefit ratio [1,2,3]. Increasingly, clinical trials have endpoints which are a composite of two or more adverse outcomes with quite different pathophysiological mechanisms or end organ involvement. As a consequence, the particular endpoints in a clinical trial can trend in quite opposite directions, making the interpretation difficult.

An example of possible causes of death trending in different directions was encountered during the conduction of the recently published CULPRIT-SHOCK trial [4]. This trial was the first large randomized trial which proved that the hitherto preferred strategy of immediate complete revascularization with PCI in cardiogenic shock led to more harm defined by the primary endpoint of death and need for renal replacement therapy with an absolute difference of 9.5%. These 30-day results were recently confirmed in the 12-month follow-up analysis [5]. The pre-specified interim analysis after inclusion of 342 patients presented to the DSMB already showed a significant difference in 30-day total mortality favoring culprit-lesion only versus immediate multivessel PCI (44.2% vs. 57.1%; p = 0.017). The primary outcome of the trial, which also included renal replacement therapy, displayed a similar difference (47.1% vs. 61.2%; p = 0.009). The protocol stated that the interim analysis should be performed according to the O’Brian-Fleming method. The trial was to be stopped “if the null hypothesis of equal event rates can be rejected with a significance level of 0.005”. Thus, the attained p value for the primary endpoint at the time when half the patients had been enrolled was close to the stopping rule with a 0.004 margin. Although not formally reaching the pre-specified boundary, the difference in total mortality was of concern to the DSMB. The main reason for not recommending a termination of the trial for safety reasons at this time point were mixed signals in the causes of death, possibly attributed to the fact that independent endpoint adjudication was lagging behind. The most common investigator reported cause of death, “refractory cardiogenic shock” was evenly distributed in the entire cohort: 49/170 in the multivessel PCI arm (28.8%) and 52/172 (30.2%) in the culprit only arm. However, among the fatal cases, “refractory cardiogenic shock” was significantly less common with multivessel PCI (50.5% (49/97) vs. 68.4% (52/76) of total death, p = 0.018) in line with the previous pathophysiological mainstream thinking that multivessel PCI would serve as a guardian against myocardial ischemia, re-infarctions and thus prevention of shock progression. In support of this possible pathophysiological mechanism, there was a numerically lower rate of deaths from re-infarctions in the multivessel PCI arm (0% (0/97) vs. 2.6% (2/76); p = 0.11). Furthermore, in this phase of the trial with incomplete adjudication, there was an imbalance in the deaths attributed to “other” or “unknown” causes. The increase in total mortality in the immediate multivessel PCI arm resulted from an excess in “unknown” (5.2% (5/97) vs. 0% (0/76); p = 0.05) and “other causes” of death (20.6% (20/97) vs. 11.8% (9/76); p = 0.13). The aforementioned values are depicted in Fig. 1.

Fig. 1
figure 1

Comparison of adjudicated and un-adjudicated mortality events during the life cycle of the CULPRIT-SHOCK study, as reported to the DSMB

These different trends made it difficult to assess whether there were inacceptable risks for the study population which prompted the DSMB to address the steering committee and recommend a continuation of the study under the condition that an immediate plan be set forward for the CEAC to promptly adjudicate the reported events. During this endeavor, to provide an updated adjudication of causes of death report to the DSMB, inclusion in the trial also picked up. In the follow-up report which included 676 patients, the absolute difference in total 30-day mortality decreased from 13% to 7% and was no longer significant. Also, the primary endpoint was not significantly different between the groups at this timepoint (61.4 vs 53.8%; p = 0.07). Death from refractory cardiogenic shock however remained, albeit less pronounced, numerically in favor of multivessel PCI (49.7 vs 62.9%). At this point, the DSMB advised that the study be completed as originally set out, and the DSMB saw no convincing argument to stop the trial for safety reasons. With respect to CULPRIT-SHOCK, the DSMB did not perform a second interim analysis and was not concerned about different trends in the two components of the primary endpoint, but about different trends in the causes of death which can be understood as components of the most important safety variable, mortality. The thorough discussion of the safety risks after 676 patients were included was not a second interim analysis and was not restricted to the primary endpoint. Ultimately, the percentage of un-adjudicated mortality events decreased and was comparable between both study arms while the finding of a superiority of the culprit-lesion-only PCI strategy prevailed.

Stopping a trial prematurely always leads to discussions regarding the appropriateness. Often very spectacular differences early in a trial tend to show some regression toward the mean with time and more patients included, as observed in CULPRIT-SHOCK. Furthermore, one should be particularly careful when stopping relatively small studies in complex patients, where a second pivotal trial is unlikely to emerge to guide physicians, authorities, and ultimately treatment guidelines. A formal stopping rule should in such cases have the narrowest possible confidence interval. In the case of CULPRIT-SHOCK the DSMB was also aware of the difficulties in attributing a cause of death in patients with multiorgan failure, and thus competing causes of death, with most patients dying within 48 h of admission.

With this letter, we want to stress the importance of reliable and fast event adjudication to ensure adequate interpretation of data for the safety of participants and proper reporting between boards in all clinical trials, not only cardiology. Every DSMB wants the particular trial to succeed but also has to remain strict and keep patient safety as the primary goal. The discussion remains open whether DSMB’s are bound by stopping rules or merely guided, especially in trials where clinical adverse events may trend in different directions.

Availability of data and materials

Not applicable.

References

  1. De Bruyne B, Pijls NHJ, Kalesan B, Barbato E, Tonino PAL, Piroth Z, Jagic N, Möbius-Winkler S, Rioufol G, Witt N, Kala P, MacCarthy P, Engström T, Oldroyd KG, Mavromatis K, Manoharan G, Verlee P, Frobert O, Curzen N, et al. Fractional flow reserve–guided PCI versus medical therapy in stable coronary disease. N Engl J Med. 2012;367(11):991–1001. https://doi.org/10.1056/NEJMoa1205361.

    Article  CAS  PubMed  Google Scholar 

  2. Carcillo JA, Michael Dean J, Holubkov R, Berger J, Meert KL, Anand KJS, Zimmerman J, Newth CJL, Harrison R, Burr J, Willson DF, Nicholson C. Eunice Kennedy Shriver National Institute of Child Health and Human Development (NICHD) Collaborative Pediatric Critical Care Research Network (CPCCRN). The randomized comparative pediatric critical illness stress-induced immune suppression (CRISIS) prevention trial*. Pediatr Crit Care Med. 2012;13(2):165–73. https://doi.org/10.1097/PCC.0b013e31823896ae.

    Article  PubMed  PubMed Central  Google Scholar 

  3. Combes A, Hajage D, Capellier G, Demoule A, Lavoué S, Guervilly C, Da Silva D, Zafrani L, Tirot P, Veber B, Maury E, Levy B, Cohen Y, Richard C, Kalfon P, Bouadma L, Mehdaoui H, Beduneau G, Lebreton G, et al. Extracorporeal membrane oxygenation for severe acute respiratory distress syndrome. N Engl J Med. 2018;378(21):1965–75. https://doi.org/10.1056/NEJMoa1800385.

    Article  PubMed  Google Scholar 

  4. Thiele H, Akin I, Sandri M, Fuernau G, de Waha S, Meyer-Saraei R, Nordbeck P, Geisler T, Landmesser U, Skurk C, Fach A, Lapp H, Piek JJ, Noc M, Goslar T, Felix SB, Maier LS, Stepinska J, Oldroyd K, et al. PCI strategies in patients with acute myocardial infarction and cardiogenic shock. N Engl J Med. 2017. https://doi.org/10.1056/NEJMoa1710261.

  5. Thiele H, Akin I, Sandri M, de Waha-Thiele S, Meyer-Saraei R, Fuernau G, Eitel I, Nordbeck P, Geisler T, Landmesser U, Skurk C, Fach A, Jobs A, Lapp H, Piek JJ, Noc M, Goslar T, Felix SB, Maier LS, et al. One-year outcomes after PCI strategies in cardiogenic shock. N Engl J Med. 2018;379(18):1699–710. https://doi.org/10.1056/NEJMoa1808788.

    Article  PubMed  Google Scholar 

Download references

Acknowledgements

Not applicable.

Funding

The CULPRIT-SHOCK trial was supported by a grant (FP7/2007–2013) from the European Union 7th Framework Program and by the German Heart Research Foundation and the German Cardiac Society.

Author information

Authors and Affiliations

Authors

Contributions

This letter was jointly written by all co-authors. The authors read and approved the final manuscript.

Corresponding author

Correspondence to Benedikt Schrage.

Ethics declarations

Ethics approval and consent to participate

The CULPRIT-SHOCK trial was approved by a multi-site ethics committee and for all eligible patients, written informed consent was obtained with the use of a prespecified process that varied slightly according to country.

Consent for publication

Not applicable.

Competing interests

There are no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Clemmensen, P., Schrage, B., Zeymer, U. et al. Importance of swift event adjudication of endpoints for adequate reporting to data and safety monitoring boards in clinical trials—lessons from CULPRIT-SHOCK. Trials 22, 197 (2021). https://doi.org/10.1186/s13063-021-05129-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s13063-021-05129-4