Development of a core outcome set for clinical trials in squamous cell carcinoma: study protocol for a systematic review of the literature and identification of a core outcome set using a Delphi survey
Trials volume 18, Article number: 321 (2017)
Squamous cell carcinoma (SCC) is a common skin cancer that poses a risk of metastasis. Clinical investigations into SCC treatment are common, but the outcomes reported are highly variable, omitted, or clinically irrelevant. The outcome heterogeneity and reporting bias of these studies leave clinicians unable to accurately compare studies. Core outcome sets (COSs) are an agreed minimum set of outcomes recommended to be measured and reported in all clinical trials of a given condition or disease. Although COSs are under development for several dermatologic conditions, work has yet to be done to identify core outcomes specific for SCC.
Outcome extraction for COS generation will occur via four methods: (1) systematic literature review; (2) patient interviews; (3) other published sources; and (4) input from stakeholders in medicine, pharmacy, and other relevant industries. The list of outcomes will be revaluated by the Measuring PRiority Outcome Variables via Excellence in Dermatologic surgery (IMPROVED) Steering Committee. Delphi processes will be performed separately by expert clinicians and patients to condense the list of outcomes generated. A consensus meeting with relevant stakeholders will be conducted after the Delphi exercise to further select outcomes, taking into account participant scores. At the end of the meeting, members will vote and decide on a final recommended set of core outcomes. The Core Outcome Measures in Effectiveness Trials (COMET) organization and the Cochrane Skin Group - Core Outcome Set Initiative (CSG-COUSIN) will serve as advisers throughout the COS generation process.
Comparison of clinical trials via systematic reviews and meta-analyses is facilitated when investigators study outcomes that are relevant and similar. The aim of this project is to develop a COS to guide use for future clinical trials.
Squamous cell carcinoma (SCC) is the second most common cutaneous malignancy. Although nodal metastasis of SCC is rare, “high-risk” tumors have an elevated 10–20% risk of metastasis [1,2,3,4]. High-risk SCC consists of lesions with size >2 cm, thickness/depth of invasion >4 mm, recurrent lesions, the presence of perineural invasion, a history of burn wounds or chronic inflammation, and immunosuppression. Treatments for SCC include standard surgical excision, Mohs micrographic surgery, cryotherapy, electrodesiccation and curettage, and radiation therapy .
Although authors of Cochrane reviews and other systematic reviews have considered the efficacy of various treatments for SCC, little research has been done to determine the most appropriate outcomes to assess those treatments . Heterogeneity in outcomes measured across trials poses a concern when comparing the effects of different interventions. Selective outcome reporting bias, defined as results-based selection of outcomes for publication, is a problem in many clinical trials and affects the conclusions of a significant proportion of systematic reviews .
To address the inconsistencies present, organizations such as the Core Outcome Measures in Effectiveness Trials Initiative (COMET) bring together researchers interested in developing a standardized set of core outcomes in various health-related fields . A core outcome set (COS) is defined as an agreed minimum set of outcomes that is recommended to be measured and reported in all clinical trials of a given condition or disease. Similarly, the Cochrane Skin Group - Core Outcome Set Initiative (CSG-COUSIN) was created to address COSs in dermatology by examining outcome measures in current research [9, 10]. CSG-COUSIN builds on the experiences of the Harmonizing Outcome Measures for Eczema (HOME) initiative, which developed a roadmap to guide the process of COS development and implementation [11,12,13,14,15,16]. Although COSs are under development for several dermatologic conditions, work has yet to be done to identify core outcomes specific for SCC. To minimize duplication, this study has been registered with the COMET and CSG-COUSIN organizations so that researchers are aware of our ongoing efforts and may participate if interested.
The aim of this study is to develop an international COS relevant to clinical trials for the treatment of cutaneous SCC. Objectives include identifying the appropriate and relevant outcomes from all interventions and methods of assessment. The final core set of outcomes is recommended for inclusion but does not preclude other outcomes from being assessed.
The development of this COS adheres to the recommendations provided by the COMET and CSG-COUSIN initiatives, with reporting conforming to the Standard Protocol Items: Recommendations for Interventional Trials (SPIRIT) checklist (see Additional File 1) [8, 16]. This project has been adapted from a previously published protocol . Figure 1 provides a brief overview of our study design.
Scope of this COS
This COS is intended as the global/international standard for clinical trials examining the efficacy of various treatments for SCC. The COS may be applied to individuals of all ages, genders, races, and ethnicities. Similarly, both surgical and nonsurgical treatments may be evaluated using the outcomes generated from this study.
Identification of outcomes
The list of outcomes currently reported will be generated over four phases:
Phase I: A systematic literature review will be performed to extract outcomes assessed in published randomized controlled trials and Cochrane reviews.
Phase II: Patient interviews will be conducted to determine patient-centered outcomes.
Phase III: Other sources, such as clinical trial registries and educational/treatment brochures, will be reviewed to ensure that all outcomes have been documented.
Phase IV: Additional stakeholders in medicine, pharmacy, and other relevant industries will be invited to provide insight into further outcomes that they would like included.
A systematic literature review using PubMed, Embase, and MEDLINE will be conducted, applying search terms related to randomized controlled studies for cutaneous SCC. Identical, published studies found across all the databases will be included once. Multiple publications on a single trial that report different outcomes or different follow-up times will be included as separate records. Authors, year of publication, source of funding, and intervention type will be documented among other study characteristics. Methodology, length of follow-up, treatment duration, results, outcomes, frequency of outcomes, and outcome measures will be noted. Outcomes extracted will then be placed into appropriate domains by two Measurement of Priority Outcome Variables in Dermatologic Surgery (IMPROVED) investigators. Similar outcomes will be listed only once. Combining and collapsing of outcomes will be performed judiciously to preserve content.
Patients will be recruited from among current patients of practicing physicians and skin cancer advocacy groups via emails and phone calls. Semistructured interviews will be conducted to explore domains identified in the literature review as well as other potential patient-identified outcomes. Open-ended questions will allow for patient expression of items important to them. Approximately 10–15 patients with SCC will be interviewed. A global context will be provided by including participants both in the United States and internationally. Interviews will be audio-recorded, transcribed, and coded to allow accurate and complete capturing of outcomes mentioned.
Examination of other published sources, including clinical trial registries and Cochrane reviews, will be done to gather outcomes related to SCC. Pamphlets and brochures describing treatments and reported outcomes will be extracted with outcomes included in the final list.
Stakeholders, or those invested in the development of a COS, will also be included in the decision process (Table 1). Dermatologists, drug and device safety regulators (e.g., U.S. Food and Drug Administration, European Medicines Agency), pharmacologists, pharmacists, and industry scientists associated with drugs and devices for treatment of SCC are potential members who can provide input regarding what outcomes they feel should be represented. Nurses, physician assistants, and other health care practitioners may be included as well to enhance further discussion.
The long list of outcomes obtained from the steps described above will then be examined by the steering committee, composed of four dermatologic surgeons: Drs. Murad Alam (Northwestern University), Ian A. Maher (St. Louis University), Joseph F. Sobanko (University of Pennsylvania), and Todd V. Cartee (Pennsylvania State University). Members may add or remove outcomes prior to the Delphi process. The steering committee members will not join in the Delphi process, but members will be invited to participate in the final consensus meeting.
Delphi surveys have been used in prior COS research . The Delphi process involves a series of rounds of data collection and analysis to condense the opinions of individuals into a group consensus. Surveys can be conducted online through the use of specialized software. Responses to each round are collected, analyzed, and then redistributed to participants in successive rounds. We plan to conduct two Delphi rounds prior to the consensus meeting.
Two separate, homogeneous groups composed of patients and physicians will participate in the Delphi exercises. Groups will consist of approximately 30 individuals to provide a diversity of input and account for potential dropouts. U.S. and international participants will be included. Prior to the exercise, details of the COS will be summarized and demographic/occupational information obtained, including years of experience, field of interest, and position. Consent will be assumed if participants complete the questionnaire. Participants will have 3 weeks to complete the online survey, with email reminders sent at the 1- and 2-week marks. For each round, the number of participants invited and those who complete the surveys will be documented.
In the first Delphi round, the complete list of outcomes developed from the aforementioned steps will be presented for rating. Using a scale devised by the Grading of Recommendations Assessment, Development and Evaluation (GRADE) working group, participants will score each outcome on a scale from 1 to 9, with 9 being critically important and 1 being not that important . For the first round, the additional option of 10 will be available if participants are unsure of the outcome’s need for inclusion. Participants will be asked to focus on ranking the most valued outcomes high and excluding outcomes of lesser importance. They will also have the option to add outcomes to the list that they feel should be included. All outcomes will be carried to the subsequent round.
Descriptive statistics will be used to analyze the data from the two groups. Responses from both stakeholder groups will be summarized and fed back to the participants after the first round, allowing participants to change their scores in light of others’ insights. Additionally, participants will be asked to identify new outcomes and determine if outcomes should be combined. New outcomes will be added to the list for the next round if two or more participants suggest its inclusion. Any uncertainties will be directed to the steering committee. The second Delphi round will follow the same format as the previous round. The set of outcomes resulting from this second round will be presented at the consensus meeting.
Prior to solidifying a COS, a consensus meeting will be held to discuss all the results of the Delphi rounds and decide on the COS. Physicians, patients, and other stakeholders will be invited to the meeting to provide insight into the process. Results from each round of the Delphi survey will be presented. In terms of consensus, if 70% of participants rank the outcome 7, 8, or 9 with less than 15% scoring it 1–3, the outcome will be retained in the consensus pool . Outcomes will be removed from the consensus list if 70% or more of the participants rank the outcome 1–3 and less than 15% rank the outcome 7, 8, or 9.
Feedback regarding the consensus-derived set of outcomes will then be elicited with the assistance of a trained moderator. Using live polling software, items will anonymously be voted for inclusion into the final core set of outcomes. If there are more than ten outcomes, then the steering committee will decide which outcomes will be kept through a discussion. By the end of the meeting, the goal is to create a COS of no more than ten outcomes that can be agreed upon by all stakeholders, patients, and physicians.
Core outcome measures
Whereas the COS defines “what” to measure, the core outcome measurement instruments represent “how” to measure these domains. To define these measures, a systematic review of at least two databases will be done to identify currently used outcome instruments. The HOME roadmap will be used along with the COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN) framework for guidance . The quality of the studies will be assessed by rating their validity, reliability, responsiveness to change, and interpretability.
To determine which measurements are suitable per outcome domain, a consensus meeting with key stakeholders, patients, and clinicians will be held . Results from the systematic review will be presented so that attendees can judge the measures on the basis of how valid, reliable, and feasible they may be as assessment tools. Each core outcome domain will be paired with a corresponding outcome measurement instrument. New instruments will be developed if there is inadequate evidence supporting existing methods. At the end of the consensus meeting, relevant stakeholders will vote to determine which measures should be included.
Use of heterogeneous or clinically irrelevant outcomes and outcome omission are increasingly problematic in the study of SCC treatment. The heterogeneity of these outcomes leaves clinicians unable to accurately compare findings in studies. Organizations such as COMET were formed to develop standardized core outcomes in various health-related fields. These COSs are not intended to limit the outcomes that can be measured but instead serve as a minimum of what should be measured. No COS for cutaneous SCC currently exists. The proposed protocol of COS generation for SCC will follow the COMET methodology with the aim of reducing the inconsistency of outcomes and outcome measurements across relevant trials.
The development of the COS is active and ongoing in its initial phase of outcome extraction.
Core Outcome Measures in Effectiveness Trials
Core outcome set
COnsensus-based Standards for the selection of health Measurement INstruments
Cochrane Skin Group - Core Outcome Set Initiative
European Medicines Agency
U.S. Food and Drug Administration
Grading of Recommendations Assessment, Development and Evaluation
Harmonizing Outcome Measures for Eczema
Measurement of Priority Outcome Variables in Dermatologic Surgery
Standard Protocol Items: Recommendations for Interventional Trials
Brantsch KD, Meisner C, Schonfisch B, Trilling B, Wehner-Caroli J, Rocken M, et al. Analysis of risk factors determining prognosis of cutaneous squamous-cell carcinoma: a prospective study. Lancet Oncol. 2008;9(8):713–20.
Mourouzis C, Boynton A, Grant J, Umar T, Wilson A, Macpheson D, et al. Cutaneous head and neck SCCs and risk of nodal metastasis – UK experience. J Craniomaxillofac Surg. 2009;37(8):443–7.
Veness MJ. High-risk cutaneous squamous cell carcinoma of the head and neck. J Biomed Biotechnol. 2007;2007(3):80572.
Williamson PR, Altman DG, Blazeby JM, Clarke M, Devane D, Gargon E, et al. Developing core outcome sets for clinical trials: issues to consider. Trials. 2012;13:132.
Rowe DE, Carroll RJ, Day Jr CL. Prognostic factors for local recurrence, metastasis, and survival rates in squamous cell carcinoma of the skin, ear, and lip. Implications for treatment modality selection. J Am Acad Dermatol. 1992;26(6):976–90.
Lansbury L, Leonardi-Bee J, Perkins W, Goodacre T, Tweed JA, Bath-Hextall FJ. Interventions for non-metastatic squamous cell carcinoma of the skin. Cochrane Database Syst Rev. 2010;4, CD007869.
Kirkham JJ, Dwan KM, Altman DA, Gamble C, Dodd S, Smyth R, et al. The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews. BMJ. 2010;340:c365.
Core Outcome Measures in Effectiveness Trials (COMET) Initiative. http://www.comet-initiative.org/. Accessed 20 Apr 2016.
The Cochrane Skin Group - Core Outcome Set Initiative (CSG-COUSIN). https://www.uniklinikum-dresden.de/COUSIN/. Accessed 21 Apr 2016.
Schmitt J, Deckert S, Alam M, Apfelbacher C, Barbaric J, Bauer A, et al. Report from the kick-off meeting of the Cochrane Skin Group Core Outcome Set Initiative (CSG-COUSIN). Br J Dermatol. 2016;174(2):287–95.
Schmitt J, Williams J, HOME Development Group. Harmonising Outcome Measures for Eczema (HOME): report from the first international consensus meeting (HOME 1), 24 July 2010, Munich, Germany. Br J Dermatol. 2010;163(6):1166–8.
Schmitt J, Spuls P, Boers M, Thomas K, Chalmers J, Roekevisch E, et al. Towards global consensus on outcome measures for atopic eczema research: results of the HOME II meeting. Allergy. 2012;67(9):1111–7.
Chalmers J, Schmitt J, Spuls P, Thomas K, Simpson EL, Dohil M, et al. Minutes of the HOME III Meeting 6–7 April 2013, San Diego, CA, USA. http://www.homeforeczema.org/documents/minutes-home-iii-final.pdf. Accessed 5 Jul 2017.
Schmitt J, Spuls PI, Thomas KS, Simpson E, Furue M, Deckert S, et al. The Harmonising Outcome Measures for Eczema (HOME) statement to assess clinical signs of atopic eczema in trials. J Allergy Clin Immunol. 2014;134(4):800–7.
Chalmers JR, Schmitt J, Apfelbacher C, Dohil M, Eichenfield LF, Simpson EL, et al. Report from the third international consensus meeting to harmonise core outcome measures for atopic eczema/dermatitis clinical trials (HOME). Br J Dermatol. 2014;171(6):1318–25.
Schmitt J, Apfelbacher C, Spuls PI, Thomas KS, Simpson EL, Furue M, et al. The Harmonizing Outcome Measures for Eczema (HOME) roadmap: a methodological framework to develop core sets of outcome measurements in dermatology. J Invest Dermatol. 2015;135(1):24–30.
Iyengar S, Williamson PR, Schmitt J, Johannsen L, Maher IA, Sobanko JF, et al. Development of a core outcome set for clinical trials in rosacea: study protocol for a systematic review of the literature and identification of a core outcome set using a Delphi survey. Trials. 2016;17:429.
Moza A, Benstoem C, Autschbach R, Stoppe C, Goetzenich A. A core outcome set for all types of cardiac surgery effectiveness trials: a study protocol for an international eDelphi survey to achieve consensus on what to measure and the subsequent selection of measurement instruments. Trials. 2015;16:545.
Guyatt GH, Oxman AD, Kunz R, Atkins D, Brozek J, Vist G, et al. GRADE guidelines: 2. Framing the question and deciding on important outcomes. J Clin Epidemiol. 2011;64(4):395–400.
Harman NL, Bruce IA, Kirkham JJ, Tierney S, Callery P, O’Brien K, et al. The importance of integration of stakeholder views in core outcome set development: otitis media with effusion in children with cleft palate. PLoS One. 2015;10(6):e0129514.
This publication was supported by Merz Center for Quality and Outcomes Research in Dermatologic Surgery and the IMPROVED (Measurement of Priority Outcome Variables in Dermatologic Surgery) group. Reference numbers for the funding sources are not available. This study protocol was not fully externally peer reviewed as part of the approval process.
Availability of data and materials
Ethics approval and consent to participate
A systematic review of the literature will be performed with the help of data extractors across four institutions. Ethical approval was not applicable for the review of literature. Ethical approval and consent to participate for the interviews and the Delphi study have been granted by the Northwestern University Institutional Review Board (protocol STU00201637). Informed consent will be obtained from each participant.
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Schlessinger, D.I., Iyengar, S., Yanes, A.F. et al. Development of a core outcome set for clinical trials in squamous cell carcinoma: study protocol for a systematic review of the literature and identification of a core outcome set using a Delphi survey. Trials 18, 321 (2017). https://doi.org/10.1186/s13063-017-2069-2