- Study protocol
- Open Access
Development of a core outcome set for clinical trials in rosacea: study protocol for a systematic review of the literature and identification of a core outcome set using a Delphi survey
Trials volume 17, Article number: 429 (2016)
Rosacea is a chronic inflammatory disorder affecting millions of individuals worldwide. Diagnosis is based on signs and symptoms with management and treatment aimed to suppress inflammatory lesions, erythema, and telangiectasia. While many clinical trials of rosacea exist, the lack of consensus in outcome reporting across all trials poses a concern. Proper evaluation and comparison of treatment modalities is challenging. In order to address the inconsistencies present, this project aims to determine a core set of outcomes which should be evaluated in all clinical trials of rosacea.
This project will utilize a methodology similar to previous core outcome set research. A long list of outcomes will be extracted over four phases: (1) systematic literature review, (2) patient interviews, (3) other published sources, and (4) stakeholder involvement. Potential outcomes will be examined by the Steering Committee to provide further insight. The Delphi process will then be performed to prioritize and condense the list of outcomes generated. Two homogenous groups of physicians and patients will participate in two consecutive rounds of Delphi surveys. A consensus meeting, composed of physicians, patients, and stakeholders, will be conducted after the Delphi exercise to further select outcomes, taking into account participant scores. By the end of the meeting, members will vote and decide on a final recommended set of core outcomes. For the duration of the study, we will be in collaboration with both the Core Outcome Measures in Effectiveness Trials (COMET) and Cochrane Skin Group - Core Outcome Set Initiative (CSG-COUSIN).
This study aims to develop a core outcome set to guide assessment in clinical trials of rosacea. The end-goal is to improve the reliability and consistency of outcome reporting, thereby allowing sufficient evaluation of treatment effectiveness and patient satisfaction.
Rosacea is a chronic dermatologic disorder with a prevalence of about 2 %, affecting 16 million Americans [1, 2]. Fair-skinned individuals of Celtic and European heritage are most commonly affected, although all skin types may develop the condition [1, 3]. The etiology of rosacea is unknown and hence, diagnosis is often based on signs and symptoms . Primary features of rosacea include inflammatory papules/pustules, erythema, and telangiectasia . Secondary features consist of burning, plaques, and edema. Disease symptoms and lesions affecting the face and other visible areas may have a negative impact on health-related quality of life. Furthermore, patients with rosacea are at an increased risk for significant comorbidities such as autoimmune diseases, inflammatory bowel disease, and possibly cardiovascular disease [5–8].
In 2002, the National Rosacea Society Expert Committee classified rosacea into four main subtypes: erythematotelangiectatic, papulopustular, phymatous, and ocular [4, 9]. The erythematotelangiectatic subtype is the most common, with the phymatous subtype being rare. A variety of treatments, including topical, systemic, and laser therapies, have been utilized thus far. Topical treatments approved by the US Food and Drug Administration (FDA) include sodium sulfacetamide, ivermectin, azelaic acid, and metronidazole . Subantimicrobial-dose doxycycline is the only FDA-approved systemic therapy for rosacea , but tetracycline and isotretinoin are often used as well . Mirvaso (brimonidine tartrate) was approved in 2014 by the European Medicines Agency (EMA) to treat facial erythema caused by rosacea . Laser therapy is recommended for the treatment of telangiectasia and rhinophyma.
Several trials are underway to evaluate new treatment modalities for rosacea. According to ClinicalTrials.gov, more than 100 trials relevant to rosacea are either in progress, actively recruiting, or complete [1, 11]. The 2015 Cochrane Review by van Zuuren assessed the efficacy and safety of interventions for rosacea with impact on quality of life . Only 11 out of the 106 trials included in the review assessed the effect of rosacea treatment on quality of life, which made recommendations nearly impossible. Heterogeneity in physician-assessed secondary review outcomes, such as physician’s global evaluation, lesion counts, and duration of remission were pervasive in the trials. This inconsistency in outcomes measured across trials poses a concern when evaluating the effects of different interventions.
There are few validated methods of assessing improvement in rosacea, possibly due to the multiplicity of outcomes collected. Selective outcome reporting bias, defined as results-based selection of outcomes for publication, is a problem in many clinical trials and affects the conclusions of a significant proportion of systematic reviews .
Specific organizations have been formed to address outcome reporting bias. The Core Outcome Measures in Effectiveness Trials Initiative (COMET) brings together researchers interested in developing a standardized set of core outcomes in various health-related fields . A core outcome set (COS) is defined as an agreed minimum set of outcomes that is recommended to be measured and reported in all clinical trials of a given condition or disease. The implementation of a COS may reduce the risk of selective outcome reporting and allow for more important outcomes to be assessed.
Another group, the Cochrane Skin Group - Core Outcome Set Initiative (CSG-COUSIN), was created specifically to address COSs in dermatology by examining outcome measures in current research . CSG-COUSIN builds on the experiences of the Harmonizing Outcome Measures for Eczema (HOME) initiative, which also developed a roadmap to guide the process of COS development and implementation [16–21]. Similar to other initiatives, the CSG-COUSIN group hopes to develop standardized, evidence-based core outcome sets which can be utilized in all clinical trials.
While COS are under development for several dermatologic conditions, work has yet to be done to identify core outcomes specific for rosacea. In order to minimize duplication, this study has been registered with the COMET and CSG-COUSIN organizations so researchers are aware of our ongoing efforts and may participate if interested.
The aim of this study is to develop an international COS that is relevant to clinical trials of rosacea based on the systematic synthesis of peer-reviewed research evidence; preferences from relevant stakeholder groups, including patients, clinicians, regulators; and structured consensus processes involving all relevant perspectives . Objectives include first determining what should be measured and second, how it should be measured in rosacea clinical trials. The selected outcomes that may be recommended for inclusion in all rosacea research trials will be a minimum set that will not preclude inclusion of other outcomes in specific trials, per the preference of individual investigators..
The development of this COS adheres to the recommendations provided by the COMET and CSG-COUSIN initiatives [14, 21]. Reporting conforms to the SPIRIT (Standard Protocol Items: Recommendations for Interventional Trials) checklist (see Additional file 1). Figure 1 provides a brief overview of our study design.
Scope of this COS
This COS is intended as the global/international standard for clinical trials examining the efficacy of various treatments for rosacea as evaluated by physicians, patients, and others invested in its results. The COS to be developed will not be population-specific and may be applied to individuals of all ages, genders, skin types, races, and ethnicities. Similarly, all rosacea-specific interventions, including topical, systemic, and laser therapy, may be evaluated using the outcomes generated from this study.
Identification of outcomes
The list of outcomes currently reported will be generated over four phases.
First, a systematic literature review will be performed to extract outcomes assessed in published randomized controlled trials. Cochrane reviews will also be examined for outcome inclusion
Second, patient interviews will be conducted to determine patient-centered outcomes which should be assessed
Third, other sources, such as clinical trials registries and rosacea educational and treatment brochures, will be reviewed to ensure all outcomes have been documented
Prior to the development of a final list of outcomes, stakeholders will be invited to provide insight into further outcomes that they would like included
A systematic literature review using PubMed, Embase, CENTRAL, and CINAHL will be conducted using search terms related to rosacea. Included studies will be randomized controlled trials of rosacea, regardless of study quality or bias. Duplicate studies which cross over from different databases will be removed and included only once. Each article will then be extracted. Authors, year of publication, source of funding, and treatment vehicle will be documented among other study characteristics. In terms of study design, items such as length of follow-up, treatment duration, results, outcomes, and outcome measures will be noted. The long list of outcomes extracted from published studies will then be placed into appropriate domains by two investigators.
Domains will be determined based on the outcomes extracted and will encompass treatment efficacy, adverse events, and quality of life. Outcomes that are very similar or synonymous in nature will be collapsed and listed only once. If two outcomes are the same, the shorter version of the name will be applied. Subcategories will be listed appropriately to further stratify a domain. If several similar entries are combined, the constituents will be listed in brackets after the new category. Combining and collapsing of outcomes, while necessary, will be performed in moderation to preserve the content and significance of distinct outcomes.
In order to encompass patient-centered outcomes, a semistructured interview will be conducted to explore the domains identified in the literature review as well as other potential patient identified outcomes. Approximately 10-15 patients with rosacea will be interviewed. A global context will be provided by including participants both in the United States and internationally. Informed consent will be obtained from each participant. Open-ended questions will allow for patient expression of relevant outcomes, which may provide identify outcomes not noted in established quality of life questionnaires or prior studies. Interviews will be audio-recorded, transcribed, and coded to allow accurate and complete capturing of outcomes mentioned. Through the addition of outcomes ascertained during the interviews, we hope to provide a more complete account of issues important to patients.
Examination of other published sources, including clinical trials registries and Cochrane reviews, will be conducted to gather outcomes related to rosacea. Pamphlets and brochures describing treatments and reported outcomes will be extracted, and outcomes detected will be included in the final list of outcomes.
Stakeholders, or those invested in the development of a COS in rosacea, will also be included in the decision process (Table 1). Dermatologists, drug and device safety regulators (e.g., FDA, EMA), patient support groups (e.g., National Rosacea Society), pharmacologists, pharmacists, and industry scientists associated with drugs and devices for treatments of rosacea are potential members who can provide input regarding what outcomes that they feel should be represented. Nurses, physician assistants, and other health care practitioners may also be included to enhance further discussion.
The long list of outcomes obtained from the steps described above will then be examined by the Steering Committee, composed of four dermatologists: Drs. Murad Alam (Northwestern University), Ian A. Maher (Saint Louis University), Joseph F. Sobanko (University of Pennsylvania), and Todd V. Cartee (Pennsylvania State University). These members may add or remove outcomes as they deem suitable to be included in the list prior to the Delphi process. The Steering Committee members will not join in the Delphi process, but members will be invited to participate in the final consensus meeting.
Delphi surveys have been used in prior COS research . Surveys can be conducted online through the use of specialized software. The Delphi process involves a series of rounds of data collection and analysis to condense the opinions of individuals into a group consensus. Responses to each round are collected, analyzed, and then redistributed to participants in successive rounds. We plan to conduct two Delphi rounds prior to the consensus meeting.
Two homogenous groups composed of patients or physicians, respectively, will participate in the Delphi exercises. Groups will consist of approximately 30 individuals to provide a greater diversity of input and account for potential dropouts. A global context will be provided by including patients from both the United States and internationally; physicians from across the country and internationally will be included in the other group. Prior to the exercise, details of the COS will be summarized in plain language. Demographic and occupational information, including years of experience, field of interest, and position, will be obtained. Consent will be assumed if participants complete the questionnaire. Participants will have 3 weeks to complete the online survey with email reminders at the 1- and 2-week marks. For each round, the number of participants invited and those who completed the surveys will be documented and attrition assessed.
In the first Delphi, the complete list of outcomes gathered from the literature review, patient interviews, stakeholder interaction, and other sources will be presented for rating. Outcomes will be listed randomly after each round to avoid any influence the display order may have when scoring. Using a scale devised by the Grading of Recommendations Assessment, Development and Evaluation (GRADE) working group, participants will score each outcome on a scale from 1 to 9 with “9” being critically important and “1” being not very important . For the first round, the additional option of “10” will be available if participants are unsure of its need for inclusion.. Participants will be asked to focus on ranking the most valued outcomes high and excluding outcomes of lesser importance. They will also have the option to add outcomes to the list that they feel should be included. All outcomes will be carried to the subsequent round.
Descriptive statistics will be used to analyze the data from the two groups. Responses from both stakeholder groups will be summarized and fed back to the participants after the first round, allowing participants to change their score in light of others’ insights. At the same time, participants will be asked to identify new outcomes and determine if outcomes should be combined. New outcomes will be added to the list for the next round if two or more participants suggest its inclusion. Any uncertainties will be directed to the Steering Committee for adjudication.
In round 2 of the Delphi exercise, participants will again score the outcomes on a scale from 1 to 9, following the same format as the previous Delphi exercise. The end result of the Delphi should consist of a more simplified set of outcomes that will be further examined at the consensus meeting.
Prior to solidifying a core set of outcomes, a face-to-face consensus meeting will be held to discuss the results of the Delphi rounds. An independent facilitator will chair the meeting to finalize the outcomes. Physicians, patients, and other stakeholders will be invited to the meeting to provide insight on the process. Results from each round of the Delphi survey will be presented. In terms of consensus, if 70 % of participants rank the outcome 7, 8, or 9 with less than 15 % scoring it 1–3, the outcome will be retained in the consensus pool . Outcomes will be removed from the consensus list if 70 % or more of the participants rank the outcome 1–3 and less than 15 % rank the outcome 7, 8, or 9.
Feedback regarding the consensus-derived set of outcomes will then be elicited with the assistance of a trained moderator. Using live polling software, items will anonymously be voted “Yes” or “No” for inclusion into the final core set of outcomes. By the end of the meeting, the goal is to create a core set of outcomes which can be agreed upon by all stakeholders, patients, and physicians.
Core outcome measures
Once a COS has been developed, the Harmonizing Outcome Measures for Eczema (HOME) roadmap will be utilized for developing or selecting a core set of measures to track the outcomes selected . Initial steps include identifying outcome domains and current instruments used to measure those specific domains. Special focus will be directed at systematically identifying relevant outcome measurements through a systematic review covering multiple databases. Quality of the studies will be assessed by rating their validity, reliability, responsiveness to change, and interpretability. The COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN) framework will be utilized for guidance.
In order to determine which measurements are suitable per outcome domain, a consensus meeting with key stakeholders, patients, and clinicians will be held . Results from the systematic review will be provided to guide discussion. Attendees will then judge the measures based on how valid, reliable, and feasible they may be for assessing each core outcome domain. If evidence for a particular measurement instrument is inadequate, it may be necessary to develop a new instrument. At the end of the consensus meeting, relevant stakeholders will vote to determine which measures should be included. An outcome measurement instrument should be recommended for each core outcome domain.
There is currently no COS relevant to clinical trials of rosacea. With a lack of standardization in outcomes assessed, the potential for reporting bias exists. Further, selection of outcomes is crucial for properly comparing and evaluating the effectiveness of different treatment interventions.
The proposed COS for rosacea aims to reduce the inconsistency of outcomes and outcome measurements across relevant trials. Through the use of COSs, we hope to hasten refinement and adoption of therapies for rosacea that address factors important to key stakeholders, particularly patients and physicians.
The development of the COS is active and ongoing in its initial phase of outcome extraction.
Core Outcome Measures in Effectiveness Trials
core outcome set
COnsensus-based Standards for the selection of health Measurement INstruments
Grading of Recommendations Assessment, Development and Evaluation
Cochrane Skin Group - Core Outcome Set Initiative
European Medicines Agency
US Food and Drug Administration
Harmonizing Outcome Measures for Eczema
Gold LM, Draelos ZD. New and emerging treatments for rosacea. Am J Clin Dermatol. 2015;16(6):457–61.
Tan J, Berg M. Rosacea: current state of epidemiology. J Am Acad Dermatol. 2013;69(6 Suppl 1):S27–35.
Gupta AK, Chaudhry MM. Rosacea and its management: an overview. J Eur Acad Dermatol Venereol. 2005;19(3):273–85.
Tan J, Blume-Peytavi U, Ortonne JP, Wilhelm K, Marticou L, Baltas E, Rivier M, Petit L, Martel P. An observational cross-sectional survey of rosacea: clinical associations and progression between subtypes. Br J Dermatol. 2013;169(3):555–62.
Dosal J, Keri J. Rosacea and cardiovascular disease: is there an association? J Am Acad Dermatol. 2015;73(2):308–9.
Bagcchi S. Rosacea associated with increased risk of some cancers. Lancet Oncol. 2015;16(8):e382.
Egeberg A, Hansen PR, Gislason GH, Thyssen JP. Clustering of autoimmune diseases in patients with rosacea. J Am Acad Dermatol. 2016;74(4):667–72. e1.
Spoendlin J, Karatas G, Furlano RI, Jick SS, Meier CR. Rosacea in patients with ulcerative colitis and Crohn’s disease: a population-based case-control study. Inflamm Bowel Dis. 2016;22(3):680–7.
Tan J, Liu H, Leyden JJ, Leoni MJ. Reliability of Clinician Erythema Assessment grading scale. J Am Acad Dermatol. 2014;71(4):760–3.
Mirvaso. European Medicines Agency. 2014. http://www.ema.europa.eu/docs/en_GB/document_library/EPAR_-_Summary_for_the_public/human/002642/WC500163123.pdf. Accessed 8 Aug 2016.
ClinicalTrials.gov. https://clinicaltrials.gov. Accessed 20 Apr 2016.
van Zuuren EJ, Fedorowicz Z, Carter B, van der Linden MM, Charland L. Interventions for rosacea. Cochrane Database Syst Rev. 2015;4:CD003262.
Kirkham JJ, Dwan KM, Altman DA, Gamble C, Dodd S, Smyth R, Williamson PR. The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews. BMJ. 2010;340:c365.
COMET Initiative. http://www.comet-initiative.org/. Accessed 20 Apr 2016.
Cochrane Skin Group - Core Outcome Set Initiative (CSG-COUSIN). https://www.uniklinikumdresden.de/COUSIN/. Accessed 21 Apr 2016.
Schmitt J, Williams HC. HOME Development Group. Harmonising Outcome Measures for Eczema (HOME). Report from the first international consensus meeting (HOME 1), 24 July 2010, Munich, Germany. Br J Dermatol. 2010;163(6):1166–8.
Schmitt J, Spuls P, Boers M, Thomas K, Chalmers J, Roekevisch E, Schram M, Allsopp R, Aoki V, Apfelbacher C, Bruijnzeel-Koomen C, Bruin-Weller M, Charman C, Cohen A, Dohil M, Flohr C, Furue M, Gieler U, Hooft L, Humphreys R, Ishii HA, Katayama I, Kouwenhoven W, Langan S, Lewis-Jones S, Merhand S, Murota H, Murrell DF, Nankervis H, Ohya Y, Oranje A, Otsuka H, Paul C, Rosenbluth Y, Saeki H, Schuttelaar ML, Stalder JF, Svensson A, Takaoka R, Wahlgren CF, Weidinger S, Wollenberg A, Williams H. Towards global consensus on outcome measures for atopic eczema research: results of the HOME II meeting. Allergy. 2012;67(9):1111–7.
Harmonising Outcome Measures for Eczema. (2013). Minutes of the HOME III Meeting. Retrieved from http://www.nottingham.ac.uk/homeforeczema/documents/minutes-home-iii-final.pdf.
Schmitt J, Spuls PI, Thomas KS, Simpson E, Furue M, Deckert S, Dohil M, Apfelbacher C, Singh JA, Chalmers J, Williams HC. The Harmonising Outcome Measures for Eczema (HOME) statement to assess clinical signs of atopic eczema in trials. J Allergy Clin Immunol. 2014;134(4):800–7.
Chalmers JR, Schmitt J, Apfelbacher C, Dohil M, Eichenfield LF, Simpson EL, Singh J, Spuls P, Thomas KS, Admani S, Aoki V, Ardeleanu M, Barbarot S, Berger T, Bergman JN, Block J, Borok N, Burton T, Chamlin SL, Deckert S, DeKlotz CC, Graff LB, Hanifin JM, Hebert AA, Humphreys R, Katoh N, Kisa RM, Margolis DJ, Merhand S, Minnillo R, Mizutani H, Nankervis H, Ohya Y, Rodgers P, Schram ME, Stalder JF, Svensson A, Takaoka R, Teper A, Tom WL, von Kobyletzki L, Weisshaar E, Zelt S, Williams HC. Report from the third international consensus meeting to harmonise core outcome measures for atopic eczema/dermatitis clinical trials (HOME). Br J Dermatol. 2014;171(6):1318–25.
Schmitt J, Apfelbacher C, Spuls PI, Thomas KS, Simpson EL, Furue M, Chalmers J, Williams HC. The Harmonizing Outcome Measures for Eczema (HOME) roadmap: a methodological framework to develop core sets of outcome measurements in dermatology. J Invest Dermatol. 2015;135(1):24–30.
Moza A, Benstoem C, Autschbach R, et al. A core outcome set for all types of cardiac surgery effectiveness trials: a study protocol for an international eDelphi survey to achieve consensus on what to measure and the subsequent selection of measurement instruments. Trials. 2015;16:545.
Guyatt GH, Oxman AD, Kunz R, et al. GRADE guidelines: 2. Framing the question and deciding on important outcomes. J Clin Epidemiol. 2011;64(4):395–400.
Harman NL, Bruce IA, Kirkham JJ, Tierney S, Callery P, O’Brien K, Bennett AMD, Chorbachi R, Hall PN, Harding-Bell A, Parfect VH, Rumsey N, Sell D, Sharma R, Williamson PR. The importance of integration of stakeholder views in core outcome set development: otitis media with effusion in children with cleft palate. PLoS One. 2015;10(6):e0129514.
This publication was supported by Merz Center for Quality and Outcomes Research in Dermatologic Surgery and the IMPROVED (Measurement of Priority Outcome Variables in Dermatologic Surgery) Group.
Availability of data and materials
MA and SI had full access to the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. MA, SI, PW, JS, and EP were responsible for the study concept and design. SI was responsible for the acquisition of data. SI, MA, and EP were responsible for analysis and interpretation of the data. SI and MA were responsible for drafting of the manuscript. SI, MA, PW, JS, EP, IAM, JFS, LJ, TVC, and DS were responsible for critical revision of the manuscript for important intellectual content. MA obtained funding and was the study supervisor. All authors read and approved the final manuscript.
The authors declare that they have no competing interests.
Consent for publication
All authors consent.
Ethics approval and consent to participate
Ethical approval and consent to participate for the study has been granted from the Northwestern University Institutional Review Board (IRB) protocol ID: STU00201637.
About this article
Cite this article
Iyengar, S., Williamson, P.R., Schmitt, J. et al. Development of a core outcome set for clinical trials in rosacea: study protocol for a systematic review of the literature and identification of a core outcome set using a Delphi survey. Trials 17, 429 (2016). https://doi.org/10.1186/s13063-016-1554-3
- Core outcome set
- Systematic review