Developing a core outcome set for chronic rhinosinusitis: a systematic review of outcomes utilised in the current literature
Trials volume 18, Article number: 320 (2017)
A core outcome set (COS) is an agreed standardised collection of outcomes that should be measured and reported by all trials for a specific clinical area, in this case chronic rhinosinusitis. These are not restrictive and researchers may continue to explore other outcomes alongside these that they feel are relevant to their intervention. The aim of this systematic review was to identify the need for a COS for chronic rhinosinusitis.
A sensitive search strategy was used to identify all published Cochrane systematic reviews and randomised control trials of intervention for adult patients with chronic rhinosinusitis. Two independent authors reviewed these to obtain a list of outcomes and outcome measures reported by each clinical trial.
Sixty-nine randomised control trials and eight Cochrane systematic reviews were included in this study. They reported 68 individual outcomes and outcome measures, with an average of four to ten outcomes per clinical trial. These outcomes were mapped to 23 subcategories belonging to eight core categories.
The key finding of this review was the heterogeneity of outcomes reported and measured by clinical trials of patients with chronic rhinosinusitis, precluding meaningful meta-analysis of data. This review supports the need for development of a COS, to be used in future trials on adult patients with chronic rhinosinusitis.
Chronic rhinosinusitis (CRS) is a disease in which patients develop inflammation of their nose and paranasal sinuses, with symptoms persistent for longer than 12 weeks. It may exist with or without nasal polyps . It represents a common and widespread source of ill health in the UK with 11% of UK adults reporting symptoms of CRS . It is also a source of substantial economic burden, through both direct health care costs and indirect societal costs secondary to lost economic productivity [3, 4].
Symptoms of CRS include nasal obstruction, nasal discharge, facial pain, loss of smell and sleep disturbance. When measured using the generic quality of life outcome measure, the Short-form-36, these symptoms have a major impact on patient’s quality of life, worse in some domains than other chronic diseases including angina and chronic respiratory disease . Acute exacerbations, inadequate symptom control and respiratory disease exacerbation are common amongst this population. Complications are rare, but may include visual impairment and intracranial infection.
There is considerable variation in the way that CRS is managed. This relates, in part, to the lack of strong recommendations in treatment guidelines. There are a number of Cochrane reviews evaluating the effectiveness of treatments in CRS, but they are limited both by a paucity of high-quality randomised trials, and the heterogeneity of outcomes in those that have been reported which precludes meaningful meta-analysis. In order to overcome this, both the European Position Paper on Rhinosinusitis and Nasal Polyps, and the recently revised Cochrane systematic reviews, recognise the need to define a core outcome set for CRS [1, 6,7,8,9,10].
A COS is an agreed standardised collection of outcomes that should be measured and reported by all trials for a specific clinical area, in this case CRS . The outcomes defined in a COS are not restrictive and trialists may continue to explore other outcomes alongside these that they feel are relevant to their intervention. However, the primary outcome should be one contained within the COS and if the COS is not being implemented then researchers should explain this decision in their findings .
The use of a COS in future trials of intervention in adult patients with CRS serves to minimise heterogeneity between outcomes reported by trials, allowing research data to be pooled for more meaningful meta-analysis, increasing both numbers of patients available and statistical power. The other advantage of a COS is that the outcomes are reflective of all health service users, including patients. The use of COS is supported by the World Health Organisation  and the Cochrane reviews of effects of health care interventions and have already been developed and adopted by multiple medical and surgical specialties .
The aim of this systematic review is to summarise the reporting standards of surgical outcomes in trials of intervention in adult patients with CRS with or without nasal polyposis, providing evidence in support of further development of a COS. It also aims to identify a list of outcomes required in the development of a COS.
This systematic review was conducted in accordance with the published Preferred Reporting Items for Systematic Reviews and Meta-analysis (PRISMA) guidelines . The study protocol was registered with the Core Outcome Measures in Effectiveness Trials (COMET) initiative (www.comet-initiative.org/) .
At the suggestion of COMET, high-quality randomised control trials for inclusion in this systematic review were obtained through a systematic search of the Cochrane Database of Systematic Reviews (16 Aug 2015) and the Cochrane Central Register of Controlled Trials (CENTRAL) (16 Aug 2015). The search was performed using the search terms, rhinosinusitis or ENT or Otolaryngology. Language, age and date restrictions were not applied to the searches.
Systematic reviews of trials of intervention in patients, 18 years or older, with CRS with or without nasal polyps were included. Those systematic reviews that had included patients with asthma, aspirin sensitivity and allergic fungal rhinosinusitis were also included.
Studies looking at CRS in a paediatric population, defined as age less than 18 years, studies of patients with allergic rhinitis or those with patients suffering with secondary CRS (patients with cystic fibrosis, granulomatosis with polyangiitis and other systemic diseases other than the ones listed above) were excluded. Published Cochrane systematic review protocols of on-going reviews were excluded.
The full-text articles for the Cochrane systematic reviews were obtained and analysed for eligibility by two independent reviewers, with a third reviewer available to adjudicate discrepancies.
As each systematic review presents the results of a number of pooled clinical trials, the published papers for each individual trial were obtained. The bibliographies of the trial papers were further evaluated to identify additional studies for inclusion, not included within the Cochrane systematic reviews themselves. If we could not find the published full-text article of each randomised control trial, the authors were contacted directly to see if they could provide a copy of their original research paper or thesis.
Two independent reviewers extracted data from the randomised controlled trials (RCTs). Two independent spreadsheets were created and the primary outcome (if defined), secondary outcome and the Outcome Measure Instrument (OMI) used to measure both the primary and secondary outcome in each trial listed. These outcomes and outcome measures were obtained from the method and results section of each paper. If the paper did not explicitly mention the outcomes that they measured, then the reviewer inferred these from the given data. At the end of the data extraction process, a final list of outcomes was compiled by consensus between both reviewers and a single spreadsheet created, containing details of each study, including sample size, study type, outcome and validity of outcome measure.
Previous work perfumed as part of the OMIPP project, commissioned by Cochrane UK, surveyed Public, Patients and Practitioners to identify a list of outcomes that they felt were important in Cochrane reviews and trials of intervention for patients with CRS. We employed the categories identified as part of this project, using them to map our outcomes and OMIs, as appropriate . The OMIPP project identified 23 predetermined core categories belonging to eight main categories, namely ‘Changes in patient-related symptom severity’, ‘Quality of life’, ‘Physiological assessments’, ‘Microbiological’, ‘Biomarkers’, ‘Lower airway disease’, ‘Side effects’ and ‘Acceptability of treatment’.
Each individual outcome from this study was mapped to a core subcategory. These subcategories were then grouped into main categories which were then placed into the Outcome Measures in Rheumatoid Arthritis Clinical Trials (OMERACT) filter 2.0. The OMERACT filter 2.0 has been developed by a group of international health care professionals involved in improving outcomes reporting in rheumatology and one of the first groups to develop a COS. They have provided a framework to be considered when developing a COS which includes death, life impact and resource use/emotional impact, pathophysiological manifestations and adverse events .
Following a systematic search of the Cochrane database, using the PRISMA guidelines, a total of 49 systematic Cochrane review articles were identified Once the inclusion and exclusion criteria were applied, this number reduced to 18. Duplicate entries and published protocols of on-going studies were excluded to leave ten systematic reviews which were included in this study. A summary flowchart is provided in Fig. 1 below.
A total of 83 RCT papers were included within these ten Cochrane systematic review papers. Of these, 14 were excluded as they were trials of paediatric patients, those with allergic rhinitis or published conference abstracts. A total of 69 RCTs were finally obtained from these systematic reviews and included within this study. No further references were identified through examination of the bibliography of the published systematic reviews or trials. All included articles were published in the English language. The included clinical trials were published between the years 1975 and 2012. Table 1 provides the details of the systematic reviews and the number of trials extracted from each of these.
Three hundred and sixty-five individual outcomes were extracted from the clinical trials. The trials overlapped in their use of outcomes and between them had used 68 different outcomes and OMIs. Initially these were divided into two broad categories, namely category 1 ‘outcome’ and category 2 ‘Outcome Measure Instrument (OMI) used to assess the specific outcome’. This produced an extensive list, with multiple OMIs used to measure individual outcomes (Table 2). We employed the categories identified as part of the OMIPP project (previously discussed in the ‘Methods’ section), using them to map our outcomes and OMIs, as appropriate . Hence, our long-list was mapped onto 23 predetermined ‘core categories’, belonging to eight ‘main categories’, namely ‘Changes in patient-related symptom severity’, ‘Quality of life’, ‘Physiological assessments’, ‘Microbiological’, ‘Biomarkers’, ‘Lower airway disease’, ‘Side effects’ and ‘Acceptability of treatment’ (Table 3).
These main categories were then mapped into the five OMERACT subheadings, ‘Life Impact’, ‘Pathophysiology’, ‘Resource use/Economic impact’, ‘Adverse Events’ and ‘Death’ (Table 4). No clinical trial in our study had included an outcome for the OMERACT core area ‘Death’ .
Most trials reported between four and ten outcomes each. The most frequently reported outcome was the patient’s individual perception of their health and severity, measured using either a Visual Analogue Scale (VAS) or the patient’s history or patient diary (n = 51 RCTs). The most frequently used outcome measure was an endoscopic assessment of the patient’s sinonasal cavity (n = 55 RCTs).
The COMET initiative was supportive of this study and, at the recommendation of their lead biostatistician, we extracted outcomes from RCTs that had been included in Cochrane systematic reviews, hopefully reflecting high-quality, well-designed research studies.
The key finding of this systematic review was the heterogeneity of outcomes reported and measured by Cochrane systematic reviews and clinical trials of adult patients with CRS. Most trials had not explicitly defined the primary outcome for their study or provided definitions for the outcomes that they were measuring, with many simply providing a list of outcome measures, such as nasoendoscopy or radiology, that were being used by them to evaluate effectiveness of treatment in their study. Sixty-nine RCTs had employed 68 individual outcomes between them to report effectiveness of various treatments being studied on patients with CRS. The biggest drawback of this is the inability to pool patient data and perform meaningful meta-analysis.
A positive finding was that most studies had included a measure of ‘life impact’ reflecting the patient’s perception of their own health and its impact on their quality of life. However this was achieved using a variety of VASs or simply based on the patients history, hence making comparison between studies more difficult.
The most commonly reported subgroup of outcomes was ‘Pathophysiology’. Some of the outcomes included in this group were the outcome measures ‘nasoendoscopy’, ‘radiology’, ‘mucociliary function’, ‘nasal volume’ and ‘resistance’. This is unsurprising as these are frequently used clinical measures that allow clinicians to assess objective response to treatment. However, 16 different outcomes measures had been used in this category alone, hence once again making direct comparison between studies difficult.
None of the included studies had assessed ‘resource use/economic impact’ of disease and only a handful had asked about patient preference and acceptability of the treatment to the patient. These may be of importance when considering compliance and longevity of treatment. They may also be important to stakeholders other than physicians, including patients and third parties such as health care providers, insurance companies and drug companies.
The OMERACT category ‘death’ was not reported by any of our trials. Patients with CRS are generally fit patients undergoing elective surgery or medical therapy and hence we would not expect that patients should die as a direct result of the treatment being offered . We must remember that the OMERACT filter 2.0 was developed for patients with rheumatoid arthritis, and although the breadth of outcomes it offers is excellent, it must perhaps be modified for our cohort of patients with CRS, with the exclusion of the death category.
We found that historical trials reported fewer outcomes, rarely defined the outcome and used simpler OMIs such as nonvalidated VAS scores and facial roentgenograms. After the late nineties these evolved to include better-defined outcomes, with multiple validated outcome measures. Also, the trend for inclusion of qualitative outcome measures changed with time; with newer studies including patient-reported outcome measures alongside qualitative outcomes.
The key strength of this study is the large number of clinical trials included, providing an exhaustive list of outcomes that have been reported in the academic literature. However, a limitation of this study is that we have only included outcomes that are important to researchers and clinicians. We have not captured outcomes important to patients themselves or other stakeholders such as primary care physicians.
To improve the quality of trials included in this review, with minimal bias, only those studies published within Cochrane systematic reviews were included in this study. Hence, a study of other search engines, as would be routine practice when researching a systematic review, was not performed. This does risk missing some good trials published independently of the Cochrane reviews.
In order to overcome the two limitations discussed, the outcomes obtained through this review, will be combined with (a) outcomes obtained through patient focus groups, (b) outcomes obtained through Public, Patient and Practitioner surveys  and (c) outcomes obtained through interviews with family practitioners and otolaryngologists. It is this final long-list of outcomes, representing all stakeholders, that will be used in the development of a COS via the Delphi process [12, 19].
The Cochrane Collaboration has recently updated a few of the published systematic reviews included in this study [6,7,8, 10, 20]. For completeness, full-text articles of these updated reviews and any trials not previously reviewed were analysed. This has not revealed any adjunctive outcomes, for inclusion in the COS development process.
Once a final list of outcomes has been formulated, we will ask patients and clinicians to define the outcomes that have been extracted from these trials, providing standardised definitions that we, both clinicians and patients, understand. The involvement of patients in contributing and then defining outcomes will ensure that the target audience for the treatment is an active participant in the development of the COS.
As previously explored, developing a COS for this subgroup of patients will allow meaningful comparisons to be made between future clinical trials of novel and existent therapies, with the ability to combine results and produce meaningful, high-quality research data, with an overall positive impact on patient management.
This systematic review supports the need for development of a COS in CRS and has identified a potential list of outcomes to be used in this process.
Core Outcome Measures in Effectiveness Trials
Core outcome set
Outcome Measures in Rheumatoid Arthritis Clinical Trials
Preferred Reporting Items for Systematic Reviews and Meta-analysis
Randomised control trial
Fokkens WJLV, Mullol J, Bachert C, Alobid I, Barody F, et al. European position paper on rhinosinusitis and nasal polyps. Rhinology. 2012;50 Suppl 23:1–298.
Hastan D, Fokkens WJ, Bachert C, Newson RB, Bislimovska J, Bockelbrink A, et al. Chronic rhinosinusitis in Europe—an underestimated disease. A GA(2) LEN study. Allergy. 2011;66(9):1216–23.
Caulley L, Thavorn K, Rudmik L, Cameron C, Kilty SJ. Direct costs of adult chronic rhinosinusitis by using 4 methods of estimation: results of the US Medical Expenditure Panel Survey. J Allergy Clin Immunol. 2015;136(6):1517–22.
Smith KV, Orlandi RR, Rudmik L. Cost of adult chronic rhinosinusitis: a systematic review. Laryngoscopy. 2015;125(7):1547–56.
Gliklich RE, Metson R. The health impact of chronic sinusitis in patients seeking otolaryngologic care. Otolaryngol Head Neck Surg. 1995;113(1):104–9.
Chong LY, Head K, Hopkins C, Philpott C, Glew S, Scadding G, et al. Saline irrigation for chronic rhinosinusitis. Cochrane Database Syst Rev. 2016;4:Cd011995.
Chong LY, Head K, Hopkins C, Philpott C, Schilder AG, Burton MJ. Intranasal steroids versus placebo or no intervention for chronic rhinosinusitis. Cochrane Database Syst Rev. 2016;4:Cd011996.
Chong LY, Head K, Hopkins C, Philpott C, Burton MJ, Schilder AG. Different types of intranasal steroids for chronic rhinosinusitis. Cochrane Database Syst Rev. 2016;4:Cd011993.
Head K, Chong LY, Hopkins C, Philpott C, Schilder AG, Burton MJ. Short-course oral steroids as an adjunct therapy for chronic rhinosinusitis. Cochrane Database Syst Rev. 2016;4.
Head K, Chong LY, Piromchai P, Hopkins C, Philpott C, Schilder AG, et al. Systemic and topical antibiotics for chronic rhinosinusitis. Cochrane Database Syst Rev. 2016;4:Cd011994.
Clarke M. Standardising outcomes for clinical trials and systematic reviews. Trials. 2007;8:39.
Williamson PR, Altman D, Blazeby J, Clarke M, Devane D, Gargon E, et al. Developing core outcome sets for clinical trials: issues to consider. Trials. 2012;13:132.
WHO: WHO handbook for guideline development 2010. Available from: www.who.int/hiv/topics/grc_mar2010_1.pdf.
Rosenbaum SE, Glenton C, Nylund HK, Oxman AD. Summary-of-findings tables in key information. J Clin Epidemiol. 2010;63:620–6.
Moher D, Liberati A, Tetzlaff J, Altman DG, Group P. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ. 2009;339:b2535.
Gargon E, Willamson PR, Altman DG, Blazeby JM, Clark M. The COMET initiative database: progress and activities update (2014). Trials. 2015;16:515.
Hopkins C, Philpott C, Crowe S, Regan S, Degun A, Papachristou I, et al. Identifying the most important outcomes for systematic reviews of interventions for rhinosinusitis in adults: working with Patients, Public and Practitioners. Rhinology. 2016;54(1):20–6.
Boers M, Kirwan JR, Gossec L, Philip GC, D’Agostino M, Bingham CO, et al. How to choose core outcome measurement sets for clinical trials: OMERACT II Approves Filter 2.0. J Rheumatol. 2014;41:1025–30.
Sinha IP, Smyth RL, Williamson PR. Using the Delphi Technique to determine which outcomes to measure in clinical trials: recommendations for the future based on a systematic review of existing studies. PLoS Med. 2011;8(1), e1000393.
Rimmer J, Fokkens W, Chong LY, Hopkins C. Surgical versus medical interventions for chronic rhinosinusitis with nasal polyps. Cochrane Database Syst Rev. 2014;12, CD006991.
Khalil HS, Nunez DA. Functional endoscopic sinus surgery for chronic rhinosinusitis. Cochrane Database Syst Rev. 2006;3, CD004458.
Harvey R, Hannah SA, Badia L, Scadding G. Nasal saline irrigations for the symptoms of chronic rhinosinusitis. Cochrane Database Syst Rev. 2007;3:CD006394.
Piromchai P, Thanaviratananich S, Laopaidboon M. Systemic antibiotics for chronic rhinosinusitis without nasal polyps in adults. Cochrane Database Syst Rev. 2011;5:CD008233.
Sacks PL, Harvey RJ, Rimmer J, Gallagher RM, Sacks R. Topical and systemic antifungal therapy for the symptomatic treatment of chronic rhinosinusitis. Cochrane Database Syst Rev. 2011;8, CD008263.
Snidvongs K, Kalish L, Sacks R, Craig JC, Harvey RJ. Topical steroid for chronic rhinosinusitis without polyps. Cochrane Database Syst Rev. 2011;8:CD009274.
Ahmed J, Pal S, Hopkins C, Jayaraj S. Functional endoscopic balloon dilation of sinus ostia for chronic rhinosinusitis. Cochrane Database Syst Rev. 2011;7, CD008515.
Kalish L, Snidvongs K, Sivasubramaniam R, Cope D, Harvey RJ. Topical steroids for nasal polyps. Cochrane Database Syst Rev. 2012;12, CD006549.
Huang Z, Hwang P, Sun Y, Zhou B. Steroid-eluting sinus stents for improving symptoms in chronic rhinosinusitis patients undergoing functional endoscopic sinus surgery (Review). The Cochrane library. 2015;6:CD010436.
Sharma R, Lakhani R, Rimmer J, Hopkins C. Surgical interventions for chronic rhinosinusitis with nasal polyps (review). The Cochrane library. 2014;11:CD006990.
No external or internal funding was received to support this systematic review.
Availability of data and materials
The datasets used and analysed during the current study are available from the corresponding author on reasonable request.
Ethical approval and consent to participate
Ethical approval for this study was obtained from the West of Scotland Research Ethics Service (WoRES). REC reference number 16/WS/0097; IRAS project ID 200646. Patients did not participate in this study.
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Soni-Jaiswal, A., Lakhani, R. & Hopkins, C. Developing a core outcome set for chronic rhinosinusitis: a systematic review of outcomes utilised in the current literature. Trials 18, 320 (2017). https://doi.org/10.1186/s13063-017-2060-y