- Study protocol
- Open Access
- Open Peer Review
This article has Open Peer Review reports available.
Standardising the reporting of outcomes in gastric cancer surgery trials: protocol for the development of a core outcome set and accompanying outcome measurement instrument set (the GASTROS study)
© The Author(s). 2017
Received: 28 March 2017
Accepted: 9 July 2017
Published: 9 August 2017
Gastric cancer is one of the leading causes of cancer-related deaths worldwide. Whilst surgery is the mainstay of curative treatment, it is associated with significant risks. Surgical strategies for treating gastric cancer should be based on evidence from systematic reviews of well-designed randomised controlled trials. However, inconsistencies in the reporting of outcomes from these trials makes evidence synthesis unreliable. We present a protocol for an international consensus study to develop a standardised set of outcomes and measurement tools – a ‘core outcome set’ (COS) – to be used by all future trials examining therapeutic surgical interventions for gastric cancer. The GASTROS study aims to standardise the reporting of outcomes in gastric cancer surgery trials through an international consensus process of key stakeholders including health care professionals and patients.
The first of three stages in the study will identify a ‘long-list’ of potentially important outcomes to be prioritised. These will be extracted from a systematic review of relevant academic literature and patient interviews. Stage 2 will comprise an eDelphi survey which will consider the views of patients, nurse specialists and surgeons to prioritise the most important outcomes. A meeting of stakeholder representatives will ratify the COS.
Stage 3 will focus on identifying appropriate instruments to measure the prioritised outcomes by means of quality assessment of available measurement instruments and stakeholder consultation.
This study aims to standardise the reporting of outcomes in future trials examining therapeutic surgical interventions for gastric cancer. It is anticipated that standardisation of outcome reporting in these surgical effectiveness trials will enhance the evidence base for clinical practice. Highlighting outcomes of greatest importance to patients will ensure that their perspectives are central to research in this field.
Gastric cancer is one of the leading causes of cancer-related deaths worldwide  and despite developments in multimodal treatment approaches, overall survival rates have not improved significantly over the last four decades . Surgery to remove part or all the stomach continues to be the mainstay of treatment that offers a potential cure; however, this is associated with significant risks of short and long-term complications [3, 4]. Variations in surgical approaches aim to minimise these risks without compromising the oncological resection of the tumour. These variations can be broadly categorised into those related to accessing the stomach (e.g. open, laparoscopic or robotic surgery) and those related to the extent of surgery (e.g. partial or total gastrectomy, level of lymphadenectomy and whether splenectomy is required).
In principle, assessing the optimal surgical strategies for gastric cancer should involve analyses of well-designed and well-conducted randomised controlled trials (RCTs) with systematic reviews and meta-analyses of data. However, trials are often methodologically heterogenous, report and measure their outcomes differently and preclude comprehensive evidence synthesis. Consequently, strong recommendations for clinical practice can seldom be made [5, 6]. In instances where trials may report the same outcomes, the definitions of these outcomes are often inconsistent and it is not known to what degree these outcomes may be relevant to key stakeholders such as patients.
In preparing this protocol, a rapid review of RCTs (published between January 2014 and January 2016) examining therapeutic surgical interventions for gastric cancer was undertaken using a structured search strategy applied to MEDLINE and EMBASE via Ovid. In the six trials identified, a total of 102 outcomes were reported, only 15% of which were defined. No single outcome was reported by every trial and only one trial described patient-reported outcomes. No trial measured quality of life after surgery.
Many groups have now demonstrated similar, widespread inconsistencies in outcome reporting [7–10]. Consequently, there has been a drive, with the support of initiatives such as COMET (Core Outcomes Measurement in Effectiveness Trials) and the Medical Research Council’s Hubs for Trials Methodology Research, to standardise the reporting of outcomes as an important step in improving trial design and reducing research waste.
Aims and objectives
One solution to this problem is through the development of a ‘core outcome set’ (COS). A COS is defined as an agreed minimum set of outcomes that should be measured and reported in all trials in a specific condition . The aim of the GASTROS study (GAstric cancer Surgery Trials Reported Outcomes Standardisation) is to develop a COS to be used by all trials examining therapeutic surgical interventions for gastric cancer, which reflects the interests of both patients and health care professionals.
To determine the degree of variation in the reporting of outcomes in the academic literature
To identify a list of potentially important outcomes from published trials and trial protocols
To identify a list of potentially important outcomes reported by patients, in semi-structured interviews, who have been treated for gastric cancer, to augment the list generated in item 2
To reach consensus regarding the most important outcomes from the perspective of patients and health care professionals into a COS
To identify appropriate outcome measurement instruments (OMIs) to be used in the reporting of the COS and at what time points the outcomes should be measured
This study will draw its methodological principles from recommendations developed by initiatives such as COMET and COSMIN (COnsensus-based Standards for the selection of health Measurement INstruments) and modified where necessary and appropriate [12–17]. The COMET initiative has been instrumental in propagating the agenda for change in relation to outcomes reporting internationally and has amassed a wealth of knowledge and experience during the last 6 years. Whilst the field of COSs is still relatively new, COMET is supporting the development of hundreds of COSs demonstrated by over 400 completed, ongoing or planned studies across a wide spectrum of clinical specialties referenced in its online database . COSMIN, whose focus lies on developing rigorous methods of OMI selection, is working closely with COMET and has developed standards for selecting instruments used in the reporting of COSs .
This COS is primarily aimed at pragmatic trials examining therapeutic surgical interventions for gastric cancer. The target population is male and female adults. We foresee that the COS will also be beneficial for the design of non-randomised studies and will inform the design of databases and national audits by identifying the priorities of patients and health care professionals.
Given that there is now a greater acceptance that the management of gastric cancer is often multimodal (involving a combination of surgical excision and chemotherapy or chemo-radiotherapy) , it may be argued that a COS would be more relevant if it were to encompass all therapeutic interventions and not just surgical ones. A structured search of the World Health Organisation’s International Clinical Trials Registry Portal (http://apps.who.int/trialsearch/, last accessed 3 August 2016) and ClinicalTrials.Gov (https://clinicaltrials.gov/, last accessed 3 August 2016) has identified 24 ongoing surgical gastric cancer trials planning to recruit approximately 11,000 patients. The rate at which these surgical trials are being set up does not show signs of slowing. As such, a surgically focused COS is highly relevant given the research activity within this field. Furthermore, there is a significant proportion of patients who do not require multimodal therapy due to early stage disease or on the account that they are unfit for additional therapies. In addition, given the large variation in surgical practice that already exists, and the range of therapeutic surgical interventions which have been, and are being, investigated, we believe that a surgically orientated COS is both desirable and necessary. Nonetheless, our group recognises that future work is required to develop a COS which will be relevant to non-surgical interventions and this is within our planned programme of work in conjunction with endoscopic and medical and clinical oncology groups.
Definition of terms used in the GAstric cancer Surgery Trials Reported Outcomes Standardisation (GASTROS) study
Core outcome set (COS)
An agreed minimum set of outcomes that should be measured and reported in all trials in a specific condition 
A unique endpoint which attempts to describe health-related changes that occur secondary to a therapeutic intervention, e.g. hospital-acquired pneumonia
A collection of ‘outcomes’ which share common features, e.g. the outcome domain ‘respiratory complications’ would include outcomes such as ‘pleural effusion’, ‘hospital-acquired pneumonia’ and ‘atelectasis’
Outcome measurement instrument (OMI)
A method or tool used to measure an ‘outcome’ or an ‘outcome domain’
Outcome measurement instrument set
A collection of OMIs which are used to measure outcome domains in a COS
An important aspect of the GASTROS study design is ensuring that key stakeholder opinion is represented at every stage of the COS development. Our primary stakeholder groups include patients (with ‘lived experience’ of the condition and its management), surgeons (those directly delivering and developing the clinical interventions) and clinical nurse specialists (as they have an important dual role as health care professionals and patient advocates). In addition to their participation highlighted below (see ‘Study design’), representatives from each group have been recruited to our Study Advisory Group to support the general delivery of the study against its stated objectives and ensure that the viewpoints of all stakeholders are considered throughout the process.
Most surgical gastric cancer trials are undertaken in the Far East where the incidence and prevalence is highest . It is, therefore, essential that aspects of the COS development take this international perspective into account. As such, representatives from all stakeholder groups will be invited to participate in the eDelphi survey. They will be drawn from a broad network of national and international patient groups, charities, professional associations and institutions. This is further elaborated upon below (see ‘Dissemination and implementation strategy’).
Stage 1. Generation of ‘long-list’ of outcomes
Stage 2. Prioritisation of outcomes and finalisation of a COS
Stage 3. Identification of OMIs
Stage 1. Generation of ‘long-list’ of outcomes
A long-list of potentially important outcomes will be identified by means of a systematic academic literature search and semi-structured patient interviews. This will be followed by a consultation exercise to finalise a list of outcomes to be prioritised by stakeholders in stage 2 of the study.
Systematic academic literature search
A systematic review of randomised control trials (RCTs) and protocols of RCTs examining therapeutic surgical interventions for gastric cancer will be undertaken. Systematic reviews of RCTs will be scrutinised to identify publications not previously identified. We will limit our analysis to RCTs as the primary purpose of the GASTROS study is to influence future RCTs.
Abstracts will be screened by two researchers and relevant publications identified. All reported outcomes will be extracted verbatim in addition to definitions and OMIs used. Validated Patient-reported Outcome Measurements (PROMs), such as those used to report ‘quality of life’, will be critically reviewed to identify further outcomes to be added to the ‘long-list’ . Whilst overall quality of life may be deemed an outcome that should be prioritised and included in a COS, there may be components related to eating and drinking, for example, that may be deemed important outcomes within their own right.
Previous reports have highlighted that patients often have differing priorities and perspectives relating to outcomes measured in trials [21–24]. To ensure that the views of patients are adequately considered, a series of semi-structured qualitative interviews, with patients who have previously undergone surgery, will be undertaken. All interviews will be audio-recorded, transcribed and interrogated for themes which may supplement the outcomes already gathered from the systematic academic literature review. There is no ‘sample size’ calculation for qualitative research. The total number of participants should be guided by the concept of ‘saturation’, whereby further interviews do not result in the identification of new outcomes, and can range from between 5 and 50 participants [25, 26]. Based on the authors’ experience in qualitative research methods in COS development, we expect that between 15 and 30 patients will need to be interviewed before ‘saturation’ is reached.
To ensure that a broad range of views are expressed during the interviews, we aim to purposefully sample patients based on several characteristics. These include age, sex, time since surgery, type of surgical approach (open or minimally invasive) and whether patients have undergone other perioperative therapies (chemotherapy or chemo-radiotherapy).
Stage 2: Prioritisation of outcomes to finalise a COS overview
Delphi surveys have been used in many COS projects to reach consensus on the most important outcomes to include [14, 27, 28]. One of the main benefits of this approach is that the views of all participants are equally heard. This may not be the case in a face-to-face forum where the views of one individual or group of participants may be more vociferously asserted. There is no fixed methodological approach to undertake a Delphi survey. Some groups have retained all potential outcome domains in each round and used the participant responses to inform a final consensus meeting [27, 29], whilst others have only retained outcome domains deemed important in each round . We intend to use a hybrid approach over three rounds as described in greater detail below. Following the Delphi survey, a meeting of key stakeholder representatives will take place to ratify the prioritised outcomes into a COS.
Organising the outcome list in preparation for stage 2
Once potentially important outcomes have been identified from the systematic review and patient interviews, a final long-list of items will be compiled for the eDelphi survey. We plan to recruit at least 100 participants and so to minimise non-response and attrition between survey rounds, the initial number of items submitted to the Delphi survey will need to be carefully managed. Previous COS developers have aimed for less than 100 initial items for participants to prioritise . To achieve this, individual outcomes will be organised into ‘outcome domains’ (Table 1) whilst ensuring that domains do not become too broad. For example, the outcomes ‘hospital-acquired pneumonia’, ‘pleural effusion’ and ‘atelectasis’ grouped together under the outcome domain ‘respiratory complications’ may be appropriate, whereas grouping the same outcomes under the domain ‘complications’ may be too specific. The process of compiling and finalising the outcome domains will be undertaken during a meeting of key stakeholder representatives to ensure transparency. This meeting will involve open discussion of each outcome domain, including information relating to how the outcome domain was formulated, to ensure that it is not too broad or specific. An outcome domain will be admitted into the long-list for the subsequent eDelphi survey once agreement by majority regarding its appropriateness has been reached by all stakeholder representatives.
Each item entered into the survey will be described in lay terms with an additional scientific description. For example, an ‘anastomotic leak’ may be described as ‘a leak from the join between the stomach and the bowel’. All item descriptions will be reviewed by the study group and patient representatives. Items will be presented to participants as collections with similar characteristics (e.g. outcomes related to ‘adverse events’ or ‘technical aspects of surgery’).
Participants and sample size
Representatives from our three primary stakeholder groups – patients, clinical nurse specialists and surgeons – will be invited to participate in the Delphi survey. Whilst there is no accepted or required ‘sample size’ requirement for a Delphi survey , we aim to recruit at least 100 participants in total. The views of each stakeholder group will be considered separately which will enable intra- and inter-stakeholder group variability to be explored. As explained previously, gastric cancer is a worldwide disease, and as such, participants will be sought internationally through a network of patient groups, organisations, professional associations and cancer institutes. The Delphi survey will be Internet-based; however, we will give participants the opportunity to complete hard copies of the surveys so as not to exclude those with limited Internet access or knowledge of the Internet.
All items in addition to further new outcome domains identified by participants in round 1 will be carried forward for consideration in round 2. Descriptive statistics will be used to summarise the scores from round 1 and presented to participants. Participants will see the results of their individual score for each outcome in addition to the median score of each stakeholder group. The rationale for showing participants the scores from other groups is that it may improve consensus between the stakeholder groups . In addition, by carrying all items forward from round 1, it may be possible to identify changes in scoring patterns as a result of viewing other scores. Participants will be asked to score all items once again using the 9-point scale.
Outcome domains which are scored ‘critical’ by greater than 70% of participants from any stakeholder group, and ‘unimportant’ by less than 15% of the group, will be carried forward for further consideration in round 3. The rationale for this threshold is that for an outcome domain to be included in the COS, it requires agreement by the majority regarding the critical importance of the outcome, with only a small minority considering it to have little importance. By carrying forward outcomes relevant to at least one stakeholder group, participants will be given another opportunity to reflect on the importance of the outcome domain in the final round. As the scope of this study is to identify the most important outcomes, all other outcomes will be discarded.
All retained outcomes will be summarised and participants will view both their individual scores and those of the other groups before being asked to score items a final time using the 9-point scale. Outcome domains which are scored as ‘critical’ by greater than 70% and ‘unimportant’ by less than 15% of participants from all three groups will be retained for inclusion in the COS.
If a participant does not complete a subsequent round of the Delphi survey, their scores from previous rounds will be counted as valid and retained in the study. Similarly, if a participant fails to score a specific item during a survey round, the answers to other items will be held as valid and retained. The rate of missing responses will be reported with the results of the Delphi survey.
Following the Delphi survey, a meeting of stakeholder representatives will take place to review the results and recommend the outcome domains as a COS.
Stage 3: Identification of outcome measurement tools
Finding existing OMIs
Quality assessment of OMIs
Recommendations on the selection of OMIs for a COS and at what time points they should be used
1. Conceptual considerations
The first step involves identifying the scope of outcomes to be measured. These will be identified in stage 2 of the study. The scope of the COS has been described earlier in this protocol.
2. Finding existing outcome measurement instruments (OMIs)
Existing OMIs will be identified through several approaches. A structured search of MEDLINE and EMBASE via Ovid will identify systematic reviews of OMIs for the outcome domain concerned. If the systematic reviews are of high quality and have undertaken a quality assessment of the OMIs, then one OMI will be selected and presented to a group of key stakeholder representatives at the end of the process.
If there are no systematic reviews or they are of poor quality, then a new or updated academic literature search will be performed. We will search MEDLINE and EMBASE via Ovid to identify studies of OMIs. We will also interrogate reference lists and examine trials and protocols identified in stage 1 of the GASTROS study to identify further OMIs. Studies describing these OMIs will be quality assessed as described below. OMIs identified through up-to-date systematic reviews where quality assessments were not undertaken will also be assessed by the predefined standards below.
3. Quality assessment of OMIs
Each OMI-related study identified will undergo an evaluation of its methodological quality using the COSMIN Checklist . In addition, an evaluation of the measurement properties of the OMI will be undertaken against several predefined criteria  including ‘content validity’ and ‘internal structure’. Content validity is defined as ‘the degree to which the content of a measurement instrument is an adequate reflection of the outcome to be measured’. ‘Internal structure’ is comprised of two aspects – ‘internal consistency’ (the degree of interrelatedness among the items within the OMI) and ‘structural validity’ (the degree to which the scores of a measurement instrument are an adequate reflection of the dimensionality of the outcome to be measured). If either ‘content validity’ or ‘internal structure’ are considered poor or unknown, then the OMI will not be assessed further. The results of this quality assessment will be combined in a ‘best evidence synthesis’ exercise against criteria defined by the COMET-COSMIN guidance.
It is also essential that OMIs are assessed in terms of their feasibility of use. The COMET-COSMIN guidance provides 17 different factors against which feasibility can be assessed. These include ‘patient comprehensibility’, ‘interpretability’, ‘ease of administration’ and ‘completion time’.
4. Generic recommendations on the selection of outcome measurement instruments for a COS
There is ‘high-quality’ evidence of ‘good’ content validity and ‘good’ internal structure
The OMI is feasible to use
‘High-quality evidence’ is defined as consistent findings in multiple studies of at least ‘good’ quality or in one study of ‘excellent’ quality and a total sample size of 100 patients or more (see the COSMIN Checklist  for clarification of the terms ‘good’ and ‘excellent’ quality).
It is possible that more than one OMI can be recommended for an outcome domain. Conversely, it is possible that no OMIs are recommended. This scenario may form the basis of future work to develop an OMI for that domain.
Key stakeholder meeting
Following quality assessment for OMIs for each outcome domain included in the COS, we will invite representatives from each key stakeholder group to review the evidence from this stage of study and ratify the recommended OMIs as an OMI set. The primary function of the stakeholder meeting is to ensure transparency of the process, raise further questions and seek further clarifications (if any). The evidence considered will also inform recommendations made through the stakeholder meeting regarding when these OMIs should be used to measure the core outcomes.
A COS must be implemented widely within its clinical field to have its intended benefit. Whilst grant-awarding bodies and international research groups are increasingly promoting the use of COSs, researchers must be willing to incorporate them into trial designs. Our approach to maximise the use of our COS is one of inclusion of key stakeholders in designing and delivering our study and dissemination of the findings at every stage. Given that most surgical gastric cancer trials are being undertaken in the Far East, this inevitably means involvement of international stakeholders. We are working with several groups in South Korea, Japan and China in addition to European and North and South American teams to ensure that this aspect of our study is facilitated. It is not yet fully understood how language or cultural differences may affect the results of consensus processes such as the one we propose. Our study will provide the opportunity to explore this question further.
Registration of our study with COMET database (http://www.comet-initiative.org/studies/details/764?result=true)
Development of our study website (www.GASTROSstudy.org) where key stakeholders and interested parties can find regular updates and register for participation
Development of our social media identity, e.g. Twitter (@GASTROSstudy)
- 4.Widespread dissemination of our work at every stage of the study through:
National and international scientific meetings
Regular updates to our network of international patient groups and charities, professional associations and cancer centres
Engagement with, and ‘ownership’ of, the COS by professional bodies will also be an important way to facilitate the necessary regular review of the COS. Such reviews are needed to ensure that individual outcomes remain relevant and to add new outcomes as appropriate. No recommendation exists regarding the time interval between reviews, but we anticipate the need for review within 3–5 years.
Improve the reliability of evidence synthesis on which robust clinical guidelines can be based
Improve shared decision-making and the preoperative consent process as outcomes from surgical interventions which are relevant to both clinicians and patients become more apparent
Better equip health care providers how best to prioritise funding for interventions that reflect the needs and priorities of patients
The COS will also inform non-RCT trial design and, additionally, provide a minimum set of outcomes relevant to key stakeholders which can be collected by health care providers and organisations designing national audits and prospective databases.
The GASTROS study will also provide a platform for future work which includes the development of PROMs where they are deficient and the further development of a COS which is relevant to multimodal therapies. Once core outcomes are identified, work can also commence on developing a minimum dataset of factors which can influence these outcomes so that risk adjustment of outcomes and ultimately the external validity of trials can improve.
This study is funded by the National Institute for Health Research (NIHR) Doctoral Research Fellowship Grant programme (DRF-2015-08-023). The funding body has no role in the study design and implementation, writing the report, or decision to submit the report for publication.
Availability of data and materials
Upon the completion of the study, supporting data will be available upon request.
BA contributed to the study design and drafted the manuscript. AMG, JMB, PRW and IAB contributed to the study design and revised the manuscript. All authors read and approved the final manuscript.
Ethics approval and consent to participate
This will be sought for each aspect of the study where ethical approval is required. At the time of writing, the GASTROS study is at stage 1 (Fig. 1). Ethical approval has been authorised for this stage by the NHS Health Research Authority (reference 16/ES/0151).
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
- Fact Sheets by Cancer. http://globocan.iarc.fr/Pages/fact_sheets_cancer.aspx. Accessed 3 Feb 2016.
- Cancer survival for common cancers | Cancer Research UK. http://www.cancerresearchuk.org/health-professional/cancer-statistics/survival/common-cancers-compared#heading-Three. Accessed 16 May 2016.
- Centre H & SCI. National Oesophago-Gastric Cancer Audit 2015. http://www.hscic.gov.uk/article/1165/searchcatalogue?q=%22National+Oesophago-Gastric+Cancer+Audits%22&sort=Date&size=10&page=1#top. Published 2015. Accessed 16 May 2016.
- Messager M, de Steur WO, van Sandick JW, et al. Variations among 5 European countries for curative treatment of resectable oesophageal and gastric cancer: a survey from the EURECCA Upper GI Group (EUropean REgistration of Cancer CAre). Eur J Surg Oncol. 2016;42(1):116–22. doi:10.1016/j.ejso.2015.09.017.View ArticlePubMedGoogle Scholar
- Gurusamy KS. Laparoscopic versus open gastrectomy for gastric cancer. Cochrane Database Syst Rev. 2014;9(2):e88753. doi:10.1002/14651858.CD011389.Google Scholar
- Mocellin S, McCulloch P, Kazi H, Gama-Rodrigues JJ, Yuan Y, Nitti D. Extent of lymph node dissection for adenocarcinoma of the stomach. Cochrane Database Syst Rev. 2015;8:CD001964. doi:10.1002/14651858.CD001964.pub4.Google Scholar
- Blencowe NS, Strong S, McNair AGK, et al. Reporting of short-term clinical outcomes after esophagectomy: a systematic review. Ann Surg. 2012;255(4):658–66. doi:10.1097/SLA.0b013e3182480a6a.View ArticlePubMedGoogle Scholar
- Currie A, Brigic A, Blencowe NS, et al. Systematic review of surgical innovation reporting in laparoendoscopic colonic polyp resection. Br J Surg. 2015;102(2):e108–16. doi:10.1002/bjs.9675.View ArticlePubMedGoogle Scholar
- Whistance RN, Forsythe RO, McNair AGK, et al. A systematic review of outcome reporting in colorectal cancer surgery. Colorectal Dis. 2013;15(10):e548–60. doi:10.1111/codi.12378.View ArticlePubMedGoogle Scholar
- Hopkins JC, Howes N, Chalmers K, et al. Outcome reporting in bariatric surgery: an in-depth analysis to inform the development of a core outcome set, the BARIACT Study. Obes Rev. 2015;16(1):88–106. doi:10.1111/obr.12240.View ArticlePubMedGoogle Scholar
- Glossary: Core Outcome Set (COS). http://www.comet-initiative.org/glossary/cos/. Accessed 19 May 2016.
- Williamson PR, Altman DG, Blazeby JM, et al. Developing core outcome sets for clinical trials: issues to consider. Trials. 2012;13(1):132. doi:10.1186/1745-6215-13-132.View ArticlePubMedPubMed CentralGoogle Scholar
- Keeley T, Williamson P, Callery P, et al. The use of qualitative methods to inform Delphi surveys in core outcome set development. Trials. 2016;17(1):230. doi:10.1186/s13063-016-1356-7.View ArticlePubMedPubMed CentralGoogle Scholar
- Sinha IP, Smyth RL, Williamson PR, et al. Using the Delphi technique to determine which outcomes to measure in clinical trials: recommendations for the future based on a systematic review of existing studies. PLoS Med. 2011;8(1):e1000393. doi:10.1371/journal.pmed.1000393.View ArticlePubMedPubMed CentralGoogle Scholar
- Brookes ST, Macefield RC, Williamson PR, et al. Three nested randomized controlled trials of peer-only or multiple stakeholder group feedback within Delphi surveys during core outcome and information set development. Trials. 2016;17(1):409. doi:10.1186/s13063-016-1479-x.View ArticlePubMedPubMed CentralGoogle Scholar
- Prinsen CAC, Vohra S, Rose MR, et al. How to select outcome measurement instruments for outcomes included in a ‘Core Outcome Set’ – a practical guideline. Trials. 2016;17(1):449. doi:10.1186/s13063-016-1555-2.View ArticlePubMedPubMed CentralGoogle Scholar
- Kirkham JJ, Gorst S, Altman DG, et al. COS-STAR: a reporting guideline for studies developing core outcome sets (protocol). Trials. 2015;16(1):373. doi:10.1186/s13063-015-0913-9.View ArticlePubMedPubMed CentralGoogle Scholar
- Database Search—Core Outcome Measures in Effectiveness Trials Initiative (COMET). http://www.comet-initiative.org/studies/search. Accessed 19 May 2016.
- Allum WH, Blazeby JM, Griffin SM, Cunningham D, Jankowski JA, Wong R. Guidelines for the management of oesophageal and gastric cancer. Gut. 2011;60(11):1449–72. doi:10.1136/gut.2010.228254.View ArticlePubMedGoogle Scholar
- Macefield RC, Jacobs M, Korfage IJ, et al. Developing core outcomes sets: methods for identifying and including patient-reported outcomes (PROs). Trials. 2014;15(1):49. doi:10.1186/1745-6215-15-49.View ArticlePubMedPubMed CentralGoogle Scholar
- Crowe S, Fenton M, Hall M, et al. Patients’, clinicians’ and the research communities’ priorities for treatment research: there is an important mismatch. Res Involv Engagem. 2015;1(1):2. doi:10.1186/s40900-015-0003-x.View ArticleGoogle Scholar
- Tallon D, Chard J, Dieppe P, et al. Relation between agendas of the research community and the research consumer. Lancet (London, England). 2000;355(9220):2037–40. doi:10.1016/S0140-6736(00)02351-5.View ArticleGoogle Scholar
- Rosenheck R, Stroup S, Keefe RSE, et al. Measuring outcome priorities and preferences in people with schizophrenia. Br J Psychiatry. 2005;187(6):529–36.View ArticlePubMedGoogle Scholar
- Urquhart-Secord R, Craig JC, Hemmelgarn B, et al. Patient and caregiver priorities for outcomes in hemodialysis: an international nominal group technique study. Am J Kidney Dis. 2016;68(3):444–54. doi:10.1053/j.ajkd.2016.02.037.View ArticlePubMedGoogle Scholar
- Mason M. Sample size and saturation in PhD studies using qualitative interviews. Forum Qual Sozialforschung/Forum Qual Soc Res. 2010;11(3). ISSN 1438-5627. Available at: http://dx.doi.org/10.17169/fqs-11.3.1428. Accessed 30 July 2017.
- Baker SE, Edwards R. How many qualitative interviews is enough? Expert voices and early career reflections on sampling and cases in qualitative research. 2012.Google Scholar
- Harman NL, Bruce IA, Callery P, et al. MOMENT—Management of Otitis Media with Effusion in Cleft Palate: protocol for a systematic review of the literature and identification of a core outcome set using a Delphi survey. Trials. 2013;14:70. doi:10.1186/1745-6215-14-70.View ArticlePubMedPubMed CentralGoogle Scholar
- McNair AGK, Whistance RN, Forsythe RO, et al. Core outcomes for colorectal cancer surgery: a consensus study. PLoS Med. 2016;13(8):e1002071. doi:10.1371/journal.pmed.1002071.View ArticlePubMedPubMed CentralGoogle Scholar
- MacLennan S, Bekema HJ, Williamson PR, et al. A core outcome set for localised prostate cancer effectiveness trials: protocol for a systematic review of the literature and stakeholder involvement through interviews and a Delphi survey. Trials. 2015;16:76. doi:10.1186/s13063-015-0598-0.View ArticlePubMedPubMed CentralGoogle Scholar
- Powell C. The Delphi technique: myths and realities. J Adv Nurs. 2003;41(4):376–82. doi:10.1046/j.1365-2648.2003.02537.x.View ArticlePubMedGoogle Scholar
- Terwee CB, Mokkink LB, Knol DL, Ostelo RWJG, Bouter LM, De Vet HCW. Rating the methodological quality in systematic reviews of studies on measurement properties: a scoring system for the COSMIN checklist. Qual Life Res. 2012;21(4):651–7. doi:10.1007/s11136-011-9960-1.View ArticlePubMedGoogle Scholar