- Open Access
- Open Peer Review
Survey indicated that core outcome set development is increasingly including patients, being conducted internationally and using Delphi surveys
Trials volume 19, Article number: 113 (2018)
The Correction to this article has been published in Trials 2018 19:430
There are numerous challenges in including patients in a core outcome set (COS) study, these can vary depending on the patient group. This study describes current efforts to include patients in the development of COS, with the aim of identifying areas for further improvement and study.
Using the COMET database, corresponding authors of COS projects registered or published from 1 January 2013 to 2 February 2017 were invited via a personalised email to participate in a short online survey. The survey and emails were constructed to maximise the response rate by following the academic literature on enhancing survey responses. Personalised reminder emails were sent to non-responders. This survey explored the frequency of patient input in COS studies, who was involved, what methods were used and whether or not the COS development was international.
One hundred and ninety-two COS developers were sent the survey. Responses were collected from 21 February 2017 until 7 May 2017. One hundred and forty-six unique developers responded, yielding a 76% response rate and data in relation to 195 unique COS (as some developers had worked on multiple COS). Of focus here are their responses regarding 162 COS at the published, completed or ongoing stages of development. Inclusion of patient participants was indicated in 87% (141/162) of COS in the published completed or ongoing stages and over 94% (65/69) of ongoing COS projects. Nearly half (65/135) of COS included patient participants from two or more countries and 22% (30/135) included patient participants from five or more countries. The Delphi survey was reported as being used singularly or in combination with other methods in 85% (119/140) of projects. Almost a quarter (16/65) of ongoing studies reported using a combination of qualitative interviews, Delphi survey and consensus meeting.
These findings indicated that the Delphi survey is the most popular method of facilitating patient participation, while the combination of qualitative interviews, Delphi survey and consensus meetings is the most popular combination of methods. The increased inclusion of patient participants in the development of COS is encouraging, as is the international approach to COS development that some developers are adopting.
Evidence enables treatment decisions to be made according to the needs of the individual patient. This evidence comes from numerous studies that record and measure the effects that different illnesses, conditions and treatments have on patients. These measurements are known as ‘outcomes’. Outcomes can include quality of life, treatment costs, fatigue, white blood cell count and pain. However, across different studies of the same condition or illness there is considerable variability in the outcomes measured. This has given rise to difficulties such as: summarising the evidence as the results cannot be adequately compared and contrasted  and authors selectively reporting outcomes . There is also the possibility that outcomes currently being measured may not accurately reflect the priorities of relevant stakeholder groups, including patients and the public. In turn, the usefulness of studies in advancing research, informing clinical practice and empowering clinicians and patients with knowledge regarding interventions is limited , rendering the research wasteful in many instances [4, 5].
One answer to this problem is the development of core outcome sets (COS). A COS is a minimum set of agreed standardised outcomes which should be measured and reported in all trials of a specific condition. It is considered a fundamental list of outcomes , not an exhaustive list, and researchers can measure additional outcomes in their trials if they wish . The same set may also be relevant to systematic reviews of trials. The Core Outcome Measures in Effectiveness Trials (COMET) Initiative was launched in 2010 in response to the recognised value of COS development. COMET aims to tackle the problem of heterogeneity in reported outcomes by promoting and facilitating the development and application of COS. COMET also collates and stimulates the production of resources for COS development, and facilitates the exchange of ideas and methodological research to enhance the quality and uptake of COS. COMET’s key resource is a publicly accessible database (www.comet-initiative.org) of planned, ongoing and completed COS studies. The database of published COS is updated annually via a systematic review. As indicated by the multiple individuals and organisations, including trialists, funders, registries, regulatory authorities, systematic review groups and journal editors now endorsing the uptake of COS and use of the COMET database (a list of these organisations is available at: www.comet-initiative.org/cosuptake), the usefulness and importance of this resource, and COS development more generally, is accepted.
Three stakeholder groups who are important to the development of all COS are those who will use the COS in research, health professionals and patients, as recently identified by the consensus-based recommendations in the Core Outcome Set – STAndards for Development: The COS-STAD recommendations [7, 8]. Thus, the inclusion of patients and the public is key to the development of COS .
Moreover, to increase reporting completeness and transparency in COS development COMET have produced the Core Outcome Set – STAndards for Reporting COS-STAR checklist. This checklist states that authors should describe the participant groups involved, the rationale for including them and the capacity in which they participated in the COS development .
When using the term ‘patient’ in this article we refer to patients, carers, health and social care service users and people from organisations who represent these groups. Researchers are increasingly including patients alongside other stakeholders in identifying what outcomes to measure in clinical trials. While a 2013 systematic review found that only 18% of published COS studies reported patient input , subsequent updates of this review in 2014 and 2015 indicated patient input in 59% and 61% of published COS developments, respectively [12, 13]. For most conditions there are many different outcomes that could be included in a COS. When patients have not been included in the COS development process, important outcomes have been overlooked , while other evidence indicates that patients and families differ in the priority they give to certain outcomes compared to clinicians . Including patient participants in deciding which of these outcomes should be in a COS reduces the danger of omitting important outcomes. More broadly, patient participation in COS development enhances the value of research as it helps to ensure that the outcomes reported are relevant to patients.
However, the best methods for facilitating the participation of patients is unknown. There are numerous challenges in enabling participation in a COS study and these will vary depending on the participants, the research team and the condition being researched. The COMET handbook brings together current thinking and methodological research regarding these challenges  including: selecting an appropriate recruitment method, finding the best way to explain the concept of a COS, using a suitable method to elicit perspectives of patients and health professionals, maintaining participant input over time, and enabling the inclusion of patients in face-to-face meetings with health professionals and academics . Previous COS studies have reported variable rates of recruitment of participants in the development of the COS , while COS developers have also reported limited experience of engaging with participants in the development of important COS .
This study examined the frequency of patient participation, which types of patient stakeholder are included, the methods employed to facilitate patient participation and the number of countries from which patients have been sampled in recently published and ongoing COS projects. By describing the current practice in the development of COS, the survey will help to identify areas for further improvement and study.
The survey was conducted in English and included some brief demographic questions before enquiring about patient participation in COS development. Patient participation was defined as: ‘where patients or the wider public (family members, carers, health and social care service users and people from organisations who represent these groups) or both, take part in the development of a core outcome set by giving data on their opinions regarding what outcomes are important.’ A full list of the questions can be found in Supplementary Information: Additional file 1. Survey Questions. The survey was constructed using the SurveyMonkey software . The benefits of using this software include the facility to incorporate filtre questions (whereby depending on the responses, questions are automatically skipped to the next appropriate question), it also allowed flexibility in the answer options which was of vital importance to obtaining the responses in the most appropriate manner.
A survey was deemed appropriate for this particular phase of the project as it is inexpensive and allowed us to engage with a large number of COS developers. Studies have shown that questionnaire length has a substantial effect on the number of non-responders , this questionnaire was purposely kept short to avoid this issue and to not overburden any prospective respondents. Other factors thought to influence the overall response rate include readability of questionnaires, such as the number of syllables per word, words per sentence, typeface and font size. We therefore followed what is considered best practice in the academic literature (The National Institute of Adult Continuing Education guideline ‘Readability: How to produce a clear written materials for a range of readers’) when building this survey .
Participant selection and recruitment
The COS developers were identified via a search (2 February 2017) of all studies published from 2013 and ongoing COS projects in the COMET Initiative database.
The survey was sent to COS developers as a link within a personalised email, inviting them to visit the SurveyMonkey website where the survey was hosted. Adopting a personalised approach and follow-up contact with those who do not respond to the initial email has been suggested to increase the odds of response by more than a quarter ; therefore, personalised emails were sent and three further emails were sent to non-responders.
Analysis of survey responses
Responses relating to published COS projects were validated by reading the appropriate publications where these were available, and emailing COS developers for clarification where necessary. The data were analysed descriptively.
The survey was sent to 192 COS developers. Some developers were involved in multiple COS projects and we asked them to complete the survey for each relevant COS. We contacted 59 developers for 59 published COS projects, 129 developers for 150 ongoing COS projects and 4 developers for 16 published and 19 ongoing COS projects. Responses were collected from February until May 2017.
There were 146 respondents yielding a 76% response rate and providing data regarding 195 projects. The breakdown of respondents and their projects is as follows: 37 responders for 37 published COS projects, 29 responders for 29 completed COS projects, 49 responders for 52 ongoing COS projects, 25 responders for 27 planned COS projects, 6 responders for a mixture of 15 published, 12 completed, 17 ongoing and 6 planned COS projects.
Table 1 summarises the frequency of patient participation in 162 COS projects since 2013, from published, completed and ongoing studies, after excluding 33 studies still in the planning stage. Overall, respondents indicated that 141/162 (87%) COS projects had included patient participants in the development of their COS (Table 1).
Survey responses for patient participation matched published information for 51 COS; in the remaining published study it was not possible to make this comparison as the developer did not provide their name in their survey response. Of 24 published COS for which no survey response was received or could be matched, five (21%) of the journal articles reported patient participation.
Developers reported input from a variety of patient stakeholder groups (Table 2): 101 (72%) projects included both patients (healthcare patients, healthcare users, consumers, family members, spouse, carers, etc.) and patient organisations (patient support groups and patient charity representatives).
For projects including patient participants, Table 3 shows how many countries were involved in the 135 studies where a response was provided. Half of COS projects included patient participants from only one country, and this was usually the United Kingdom (41/70, 59%). Where the study was international, typically COS developers involved participants from five or more countries (n = 30/135, 23% of total COS).
Table 4 summarises COS developers’ responses regarding the methods that they had used to facilitate patient participation. Developers responded via a fixed response option that included five commonly used methods (Delphi survey, questionnaire, focus group, qualitative interview and consensus meeting) and an additional ‘other’ option, which prompted respondents to state the method in a free-text box. All method combinations can be found in Supplementary Information: Additional file 2. Method Combinations.
As Table 4 shows, the Delphi survey was the most popular method, having been used singularly or in combination with other methods in over 119 (85%) of the 140 projects with patient participation. A multiple methods approach was used in 110 (79%) of the 140 projects with patient participation, of which the most popular method of was the combination of (1) Delphi survey, qualitative interviews and consensus meeting (22/140, 16%), followed by (2) Delphi survey singularly (21/140, 15%). In ongoing studies the most popular methods used were the combinations of (1) Delphi survey, consensus meeting and qualitative interviews (16/65, 25%), followed by (2) Delphi survey, consensus meeting, focus group and qualitative interviews (9/65, 14%) and finally (3) Delphi survey and consensus meeting (7/65, 11%).
This survey indicates that COS developers are increasingly including patients as participants in COS project development despite reports of COS developers finding patient participation difficult to facilitate in comparison to the participation of other stakeholder groups .
While many will welcome the increased inclusion of patients and patient organisations in COS development, it could also be argued that patient participants should exclusively be people who have personal experience of the condition or situation, as they are best placed to offer insight into what outcomes are important to someone living with a condition. This would exclude people working for patient organisations as COS study patient participants or others without personal experience of what it is like to live with a condition, as their perspectives may be closely aligned with that of a healthcare professional or researcher. Further research could examine what should constitute patient participation in COS development and explore the roles that these groups have and the similarities and differences in the input they provide.
The principle behind the development of a COS is that all researchers working on the same condition, illness or treatment will use that COS in their research. Therefore, COS need to be relevant for use across different countries if they are to improve the power of research to benefit patients . The findings of this survey are encouraging, with several COS projects being run in two or more countries with patient participants. However, the majority of COS projects mainly included patient participants from only one country, usually the UK. Previous research has indicated that COS developers have concerns regarding the practicalities and resources surrounding international COS development. Concerns were also raised in relation to ‘heterogeneity of views that might arise when participants are included from multiple countries’ . Future research could explore methods of developing COS with patients and health professionals from multiple countries in a practical and feasible manner.
A key challenge in patient participation is enabling patients to contribute their perspectives in ways that are meaningful and sustainable. It is vital that the methods suit the patient group concerned. Patient and public involvement where patients and the public are involved as active research partners in a COS project, can provide a patient and public perspective on the suitability of different methods from the design to conclusion of a COS project. The collaboration of researchers and patient and public involvement partners can help to ensure the appropriate design and conduct of a COS project. The survey responses indicated that the use of combinations of different methods, such as the Delphi survey, questionnaires, interviews, focus groups and consensus meetings, is not unusual. It was also evident that the Delphi survey was the most popular of all methods of participation in COS development. Delphi surveys can widen patient participation, promote transparency and offer anonymity. However, these surveys can be lengthy, and some believe these are intimidating for patient participants . COS developers have acknowledged a need for guidance on conducting Delphi surveys and consensus meetings . The COS-STAD recommendations identify minimum standards that should be met during the COS development .
A strength of this study was the relatively high response rate. However, non-response bias is a potential issue within this survey. The validation work shows non-respondents for published projects had a lower patient participation rate than that of those who responded. This is likely to also be true for non-respondents of ongoing studies, resulting in an over-estimate of patient participation reported in the survey. Full and accurate reporting of COS projects, including details of patient participation, should continue to improve if developers use the recently published COS-STAR reporting guideline .
The ongoing inclusion of patient participants in the development of COS is encouraging, as is the international approach that some developers are adopting, despite the academic literature suggesting that there are barriers to be overcome in developing international COS projects.
Core Outcome Measures in Effectiveness Trials
Core outcome set
Clarke M, Williamson PR. Core outcome sets and systematic reviews. Syst Rev. 2016;5(1):1.
Page MJ, McKenzie JE, Kirkham J, Dwan K, Kramer S, Green S, Forbes A. Bias due to selective inclusion and reporting of outcomes and analyses in systematic reviews of randomised trials of healthcare interventions. Cochrane Database Syst Rev. 2014;10:Mr000035.
Williamson P. Core outcome sets will improve the quality of obstetrics research. BJOG. 2014;121(10):1196.
Chalmers I, Bracken MB, Djulbegovic B, Garattini S, Grant J, Gülmezoglu AM, Howells DW, Ioannidis JP, Oliver S. How to increase value and reduce waste when research priorities are set. Lancet. 2014;383(9912):156–65.
Chalmers I, Glasziou P. Avoidable waste in the production and reporting of research evidence. Obstet Gynecol. 2009;114(6):1341–5.
Williamson P, Altman D, Blazeby J, Clarke M, Gargon E. Driving up the quality and relevance of research through the use of agreed core outcomes. J Health Serv Res Policy. 2012;17(1):1–2.
Williamson PR, Altman DG, Bagley H, Barnes KL, Blazeby JM, Brookes ST, Clarke M, Gargon E, Gorst S, Harman N. The COMET handbook: version 1.0. Trials. 2017;18(3):280.
Kirkham JJ, Davis K, Altman DG, Blazeby JM, Clarke M, Tunis S, Williamson PR. Core Outcome Set-STAndards for Development: The COS-STAD recommendations. PLoS Med. 2017;14(11):e1002447.
Young B, Bagley H. Including patients in core outcome set development: issues to consider based on three workshops with around 100 international delegates. Res Involvement Engagement. 2016;2(1):25.
Kirkham JJ, Gorst S, Altman DG, Blazeby JM, Clarke M, Devane D, Gargon E, Moher D, Schmitt J, Tugwell P. Core Outcome Set–STAndards for reporting: the COS-STAR statement. PLoS Med. 2016;13(10):e1002148.
Gargon E, Gurung B, Medley N, Altman DG, Blazeby JM, Clarke M, Williamson PR. Choosing important health outcomes for comparative effectiveness research: a systematic review. PLoS One. 2014;9(6):e99111.
Gorst SL, Gargon E, Clarke M, Blazeby JM, Altman DG, Williamson PR. Choosing important health outcomes for comparative effectiveness research: an updated review and user survey. PLoS One. 2016;11(1):e0146444.
Gorst SL, Gargon E, Clarke M, Smith V, Williamson PR. Choosing important health outcomes for comparative effectiveness research: an updated review and identification of gaps. PLoS One. 2016;11(12):e0168403.
Kirwan JR, Minnock P, Adebajo A, Bresnihan B, Choy E, De Wit M, Hazes M, Richards P, Saag K, Suarez-Almazor M. Patient perspective: fatigue as a recommended patient centered outcome measure in rheumatoid arthritis. J Rheumatol. 2007;34(5):1174–7.
Sinha IP, Gallagher R, Williamson PR, Smyth RL. Development of a core outcome set for clinical trials in childhood asthma: a survey of clinicians, parents, and young people. Trials. 2012;13(1):1.
Gargon E, Williamson PR, Young B. Improving core outcome set development: qualitative interviews with developers provided pointers to inform guidance. J Clin Epidemiol. 2017;86:140–52.
SurveyMonkey LLC. SurveyMonkey®. Palo Alto: SurveyMonkey, LLC; 2012.
Edwards PJ, Roberts I, Clarke MJ, DiGuiseppi C, Wentz R, Kwan I, Cooper R, Felix LM, Pratap S. Methods to increase response to postal and electronic questionnaires. Cochrane Database Syst Rev. 2009;(3). Art. No.: MR000008. https://doi.org/10.1002/14651858.MR000008.pub4.
The authors would like to thank the COS developers who took the time to respond to this survey.
The authors would also like to thank Dr. Sarah Gorst and Ms. Katy Davis of the COMET Initiative, University of Liverpool for their assistance in relation to searching the COMET database.
This project is a part of a MiRoR (Methods in Research on Research)-funded PhD, which is being undertaken by AB. MiRoR has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No 676207.
Availability of data and materials
Ethics approval and consent to participate
Ethical approval was granted from Health and Life Sciences Committee on Research Ethics (Human participants, tissues and databases) at The University of Liverpool on 16 February 2017 (reference 1339). All survey participants were provided with full written information prior to survey commencement. Consent was assumed upon entering the survey and participants were free to leave at any time without providing a reason.
Consent for publication
Paula R Williamson chairs the Management Group of the COMET Initiative.
Bridget Young is a member of the COMET PoPPIE (People and Public Participation, Involvement and Engagement) Working Group.
No other authors have any competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
The original version of this article was revised: In the original publication the abbreviation for Core outcome sets (COS) was pluralized unnecessarily throughout the next. The article was updated to rectify this error.
About this article
Cite this article
Biggane, A.M., Brading, L., Ravaud, P. et al. Survey indicated that core outcome set development is increasingly including patients, being conducted internationally and using Delphi surveys. Trials 19, 113 (2018) doi:10.1186/s13063-018-2493-y
- Patient engagement
- Patient participation
- Core outcome sets