We will use the Delphi process to achieve a consensus of opinion from broadly representative and international expert stakeholder groups [15]. This is an observational study design comprising three independent Delphi processes, sponsored by the University of Nottingham and managed by the National Institute for Health Research (NIHR) Nottingham Biomedical Research Centre. A prospective study protocol was registered in the Core Outcome Measures in Effectiveness Trials (COMET) database [18]. This protocol describes version 2.0 (dated 14 March 2017), that was approved by the Proportionate Review Sub-committee of the West Midlands – Solihull Research Ethics Committee (REC reference 17/WM/0095, IRAS reference 220112) on 21 March 2017.
Research steering group
A Research Steering Group has been appointed to oversee and manage the project. The group comprises international colleagues representing the identified intervention approaches (HFH, BM, AL), a patient and public involvement and engagement manager (AH) two health care users with lived experience of tinnitus, referred to as Public Research Partners (VC, BT), and the Study Management Team (KF, HS, DAH). The role of the Research Steering Group is to:
-
1.
Support the development of the study protocol, specifically commenting on the feasibility of the Delphi process, reviewing study documentation (e.g. advertisements, Information Sheets, video instructions for the survey, website content) and to participate in a pilot of round 1 of the Delphi survey
-
2.
Review the list of outcome domains and associated descriptions, specifically commenting on the readability of the outcome descriptions, the appropriateness of the grouping of outcomes into categories and providing any additional outcomes that they believe should be included in the first round.
Eligibility criteria for the Delphi panels
A range of expertise within the panel is considered to be an important quality criterion for development of a Core Outcome Domain Set [19]. We will, therefore, seek to include representatives from four key stakeholder groups that may be particularly interested in managing chronic subjective tinnitus in adults. Specific inclusion criteria have been defined for each stakeholder group:
-
1.
Health care users must have the experience of living with tinnitus for 3 months or more and have had experience with at least one sound-, psychology-, or pharmacology-based intervention for their tinnitus, or have the intention of undergoing that treatment in the future
-
2.
Health care practitioners must have a clinical qualification, be currently employed by a public or private institution that provides a tinnitus service to patients, have specific experience in assessing, diagnosing or managing chronic subjective tinnitus in adults who receive, or have received, a sound-, psychology-, or pharmacology-based intervention.
-
3.
Clinical researchers must have an academic qualification, be currently employed by a research organisation, have current or ‘recent-past’ experience in clinical research with studies that focus on questions of clinical efficacy (benefit) of a sound-, psychology-, or pharmacology-based tinnitus intervention in humans (i.e. a co-author on a relevant peer-reviewed journal publication in the past 3 years)
-
4.
Commercial representatives must be currently employed by a company that develops, manufactures or sells relevant sound-, or pharmacology-based product(s) that may be used for alleviating tinnitus. Funders must be currently contracted by an organisation that has funded tinnitus research projects addressing sound-, psychology-, or pharmacology-based interventions in the last 3 years
General eligibility for participation in the Delphi process includes men and women aged 18 years or older, with sufficient command of English to read, understand and independently complete the questionnaires. An online screening stage will ask all participants to self-certify that they are an ‘expert’ in at least one of the three tinnitus-intervention strategies.
All enrolled Delphi panellists will be eligible to register their interest in attending a 1-day consensus meeting. However, allocation of places will be limited to those respondents who complete all three rounds of the online survey. None of the Research Steering Group members will be allowed to vote on domains in the consensus meetings because this risks inadvertently introducing a power differential across participants, but they can enrol in the online Delphi surveys (with the exception of KF and HS who will have access to the dataset for all analyses). Each expert panellist is permitted to enrol for more than one Delphi process if they meet the eligibility criteria, but would be invited to attend only one consensus meeting.
Panel size and justification
There is no agreed method to statistically calculate a sample size for online Delphi surveys or for consensus meetings and no criteria against which a sample size choice can be judged (e.g. [20, 21]). Some individual studies indicate that stakeholder groups of around 20 can provide stable results [21]; and this defines our minimum sample size. However, one of the key deciding factors is that panel membership should adequately represent their corresponding stakeholder group (health care users, clinical researchers, etc.). Another key deciding factor is pragmatic; the global pool of experts meeting our eligibility criteria differs from one intervention to another. So it is not feasible for all three online Delphi surveys to have the equivalent sample size or target number of stakeholders.
Across the three Delphi processes, we aim to recruit a minimum target of 260 and an upper target of 420 experts spanning the range of stakeholder roles. Sound-based tinnitus interventions are the most accessible intervention in the UK [6] and devices and smartphone apps are widely available for self-management. Thus, a large pool of stakeholders is expected. Therefore, the intention is to enrol into the sound-based Delphi survey 60–90 health care users, 20–30 health care practitioners, 20–30 clinical researchers, and 20–30 commercial representatives and/or funders. For psychology-based tinnitus interventions, there are no specific commercial avenues or funding streams. Therefore, for this Delphi survey, we will enrol 40–60 health care users, 20–30 health care practitioners, and 20–30 clinical researchers. Pharmacological interventions are not widely accessible because there are no licensed drugs for tinnitus [22]. Instead, there are some established medications for comorbid symptoms such as insomnia, depression, and anxiety. Therefore, the number of professionals and members of the public with knowledge and experience of these interventions is expected to be more limited. For the pharmacology-based survey, the intention is to enrol 30–60 health care users, 10–20 health care practitioners, 10–20 clinical researchers, and 10–20 commercial representatives and/or funders. Experts will be allocated to the appropriate online survey based on their self-declared expertise.
The consensus meetings require in-depth discussions with smaller groups. Up to 20 expert panellists will be invited to participate in each of three face-to-face group consensus meetings (corresponding to the sound-, psychology-, or pharmacology-based online Delphi surveys). Enrolment will be balanced across stakeholder groups, where possible.
Recruitment methods
Taking a business-informed approach has been shown to be beneficial to recruitment in several multicentre clinical trials [23]. Adopting an explicit marketing plan, engaging charities or participants to act as champions, delivering effective messages to multiple audiences at multiple levels and achieving clinician and public buy-in are all known effective ‘business’ comp-style components [24]. We are using a purposive sampling approach to recruit health care users from both clinical and non-clinical settings, and to recruit professional experts who maximise the international relevance of the study findings (see [13]). We have created a study recruitment plan with adequate resource allocation at the outset. Recruitment strategies include a number of planned methods approved by the Research Ethics Committee, some of which will be general and others will be targeted at particular stakeholder audiences. General e-promotion routes include a study webpage (http://www.hearing.nihr.ac.uk/research/outcome-measures-for-clinical-trials), a ‘research news’ feature on the health care user community website (https://www.tinnitustalk.com/), and regular updates on study progress via social media channels such as Twitter (e.g. @COMITIDStudy) and Facebook (e.g. hearingnihr).
Health care users will be targeted using several different planned recruitment routes. UK ‘Tinnitus Awareness Week’ (6–12 February 2017) shortly preceded the study launch and provided a platform to inform patients about this study and the importance of accurately measuring tinnitus complaints. An introductory feature published in the British Tinnitus Association Quiet magazine highlighted the upcoming study and enabled people to register for the study [25]. In addition, health care users in the UK are to be targeted using a traditional NHS recruitment route with five audiology centres designated as Participant Identification Centres, in addition to the lead site in Nottingham. Sites will display study posters in the audiology clinic waiting room, and hand out Participant Information Sheets, as appropriate. A small number of health care professionals in Europe will do the same in their tinnitus clinics, dependent on gaining appropriate local regulatory approval. Specific e-promotion routes include several patient organisations that have agreed to support the project by publishing newsletter articles and announcements to their members (e.g. the British Tinnitus Association, Action on Hearing Loss, the Deutsche Tinnitus-Liga e.V., the German Tinnitus Foundation Charité, and Association France Acouphènes). And finally, the lead study site has a participant database containing email contacts for approximately 1000 health care users who have self-declared tinnitus.
To recruit health care professionals, a number of professional networks and organisations will also circulate invitations to their membership (British Society of Audiology, British Academy of Audiology, ENT and Audiology News, the TINNET network funded by an EU COST Action BM1306, the International Collegium of Rehabilitative Audiology, and the Pharmacological Interventions for Hearing Loss Working Group at the Hearing Centre of Excellence in the USA). Collectively, these networks reach out to tens of thousands of professionals across the globe, but the disadvantage is that the approach can be somewhat impersonal. Parallel routes for recruiting health care professionals will, therefore, involve personal invitation via email or face-to-face contact. The Study Management Team has created a list of potential participants with relevant expertise using manual searches of the following information sources: first and last authors of relevant conference proceedings in last 3 years (e.g. Tinnitus Research Initiative, International Tinnitus Seminar, British Tinnitus Association conference); corresponding authors for each clinical trial of tinnitus identified in our previous systematic review [6]; and all authors of systematic reviews of tinnitus (Cochrane or otherwise) published in the preceding 5 years. An email request for relevant staff nominations has also been sent to members of the Hearing Industry Research Consortium representing the hearing-aid sector.
We will also personally approach editors of scholarly journals in the fields of audiology and otology. They will not be considered as a separate stakeholder because this role tends to be secondary to one of the above professional occupations. But participants who are journal editors will be recorded, allowing for this subgroup of data to be examined for any notable response differences during the analysis phase.
The Delphi survey
Three (parallel, independent) international Delphi surveys will be managed online using DelphiManager software maintained by the Core Outcome Measures in Effectiveness Trials (COMET) initiative (University of Liverpool: http://www.comet-initiative.org/). Each panellist will receive a Unique Identification Code and an e-link to the webpage for whichever survey matches their self-nominated expertise. Confirmation of eligible expertise will be collected online. A video explanation will illustrate how to complete the online tool. A flowchart of the study is shown in Fig. 1.
We will use a modified Delphi technique whereby we identify a ‘long-list’ of candidate outcomes prior to the first survey round, through systematic procedures that involve professional and public stakeholders and pool several independent sources of information [13, 26, 27]. For each outcome, we will provide the same plain-language description to participants in each stakeholder group. Each survey will comprise the same three sequential rounds, but will always be completed from the perspective of one of the interventions. All outcome domains will be retained from one round to the next presented in the same fixed order.
The pre-Delphi stage involving health care users as Public Research Partners
We have completed the pre-Delphi stage and this created a finalised set of 66 outcome domains, each with a short plain-language description, grouped into 13 domain categories; the main content of the online Delphi surveys. Patient involvement has been an integral and transformative step of the study design process. To improve the appeal to health care users and to reduce attrition, two Public Research Partners on the Research Steering Group and one independent member of the public commented on the feasibility of the Delphi survey design, and reviewed study documentation (advertisements, Information Sheets, video instructions for the survey). In line with recommended considerations [10; 13], we also used qualitative research data gathered from health care users to make informed decisions about the main content of the online Delphi surveys and to ensure that outcome domain concepts are explained in ways that patient participants can understand and generate meanings which are consistent across all Delphi stakeholder groups. This goal is also highly relevant for any participants whose native language is not English; a factor for consideration in our international Delphi process.
The COMIT’ID Study Management Team scoped out information from three different sources. The first information source was a systematic review of outcome domains used in clinical trials of tinnitus treatments in adults which identified 62 outcome domains [6]. The second information source was an ongoing study that identified 64 outcome domains through thematic analysis of the items taken from 23 commonly reported tinnitus questionnaires [Fackrell, personal communication]. Both of these sources largely reflect outcomes that researchers have thought important to measure. The third source was a systematic review of dimensions of tinnitus-related complaints reported by patients and their significant others using questionnaire- and interview-based methods [28]. This source identified 43 outcome domains that reflected outcomes which patients considered important to them. This scoping process identified a total of 124 distinct outcome domains, after duplicates had been consolidated (see Fig. 2). For each outcome domain, associated examples were taken from each data source to aid interpretation of meaning for each outcome domain.
Our Research Steering Group, in particular our two Public Research Partners, informed our decision to reduce the long-list of outcome domains in order to improve the likelihood of completing all items in the Delphi survey. Discussion of each outcome domain and examples reduced the list of 124 by 57. Fifteen outcome domains were comorbidities that were associated with, but not specific to, tinnitus and so these can be considered out of the scope of this Core Outcome Domain Set and consequently were removed. Comorbidities were primarily related to hearing and mental health difficulties with examples including ‘hearing handicap’, ‘speech perception’, ‘reduced sound tolerance’, and ‘depression’.
Our Public Research Partners strongly argued that 13 outcome domains were broad concepts that were already encapsulated by other outcome domains in the list. Such hierarchies of constructs were considered a risk by confusing participants when rating the importance of individual outcome domains in the online Delphi survey. For example, ‘cognitive difficulties’ was already captured by ‘concentration’, ‘confusion’, ‘ability to ignore’ and ‘tinnitus-related thoughts’ and ‘health-related quality of life’ was already covered by ‘impact on relationships’, ‘impact on individual activities’, ‘impact on social life’, ‘impact on work’, and ‘sexual difficulties’. The Public Research Partners recommended that such broad concepts should be removed from the long-list and instead used as the category label.
Eight domains were not considered to be suitable as tinnitus-specific outcomes; two because they were restricted to the clinician’s diagnostic assessment and six because they were highly specific to the fidelity of the intervention. Diagnostic items were ‘causes of tinnitus’ and ‘tinnitus duration’. Intervention items were ‘adequacy of blinding’, ‘credibility of sham intervention (control)’, ‘credibility of real intervention’, ‘needling sensation (acupuncture)’, ‘masking (sound therapy)’, and ‘therapeutic alliance’. It is interesting to note that all of these candidates originated from the two information sources which reflected what researchers have thought important to measure, mostly from the systematic review of clinical trials [6].
A number of outcome domains were judged to relate to specific ways in which an outcome domain could be measured and thus, on a conceptual/semantic level at least, to be broadly equivalent to one another. For example, an ‘active task to distract or cope with tinnitus’ and ‘purposefully protecting or reducing the chance of potential problems’ were both considered different ways to measure the same health construct ‘coping’. This recommendation resulted in ten outcome domains being consolidated in the final version of the long-list under existing outcome domains ‘tinnitus loudness’, ‘tinnitus quality’, ‘coping’, and the category ‘body structures and functions’. Three further outcome domains were consolidated with the existing outcome domain ‘fear’ because they were considered by the Public Research Partners to be overlapping examples of the same concept (i.e. ‘fear for health’, ‘fear for quality of life’, and ‘fear of tinnitus becoming worse’).
Finally, eight outcome domains were consolidated because they were specific examples of what could be considered as a form of adverse reaction (i.e. ‘drug safety and tolerability’, ‘safety’, ‘safety and tolerability’, ‘side effects’, ‘tolerability’, ‘headache’, ‘pain frequency’, and ‘pain intensity’). ‘Adverse reaction’ was, therefore, created as a new outcome domain term.
These health care user-led decisions established a revised long-list of 68 distinct outcome domains. The Research Steering Group next took careful steps to identify the appropriate language for use in the Delphi survey. The task of phrasing each outcome domain, writing plain language concept definitions, and phrasing the names for domain category groupings was an iterative procedure conducted over two half-day workshop sessions with members of the Study Management Team working closely with the two Public Research Partners. Resulting outcome domain terms, concept definitions and category labels were reviewed by 14 members of the British Tinnitus Association’s Users’ panel and five clinical experts who were members of the British Tinnitus Association Professional Advisory Committee, and all members of the Research Steering Group for face validity, understanding, and acceptability.
At this stage, two further revisions were made to the long-list, arising from both health care user-led and professional-led recommendations about semantic equivalence. The first was that the outcome domain ‘bothered’, defined as ‘being disturbed by tinnitus or finding it a nuisance’, was consolidated with ‘annoyance’. The second was that the outcome domain ‘frustration’, defined as ‘feeling unable to change or achieve something because of or in relation to tinnitus’, was felt to be already represented by ‘annoyance’ and ‘upset’ and, therefore, was consolidated with these. Additional file 1 reports the finalised set of 13 domain categories, 66 outcome domains, and short plain language descriptions. Categories and outcome domains will be presented within the Delphi surveys in alphabetical order to avoid potential weighting [26].
Round 1
For each outcome domain, participants will be asked to think about the importance of each tinnitus outcome domain and indicate how important it is to measure when deciding if a sound-based/psychology-based/pharmacology-based tinnitus treatment is working. Participants will be asked to score each outcome domain using the GRADE scale of 1–9, where 1 represents least important and 9 represents most important [29]. Selecting response options 1–3 indicates that the domain is ‘not important’, whilst 7–9 indicates that the domain is ‘critically important’ in deciding whether a tinnitus treatment is effective. Scores 4, 5, and 6 indicate that the outcome domain is ‘important but not critical’. If a participant feels that they did not understand a particular outcome, they will be able to select ‘unable to score’. Following each outcome and at the end of the round, optional open-text boxes will enable participants to add any comments. In round 1, participants will be able to propose additional outcome domains. These additional outcome domains will be reviewed and coded by two Study Management Team members, with appropriate plain-language concept definitions, to ensure that they represent new items for inclusion in rounds 2 and 3. Where uncertainty exists the Research Steering Group will be consulted, and all new outcome domain terms, concept definitions and category labels will be reviewed.
Rounds 2 and 3
Participants will be eligible to continue to rounds 2 and 3 if they have responded to at least 40% of the outcome domains in the previous round. Corresponding data from those participants who responded to less than 40% will be removed. In rounds 2 and 3, all participants will receive the same list of outcomes with feedback tailored according to their Delphi allocation. The purpose of round 2 will be to enable participants to reflect on their scores in light of the viewpoint of their stakeholder peers, and to score the outcomes again. The purpose of round 3 will be to enable participants to reflect on their scores in light of the viewpoint of their stakeholder peers and all other stakeholder groups, and to score the outcomes again. Results will be presented graphically as well as numerically to improve visual appeal (Fig. 2).
After round 3, a short questionnaire with open- and closed-text response options will be emailed to all participants to collect feedback on their experience of being a participant, in addition to their perceptions of our strategies for recruiting and for reducing attrition of professional and health care users’ stakeholders.
Consensus meeting
Professionals and health care users who have completed all three rounds of the Delphi survey and responded to at least 90% of the outcome domains in round 3 will be eligible to participate in the consensus meeting. As far as possible, allocated places will maintain the balance across stakeholder groups (e.g. 50% health care users/50% professionals) and will include non-UK, non-native, English language speakers. The purpose will be to agree a final Core Outcome Domain Set for each intervention category. Separate discussion within each meeting will include anonymised voting on each outcome as either ‘In’ or ‘Out’ (e.g. using electronic keypads which will create histograms and descriptive statistics ‘live’, to be displayed in the meeting).
Consensus criteria
Consensus recommendations for each intervention category will be guided by the respective round-3 results. We will include domains for which 70% or more of the participants in each stakeholder groups score 7–9, and fewer than 15% score 1–3 [30]. We will exclude domains for which 70% or more of the participants in each stakeholder groups score 1–3, and fewer than 15% score 7–9 [30]. These decisions will be taken without further discussion because time at the consensus meeting will be limited. At the consensus meeting, the moderated discussion, voting and decisions will be guided as follows:
-
For outcomes recommended to be included based on the round-3 analysis (70% scored 7–9), domains will be included if at least 70% of participants vote ‘In’
-
For outcomes where at least 50% of more than one stakeholder group scored 7–9 on the round-3 analysis, domains will be included if at least 70% of participants vote ‘In’
-
For outcomes where less than 50% of the participants in all stakeholder groups or at least 50% of only one stakeholder group scored 7–9 on the round-3 analysis, domains will be included only if at least 70% of participants vote ‘In’
If consensus is not reached after two rounds of voting, a ‘majority rules’ approach will be applied.
In the very final step, the Study Management Team will seek to establish a common cross-cutting Core Outcome Domain Set by identifying those outcome domains that were voted for inclusion in all three intervention-specific Core Outcome Domain Sets.
Observational cohort study
We aspire to conduct a prospective observational cohort study to determine if participation of clinical researchers in the Delphi process increases uptake of the recommended Core Outcome Domain Sets for trials of sound-, psychology-, and pharmacology-based interventions for chronic subjective tinnitus in adults, compared with researchers who did not participate. This investigation will be conducted on the basis of the implementation of the recommendations in published reports of clinical trials in the 7 years after the Core Outcome Domain Sets are published.
Analyses
The data from each Delphi round will be subjected to descriptive statistics, such as number completing each round, gender, country, region, native English language speaker (or not), and retention rate at rounds 2 and 3. We will plot the distribution of rating scores within each stakeholder group (including the ‘unable to score’ option and any missing data), and we will investigate attrition bias by comparing these plots across participants who complete successive rounds versus those who withdraw at rounds 2 or 3 [31]. We will also compare the distribution of rating scores for native and non-native English language speakers. Withdrawal and language are both factors that may affect how representative the final consensus might be of the target population. A separate analysis will assess the shifts in scores across rounds as a consequence of considering the anonymised feedback from other participants. The distribution of the scores for each outcome domain will be calculated for each stakeholder group within each intervention-specific Delphi survey.
The data from the feedback questionnaire will comprise open-text responses and these will be analysed using a thematic analysis approach to address one of the secondary objectives. Uptake of the recommendations will be analysed by calculating the number and proportion of authors who have implemented and referenced the COMIT’ID study recommendations in trials reported up to 7 years after their publication, comparing data between groups using non-parametric statistics (p < 0.05).
Dissemination
Any important changes to eligibility criteria that are approved as a protocol modification will be communicated to our list of potential participants and advertised via all general e-promotion routes. Data from the final analyses of the Delphi surveys, consensus meetings and feedback questionnaire will be presented at relevant national and international conferences, including the British Tinnitus Association, the British Society of Audiology, the Tinnitus Research Initiative and the International Tinnitus Seminar. Several peer-reviewed publications are also planned; one addressing all primary objectives and one addressing the secondary objective for recruiting and for reducing attrition of health care users. Our findings will be further disseminated to members of the public and clinicians through specialist magazine articles, and our patient organisation supporters. Participants will not be identified in any publications.
