Skip to main content

Co-producing a multi-stakeholder Core Outcome Set for distal Tibia and Ankle fractures (COSTA): a study protocol



Ankle fracture is a common injury with a strong evidence base focused on effectiveness of treatments. However, there are no reporting guidelines on distal tibia and ankle fractures. This has led to heterogeneity in outcome reporting and consequently, restricted the contribution of evidence syntheses. Over the past decade, core outcome sets have been developed to address this issue and are available for several common fractures, including those of the hip, distal radius, and open tibial fractures. This protocol describes the process to co-produce—with patient partners and other key stakeholders—a multi-stakeholder derived Core Outcome Set for distal Tibia and Ankle fractures (COSTA). The scope of COSTA will be for clinical trials.


The study will have five-stages which will include the following: (i) systematic reviews of existing qualitative studies and outcome reporting in randomised controlled trial studies to inform a developing list of potential outcome domains; (ii) qualitative interviews (including secondary data) and focus groups with patients and healthcare professionals to explore the impact of ankle fracture and the outcomes that really matter; (iii) generation of meaningful outcome statements with the study team, international advisory group and patient partners; (iv) a multi-round, international e-Delphi study to achieve consensus on the core domain set; and (v) an evidence-based consensus on a core measurement set will be achieved through a structured group consensus meeting, recommending best assessment approaches for each of the domains in the core domain set.


Development of COSTA will provide internationally endorsed outcome assessment guidance for clinical trials for distal tibia and ankle fractures. This will enhance comparative reviews of interventions, potentially reducing reporting bias and research waste.

Peer Review reports


Fractures are a common problem with incidence rates in the UK reported to be 73.3 per 10,000 in adults aged 18–49 years, increasing to 116.3 per 10,000 in adults aged 50+ years [1]. Recent evidence indicates that 14% of hospital fracture admissions in England were for distal tibia and ankle fractures [2]. Lower-limb fractures are associated with both short and long-term disability and pain [3, 4]. Evidence suggests that recovery can be slow [5] with substantial variation in how quickly or successfully patients return to their preinjury lives [5, 6]; many patients report experiencing prolonged disability [7]. However, evidence exploring the recovery and impact of such injuries is limited. Growing evidence suggests that an ankle fracture can affect physical, social and psychological functioning with indications of fatigue, depression, anxiety, and disturbed sleep [8]. However, due to inconsistent outcome selection and reporting, it is difficult to collate and synthesise evidence to scrutinise these findings [9,10,11].

Evidence syntheses are essential for collating and determining the strength of evidence for interventions, informing, and advancing healthcare provision. However, such evidence syntheses (e.g. meta-analysis) rely on homogeneity in outcome selection and reporting to draw evidence together [12]. The need and benefits of uniformity in outcome selection and reporting have been well described [13,14,15,16], including the following: (i) increased relevance of outcomes included in trials (to healthcare services users and professionals); (ii) enhanced homogeneity in outcome reporting between studies, thus strengthening meta-analyses through wider inclusion of research studies; (iii) reducing outcome reporting bias; and (iv) reducing research waste [16].

Significant progress has been made towards developing reporting guidance across a range of fractures, with core outcome sets in development or available for hip [17], distal radius [18], open lower limb [19], and shoulder disorders (including proximal humeral fractures) [20]. However, there is currently no guidance for distal tibia and ankle fractures. In a 2012 Cochrane review of rehabilitation for adults with ankle fractures, researchers highlighted that clinical and statistical heterogeneity between studies was a substantive challenge when conducting meta-analyses [9]. This issue has persisted with subsequent reviews examining return to sport in adults [10] and managing low risk ankle fractures in children [11] also identifying the challenge of heterogeneity in outcome selection and reporting and its detrimental effect on conducting evidence synthesis. The need for a core outcome set for distal tibia and ankle fractures is clear and has been recently recommended to increase outcome reporting homogeneity in ankle fracture research [21]. Currently, there is work ongoing to develop a core outcome set for ankle fractures specifically for children [22] but no guidance for adults.

A Core Outcome Set (COS) is an agreed minimum number of outcomes that should be measured and reported in all trials for a specific clinical area [16]. Two key development stages are described [23,24,25].

  1. 1.

    Clarifying the outcome domains that matter to key stakeholders and should be minimally assessed in future clinical trials (Core Domain Set; CDS).

  2. 2.

    Confirming the assessment method(s) for each outcome domain (Core Measurement Set; CMS).

Current best practice guidance [24, 25] describes a mixed-methods approach, integrating evidence from literature reviews, patient interviews, and active engagement with key stakeholders to reach consensus. Additionally, establishing advisory groups with key stakeholders is recommended and widely described [26,27,28,29], contributing to improved COS uptake [16]. Similarly, public involvement groups can enhance the full research process, including identification of unique outcomes for incorporation within the COS [16, 30].

This protocol describes the process for co-producing a multi-stakeholder derived Core Outcome Set for distal Tibia and Ankle fractures (COSTA).


A five-stage approach will be adopted (Fig. 1) [24, 25]. The first three stages include preparatory activities to inform the generation of potential outcome domains for consideration in the e-Delphi study: systematic reviews of existing qualitative studies and outcome reporting in randomised controlled trial studies (stage 1) qualitative interviews (including secondary data) and focus groups with both patients and healthcare professionals to explore the impact of ankle fracture and the outcomes that really matter (stage 2) and generating meaningful outcome statements with the research team, international advisory group, and patient partners (stage 3). A multi-round, international e-Delphi study will seek to achieve consensus on the Core Domain Set (CDS) for the distal tibia and ankle fracture core outcome set (COSTA) (Stage 4) and, finally, a consensus meeting to agree a Core Measurement Set (CMS) and recommend the COS for distal Tibia and Ankle fracture (COSTA) (Stage 5). The COSTA project is registered with the Core Outcome Measures in Effectiveness Trials (COMET) initiative (

Fig. 1

Flow diagram illustrating the process of developing COSTA


A protocol for the development of a core outcome set for distal tibia and ankle fractures (COSTA) for use in clinical trials.


  1. 1.

    To develop a long list of important outcome domains, incorporating the perspectives of multiple stakeholders (researchers, patients, healthcare professionals) (stages 1 and 2).

  2. 2.

    To generate meaningful statements for identified outcome domains (stage 3).

  3. 3.

    To achieve consensus on the core domain set for COSTA (stage 4).

  4. 4.

    To achieve consensus on a core measurement set for COSTA (stage 5).


Three groups will be actively involved in the COSTA project:

  1. 1.

    Core research team: consisting of methodologists, clinicians, and a patient partner, will meet monthly to discuss project design issues and progress.

  2. 2.

    International advisory group: consisting of methodologists (core outcome set, health measurement), clinicians, and researchers with experience in distal tibia and ankle fractures and clinical trials, with representation from funding bodies and journals. This group will meet every 6 months to discuss COSTA progress.

  3. 3.

    Patient and public involvement (PPI) group: will be established to work collaboratively with the core research team in delivering the project. Their involvement will ensure that COSTA is relevant and acceptable to distal tibia and ankle patients. Regular meetings and email communication will be initiated to achieve and support active engagement.


The purpose of COSTA is to provide outcome selection and reporting guidance for:

Research or practice settings: For adoption in clinical trials.

Health condition: Fracture of the distal tibia and ankle.

Population: Adults (aged 18 years or older with closed growth plates) who have experienced a distal tibia (tibia, distal end segment type 43) and ankle fracture (malleolar segment type 44) as defined by the AO/OTA classification [31].

Intervention: Any surgical or non-surgical intervention including physical rehabilitation.

Incorporating patient and public involvement (PPI) into COS development

A COSTA-specific PPI group will be established from existing PPI networks. Members will include individuals who have experienced distal tibia and ankle fractures or been involved in caring for someone following an ankle fracture. Evidence suggests that failure to incorporate patients can lead to important outcomes being overlooked [32]. Moreover, active PPI can enrich the research process through improved outcome identification [33, 34] and greater participant retention in key stages of COS development, e.g. e-Delphi surveys [35]. Active collaboration with PPI members as patient research partners in the research process will ensure that outcomes that are important and relevant to patients are reflected in the COS [30].

A ‘co-production’ approach to COS development will be adopted [36], underpinned by the COMET People and Patient Participation, Involvement and Engagement (PoPPIE) guidance [37] and the six UK standards for public involvement [38]. Our patient research partners will have roles that are comparable to the research team but without the day-to-day responsibilities of running the project. They will be supported to contribute throughout the study, including shaping ethics applications, study materials (e.g. interview schedules), data analysis and interpretation, reviewing, modifying, and generating meaningful statements for the outcomes list and ratifying the core domain set. Their involvement will be reported using the Guidance for Reporting Involvement of Patients and the Public (GRIPP2) short-form checklist [39].

Stage one: reviews of outcome reporting in clinical trials and of patient experiences of recovery

We will conduct two systematic literature reviews to identify reported outcomes in (1) intervention randomised control trial (RCT) studies and (2) qualitative studies exploring the patient experience and recovery following an ankle fracture. Both reviews will use the Preferred Reporting Items for Systematic Reviews and Meta-Analyses [40] guidance to support transparency in reporting.

Review of trial outcome reporting

This review will support the development of an outcomes list that incorporates what is important to researchers. Searches will be conducted across five databases (from 2000 to 2021): Medline (OVID), EMBASE (OVID), PsycINFO (OVID), CINAHL, and AMED. Additionally, three trial registries (ISRCTN, ICTRP, and and the Cochrane Central Register of Controlled Trials (CENTRAL) will be searched. One experienced reviewer (NP) will screen titles and abstracts and determine whether they meet eligibility criteria (Table 1); a 10% subset will be double assessed by another experienced reviewer (KH) to check for accuracy and consistency. Any disagreements will be resolved through discussion and if necessary, a third reviewer (ET) will be consulted. NP and KH (10% subset) will extract information which will include the study reference, design, population and sample size, interventions studied, and outcomes reported (e.g. health, clinical, social, economic) and measurement approaches (e.g. patient or clinician reported outcome). Additionally, the review will also examine clarity of reporting and extract information on whether used measures are reproducible, include appropriate references, and if they have been modified for the study. Outcomes will be grouped using the World Health Organisation (WHO) International Classification of Functioning, Disability and Health model (ICF) [41].

Table 1 Eligibility criteria for inclusion in reviews

Review of qualitative reports of patient experiences of recovery following ankle fracture

This review aims to examine and synthesise evidence from existing qualitative studies exploring patient experience of injury and recovery from distal tibia or ankle fracture, ensuring relevant outcomes to patients are reflected in the developing outcomes list. Additionally, this review will contribute to our wider understanding of ankle fracture experience and recovery.

Searches will be conducted across five databases (from 2000 to 2020): Medline (OVID), EMBASE (OVID), PsycINFO (OVID), CINAHL, and AMED. One experienced reviewer (NP) will screen titles and abstract to determine eligibility with a 10% subset double-assessed by a second experienced reviewer (ET) to check for accurate and consistent application of the eligibility criteria (Table 1). Disagreements will be resolved, where necessary, through discussion or in consultation with a third reviewer (KH).

Data will be extracted and analysed by NP and ET and managed using NVivo software. Data will be analysed using thematic synthesis [42] to elucidate higher-order analytical themes; this will supplement the extraction of important outcomes. Outcomes will be grouped using the WHO ICF framework [41] and incorporated into the developing outcomes list.

Stage two: Understanding experiences and ‘what matters’ to patients and healthcare professionals

Part A: Interviews with patients

We will conduct semi-structured interviews with a purposive sample of up to 30 individuals who have experienced distal tibia and ankle fractures. This sample size will allow for a range of views to be expressed across injuries (distal tibia, ankle) and age groups (younger and older adults) to be captured. Interviews will be guided by an interview schedule co-produced between the research team and patient partners (Table 2) and are anticipated to last up to one 1 h. All interviews will be audio-recorded and transcribed verbatim. Notes will be taken during and after interview to support analysis, interpretation, and reflexivity.

Table 2 Patient co-produced interview questions and prompts

Sampling criteria will include their treatment strategy (operative or non-operative), type of injury, time since injury, and geographic location. Recognising that there is a bimodal incidence of ankle fractures in adults by age, with younger patients often experiencing high energy fractures and older patients more often experiencing low energy fractures [43], sampling will ensure a range of ages are included to ensure a breadth of perspectives are captured in interviews. Participants will be aged 18 years or older, will have experienced distal tibia and ankle fractures within the last 9 months, and be able to communicate (written and verbally) in English. Participants will be identified through clinical lists maintained by their local hospital trauma department. Given the evolving situation with COVID-19, interviews will either be conducted face-to-face, virtually (via webcam) or by telephone, depending on safety guidance. Patients will be unknown to the researcher prior to interview.

The interviews will draw on interpretative phenomenological analysis (IPA) as a methodological framework [26], an analysis process developed specifically to provide methodological direction for research that focuses on understanding the lived experience of others [26]. The analysis consists of two elements:

  1. 1.

    A thematic analysis [44], drawing on the impact triad of severity, importance, and self-management [45], will be conducted to develop a framework of meaningful patient outcomes. These outcomes will be mapped to the ICF framework [41] and contribute to the developing outcomes list (from the systematic reviews).

  2. 2.

    A separate IPA will be conducted on a homogeneous subset of up to 10 interviews. This will make a separate contribution to the evidence base by developing our understanding of the lived experiences of patients with distal tibia and ankle fractures and their recovery. The analysis will be inductive and continuous, following the process described by Smith, Flowers, and Larkin [46].

Interviews will be conducted and coded by NP, ET, and KH, none of whom are involved in routine patient care. All researchers are experienced in conducting interviews and have used IPA and thematic analysis. NVivo software will be used to manage the data. NP is experienced in mixed methods, a male Research Fellow with a PhD in Health Sciences, and a background in Psychology and Cognitive Neuroscience. ET is an experienced, qualitative, female researcher with a PhD in Health Sciences and a focus on traumatic orthopaedic injuries. KH is a female with a DPhil in Health Sciences and Clinical Evaluation; she is a mixed methods researcher with experience across a range of patient populations.

Rigour will be guided by the principles proposed by Yardley [47]. These criteria reflect sensitivity to context, commitment and rigour, transparency and coherence, and impact and importance. Sensitivity to context will be demonstrated through consideration of relevant literature, theoretical understanding and understanding the perspective of participants. Commitment and rigour will be demonstrated through in-depth engagement with the topic, thorough data collection and the depth of analysis. Transparency and coherence will be demonstrated through the detailed accounts from participants, reflexivity, and the production of a clear audit trail that captures key quotes that inform theme derivation. Finally, impact and importance will be demonstrated through development of our theoretical and practical understanding of fracture experience and recovery.

Secondary data analysis of existing interview data

A secondary analysis of existing interview data, garnered from a qualitative sub study of the “Ankle Fracture Treatment: Enhancing Recovery” (AFTER) [48] study, will be conducted (ET). The main study included 60 adults, aged 50 years and above, who had sustained an ankle fracture and consented to take part in a study of progressive functional exercise versus best practice advice. The qualitative study explored the lived experience and 6-month recovery in twenty patients. Important outcomes will be extracted from this data as part of the ongoing phenomenological analysis. The AFTER study is wholly independent of the COSTA initiative and will be reported separately by the respective authors (DK, ET).

Part B: Focus groups or interviews with healthcare professionals

Given the evolving circumstances and current pressures on healthcare professionals regarding COVID-19, the conducting of focus groups may not be feasible at the point of study commencement. Consequently, alternative arrangements have been developed which would use interviewing as per the approach taken in Part A with patients. Here, we will explore the experiences of professionals working with or treating ankle fracture patients before reviewing and potentially adding to the outcomes list derived from the systematic reviews. Whilst the benefit of group discussion and sharing experiences may be lost using an interview approach, they will ensure that the healthcare professional perspective is incorporated into the developing outcomes list.

Should it be possible to host focus groups, up to four will be held and co-facilitated by members of the research team with experience in conducting focus groups. Previous experience suggests that focus group attendance tends to be difficult to predict. Current guidance recommends between 5 and 12 participants are required for an effective focus group discussion. Therefore, this study will seek to recruit at the upper end (n = 12) of this guidance to accommodate likely attrition [49,50,51]. Professionals will be purposively sampled based on their professional role (e.g. clinicians, occupational therapists, physiotherapists, nurses). It is anticipated that the focus group will take up to 1 h, including a short 10-min comfort break. To participate, healthcare professionals will have experience working within a trauma setting, including with patients who have experienced relevant fractures to the study (distal tibia or ankle). Professionals will be identified and approached by site principal investigators, snowballing, and personal contacts.

Data will be analysed using thematic analysis [44] to identify key outcomes and themes as per the process described in Part A interviews. The findings of the focus groups/ interviews will support identification of distal tibia and ankle fracture outcomes that are considered important to healthcare professionals. The process of ensuring rigour and transparency will again follow the principles proposed by Yardley [47].

Stage three: Generating meaningful outcome statements

The long list of outcomes generated in stages 1 and 2 will be discussed with members of the core, international advisory, and PPI groups. Working collaboratively with all members, outcome statements will be generated for each outcome domain and mapped to the World Health Organisation (WHO) International Classification of Functioning, Disability and Health model (ICF) [41]—a structured and widely recognised outcomes framework that will support greater transparency and consistency in the language used to describe potential outcome domains [52].

A face-to-face (or digitally hosted) meeting will be scheduled across one full day (if possible) or alternatively, two half-day sessions, and materials sent to participants in advance of the meeting for their consideration. These materials will include research findings (from reviews and qualitative data) and a questionnaire for participants to complete and return in advance of the meeting, listing all outcomes with potential definitions. When completing the questionnaire, participants will consider outcome domains, definitions, and whether any important outcome domains are missing. The meeting will be structured using a modified nominal group technique to facilitate group discussions. This will consist of three stages:

  1. 1.

    Reviewing the research findings: work completed will be discussed; the results of the pre-meeting survey will be discussed.

  2. 2.

    Refining outcome priorities: the list of outcome domains will be reviewed and refined to reduce repetition, ensure appropriate grouping of outcome domains, and review the mapping of outcome domains to the WHO-ICF framework.

  3. 3.

    Producing meaningful outcome statements: we will consider, and where necessary, revise outcome statements for each outcome domain through discussion to achieve a list of outcome domains with meaningful statements.

The revised outcome domains list and meaningful statements will be written in plain English, using both open and closed-format questions, forming the basis for an on-line e-Delphi questionnaire (stage 4). The questionnaire will be piloted with the core team, PPI partners, and researchers naïve to the study (n = 10).

Stage four: Achieving outcome prioritisation and core domain confirmation in an international, multi-stakeholder e-Delphi study

Delphi studies utilise a process of sequential questionnaire completion and feedback to establish expert consensus between a panel of experts [53]. To ensure that the COSTA reflects the perspectives of experts in the field of tibial and ankle fracture, two panels will be defined: (1) patients who have experienced a distal tibia or ankle fracture and (2) health professionals and researchers who are active in this field, representative of their professional groups, and well-positioned to implement the COSTA recommendations [54].

Consensus or accepted standards for sample sizes for Delphi studies are currently lacking [53], with expert panel sizes described in COS development ranging from 15 [55] to over 200 panellists [56]. We refer to recent examples of Delphi studies where pre-identified expert panels of between 60 and 70 panellists per panel are described [57,58,59]. This will facilitate a wide expert view and accommodate for attrition across rounds. Eligibility criteria for panellists are presented in Table 3.

Table 3 Eligibility criteria for participants in the e-Delphi

Expert panel 1: We will identify patients—adults aged 18 years and over who have sustained a fracture of the distal tibia or ankle within the 2-year period of the start of the e-Delphi—as per the recruitment approach specified in stage two. This will be a UK-based sample. A ‘pre-agreed’ list of participants will be established and invited to participate in the e-Delphi.

Expert panel 2: A selected group of international health professionals (clinicians / surgeons, physiotherapists, occupational therapists, nurses), and researchers (trialists, reviewers, measurement experts) known to be actively involved in delivering fracture care, or in fracture research of relevance to distal tibia and ankle fractures, will be identified through national and international professional networks (e.g. AO Trauma Network) and societies (e.g. the British Orthopaedic Foot and Ankle Society, Association of Foot and Ankle Physiotherapists), published research and involvement in clinical trials. Recruitment approaches will be supplemented by snowballing and personal contacts. Potential participants will be invited by e-mail to consider participation in the study. A pre-agreed list will be established and invited to participate in the e-Delphi.

Modified e-Delphi method

The modified e-Delphi will be conducted using the COMET DelphiManager software (University of Liverpool). It will consist of three sequential rounds with the same group of panellists: participants completing round 1 will be eligible to complete round 2, and those completing round 2 will be eligible to complete round 3. Participants will have up to 2-weeks to complete each round, with reminders sent after 1 week and again 24 h before the round is closed. Data will be analysed using descriptive statistics and presented using measures of central tendency, and as graphs, where necessary. Missing data will be examined across each outcome domain to identify potential patterns in non-response (e.g. by panel, or other characteristics). If an item-level pattern is observed (i.e. > 10% of responses are missing), we will check this against qualitative feedback provided by participants.

Round 1: Participants will be invited to rate the relative importance of each outcome domain for ‘inclusion in future distal tibia and ankle fracture research studies’ using a nine-point numeric rating scale (range: 1–3 ‘not at all important’, 4–6 ‘uncertain’ and 7–9 ‘very important’). An additional option ‘unable to rate’ will also be provided. Participants can elaborate on their decisions, providing additional qualitative comments and feedback for consideration in subsequent rounds. Evidence suggests that attrition can be reduced with such active engagement [53, 60].

A reduction in the number of outcome domains is a key objective of the Delphi study, seeking to achieve consensus on a minimum number of outcome domains (core domain set) for the COS. Therefore, a bespoke grading system, described in the development of a COS for migraine (COSMIG), will be adopted to provide greater clarity where participants from different sub-groups either agree or disagree in their judgements [61]. This approach defines clear criteria and decision rules for handling differing levels of consensus e.g. little or no consensus (grade C or D), uncertainty (grade A and B), and clear consensus (grade A** and A*) per outcome domain. Only outcome domains judged most favourably by one or both expert panels will be included in round two (Table 4).

Table 4 Grading system for determining consensus in round 1 of the e-Delphi study

Round 2: Responses to round one will be summarised (individual and group median scores for each outcome). Qualitative feedback will inform the inclusion of additional or edits to existing outcome domains.

Further prioritisation will be sought by inviting panellists to allocate ‘points’ to illustrate how important they feel an outcome domain is for inclusion in the core domain set. Each outcome domain will be assessed using an 11-point numeric rating scale, where 0 is not a priority, and 10 an absolute priority. A maximum of 70 points can be spent, with a maximum of 10 points can be assigned to any one outcome domain.

The sums of priority ratings will be calculated (per sub-panel and combined), supporting the identification of both the top 10 and top 50% of prioritised outcome domains. These will be discussed by the core research team. Outcome domains will be retained as they are, or if considered to be similar concepts of health, grouped into a new higher order of ‘meaningful’ outcome domains [61, 62].

Round 3: Participants will be asked to consider if they are happy with a series of decisions informed by earlier rounds. First, they will be advised of the top 50% of prioritised outcome domains from round 2. They will then be advised of any between-panel discrepancies. Where outcome domains are prioritised by just one panel, respondents will be asked to consider if they should be included in the prioritised list (Yes/No); outcome domains voted for inclusion by ≥70% of respondents will be included. Finally, respondents will be asked to specify if they agree (Yes/No) with the grouping of prioritised outcome domains, and the meaningful outcome domains and definitions. An option to provide additional comments will be available. A frequency distribution of responses will be computed.

Achieving a core domain set

It is anticipated that a core domain set will be achieved by the end of round 3. Whilst there is no agreed ‘ideal’ number of core domains, to be feasible and acceptable, a smaller core domain set is recommended (i.e. fewer than 8 outcome domains) [63]. However, numerous examples of COS development have required further consensus activities to clarify and ratify findings from the Delphi process [20, 27, 61]. Recent protocols to develop core outcome sets have also outlined plans for consensus meetings to ratify a core domain set [64, 65]. Should a core domain set not be achieved following the e-Delphi, then the findings will be taken to a consensus meeting for final voting and ratification (stage 5) before seeking consensus on a core measurement set.

Stage 5: Consensus meeting

The aim of this meeting will be to confirm the core domain set (stage 4), agree a core measurement set (CMS) and recommend the COS for distal Tibia and Ankle fracture (COSTA). Outcome domains that are considered important but do not have an available measurement approach will not be included within the COS.

We will invite health professionals and patients who participated in the e-Delphi study to take part in this consensus meeting. In advance of the meeting, participants will be provided with an information pack outlining the objectives of the meeting—this will include the CDS and short-listed outcome measures to be considered for the CMS.

Developing the core measurement set

In advance of the meeting, we will identify available guidance or existing consensus on how best to measure short-listed outcome domains. Where there is no existing consensus for potential outcome measures, the core research team will review key evidence sources to determine measurement quality [66], acceptability, and feasibility.

The meeting will begin with an overview of the e-Delphi and the prioritised outcome domains, with accompanying information on the evidence underpinning identified measures. Participants will be asked to consider ‘placement’ of the outcome domains within the final COS [61, 62]:

  1. 1.

    Core ‘inner’ circle: outcome domain is clear with an acceptable, method of assessment.

  2. 2.

    Middle circle: whilst important, the inclusion of this outcome domain in all trials is not feasible (e.g. lack of available assessment method).

  3. 3.

    Outer circle: whilst important, the current available evidence for the conceptualisation of the outcome domain or assessment method is limited.

The meeting will be structured into three parts:

  1. i.

    Initial presentations to the large group will be followed by facilitated small-group discussions with a mixed group of stakeholders. Groups will discuss each prioritised outcome domain and potential method of assessment, considering evidence of quality, acceptability, feasibility, and importance. Findings between groups will then be shared to facilitate further discussion.

  2. ii.

    At the end of each small-group discussion, participants will anonymously complete a questionnaire to confirm the inclusion of each outcome domain (Yes/No/Don’t know) and their preferred method of assessment (selecting from a short-list). Agreement will be defined as ≥ 70% participants endorsing an outcome domain and/or method of assessment.

  3. iii.

    A whole group discussion will be facilitated to discuss outcome domain priorities or assessment methods. Where there is agreement, no further discussion will be required. Subsequent discussions will focus on areas of disagreement and where further refinement is required. Finally, participants will be asked to anonymously vote electronically to confirm ‘placement’ of all core domains (inner, middle or outer circles), and, where previously not confirmed, the preferred method of assessment.

Written notes will be taken throughout the sessions, and the voting will be recorded.

The outcome of this consensus meeting will be to ratify a COS for distal tibia and ankle trials, identifying both the core outcome domains that should be minimally assessed, alongside evidence-based recommendations on current best methods for assessment.


In line with recommendations, ongoing work will be necessary to maximise the uptake and implementation of the COS [16]. We will seek to actively address this challenge through our close active collaboration with our PPI group and engagement with our international advisory group.

Participants in the research will be informed of project outcomes through the sharing of a summary document following completion of each stage. This will be shared directly with participants via email. COS users (including funders and journal editors) will be reached through a range of methods, including wide dissemination through publications and both national and international conference presentations. Additionally, AO Trauma’s international reach through their leading expertise in trauma will enable the team to directly inform the international trauma community about COSTA.


Currently, outcome reporting guidance for distal tibia or ankle fractures is lacking, despite the need for such guidance being highlighted in recent reviews [9,10,11]. The development and uptake of a core outcome set for tibia and ankle fractures (COSTA) will ensure that outcomes viewed as important to key stakeholders are included in clinical trials, supporting the development of an evidence base that can be synthesised and better examined to support healthcare decisions.

A limitation of the COS is that it will rely on a UK sample of patients throughout. This is due to project time constraints which are amplified by COVID. However, systematic reviews of international qualitative and quantitative literature will be used to inform an initial long list of potential outcome domains.

A well-developed COSTA will address the current challenges associated with heterogeneity in outcome selection and reporting in ankle fracture research [9,10,11] through improving opportunities for evidence synthesis, creating opportunity for comparative reviews of care provision across different fracture types, reducing reporting bias, and reducing research waste [13,14,15]. To ensure these benefits are brought to fruition, ongoing work will be necessary to ensure the uptake and implementation of COSTA. This will involve maximising dissemination of the COS and highlighting its value in application for both clinical practice and research.

Trial status

Protocol version number: 1.0

Protocol date: March 8, 2021

Recruitment start date: delayed due to COVID

Planned recruitment end date: January 2022

Availability of data and materials

Data from this study will be accessible to the research team, advisory group, and patient research partners.


  1. 1.

    Curtis EM, van der Velde R, Moon RJ, van den Bergh JPW, Geusens P, de Vries F, et al. Epidemiology of fractures in the United Kingdom 1988-2012: variation with age, sex, geography, ethnicity and socioeconomic status. Bone. 2016;87:19–26 Available from:

    Article  Google Scholar 

  2. 2.

    Jennison T, Brinsden M. Fracture admission trends in England over a ten-year period. Ann R Coll Surg Engl. 2019;101(3):208–14 Available from:

    CAS  Article  Google Scholar 

  3. 3.

    Forsberg A, Söderberg S, Engström Å. People’s experiences of suffering a lower limb fracture and undergoing surgery. J Clin Nurs. 2014;23(1–2):191–200.

    Article  PubMed  Google Scholar 

  4. 4.

    Stufkens SAS, van den Bekerom MPJ, Kerkhoffs GMMJ, Hintermann B, van Dijk CN. Long-term outcome after 1822 operatively treated ankle fractures: a systematic review of the literature. Injury. 2011;42(2):119–27.

    Article  PubMed  Google Scholar 

  5. 5.

    Trickett RW, Mudge E, Price P, Pallister I. A qualitative approach to recovery after open tibial fracture: the road to a novel, patient-derived recovery scale. Injury. 2012;43(7):1071–8.

    CAS  Article  PubMed  Google Scholar 

  6. 6.

    Rees S, Tutton E, Achten J, Bruce J, Costa ML. Patient experience of long-term recovery after open fracture of the lower limb: a qualitative study using interviews in a community setting. BMJ Open. 2019;9(10):e031261 Available from:

    Article  Google Scholar 

  7. 7.

    Van Son MAC, De Vries J, Roukema JA, Den Oudsten BL. Health status, health-related quality of life, and quality of life following ankle fractures: a systematic review. Injury. 2013;44(11):1391–402.

    Article  PubMed  Google Scholar 

  8. 8.

    van der Sluis CK, Eisma WH, Groothoff JW, ten Duis HJ. Long-term physical, psychological and social consequences of a fracture of the ankle. Injury. 1998;29(4):277–80.

    Article  PubMed  Google Scholar 

  9. 9.

    Lin CWC, Donkers NAJ, Refshauge KM, Beckenkamp PR, Khera K, Moseley AM. Rehabilitation for ankle fractures in adults. Cochrane Database Syst Rev. 2012;(11). Available from:.

  10. 10.

    Del Buono A, Smith R, Coco M, Woolley L, Denaro V, Maffulli N. Return to sports after ankle fractures: a systematic review. Br Med Bull. 2012;106(1):179–91. Available from.

    Article  PubMed  Google Scholar 

  11. 11.

    Marson BA, Ng J, Myint Y, Grindlay DJC, Ollivere BJ. Management of ‘low-risk’ ankle fractures in children: a systematic review. Ann R Coll Surg Engl. 2019;101(8):539–45. Available from.

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  12. 12.

    Ollivere BJ, Marson BA, Haddad FS. Getting the right answer: core outcome sets in orthopaedics. Bone Joint J. 2019;101-B:233–5.

  13. 13.

    Kirkham JJ, Altman DG, Chan A-W, Gamble C, Dwan KM, Williamson PR. Outcome reporting bias in trials: a methodological approach for assessment and adjustment in systematic reviews. BMJ. 2018;362:k3802 Available from:

    Article  Google Scholar 

  14. 14.

    Kirkham JJ, Dwan KM, Altman DG, Gamble C, Dodd S, Smyth R, et al. The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews. BMJ. 2010;340:c365 Available from:

    Article  Google Scholar 

  15. 15.

    Kirkham JJ, Gargon E, Clarke M, Williamson PR. Can a core outcome set improve the quality of systematic reviews?--a survey of the Co-ordinating Editors of Cochrane Review Groups. Trials. 2013;14(1):21.

    Article  PubMed  PubMed Central  Google Scholar 

  16. 16.

    Williamson PR, Altman DG, Blazeby JM, Clarke M, Devane D, Gargon E, et al. Developing core outcome sets for clinical trials: issues to consider. Trials. 2012;13(1):132. Available from.

    Article  PubMed  PubMed Central  Google Scholar 

  17. 17.

    Haywood KL, Griffin XL, Achten J, Costa ML. Developing a core outcome set for hip fracture trials. Bone Joint J. 2014g;96-B(8):1016–23.

    CAS  Article  PubMed  Google Scholar 

  18. 18.

    Goldhahn J, Beaton D, Ladd A, Macdermid J, Hoang-Kim A. Recommendation for measuring clinical outcome in distal radius fractures: a core set of domains for standardized reporting in clinical practice and research. Arch Orthop Trauma Surg. 2014;134(2):197–205.

    Article  PubMed  Google Scholar 

  19. 19.

    Aquilina A, Costa M, Griffin X, Tutton L, Fitzpatrick R, Khan U, et al. Core Outcomes for research on Open Lower Limb Fractures (CO-OLLF): development of a core outcome set and core outcome measurement set . [cited 2020 Jan 15]. Available from:

  20. 20.

    Ramiro S, Page MJ, Whittle SL, Huang H, Verhagen AP, Beaton DE, et al. The OMERACT core domain set for clinical trials of shoulder disorders. J Rheumatol. 2019;46(8):969 LP–975 Available from:

    Article  Google Scholar 

  21. 21.

    McKeown R, Rabiu A-R, Ellard DR, Kearney RS. Primary outcome measures used in interventional trials for ankle fractures: a systematic review. BMC Musculoskelet Disord. 2019;20(1):388 Available from:

    Article  Google Scholar 

  22. 22.

    Marson BA, Manning JC, James M, Craxford S, Deshmukh SR, Ollivere BJ. CORE-Kids: a protocol for the development of a core outcome set for childhood fractures. BMJ Open. 2020;10(2):e036224 Available from:

    Article  Google Scholar 

  23. 23.

    Boers M, Beaton DE, Shea BJ, Maxwell LJ, Bartlett SJ, Bingham CO 3rd, et al. OMERACT Filter 2.1: Elaboration of the conceptual framework for outcome measurement in health intervention studies. J Rheumatol. 2019 Aug;46(8):1021–7.

    Article  PubMed  Google Scholar 

  24. 24.

    Williamson PR, Altman DG, Bagley H, Barnes KL, Blazeby JM, Brookes ST, et al. The COMET Handbook: version 1.0. Trials. 2017;18(3):280. Available from.

    Article  PubMed  PubMed Central  Google Scholar 

  25. 25.

    Boers M, Kirwan JR, Tugwell P, Beaton D, Bingham CO, Conaghan PG, et al. The OMERACT Handbook . 2017 [cited 2020 Jan 15]. Available from:

  26. 26.

    Duffy JMN, Hirsch M, Vercoe M, Abbott J, Barker C, Collura B, et al. A core outcome set for future endometriosis research: an international consensus development study. BJOG An Int J Obstet Gynaecol. 2020;127(8):967–74. Available from.

    Article  Google Scholar 

  27. 27.

    Haywood KL, Whitehead L, Perkins GD. An international, consensus-derived Core Outcome Set for Cardiac Arrest effectiveness trials: the COSCA initiative. Curr Opin Crit Care. 2019;25(3):226–33.

    Article  PubMed  Google Scholar 

  28. 28.

    Chiarotto A, Boers M, Deyo RA, Buchbinder R, Corbin TP, Costa LOP, et al. Core outcome measurement instruments for clinical trials in nonspecific low back pain. Pain [Internet]. 2018;159(3):481–95. Available from:

    Article  PubMed  PubMed Central  Google Scholar 

  29. 29.

    Fackrell K, Smith H, Colley V, Thacker B, Horobin A, Haider HF, et al. Core Outcome Domains for early phase clinical trials of sound-, psychology-, and pharmacology-based interventions to manage chronic subjective tinnitus in adults: the COMIT’ID study protocol for using a Delphi process and face-to-face meetings to establis. Trials. 2017;18(1):388 Available from:

    Article  Google Scholar 

  30. 30.

    Young B, Bagley H. Including patients in core outcome set development: issues to consider based on three workshops with around 100 international delegates. Res Involv Engagem. 2016;2(1):25. Available from.

    Article  PubMed  PubMed Central  Google Scholar 

  31. 31.

    Meinberg EG, Agel J, Roberts CS, Karam MD, Kellam JF. Fracture and Dislocation Classification Compendium—2018. J Orthop Trauma. 2018:32 Available from:

  32. 32.

    Kirwan JR, Minnock P, Adebajo A, Bresnihan B, Choy E, de Wit M, et al. Patient perspective: fatigue as a recommended patient centered outcome measure in rheumatoid arthritis. J Rheumatol, Canada. 2007;34:1174–7.

  33. 33.

    Clarke M. Standardising outcomes for clinical trials and systematic reviews. Trials. 2007;8(1):39. Available from:.

    Article  PubMed  PubMed Central  Google Scholar 

  34. 34.

    Sanderson T, Morris M, Calnan M, Richards P, Hewlett S. What outcomes from pharmacologic treatments are important to people with rheumatoid arthritis? Creating the basis of a patient core set. Arthritis Care Res (Hoboken). 2010;62(5):640–6.

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  35. 35.

    Smith H, Horobin A, Fackrell K, Colley V, Thacker B, Hall DA, et al. Defining and evaluating novel procedures for involving patients in Core Outcome Set research: creating a meaningful long list of candidate outcome domains. Res Involv Engagem. 2018, 2018;(1):4, 8. Available from.

  36. 36.

    INVOLVE. Guidance on co-producing a research project [Internet]. 2019 [cited 2019 Apr 23]. Available from:

  37. 37.

    COMET PoPPIE Group. Checklist for Public Research Partners and COS Study Developers Involved in Designing a COS Study [Internet]. 2015 [cited 2020 Jan 15]. Available from:

  38. 38.

    NIHR. UK Standards for Public Involvement [Internet]. 2021 [cited 2021 May 18]. Available from: = 0

  39. 39.

    Staniszewska S, Brett J, Simera I, Seers K, Mockford C, Goodlad S, et al. GRIPP2 reporting checklists: tools to improve reporting of patient and public involvement in research. BMJ. 2017;358:j3453 Available from:

    CAS  Article  Google Scholar 

  40. 40.

    Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ. 2009;339:b2535 Available from:

    Article  Google Scholar 

  41. 41.

    WHO. International classification of functioning, disability and health: ICF. Geneva: World Health Organisation; 2001. [cited 2019 Oct 1]. Available from:

    Google Scholar 

  42. 42.

    Thomas J, Harden A. Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Med Res Methodol. 2008;8(1):45. Available from.

    Article  PubMed  PubMed Central  Google Scholar 

  43. 43.

    Court-Brown CM, McBirnie J, Wilson G. Adult ankle fractures--an increasing problem? Acta Orthop Scand. 1998;69(1):43–7.

    CAS  Article  PubMed  Google Scholar 

  44. 44.

    Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. 2006;3(2):77–101.

    Article  Google Scholar 

  45. 45.

    Sanderson TC, Hewlett SE, Flurey C, Dures E, Richards P, Kirwan JR. The impact triad (severity, importance, self-management) as a method of enhancing measurement of personal life impact of rheumatic diseases. J Rheumatol. 2011;38(2):191 LP–194 Available from:

    Article  Google Scholar 

  46. 46.

    Smith J, Flowers P, Larkin M. Interpretative phenomenological analysis: theory, method and research; 2009.

    Google Scholar 

  47. 47.

    Yardley L. Dilemmas in qualitative health research. Psychol Health. 2000;15(2):215–28. Available from.

    Article  Google Scholar 

  48. 48.

    Keene DJ, Costa ML, Tutton E, Hopewell S, Barber VS, Dutton SJ, et al. Progressive functional exercise versus best practice advice for adults aged 50 years or over after ankle fracture: protocol for a pilot randomised controlled trial in the UK - the Ankle Fracture Treatment: Enhancing Rehabilitation (AFTER) study. BMJ Open. 2019 Nov;9(11):e030877.

    Article  PubMed  PubMed Central  Google Scholar 

  49. 49.

    Krueger R, Casey M. Focus groups: a practical guide for applied research. 4th ed. Thousand Oaks, California: Sage; 2009.

    Google Scholar 

  50. 50.

    Kitzinger J. Qualitative research: introducing focus groups. BMJ. 1995;311(7000):299 LP–302 Available from:

    Article  Google Scholar 

  51. 51.

    Bender DE, Ewbank D. The focus group as a tool for health research: issues in design and analysis. Heal Transit Rev Cult Soc Behav Determ Heal. 1994;4(1):63–80.

    CAS  Google Scholar 

  52. 52.

    Stucki G, Boonen A, Tugwell P, Cieza A, Boers M. The World Health Organisation International Classification of Functioning, Disability and Health: a conceptual model and interface for the OMERACT process. J Rheumatol. 2007;34(3):600–6 Available from:

    PubMed  Google Scholar 

  53. 53.

    Sinha IP, Smyth RL, Williamson PR. Using the Delphi technique to determine which outcomes to measure in clinical trials: recommendations for the future based on a systematic review of existing studies. PLoS Med. 2011;8(1):e1000393.

    Article  PubMed  PubMed Central  Google Scholar 

  54. 54.

    Fink A, Kosecoff J, Chassin M, Brook RH. Consensus methods: characteristics and guidelines for use. Am J Public Health. 1984;74(9):979–83. Available from:

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  55. 55.

    Eubank BH, Mohtadi NG, Lafave MR, Wiley JP, Bois AJ, Boorman RS, et al. Using the modified Delphi method to establish clinical consensus for the diagnosis and treatment of patients with rotator cuff pathology. BMC Med Res Methodol. 16(1):2016, 56. Available from.

  56. 56.

    Manera KE, Tong A, Craig JC, Shen J, Jesudason S, Cho Y, et al. An international Delphi survey helped develop consensus-based core outcome domains for trials in peritoneal dialysis. Kidney Int. 2019;96(3):699–710.

    Article  PubMed  PubMed Central  Google Scholar 

  57. 57.

    Topjian AA, Scholefield BR, Pinto NP, Fink EL, Buysse CMP, Haywood K, et al. P-COSCA (pediatric core outcome set for cardiac arrest) in children: an advisory statement from the International Liaison Committee on Resuscitation. Circulation. 2020;142(16):e246–61. Available from.

    Article  PubMed  Google Scholar 

  58. 58.

    Karas J, Ashkenazi S, Guarino A, Lo Vecchio A, Shamir R, Vandenplas Y, et al. A core outcome set for clinical trials in acute diarrhoea. Arch Dis Child. 2015;100(4):359 LP–363 Available from:

    Article  Google Scholar 

  59. 59.

    Somner JEA, Ismail R, Froud R, Azuara-Blanco A, King AJ. Consensus generation of a minimum set of outcome measures for auditing glaucoma surgery outcomes—a Delphi exercise. Graefe’s Arch Clin Exp Ophthalmol. 2018;256(12):2407–11. Available from.

    CAS  Article  Google Scholar 

  60. 60.

    Blazeby JM, Macefield R, Blencowe NS, Jacobs M, McNair AGK, Sprangers M, et al. Core information set for oesophageal cancer surgery. Br J Surg. 2015;102(8):936–43.

    CAS  Article  PubMed  Google Scholar 

  61. 61.

    Haywood K, Potter R, Froud R, Pearce G, Box B, Muldoon L, et al. A Core Outcome Set for Preventive Intervention Trials in Chronic and Episodic Migraine (COSMIG): an international, consensus-derived and multi-stakeholder initiative. BMJ. 2021.

  62. 62.

    Orbai A-M, de Wit M, Mease P, Shea JA, Gossec L, Leung YY, et al. International patient and physician consensus on a psoriatic arthritis core outcome set for clinical trials. Ann Rheum Dis. 2017 Apr;76(4):673–80.

    Article  PubMed  Google Scholar 

  63. 63.

    Kirstie H, Laura W, Nadkarni VM, Felix A, Stefanie B, Böttiger BW, et al. COSCA (Core Outcome Set for Cardiac Arrest) in adults: an advisory statement from the International Liaison Committee on Resuscitation. Circulation. 2018;137(22):e783–801. Available from.

    Article  Google Scholar 

  64. 64.

    Rose L, Agar M, Burry LD, Campbell N, Clarke M, Lee J, et al. Development of core outcome sets for effectiveness trials of interventions to prevent and/or treat delirium (Del-COrS): study protocol. BMJ Open. 2017;7(9):e016371 Available from:

    Article  Google Scholar 

  65. 65.

    Gagnier JJ, Page MJ, Huang H, Verhagen AP, Buchbinder R. Creation of a core outcome set for clinical trials of people with shoulder pain: a study protocol. Trials. 2017;18(1):336 Available from:

    Article  Google Scholar 

  66. 66.

    McKeown R, Ellard DR, Rabiu A-R, Karasouli E, Kearney RS. A systematic review of the measurement properties of patient reported outcome measures used for adults with an ankle fracture. J Patient Rep Outcomes. 2019;3(1):70 Available from:

    Article  Google Scholar 

Download references


We would like to thank AO Trauma for funding this project and Professor Kate Seers for providing peer review of the project plan and ethics applications. We also thank our COSTA collaborators Andrew Metcalfe, Joel Humphrey, Marilyn Heng, Marinis Pirpiris, Martin Underwood, Prakash Jayakumar, Rebecca Kearny, Rodrigo Pesantez, and Simon Lambert. The team contributed to the study design and writing the manuscript. We would also like to thank the study patient and public involvement group for their contributions to the study.


This work was supported by the AO Foundation via AO Trauma (no grant number assigned).

Author information




KH is the Chief Investigator on this study and conceived the study with ET, AJ, and SG. KH and NP designed the project and developed the protocol. DK is a collaborator on the project and is providing additional secondary data. NP and KH wrote the manuscript. RG is a patient research partner who reviewed the study methods, providing the patient perspective on the planned research process. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Nathan A. Pearson.

Ethics declarations

Ethics approval and consent to participate

Ethical approval to conduct the qualitative phase of study was obtained from a Research Ethics Committee (REC) proportionate review subcommittee (REC reference: 20/LO/1147) and the Health Research Authority (HRA) dated November 10, 2020. Approval to conduct the e-Delphi was obtained from a REC proportionate review subcommittee (REC reference: 21/SC/0173) and HRA dated June 3, 2021. Participants will provide written informed consent, but if not possible (e.g. with a telephone or online interview), then verbal consent will be taken, recorded in the transcript, and witnessed by a GCP trained researcher.

Consent for publication

Not applicable

Competing interests

All authors have no competing interests to declare.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit The Creative Commons Public Domain Dedication waiver ( applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Pearson, N.A., Tutton, E., Joeris, A. et al. Co-producing a multi-stakeholder Core Outcome Set for distal Tibia and Ankle fractures (COSTA): a study protocol. Trials 22, 443 (2021).

Download citation


  • Distal tibia
  • Malleolar
  • injury
  • Ankle fracture
  • Trauma
  • Core outcome sets
  • Interviews
  • Delphi study
  • Systematic review