Skip to main content

Starting a conversation about estimands with public partners involved in clinical trials: a co-developed tool

Abstract

Background

Clinical trials aim to draw conclusions about the effects of treatments, but a trial can address many different potential questions. For example, does the treatment work well for patients who take it as prescribed? Or does it work regardless of whether patients take it exactly as prescribed? Since different questions can lead to different conclusions on treatment benefit, it is important to clearly understand what treatment effect a trial aims to investigate—this is called the ‘estimand’. Using estimands helps to ensure trials are designed and analysed to answer the questions of interest to different stakeholders, including patients and public. However, there is uncertainty about whether patients and public would like to be involved in defining estimands and how to do so. Public partners are patients and/or members of the public who are part of, or advise, the research team. We aimed to (i) co-develop a tool with public partners that helps explain what an estimand is and (ii) explore public partner’s perspectives on the importance of discussing estimands during trial design.

Methods

An online consultation meeting was held with 5 public partners of mixed age, gender and ethnicities, from various regions of the UK. Public partner opinions were collected and a practical tool describing estimands, drafted before the meeting by the research team, was developed. Afterwards, the tool was refined, and additional feedback sought via email.

Results

Public partners want to be involved in estimand discussions. They found an introductory tool, to be presented and described to them by a researcher, helpful for starting a discussion about estimands in a trial design context. They recommended storytelling, analogies and visual aids within the tool. Four topics related to public partners’ involvement in defining estimands were identified: (i) the importance of addressing questions that are relevant to patients and public in trials, (ii) involving public partners early on, (iii) a need for education and communication for all stakeholders and (iv) public partners and researchers working together.

Conclusions

We co-developed a tool for researchers and public partners to use to facilitate the involvement of public partners in estimand discussions.

Peer Review reports

Background

Clinical trials are conducted to evaluate how well healthcare interventions work. However, in trials, as in routine health practice, treatments are not always taken by patients exactly as prescribed [1, 2]. When this happens, by looking at trial data in various ways, different (and potentially more relevant) questions can be addressed such as: ‘does the treatment improve health outcomes for all patients even if it is not taken as instructed?’ or ‘does the treatment improve health outcomes only for patients who take treatment as prescribed?’ Because the answers to different questions can sometimes lead to alternative conclusions on treatment benefits, when planning a trial, it is important to have clarity on exactly what questions the trial needs to address [3, 4]. The trial can be subsequently designed, conducted, and analysed to ensure the key clinical questions of interest are addressed.

To facilitate clarity on precisely what a trial intends to investigate, international trial regulatory guidelines (ICH E9(R1)) call for trialists to clearly define the treatment effects being investigated by using estimands [5]. An estimand is a precise description of what treatment effect a trial is aiming to address (for examples, see [3, 6]). It describes what the numerical result obtained in the trial will mean. By clarifying the interpretation of numerical results, estimands can help ensure these results are relevant and useful to all stakeholders, including patients and public. Since the question of most relevance to address will likely vary across different stakeholders, more than one question of interest, hence estimand, may need to be investigated in a trial.

There is increasing attention being placed on using estimands in trial design, therefore giving patients and public the possibility of getting involved in this critical part of trial design is important. Previous work has identified that patients and public want to be involved in determining numerical aspects of trials [7, 8]; however, there is uncertainty about whether they would like to be involved in discussing and defining estimands as this has not been previously explored. Moreover, there is first a need to break down the barriers of scientific jargon and explain what an estimand is to public partners. Public partners are patients and/or members of the public who are part of the research team or advise the research team (not trial participants). We aimed to (i) co-develop a practical tool with public partners that describes what an estimand is and what impact it may have in trial results interpretation and (ii) explore public partners’ perspectives on discussing estimands with public partners when designing a trial.

Methods

An online consultation meeting was held with researchers and public partners from an established statistical trial methodology project, the HEALTHY STATS public involvement group, in January 2022. The HEALTHY STATS public involvement group was initiated in January 2021 to provide advice on an NIHR funded project to develop accessible statistical methods to determine treatments effects that matter to patients and prescribing physicians in randomised controlled trials (Advanced fellowship NIHR300593). The specific remit of the HEALTHY STATS public involvement group was to support the researchers to improve the information reported from clinical trials for patients and public. The group consisted of five public partners (AH, JC, MK, PH, YR) who were chosen to represent the patient and public voice. Members were aged between 20 and 70 years of mixed ethnicities and gender, and the group was co-led by SC and AH. Four researchers with statistical expertise facilitated the breakout discussions (SC, BK, AP, and BG). All attendees were located in England or Scotland.

Prior to the meeting a practical two-page tool to explain what an estimand is and why it matters was first drafted by members of the research (SC, BK, and BG) team. This was based on knowledge of the estimand framework described in ICH E9(R1) [5] and the researchers previous clinical trial experience.

The specific objectives of the meeting were to:

  1. 1.

    Co-develop a practical tool with public partners that the study team can use to help explain to public partners involved in trial planning what an estimand is and what impact it may have on trial results.

  2. 2.

    Explore public partners perspectives on discussing and defining the precise treatment effects to be found out (i.e. estimands) with public partners when designing a trial

The online meeting used the Zoom platform and was audio-recorded and lasted 2 h. Public partners’ perspectives were collected using Zoom polls and open-ended questions. After the meeting, the poll responses on usefulness of the tool were tabulated and the audio-recording was transcribed by SC. Discussion points were sorted into four general topic areas by SC and BG.

The practical tool describing estimands was reviewed and further developed by the public partners during the meeting. Afterwards the tool was refined, and additional feedback was sought via email in two rounds of refinement.

Results

Tool to explain an estimand

All 5 of the HEALTHY STATS group voted that the tool was useful. The feedback obtained on the tool included the following: (i) public partners found the tool useful to start a discussion about what question a trial is answering in a trial design context; (ii) they recommended the use of storytelling, analogies and visual aids; (iii) it was felt that the tool should be shared and a chance provided to discuss it with the trial team/statistician; (iv) public partners raised that potential trial participants might need/want to know about the estimand of the trial; however, this tool would not be indicated for that. They reported that there is already a lot of complex information presented to trial participants in patient information sheets and felt this information on estimands could be too much.

After the meeting, the tool was updated using the feedback in an iterative process with further feedback sought via email in two rounds of refinement. Figure 1 is a screenshot of the finalised tool which explains what an estimand is and why it is important. The full tool can be found in Additional file 1 for download and use, and if you are interested in being contacted to provide feedback on the tool you can download it and provide your contact details here https://www.statsci.co.uk/ppi [9]. The tool starts with an analogy of buying a new car and explains why individuals need to ask clear and specific questions about a car to understand whether it will perform as desired for them, as different individuals have different needs. This illustrates how when testing a new treatment all stakeholders need to be clear about the precise question being asked, so that the answer is meaningful and useful.

Fig. 1
figure 1

Overview of estimand explainer tool

Page two then proceeds to further demonstrate why this matters. This introduces a fictional two-group trial setting comparing a new tablet against a dummy (placebo) treatment for headache prevention. It explains how a group of trial participants were given the dummy treatment whilst another given the new tablets which were to be taken four times a day, but how some participants in the new tablet group stopped taking treatment early because of stomach cramps. It illustrates how asking different questions in this trial can lead to different results and therefore why it is important to think carefully about which specific question we want to know the answer to.

In the headache trial, by asking question one—what was the typical reduction in the number of headaches for a participant even if they did not take all tablets each day?—there is an average difference of half a headache between the groups. By asking question two—what was the typical reduction in the number of headaches for a participant who took all 4 tablets each?—there is an average difference of 4 headaches between the groups.

The tool ends by bringing in the estimand term as a label to refer to the exact question (i.e. treatment effect) the trial is aiming to answer.

Public partner perspectives

The four key topics that came out of the public partner discussions are summarised in Fig. 2.

Fig. 2
figure 2

Four key topics raised by public partners on discussing the precise research question addressed in a trial

Topic 1: Trials should address questions that are relevant to patients and public

Public partners highlighted the importance of conducting trials that matter to patients and public. To achieve this, they raised how trials need to select outcomes that matter to the patient. They understood that we could find out the answer to many different questions in a single trial for the same outcome and felt we should be asking patients what they want from the trial. Public partners expressed that they would want a say on this, particularly if it is public funded research. They acknowledged we cannot always serve everyone and directly address all patient and public desires, but by engaging with public partners, researchers can identify their needs and ensure trials get close to what matters most to them.

Topic 2: Involve public partners early on in trial planning

To make sure trials give answers relevant to patients, it was felt important to involve public partners early on in initial trial planning meetings, e.g. in trial steering committee meetings. If this is not done early on, then there was a concern the public partner voice may be overlooked later. By involving early on, there is opportunity to shape the direction of the trial.

Topic 3: Education and communication is necessary

To achieve successful involvement in this area, education and communication from researchers was felt necessary as this is a new area unfamiliar to patients and public. They acknowledged this would take time and be labour intensive to educate people to know how to ask the right questions.

Topic 4: A change of culture is required of public partners and researchers working together

Public partners indicated it will require a change of culture of researchers and public partners working closely together to achieve this. They felt talking about the precise scientific question addressed in a trial is a complex issue for a lay person. There is a need to break barriers between researchers and public partners by ensuring lay terms are used and to make it clear why the precise definition of the research question is important. Along with discussing the precise research question in general, we considered the estimand term which is the label used to describe the precise definition of the treatment effect. Initially, the word estimand was felt to be a piece of ‘statistical jargon’ that was ‘uninteresting’ to know. However, after discussion, public partners felt that if researchers are going to use this term around them, for example in trial steering committee meetings involving other researchers, they do need to first explain what it means. The co-developed tool can be described to achieve this (Fig. 1).

Discussion

Main findings

Overall, we found there was unanimous support from the HEALTHY STATS group to involve public partners in discussions around choosing estimands. Public partners want a say on the precise research question a clinical trial addresses so that trials do find out what matters to patients and the public. If investigators are going to use the estimand term in trial design conversations involving public partners, this term needs to be explained. We have co-developed a practical tool with the HEALTHY STATS group that explains what an estimand is and why it matters, to facilitate discussions during trial planning. Researchers can share and discuss this with public partners involved in trial design, to start a conversation around estimands.

Research in context

Our findings demonstrate a desire for public partners to be involved in setting the precise research question addressed in a trial analysis, which is line with previous research that has identified patients and public wanting to be involved in other numerical aspects of clinical trials [7, 8]. The topics we found via discussion with the HEALTHY STATS public partners (Fig. 1) are in line with previous literature. For example, the communication and jargon challenge and need for education to enable meaningful patient and public involvement in research and especially around statistics [7, 10, 11]. The importance of conducting involvement early in the research process has also been recommended before [12, 13]. The calls for change of culture to enable better patient and public involvement are also highlighted in a recent survey, where public partners identified the main barrier for involvement to have an impact is tokenism and not being taken seriously [14].

A previous overview of systematic reviews of patient and public involvement in clinical trial design concluded that involvement can be beneficial but requires resources, preparation, training, flexibility and time [15]. The review highlighted a need for clarity within a common language which our public partners also clarified with respect to the estimand. We hope that the developed tool provides such clarity in this emerging area.

Strengths and limitations

This is, to our knowledge, the first workshop discussing estimands with public partners, and it addresses a main barrier to involvement in clinical trial statistics [7, 11] by introducing a tool to enable better communication. Our methodology has been described to show researchers how patient involvement can work in a trial methodology setting. The nature of the workshop and process was participatory and allowed everyone involved to have a say on the tool and shape its content and format. The fact that we went to the HEALTHY STATS group can also be seen as a strength: this group has its roles and expectations, and they clearly trusted their opinions would be heard in a non-judgemental way. The public partners included in our study had a mix of ages, genders and ethnicities. However, due financial restrictions, we were limited to 5 public partners and so results might not be generalisable. Although the example trial used within the tool is a drug trial, it can be used across medical disciplines to explain the concept of an estimand, for example including surgical settings and therapeutic settings. The intervention and medical condition in the example on page 2 of the tool could be easily edited and adapted by researchers to aid communication in non-pharmaceutical settings. Estimands are applicable in clinical trials across medical fields.

Future research

The co-developed tool can be used to start a conversation about estimands with researchers and public partners, for example in trial steering committee meetings. The next steps will be to use this tool with larger different groups of public partners and in different trial design contexts and to evaluate performance and implementation. We welcome readers who use the tool to contact the corresponding author of this article (SC) to provide feedback on its implementation.

Defining an estimand typically consists of specifying five components: the population of patients targeted, the treatment conditions being compared, how intercurrent events are being handled, the outcome variable and the population level summary measure. Understanding these individual 5 aspects was beyond the scope of this project which explored the general concept of an estimand. The best way to define estimands with public partners in a trial design context needs to be established.

Public partners’ interest in being involved in estimands could be further explored, since we only asked the HEALTHY STATS group. Further research is also required to establish whether estimand information should be made available for individuals considering taking part in a trial who are different to public partners. It is currently unclear whether this information is necessary and, if so, how? We have only considered communicating about what an estimand is with public partners involved in trial design and started the conversation in this area. We have not considered whether trial participants need estimand information available, which will involve different considerations.

Conclusions

Our experience with the HEALTHY STATS public involvement group indicates public partners want to be involved in establishing estimands early on during the planning of a clinical trial. This is to help ensure trials address what patients and the public want to know. To facilitate the involvement of public partners in estimand discussions, we co-developed a tool explaining the concept of an estimand and why it matters, which is available for researchers and public partners to use.

Availability of data and materials

All data and materials are included in this publication.

Abbreviations

ICH E9(R1):

International council for harmonisation of technical requirements for pharmaceuticals for human use addendum on estimands and sensitivity analysis in clinical trials to the guideline on statistical principles for clinical trials

References

  1. Dodd S, White IR, Williamson P. Nonadherence to treatment protocol in published randomised controlled trials: a review. Trials. 2012;13(1):84.

    Article  PubMed  PubMed Central  Google Scholar 

  2. Haynes RB, McDonald HP, Garg AX. Helping patients follow prescribed treatment: clinical applications. JAMA. 2002;288(22):2880–3.

    Article  PubMed  Google Scholar 

  3. Cro S, Kahan BC, Rehal S, Chis Ster A, Carpenter JR, White IR, et al. Evaluating how clear the questions being investigated in randomised trials are: systematic review of estimands. BMJ. 2022;378:e070146.

    Article  PubMed  PubMed Central  Google Scholar 

  4. Cro S. Time to improve the clarity of clinical trial reports by including estimands. BMJ. 2022;378:o2108.

    Article  PubMed  Google Scholar 

  5. International council for harmonisation of technical requirements for pharmaceuticals for human use. Addendum on estimands and sensitivity analysis in clinical trials to the guideline on statistical principles for clinical trials 20 november 2019. Available at https://database.ich.org/sites/default/files/E9-R1_Step4_Guideline_2019_1203.pdf. Accessed 30.06.23.

  6. Clark TP, Kahan BC, Phillips A, White I, Carpenter JR. Estimands: bringing clarity and focus to research questions in clinical trials. BMJ Open. 2022;12(1):e052953.

    Article  PubMed  PubMed Central  Google Scholar 

  7. Goulao B, Bruhn H, Campbell M, Ramsay C, Gillies K. Patient and public involvement in numerical aspects of trials (PoINT): exploring patient and public partners experiences and identifying stakeholder priorities. Trials. 2021;22(1):499.

    Article  PubMed  PubMed Central  Google Scholar 

  8. Murray EJ, Caniglia EC, Swanson SA, Hernández-Díaz S, Hernán MA. Patients and investigators prefer measures of absolute risk in subgroups for pragmatic randomized trials. J Clin Epidemiol. 2018;103:10–21.

    Article  PubMed  PubMed Central  Google Scholar 

  9. Suzie Cro, Brennan Kahan, Akshaykumar Patel, Beatriz Goulão, and the HEALTHY STATS public involvement group. Patient and public involvement tool: how to start a conversation with public partners about estimands. Available at https://www.statsci.co.uk/ppi. Accessed 4th July 2023.

  10. Gamble C, Dudley L, Allam A, Bell P, Buck D, Goodare H, et al. An evidence base to optimise methods for involving patient and public contributors in clinical trials: a mixed-methods study. Health Serv Deliv Res. 2015;3(39).

  11. Goulao B, Poisson C, Gillies K. Patient and public involvement in numerical aspects of trials: a mixed methods theory-informed survey of trialists’ current practices, barriers and facilitators. BMJ Open. 2021;11(3):e046977.

    Article  PubMed  PubMed Central  Google Scholar 

  12. McConnell T, Best P, Davidson G, McEneaney T, Cantrell C, Tully M. Coproduction for feasibility and pilot randomised controlled trials: learning outcomes for community partners, service users and the research team. Res Involv Engagem. 2018;4(1):32.

    Article  PubMed  PubMed Central  Google Scholar 

  13. Crocker JC, Pratt-Boyden K, Hislop J, Rees S, Locock L, Olszowski S, et al. Patient and public involvement (PPI) in UK surgical trials: a survey and focus groups with stakeholders to identify practices, views, and experiences. Trials. 2019;20(1):119.

    Article  PubMed  PubMed Central  Google Scholar 

  14. Minogue V, Cooke M, Donskoy A-L, Vicary P, Wells B. Patient and public involvement in reducing health and care research waste. Res Involv Engagem. 2018;4(1):5.

    Article  PubMed  PubMed Central  Google Scholar 

  15. Price A, Albarqouni L, Kirkpatrick J, Clarke M, Liew SM, Roberts N, et al. Patient and public involvement in the design of clinical trials: an overview of systematic reviews. J Eval Clin Pract. 2018;24(1):240–53.

    Article  PubMed  Google Scholar 

Download references

Acknowledgements

We are thankful to Marie Miller, Imperial Clinical trials Units’ patient and public involvement coordinator who contributed to the set-up the HEALTHY STATS public involvement group. We are also thankful to Imperial College London’s’ Patients Experience and Research Centre (PERC).

Funding

This research was funded by an MRC NIHR TMRP project funding grant and an NIHR advanced research fellowship (Suzie Cro, NIHR300593). The views expressed are those of the authors and not necessarily those of the NIHR or the Department of Health and Social Care. B.C.K. is funded by the UK MRC, grants MC_UU_00004/07 and MC_UU_00004/09.

Author information

Authors and Affiliations

Authors

Contributions

SC conceived of the idea for this study and estimand explainer tool. SC, BK and BG drafted the first version of the estimand explainer tool. SC and AH co-chaired the HEALTHY STATS group; other public partner members of the group include JC, PH, MK and YR. All researchers and public partners attended the online meeting and contributed to the development of the tool and discussions. SC wrote the first draft of this manuscript which was critically reviewed by BK, AP, BG and AH. All authors approved of the final version of this manuscript.

Corresponding author

Correspondence to Suzie Cro.

Ethics declarations

Ethics approval and consent to participate

Not applicable for this research study which consisted of an online meeting of researchers and public partners who are patients or members of the public who are part of the research team (not trial participants).

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Cro, S., Kahan, B.C., Patel, A. et al. Starting a conversation about estimands with public partners involved in clinical trials: a co-developed tool. Trials 24, 443 (2023). https://doi.org/10.1186/s13063-023-07469-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s13063-023-07469-9

Keywords