Trial Forge Guidance 4: a guideline for reporting the results of randomised Studies Within A Trial (SWATs)

Background Evidence to support decisions on trial processes is minimal. One way to generate this evidence is to use a Study Within A Trial (SWAT) to test trial processes or explore methodological uncertainties. SWAT evidence relies on replication to ensure sufficient power and broad applicability of findings. Prompt reporting is therefore essential; however, SWAT publications are often the first to be abandoned in the face of other time pressures. Reporting guidance for embedded methodology trials does exist but is not widely used. We sought therefore to build on these guidelines to develop a straightforward, concise reporting standard, which remains adherent to the CONSORT guideline. Methods An iterative process was used to develop the guideline. This included initial meetings with key stakeholders, development of an initial guideline, pilot testing of draft guidelines, further iteration and pilot testing, and finalisation of the guideline. Results We developed a reporting guideline applicable to randomised SWATs, including replications of previous evaluations. The guideline follows the Consolidated Standards for Reporting Trials (CONSORT) statement and provides example text to ensure ease and clarity of reporting across all domains. Conclusions The SWAT reporting guideline will aid authors, reviewers, and journal editors to produce and review clear, structured reports of randomised SWATs, whilst also adhering to the CONSORT guideline. Trial registration EQUATOR Network – Guidelines Under Development (https://www.equator-network.org/library/reporting-guidelines-under-development/reporting-guidelines-under-development-for-clinical-trials/#SWAT). Registered on 25 March 2021.


Background
There is a significant amount of avoidable waste in producing and reporting evidence from randomised trials [1].Some of this waste stems from uncertainty about how best to undertake specific trial processes: recruitment and retention of trial participants, for instance, are essential to nearly all trials but remain a persistent challenge [2,3].Despite this, the evidence available to support trialists' decisions about recruitment and retention is minimal [4,5].Evidence on how best to undertake other trial processes will likely be even worse.
One way to generate trial process evidence is to embed a Study Within A Trial (SWAT) within a host trial to test trial process alternatives (e.g.different trial retention or data collection strategies) or explore why processes are undertaken as they are (e.g.exploration of reasons for non-consent) [6,7].SWATs may be randomised or nonrandomised depending on the question being asked and may be completed in a single-host trial or across multiple-host trials.Evaluations in multiple host trials can either be done at the same time or individually over an extended period.A randomised evaluation of a research process may also be embedded within other research designs, e.g.within a prospective cohort (Trial Within A Cohort TWIC).
Most SWATs to date have focused on recruitment and retention strategies.The number of such SWATs is increasing, with 45 recruitment studies identified in a 2010 systematic review and 68 in the 2018 update of that review [4].For retention, there were 38 studies identified in 2014 and 72 by 2020 [5].There have been fewer SWATs in other trial process areas and so further advances would be welcome.
A central driver for the increase in SWAT activity, especially in recruitment and retention, is the promotion of SWATs through funded programmes such as MRC-Start [8], initiatives such as Trial Forge [9] and the Health Research Board-Trials Methodology Research Network (Ireland), and the availability of dedicated SWAT funding from funders such as the UK National Institute for Health and Care Research (NIHR) [10], the Health Research Board in Ireland [11] and Accelerating Clinical Trials (Canada) [12].The PROMoting the USE of SWATs (PROMETHEUS) research programme, a programme of coordinated recruitment and retention SWATs, has added further to this by overseeing 42 SWATs in 31 trials [13].
The need for prompt and transparent reporting of research findings is well known.SWAT evidence depends on replication to ensure sufficient participants are involved and to support broad applicability by including contextual variation across a wide range of trials with different clinical populations.For those replications to improve trial process decisions, SWATs need to be published and reported.However, discussions with SWAT researchers suggest that SWATs are often one of the first publications to be abandoned in the face of time pressures.More empirical evidence from the Cochrane reviews on recruitment and retention [4,5] shows that even basic information for risk of bias assessment is poorly reported in 48% of the included SWATs (i.e.risk of bias was assessed as unclear).
Reporting guidance for the reporting of embedded recruitment trials does exist [14] but is not widely used, perhaps because it seems too demanding for what is often a small study nested within a large trial.As part of the PROMETHEUS Programme, we sought to build on these guidelines to develop a more straightforward standard, which still adheres to the CONSORT guideline [15] but is more focused on consistent, concise, and rapid reporting of SWATs.Like the original guidance, our guideline is tailored to reporting randomised SWATs.

Scope of the guideline
Given that clinical trial evidence informs healthcare decision-making, it follows that evidence from SWATs has the potential to improve decision-making in trial processes.However, to realise this potential, we need to remove the barriers to effective reporting of SWATs.The use of a SWAT reporting guideline can help us to achieve this goal.
This SWAT reporting guideline was developed to aid authors in producing clear, structured reports of randomised SWATs conducted in host trials done both separately and simultaneously.Moreover, this guideline also provides a useful tool for reviewers and journal editors.

SWAT reporting guideline rationale
Development of the guideline was initiated because of several common problems identified through the PRO-METHEUS programme [13,16,17].Discussion with members of the Trial Forge SWAT Network also identified more straightforward publication of SWATs as an important, medium-term priority [18].Common problems reported by SWAT researchers concerning the publication of SWATs included: • A lack of time to write a SWAT publication.This concern stemmed from researchers assuming a SWAT publication needs to be a lengthy document like that for the host trial(s) in which the SWAT was embedded.• The SWAT publication is not considered a priority compared to the main host trial publication(s).• A lack of confidence and knowledge about how to generate and submit a SWAT publication.
• A lack of SWAT-focused journals and/or a reluctance from other non-methodological journals to publish such work.• A lack of funding to support SWAT publications in peer-reviewed open-access journals.• Reviewer feedback that reflects a misunderstanding of SWAT methodology.

Development of the SWAT reporting guideline
The PROMETHEUS programme faced challenges implementing the earlier guidance [14], which led its Programme Management group to propose a new reporting format in 2019.The goal was to make publishing SWATs easier by developing a concise reporting guideline of 1000 words or less.This new format would be simpler to write, and potentially more cost-effective, as shorter articles often have lower open-access publication charges.
A further meeting was convened in July 2019 to discuss this proposal more widely with PROMETHEUS Programme Management team members, authors of previous guidelines for reporting embedded trials [14], and a representative from the BMC journal Trials.Meeting participants were provided with example publications (one was in development for peer-reviewed submission [19], and the other was reworked from a previously published SWAT [20]), written in under 1000 words for review and consideration.It was agreed by consensus that the methodological information included was sufficiently robust for reporting the SWATs (i.e. in line with CONSORT) and would enable inclusion of the results into an aggregate meta-analysis.
Following this, a further meeting was convened with the authors of previous guidelines for reporting embedded trials [14] to discuss the proposed guideline.The consensus was that the proposed guideline should be developed to build on knowledge derived from the PRO-METHEUS programme.Suggested additional revisions included the inclusion of the term 'SWAT' as opposed to 'embedded trial' and ensuring that any developed guideline remained CONSORT compliant [15].
The PROMETHEUS Programme Management team developed a draft guideline for concise SWAT reporting, which was then reviewed and refined by the wider team.At this stage, the team conceded that a word count of 1000 words was too ambitious and arbitrary, making it challenging to include sufficient details of the host trial(s) and report on complex interventions and designs.Therefore, we dropped the word limit to allow for more comprehensive reporting, if needed.The need for an initial meta-analysis if the reported SWAT was the second replication or updated meta-analysis (for replications after that) was also added to ensure that the accumulated effect of the intervention was reported.
The guideline was then circulated to a wider stakeholder group for comment.This group included five national and one international trial methodologist, affiliated with academic institutions (n = 5), and one methodologist working for a commercial contract research organisation.The guideline was also reviewed by a patient and public involvement (PPI) contributor.The trial methodologist stakeholder group suggested that the best way to assist researchers in writing and publishing their SWAT would be to provide a reporting template that included exemplary wording for each of the guideline's sections.The PPI member recommended that technical language throughout be simplified.The guideline was updated accordingly using a CONSORTstyle tabulation, which included exemplary wording, with attempts made to simplify language where possible.
Revisions were also made to the exemplar text for randomisation and allocation concealment after it was identified in an updated Cochrane review of strategies for improving retention to RCTs that many SWATs had moderate or low-grade certainty evidence due to poor reporting of these items [5].The Cochrane review found that out of 68 studies, 42 (62%) inadequately reported allocation concealment and 28 (41%) inadequately reported sequence generation [5].Minor changes to the guideline also included encouraging the use of standard keywords in SWAT reporting, which can help users and systematic reviewers find relevant SWATs through electronic searches.

Pilot testing of the SWAT reporting guideline
Throughout the review and development process, we continued to assess the iterations of the guideline by asking colleagues at the York Trials Unit, University of York, and PROMETHEUS Programme team members and collaborators to use the most current version of the guideline when writing up a SWAT for publication [21][22][23][24][25].The Research Methods in Medicine and Health Sciences journal also provided a version of the guideline to support their SWAT special issue in September 2022 [26].
The final draft guidelines were then tested in two further SWAT publications (one recruitment SWAT, one retention SWAT) to identify any necessary further edits.Some minor clarifications were made to the exemplar text and instances of duplication removed to streamline the guideline.References to PROGRESS-PLUS criteria were also added to ensure sufficient reporting of equality, diversity, and inclusion aspects [27].
For transparency, the development of this reporting guideline was registered with the EQUATOR network on 25 March 2021.

Table 1 The SWAT reporting guideline
CONSORT 2010 item to be included in publication [15] Additional information and example text shown in italics where possible

Title and Abstract 1a
The term 'SWAT' should be used in the title The to any demographics deemed relevant by the host trial team, however, we encourage authors to refer to and report in accordance with PROGRESS-PLUS [27] where feasible

Numbers analysed 16
For each group of the SWAT, the number of participants (denominator) included in each analysis and whether the analysis was by originally assigned groups

Outcomes and estimation 17a
For each primary and secondary outcome, results for each group, and the estimated effect size and its precision (such as 95% confidence interval) Results should be presented in tables as far as possible rather than only being presented in the body of the text.To facilitate meta-analysis, SWATs should report the actual number of participants in each group in the SWAT evaluation A key element of SWAT evidence is their ability to be replicated.An important principle for reporting research is that new findings should be placed in the context of existing, relevant evidence.Therefore, we recommend, where possible, that an updated meta-analysis be included that presents the results of the current SWAT combined with previous evaluations of the SWAT intervention.Presentation as a cumulative meta-analysis is particularly helpful because it would help to inform judgements about the need for further evaluations of a SWAT intervention [7] 17b For binary outcomes, the presentation of both absolute and relative effect sizes is recommended

Data sharing
We suggest authors make the data used to generate their results available as a supplementary file or through data-sharing platforms such as OSF (https:// osf.io)

SWAT reporting guideline
The final SWAT reporting guideline is given in Table 1 and applies to all reports of randomised SWAT evaluations, including replications of previous evaluations.
For replication SWATs, it is recommended to include a cumulative meta-analysis of all replications to date in the publication, if feasible.For coordinated simultaneous SWATs (e.g.conducted across multiple host trials at the time), the report should summarise all included host trials and combine the results in a cumulative meta-analysis.
The guideline shown in Table 1 is composed of 40 individual components.The vast majority of the components (n = 35, 87.5%) correspond to items in the CONSORT checklist of 2010 [15] and were selected by Madurasinghe et al. for inclusion in their guidance for reporting embedded recruitment studies [14].Each of these 35 items has been reviewed and guidance and suggested text provided to accurately reflect the conduct of, and guide researchers in the reporting of specific nuances relevant to, SWAT design, delivery, and reporting.Of the remaining five items, four were new items: Keywords-Item 1c; Presentation of binary outcomes-Item 17b; Costs of the SWAT-Item 17c; and Implications for practice and trials research-Item 22, and one item was a modification of an existing CON-SORT 2010 checklist item (Discussion) which was amended to reorder the section structure to improve reporting flow.

Discussion
Our guideline draws on previous work by Madurasinghe et al. [14], adheres to the CONSORT 2010 guideline [15], and has been registered with the EQUATOR network.Throughout the development process, various stakeholders have been consulted, leading to iterative refinement of the guideline.
SWAT publications can be short and do not need to repeat information provided elsewhere (e.g. in the SWAT protocol on the SWAT repository at http:// www.qub.ac.uk/ sites/ TheNo rther nIrel andNe twork forTr ialsM ethod ology Resea rch/ SWATS WARIn forma tion/ Repos itori es/ SWATS tore/).This guideline ought to make them an easy write and an easy read.
It is important to note that the guidance is currently designed for randomised studies embedded within a trial.Whilst this does not therefore cover the reporting of non-randomised SWATs, or randomised studies within cohorts for example, we anticipate these guidelines could easily be applied to these SWATs, albeit with some minor adaption, for example, non-reporting of intervention details and randomisation method.This corresponds with the approach Madurasinghe et.al. took with their earlier guideline [14].
SWATs play a key role in improving the evidence base for trial process decision-making, but they can only do so if their results are made publicly available promptly.If SWATs are published, ideally with an updated cumulative meta-analysis, this will provide more complete evidence on the effectiveness of alternative trial processes and will help trialists make better decisions.

Conclusion
SWATs play a key role in improving the evidence base for trial process decision-making, but they can only do so if their results are made publicly available promptly.To ensure this, we need to remove the barriers to effective reporting of SWATs.The SWAT reporting guideline will aid authors, reviewers, and journal editors to produce and review clear, structured reports of randomised SWATs, whilst also adhering to the CON-SORT guideline [15].

Table 1 (continued) CONSORT 2010 item to be included in publication [15] Additional information and example text shown in italics where possible
[27]rence the host trial's registration number(s) and if the protocol(s) for the host trial is/are available elsewhere or include a link to the study project page(s) Provide a brief description of the host trial(s) using PICO format.At a minimum, age, gender, and ethnicity should be reported per group in addition to any demographics deemed relevant by the host trial team(s); however, we encourage authors to refer to and report in accordance with PROGRESS-PLUS[27]where feasible.If the SWAT was conducted across multiple host trials at the same time, a description of each host trial should be provided

Table 1 (continued) CONSORT 2010 item to be included in publication [15] Additional information and example text shown in italics where possible
the host trial(s) for each SWAT evaluation is likely to be different and contextual information about the host trial(s) should be providedIn addition to general information about the host trial(s) (see 'Methods'), we suggest a table of participant baseline characteristics for those allocated to each group of the SWAT evaluation if these details are available.At a minimum, age, gender, and ethnicity should be reported per group in addition

Table 1 (continued) CONSORT 2010 item to be included in publication [15] Additional information and example text shown in italics where possible
The total cost of the SWAT was [insert cost], which equates to [insert cost] per participantTabulate the additional costs to the trial incurred because of the SWAT, including total cost and cost per participant.This may include direct costs (e.g.printing, postage, animation) and indirect costs (e.g.staff time to prepare mailings).As SWAT evaluations generally need replication, it is useful for trialists to see the costs of both using the SWAT intervention and the cost of evaluating the SWAT should they wish to replicate the evaluation If a positive effect (irrespective of statistical significance) was identified, provide a cost per additional participant for whom there is a favourable result (e.g.cost per participant retained).Otherwise, note that cost per participant was not derived