Adherence to participant flow diagrams in trials on postoperative pain management after total hip and knee arthroplasty: a methodological review
Trials volume 22, Article number: 280 (2021)
The Consolidated Standards of Reporting Trials (CONSORT) statement aims to improve transparent reporting of randomised clinical trials. It comprises a participant flow diagram with the reporting of essential numbers for enrolment, allocation and analyses. We aimed to quantify the use of participant flow diagrams in randomised clinical trials on postoperative pain management after total hip and knee arthroplasty.
We searched PubMed, Embase and CENTRAL up till January 2020. The primary outcome was the proportion of trials with adequate reporting of participant flow diagrams, defined as reporting of number of participants screened for eligibility, randomised and included in the primary analysis. Secondary outcomes were recruitment (randomised:screened) and retention (analysed:randomised) rates, reporting of a statistical strategy, reasons for exclusion from the primary analysis and handling of missing outcome data. Trends over time were assessed with statistical process control.
Of the 570 included trials, we found adequate reporting in 240 (42%). Reporting with participant flow diagram increased significantly over time. Median recruitment was 73% (IQR 44–91%), and retention was 97% (IQR 93–100%). These rates did not change over time. Trials with adequate reporting of participant flow were more likely to report a statistical strategy (41% vs 8%), reasons for post-randomisation exclusions (100% vs 55%) and handling of missing outcome data (14% vs 6%).
Adherence to participant flow diagrams for RCTs has increased significantly over time. Still, there is room for improvement of adequate reporting of flow diagrams, to increase transparency of trials details.
Transparent reporting in randomised clinical trials (RCTs) is essential to trial validity and the subsequent implementation of results in clinical settings. Insufficient trial reporting has led to establishment of the first Consolidated Standards of Reporting Trials (CONSORT) statement in 1996 [1, 2].
The most recent version CONSORT 2010 provides a set of evidence-based standards for transparent and sufficient reporting of randomised trials [3,4,5]. The statement comprises a checklist and a participant flow diagram. A sufficient participant flow diagram includes the number of participants screened, randomised and analysed for the primary outcome and the number of, and reasons for, post-randomisation exclusions in each trial arm.
RCTs should be pragmatic and optimally reflect clinical settings [6,7,8]. The CONSORT participant flow diagram enables the reader to assess discrepancies between eligible and included individuals and reasons for post-randomisation exclusions. In pain management research, adequate flow diagrams are particularly important, because inadequate pain relief or adverse events can cause dropouts. Without an adequate flow diagram, the intervention may be misinterpreted as more beneficial than the true effect. Postoperative pain treatment after total hip and knee arthroplasty (THA and TKA) are among the most frequent elective procedures with high risk of postoperative pain. However, heterogenic trial designs and insufficient reporting has hampered aggregation and interpretation of intervention effects in meta-analyses, which is why methodological improvements are needed [9,10,11].
With this review, we aimed to investigate the use of participant flow diagrams and adequacy in the reporting of participant flow in RCTs on postoperative pain management after THA and TKA.
This methodological review was structured in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) statement, though leaving out meta-analyses and bias evaluation, as these were deemed irrelevant for the aim of this review . The protocol for this and a parallel review  was registered prior to study commencement at International Prospective Register of Systematic Reviews (PROSPERO) (identifier CRD42020151317).
Literature search and eligibility criteria
A systematic search was conducted in PubMed, Embase and CENTRAL. The search strategy is in Additional file 1. The last search was conducted January 7, 2020.
Eligible studies were RCTs investigating perioperative analgesic interventions for postoperative pain management in adults aged 18 years or above undergoing total hip or knee arthroplasty, written in English and regardless of year of publication. Trials comparing multiple interventions as well as trials comparing one or several interventions with a control group were included. Quasi-randomised trials, conference abstracts and observational studies were excluded. We excluded RCTs that included hemi-arthroplasties and arthroplasties due to fractures.
Records up to June 2019 were screened for a previous review (PROSPERO Identifier CRD42019125691). For the updated search, two authors (CP and AK) assessed RCTs for eligibility independently. Four authors (AK, CP, JL and TR) independently extracted data for the first 20 RCTs to secure uniformity. Data for the remaining RCTs were extracted by two independent authors (CP, JL or TR). Discrepancies were solved involving a senior author (AK or OM). Data concerning participant flow were extracted into Excel. As this review investigates reporting standards in RCTs, the authors have not been contacted for missing data.
The primary outcome was to quantify adequacy in reporting of participant flow diagrams in the included RCTs. Reporting was defined as adequate if trials reported the number of participants screened for eligibility, randomised and included in the primary analysis for each group.
Secondary outcomes were as follows: (1) recruitment (randomised:screened) and retention (analysed:randomised) rates, (2) proportion of trials reporting a statistical strategy for the primary analysis (intention-to-treat, per protocol or modified intention-to-treat analyses), (3) reasons for exclusions from the primary analysis and (4) handling of missing outcome data.
Further, we assessed trends over time and performed subgroup analyses for continental, interventional and procedural differences.
Data were described by numbers and percentages or median and interquartile range. Differences between trials with and without patient flow diagrams for choice of primary outcome, statistical strategy and handling of missing data were assessed with chi-square test with p < 0.05 as significance level. All other comparisons were assessed qualitatively.
Trends over time for adequate reporting of participant flow were evaluated with statistical process control. Control charts were used to test for non-random variation , indicated by unusually long runs at one side of the mean or unusually few crossings over the mean. Cutoff values for runs and crossings are calculated based on the number of observations in the chart and correspond to a significance level of 0.05 [15, 16].
In a post hoc multivariate regression, we analysed the effect of (i) publication year, (ii) number of participants (per trial arm), (iii) journal impact factor, (iv) multicentricity and (v) prospective online trial registration on (1) the probability of having an adequately reported flow diagram, (2) recruitment rate and (3) retention rate. We used logistic regression for adequacy in flow diagrams and linear regression for recruitment and retention rates. We used complete-case analyses and considered p values < 0.05 as significant. R i386 version 3.6. with add-on Qicharts2 was used for process control and statistical analyses [17, 18].
Characteristics of the RCTs
Characteristics of the included trials are summarised in Table 1. The majority of trials were published in Europe (40%), Asia (31%) and North America (24%). Of the included trials, 65% concerned postoperative pain management following TKA, and 59% were published between 2011 and 2020. Less than 100 participants were randomised in 75% of trials.
Reporting of participant flow
A participant flow diagram was reported in 258 trials, and 240 (42%) trials reported adequately on the number of participants screened, randomised and analysed. Adequate reporting of participant flow diagrams increased significantly over time (Fig. 2).
In total, the 240 RCTs using adequate participant flow diagrams screened 53,123 individuals, randomised 27,432 participants and analysed 25,740. The median recruitment rate was 73% (IQR 44–91%). The median retention rate was 97% (IQR 93–100%). None of these rates changed significantly over time (Table 2, Fig. 2). Subgroup analyses are listed in Additional file 4.
Reported analyses and reasons for exclusion
Trials with adequate flow diagrams were significantly better at reporting the strategies for statistical analyses and for handling of missing outcome data (Table 3).
Reported reasons for exclusion of randomised participants are listed in Table 3. Trials with a participant flow diagram were more likely to report reasons for exclusion.
Post hoc multivariate logistic regression
We found that increasing year of publication, higher number of participants and higher journal impact factor were associated with reporting of adequate flow diagrams. Higher impact factor and prospective online trial registration were associated with a lower recruitment rate. A higher number of participants were associated with a lower retention rate (Table 4).
Eighty-two values were missing for ‘Journal impact factor’ in a non-random fashion (including 68 trials published before 1999); thus, we refrained from imputing, and analyses were performed only on complete cases. The number of participants is reported per trial arm.
Adequate participant flow diagrams stating the number of participants screened, randomised and analysed were reported in 240 of the 570 included trials (42%). Adequate reporting increased significantly over time and was higher in large trials and high impact journals. Rates for recruitment and retention were stable over time. Reporting of strategy for handling of missing data was low in both trials with and without participant flow diagrams, although significantly better in trials with adequately reported participant flow. Trials with adequate flow diagrams were also significantly better to report the statistical analysis for the primary outcome, used more appropriate methods and sufficiently described reasons for exclusion of participants.
Studies of poor methodological quality are more likely to overestimate benefits . Adherence to CONSORT improves the adequacy of reporting and is associated with higher methodological quality [6, 20]. A recent review of periodontological trials published between 2011 and 2016 found that 60–79% of published trials adhered to the CONSORT checklist and reported a participant flow diagram, which is higher than for our review . Similarly, a review from 2009 investigating the reporting of recruitment and retention in top journals found that 79% of trials had flow diagrams. However, 40% of these would have been deemed insufficient in our review as they failed to report participants assessed for eligibility. We found an association between having an adequate flow diagram and publication in higher impact journals. Further, the 2009 review reported a high rate of eligible persons not wanting to participate and drop-outs for unknown reasons though retention of randomised participants was generally high . In our review, 31% of trials with adequate flow diagrams reported at least one case of withdrawal of consent. Our post hoc analysis demonstrated significant lower retention in larger trials, but the point estimate was close to zero and the results were hardly of clinical relevance. These findings indicate a potential benefit in making trial participation more participant-friendly. A systematic review from 2020 investigating burdens of participating in RCTs found that participation was associated with both psychological, physical and financial distress. The authors concluded that these burdens could be anticipated by conducting minimally disruptive clinical research, which in terms could improve trial recruitment and retention . A third of trials included in our review with flow diagrams had at least one wrongful inclusion or participant that did not receive the intervention. Recruitment, randomisation and treatment errors are common, even for trials published in top journals, and should be reported to increase transparency .
We found a median recruitment rate of 73%, which may be considered relatively high. We were unable to assess whether this was partly due to some trials having pre-excluded participants that were obviously non-includable. Trials in other research areas have been criticised for having clinically unrepresentative populations hampering the clinical relevance and implementation of trials results [7, 25]. Adequate flow diagrams facilitate detection of left-out patient subgroups, which makes it easier to evaluate trial generalisability (external validity) [22, 26, 27]. Broadening eligibility criteria in clinical pragmatic trial designs and avoiding excessive exclusion criteria can make the population more representative [28, 29].
Retention rates were overall sufficiently described in most trials. It is however remarkable that one third of the trials had more than 5% of participants left out of the primary analyses. These numbers weaken reliability of trial results, unless such patients are missing at random which most often is not the case. Therefore, trials should ideally state how missing data were handled . It is disappointing that the majority of the trials did not inform on handling of missing data, although these numbers were significantly improved for trials with participant flow diagrams.
The primary analysis of data in RCTs should preserve patients for analyses according to the assigned group. This is designated the ‘intention-to-treat’ principle. This minimises the influence of withdrawals and patients lost to follow-up and likely prevents type 1 errors . Furthermore, leaving out reporting of the statistical analyses impairs the reader’s quality assessment of trial findings. The intention-to-treat analysis is the optimal way to preserve randomisation and enhance internal validity  (though a cross-sectional study found that the modified intention-to-treat provided an equal methodological quality ). It is therefore somewhat discouraging, that nearly six of 10 trials did not report on handling of missing data in our review. Again, we found that for trials presenting patient flow diagrams, this number was significantly improved compared to trials without. Such improvement possibly does not relate to the patient flow diagram per se but can be seen as an overall improved methodological quality of trials presenting patient flow diagrams.
Strengths and limitations
We did not assess unpublished trials which could potentially skew our analyses, and we left out non-English publications which is a limitation to the continental subgroup analysis. The risk of typos using comprehensive datasets was counteracted by parallel extraction. For pragmatic reasons, some categories of reasons for exclusion of participants were broad and may cover more subgroups. Moreover, the categorisation was prone to a degree of arbitrariness. Limited by the data at hand, we reported the authors’ declaration of statistical strategy without evaluating whether the strategy was maintained. We did not investigate alternatives to CONSORT. Some trials used patient flow diagrams, without specifically stating the use of a CONSORT diagram, which is why we chose to include all trials with patient flow diagrams. We did not go into details on participant characteristics, generalisability and pre-randomisation exclusion criteria as these data were investigated in a parallel review . Obviously, we were unable to assess recruitment and retention in trials without flow diagrams, which may be different from trials with adequate flow diagrams. Where other reviews on CONSORT adherence have included trials from leading journals or a specific annual range [22, 24, 34], we included all trials investigating postoperative pain management following THA or TKA regardless of journal or publication year to make the review generalisable to the total research base. Nevertheless, we did a post hoc regression analysis to address associations with important variables.
Though improvements have been made, the general adherence to reporting guidelines is suboptimal . Continuous endorsement of the CONSORT Statement by editors and mandatory guideline checklists for journal submission could be ways to improve this [5, 36]. An expansion of the CONSORT flow diagram could facilitate higher external and internal validity, by adding more details concerning the process of inclusion and clarify reasons for exclusion following randomisation, while keeping the diagram manageable [22, 26].
Adherence to participant flow diagrams was 42% among published RCTs investigating postoperative pain management after THA and TKA with a significant increase over time. Recruitment and retention rates are high. Still, there is room for improvement in quantitative and qualitatively reporting of flow diagrams, to increase transparency of study details and facilitate better and easier quality assessment of trial results.
Availability of data and materials
The datasets are available from the corresponding author on reasonable request.
Begg C, Cho M, Eastwood S, Horton R, Moher D, Olkin I, et al. Improving the quality of reporting of randomized controlled trials. The CONSORT statement. Jama. 1996;276(8):637–9. https://doi.org/10.1001/jama.276.8.637.
Schulz KF, Altman DG, Moher D, Group C. CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. Int J Surg. 2011;9(8):672–7. https://doi.org/10.1016/j.ijsu.2011.09.004.
Moher D, Hopewell S, Schulz KF, Montori V, Gotzsche PC, Devereaux PJ, et al. CONSORT 2010 Explanation and Elaboration: updated guidelines for reporting parallel group randomised trials. BMJ. 2010;340(mar23 1):c869. https://doi.org/10.1136/bmj.c869.
Moher D, Jones A, Lepage L, Group ftC. Use of the CONSORT statement and quality of reports of randomized trials: a comparative before-and-after evaluation. JAMA. 2001;285(15):1992–5. https://doi.org/10.1001/jama.285.15.1992.
Turner L, Shamseer L, Altman DG, Schulz KF, Moher D. Does use of the CONSORT Statement impact the completeness of reporting of randomised controlled trials published in medical journals? A Cochrane review. Syst Rev. 2012;1(1):60. https://doi.org/10.1186/2046-4053-1-60.
Plint AC, Moher D, Morrison A, Schulz K, Altman DG, Hill C, et al. Does the CONSORT checklist improve the quality of reports of randomised controlled trials? A systematic review. Med J Aust. 2006;185(5):263–7. https://doi.org/10.5694/j.1326-5377.2006.tb00557.x.
Rothwell PM. External validity of randomised controlled trials: “to whom do the results of this trial apply?”. Lancet. 2005;365(9453):82–93. https://doi.org/10.1016/s0140-6736(04)17670-8.
Gabriel SE, Normand SL. Getting the methods right--the foundation of patient-centered outcomes research. N Engl J Med. 2012;367(9):787–90. https://doi.org/10.1056/NEJMp1207437.
Karlsen AP, Wetterslev M, Hansen SE, Hansen MS, Mathiesen O, Dahl JB. Postoperative pain treatment after total knee arthroplasty: a systematic review. PLoS One. 2017;12(3):e0173107. https://doi.org/10.1371/journal.pone.0173107.
Karlsen APH, Geisler A, Petersen PL, Mathiesen O, Dahl JB. Postoperative pain treatment after total hip arthroplasty: a systematic review. Pain. 2015;156(1):8–30. https://doi.org/10.1016/j.pain.0000000000000003.
Rennie D, Flanagin A. Research on peer review and biomedical publication: furthering the quest to improve the quality of reporting. Jama. 2014;311(10):1019–20. https://doi.org/10.1001/jama.2014.1362.
Moher D, Liberati A, Tetzlaff J, Altman DG, Group P. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. J Clin Epidemiol. 2009;62(10):1006–12. https://doi.org/10.1016/j.jclinepi.2009.06.005.
Laigaard J, Pedersen C, Ronsbo TN, Mathiesen O, Karlsen APH. Minimal clinically important differences in randomised clinical trials on pain management after total hip and knee arthroplasty: a systematic review. Br J Anaesth. 2021. https://doi.org/10.1016/j.bja.2021.01.021.
Anhoj J, Olesen AV. Run charts revisited: a simulation study of run chart rules for detection of non-random variation in health care processes. PLoS One. 2014;9(11):e113825. https://doi.org/10.1371/journal.pone.0113825.
Schilling MF. The surprising predictability of long runs. Mathematical Assoc Am. 2012;85(2). https://doi.org/10.4169/amer.math.monthly.118.05.404.
Chen Z. A note on the runs test. Model Assist Stat Appl. 2010;5(2):73–7. https://doi.org/10.3233/mas-2010-0142.
Jacob Anhoej. Qicharts2: quality improvement charts. R package version 0.2.1. 2017. URL https://CRAN.R-project.org/package=qicharts2. Accessed 05 May 2020.
R Core Team. R: a language and environment for statistical computing. Foundation for Statistical computing, Vienna, Austria. 2017. URL https://www.R-project.org/. Accessed 05 May 2020.
Moher D, Pham B, Jones A, et al. Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses? Lancet. 1998;352(9128):609–13. https://doi.org/10.1016/s0140-6736(98)01085-x.
Agha R, Cooper D, Muir G. The reporting quality of randomised controlled trials in surgery: a systematic review. Int J Surg. 2007;5(6):413–22. https://doi.org/10.1016/j.ijsu.2007.06.002.
Siddiq H, Pentapati KC, Acharya S. Adherence of randomized controlled trials to consolidated standards of reporting trials 2010 guidelines: a survey of randomized controlled trials published in 2011-2016 in 3 periodontology journals. J Evid Based Dent Pract. 2019;19(3):260–72. https://doi.org/10.1016/j.jebdp.2019.04.001.
Toerien M, Brookes ST, Metcalfe C, de Salis I, Tomlin Z, Peters TJ, et al. A review of reporting of participant recruitment and retention in RCTs in six major journals. Trials. 2009;10(1):52. https://doi.org/10.1186/1745-6215-10-52.
Naidoo N, Nguyen VT, Ravaud P, Young B, Amiel P, Schanté D, et al. The research burden of randomized controlled trial participation: a systematic thematic synthesis of qualitative evidence. BMC Med. 2020;18(1):6. https://doi.org/10.1186/s12916-019-1476-5.
Yelland LN, Kahan BC, Dent E, Lee KJ, Voysey M, Forbes AB, et al. Prevalence and reporting of recruitment, randomisation and treatment errors in clinical trials: a systematic review. Clin Trials. 2018;15(3):278–85. https://doi.org/10.1177/1740774518761627.
Greenhalgh T, Howick J, Maskrey N. Evidence based medicine: a movement in crisis? Bmj. 2014;348(jun13 4):g3725. https://doi.org/10.1136/bmj.g3725.
Glasgow RE, Huebschmann AG, Brownson RC. Expanding the CONSORT figure: increasing transparency in reporting on external validity. Am J Prev Med. 2018;55(3):422–30. https://doi.org/10.1016/j.amepre.2018.04.044.
Shapiro SH, Weijer C, Freedman B. Reporting the study populations of clinical trials. Clear transmission or static on the line? J Clin Epidemiol. 2000;53(10):973–9. https://doi.org/10.1016/s0895-4356(00)00227-4.
Kim ES, Bruinooge SS, Roberts S, Ison G, Lin NU, Gore L, et al. Broadening eligibility criteria to make clinical trials more representative: American Society of Clinical Oncology and Friends of Cancer Research Joint Research Statement. J Clin Oncol. 2017;35(33):3737–44. https://doi.org/10.1200/jco.2017.73.7916.
Pedersen C, Troensegaard H, Laigaard J, Koyuncu S, Schrøder HM, Overgaard S, et al. Differences in patient characteristics and external validity of randomized clinical trials on pain management following total hip and knee arthroplasty: a systematic review. Reg Anesth Pain Med. 2020;45(9):709–15. https://doi.org/10.1136/rapm-2020-101459.
Jakobsen JC, Gluud C, Wetterslev J, Winkel P. When and how should multiple imputation be used for handling missing data in randomised clinical trials - a practical guide with flowcharts. BMC Med Res Methodol. 2017;17(1):162. https://doi.org/10.1186/s12874-017-0442-1.
Fergusson D, Aaron SD, Guyatt G, Hébert P. Post-randomisation exclusions: the intention to treat principle and excluding patients from analysis. Bmj. 2002;325(7365):652–4. https://doi.org/10.1136/bmj.325.7365.652.
Polit DF, Gillespie BM. Intention-to-treat in randomized controlled trials: recommendations for a total trial strategy. Res Nurs Health. 2010;33(4):355–68. https://doi.org/10.1002/nur.20386.
Montedori A, Bonacini MI, Casazza G, Luchetta ML, Duca P, Cozzolino F, et al. Modified versus standard intention-to-treat reporting: are there differences in methodological quality, sponsorship, and findings in randomized trials? A cross-sectional study. Trials. 2011;12(1):58. https://doi.org/10.1186/1745-6215-12-58.
Shaikh AM, Mehta MM, Chawan VV. Evaluation of reporting of CONSORT flow diagrams in randomized controlled trials in a national and international pharmacology journal. Perspect Clin Res. 2019;10(4):168–71. https://doi.org/10.4103/picr.PICR_73_18.
Jin Y, Sanger N, Shams I, Luo C, Shahid H, Li G, et al. Does the medical literature remain inadequately described despite having reporting guidelines for 21 years? - A systematic review of reviews: an update. J Multidiscip Healthc. 2018;11:495–510. https://doi.org/10.2147/jmdh.S155103.
Peters JP, Hooft L, Grolman W, Stegeman I. Assessment of the quality of reporting of randomised controlled trials in otorhinolaryngologic literature - adherence to the CONSORT statement. PLoS One. 2015;10(3):e0122328. https://doi.org/10.1371/journal.pone.0122328.
No grants were received. None of the authors had conflicts of interest. All authors met the ICMJE guidelines.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Rønsbo, T.N., Laigaard, J., Pedersen, C. et al. Adherence to participant flow diagrams in trials on postoperative pain management after total hip and knee arthroplasty: a methodological review. Trials 22, 280 (2021). https://doi.org/10.1186/s13063-021-05233-5