Study design
We conducted a systematic citation analysis of the ProFaNE COS and correlational study to identify factors associated with reporting of the ProFaNE COS in a purposeful sample of RCTs examining the safety or effectiveness of interventions to prevent and/or manage falls in older people.
Theoretical underpinning
Our investigation into factors associated with reporting of ProFaNE COS domains was informed by the Consolidated Framework for Implementation Research (CFIR) [36]. Some associations were hypothesis-driven; however, as little is known about factors influencing reporting of COS in RCTs, others were driven by exploration, but based on the assumption that characteristics of the inner setting, outer setting, intervention, and individuals involved are possible determinants of knowledge use [36].
Outcomes
Our primary outcome was the percentage of ProFaNE COS domains reported. Secondary outcomes included whether falls were reported (yes, no), injuries were reported (yes, no), psychological consequences of falling were reported (yes, no), HRQoL was reported (yes, no), and physical activity was reported (yes, no).
Explanatory variables and study hypotheses
We examined associations between author-level, study-level, and manuscript-level factors and primary and secondary outcomes.
Author-level variables
We included a single author-level variable—whether the geographic affiliation of the corresponding author was European vs. other. The ProFaNE COS was developed on behalf of a European network [29]. The CFIR states that the perception of stakeholders about whether the intervention or evidence-informed practice (here, the COS) is internally or externally developed can influence implementation success [36]. It is possible investigators within Europe may be more likely to perceive the ProFaNE COS was internally developed than investigators outside Europe. We hypothesized that articles authored by a corresponding author affiliated with one or more European institutions would report a greater percentage of ProFaNE COS domains than articles authored by a corresponding author based outside of Europe.
Study-level variables
Study-level variables included the setting (community vs. institution or combined), number of trial centres (single, multiple, not reported), number of trial arms (two vs. three or more), funding source(s) (at least one source of non-industry vs. no non-industry source(s) or not reported/unclear), intervention type (exercise, multi-component/factorial, other), mean age of the sample at baseline, percentage of the sample that was female at baseline, whether the population of interest had a specific disease diagnosis (yes, no), fall risk of participants at baseline (at risk, at high risk), sample size, and length of follow-up of the primary outcome variable (12 months or more vs. less than 12 months).
The CFIR states that the compatibility of the intervention or evidence-informed practice (the COS) with the local context, or inner setting, may influence the success of implementation [36]. Lamb et al. specified the selection of outcomes in the ProFaNE COS should focus on community-dwelling populations [29]. We hypothesized that articles reporting the results of RCTs conducted in community settings would report a greater percentage of domains than articles reporting the results of RCTs set in institutions, such as hospitals, long-term care or nursing homes, and assisted-living facilities.
Outcomes included in the ProFaNE COS are generally measured by self-report and were selected to accommodate “nearly all community-dwelling older people” [29]. Copsey et al. reported that some RCTs citing the ProFaNE COS restricted their population of interest to people with dementia or cognitive impairment, Parkinson’s disease, or stroke [30]. The measurement and validity of some or all of these outcomes in persons living with specific disease diagnoses, especially cognitive impairment or dementia [37], may require additional considerations when compared to the general population of community-dwelling older adults. Previously, in an international survey of trialists on their reasons for non-adoption of the hip fracture COS, more than half (54%) of respondents stated the COS was not appropriate for trials on people living with cognitive impairment and needed revision to accommodate this population [33]. We hypothesized that articles reporting the results of RCTs in populations with specific disease diagnoses, including but not limited to cognitive impairment or dementia, would report a smaller percentage of ProFaNE COS domains than articles reporting the results of RCTs in a general population of older adults.
Manuscript-level variables
Manuscript-level variables included the year of publication, the type of journal (specialist vs. general), and whether the manuscript cited the ProFaNE COS in the introduction (yes, no), methods (yes, no), and discussion (yes, no) sections. We hypothesized that articles citing the ProFaNE COS in the methods section would report a greater percentage of domains than articles that did not cite the ProFaNE COS in the methods section.
Population of interest and eligibility criteria
Our aim was to investigate factors governing the extent to which the ProFaNE COS was implemented with fidelity as intended by developers. Consistent with this aim, we made several decisions to restrict our population of interest. First, we only included articles citing the ProFaNE COS, as we interpreted citation as evidence of adoption (i.e. uptake). To be implemented with fidelity requires that an innovation or evidence-based practice must first be adopted by an individual, organization, or setting [35]. Second, we restricted our sample to completed RCTs, defined as prospective studies that assessed healthcare interventions in human participants who were randomly allocated to study groups [30]. As stated by developers, the ProFaNE COS was “intended to promote consistency in collection and reporting of essential elements (page 1619)” in future trials and meta-analyses [29]. We excluded non-completed RCTs, such as protocols, due to the prevalence of outcome modification after trial initiation and selective outcome reporting (e.g. [38]), making it impossible to explore implementation fidelity based on reporting of pre-specified outcomes. We also excluded early developmental trials, such as feasibility and pilot studies, where the focus was to examine whether the trial should be done, and if so, how. Third, we restricted our sample to RCTs that sampled older people, defined as whether the study excluded participants younger than 60 years of age, the mean age of included participants was 60 years of age or above, or the patient population was described as ‘older’, ‘elderly’, or ‘senior’ [30]. Fourth, we restricted our sample to RCTs examining the safety, effectiveness, or cost-effectiveness of interventions to prevent and/or manage falls. Fifth, we excluded articles if they were published in a language other than English. This was a practical consideration as time and funding were limited. Last, when more than one article was published on the results of a single RCT, we included all articles if they reported different outcomes at the same follow-up point analysed after trial completion, included the first article (determined by the date of publication) only when they reported the same outcomes at the same follow-up points analysed after trial completion, and included all articles if they reported the same or different outcomes at different follow-up points analysed after trial completion. This provided insight into temporal changes in barriers and enablers to the reporting of ProFaNE COS domains over the life course of a given trial.
Data sources
To identify eligible articles, we conducted a systematic citation analysis of the ProFaNE COS between 01 October 2005 and 12 July 2021. However, our search strategy differed for articles published prior to and after 16 January 2015. As Copsey et al. previously conducted a systematic citation analysis for all articles citing the ProFaNE COS between 01 October 2005 and 16 January 2015, we screened their list of included RCTs to identify articles published prior to 17 January 2015 [30]. As our eligibility criteria only differed from Copsey et al.’s with respect to the inclusion of multiple articles reporting data from the same RCT, we also screened Copsey et al.’s [30] list of RCTs excluded because they were secondary reports of already included RCTs. To identify articles citing the ProFaNE COS after 17 January 2015 and before 12 July 2021, we searched the Web of Science Citation Index, PubMed, and Scopus databases.
Article selection
To screen articles for eligibility, we adopted a two-step approach involving the title and abstract screening followed by full-text screening. To facilitate high-quality screening, reviewers were provided with protocols for screening. All difficulties, and their associated resolutions, were logged on screening tracking forms.
Our screening process differed for articles published prior to and after 17 January 2015. A single reviewer (AMBK) independently screened the titles and abstracts and subsequently the full text of articles published prior to 17 January 2015 for eligibility. We limited screening to a single reviewer as all articles published prior to 17 January 2015 had previously undergone screening by Copsey et al. [30]. Comparatively, at least two reviewers independently screened the titles and abstracts and subsequently the full text of articles published after 17 January 2015 for eligibility, as per the following. First, after removing duplicates, at least two reviewers independently screened the title and abstract of each article to determine the study type. Then, at least two reviewers independently screened the full text of the subset of articles with study types coded as RCT and unsure. When necessary, a third reviewer screened articles with conflicts between first and second screenings to resolve conflicts. If the third reviewer disagreed with both the first and second reviewer, two reviewers (AMBK and KMS) met in person to discuss and resolve conflicts.
Data abstraction
At least two reviewers independently abstracted data on whether each domain of the ProFaNE COS was reported, as well as author-level, study-level, and manuscript-level data from all articles meeting our eligibility criteria. To do this, a single reviewer (DS) from our research team first independently abstracted data from articles included in the analysis by Copsey et al., which also fulfilled our eligibility criteria [30]. Upon request, Copsey et al. shared the data they abstracted from these articles, which constituted the second abstraction, where possible [30]. For variables not abstracted by Copsey et al. [30], a second reviewer from our research team (AMBK) abstracted data from this subset of articles. Then, four reviewers, including one member of the research team (DS) and three research assistants, independently abstracted data from the remaining articles included in our sample. Reviewers were provided with a protocol for data abstraction and were instructed to log all difficulties and their associated resolutions on data abstraction tracking forms. A third, independent reviewer (AMBK or KMS) identified and resolved conflicts between the first and second abstractions.
Statistical analysis
Statistical significance in the final models was defined as P < 0.05. For all analyses, SAS (9.4) software, Cary, NC, US, was used.
Primary outcome: percentage of ProFaNE COS domains reported
We used the penalized regression method to select important predictors. This method allowed us to create a linear regression model that is penalized, for having too many variables in the model, by adding a constraint in the Eq. [39]. We used the LASSO (Least Absolute Shrinkage and Selection Operator) method [40] using the GLMSELECT procedure in SAS [41, 42], whereby a sequence of models is obtained using the LASSO algorithm, and in this sequence, the one yielding the smallest value of the Mallow’s C(p) statistic [43] is chosen as the final model. We produced diagnostic plots to select the most parsimonious model (Additional file 1). We used the split option for categorical variables to include all levels as dummy coded variables in the model. In the LASSO selection procedure, it is not possible to produce P values for the selected variables. In lieu, we ran a generalized linear model with these selected variables to estimate the mean difference and its 95% confidence intervals (95% CI) and P values to describe the variables that have significant relationships with the outcome, as that was our main interest.
Secondary outcomes: reporting of each domain of the ProFaNE COS
We used statistical models for classification to select important variables that are useful for predicting the outcome variable. We used cost-complexity pruning [44] and estimated the misclassification rate by tenfold cross-validation to prevent overfitting. For all models, the fitted model classified the response variable well (Additional file 2). The model is estimated using the HPSPLIT procedure in SAS [42]. In addition, we ran a generalized linear model with those selected variables to estimate the odds ratio (OR) and its 95% CI and P values to describe the variables that have significant relationships with each outcome, as that was our main interest.
Missing data
We were missing data for three explanatory variables and no outcome variables. We could not determine the number of trial centres in n = 13 (15.3%) articles, funding type in n = 4 (4.7%) articles, and length of follow-up for the primary outcome variable in n = 1 (1.2%) article. We used different approaches to handle missingness in our statistical analyses. First, we list-wise deleted the single article containing missing data for the length of follow-up of the primary outcome variable. Then, to treat missing data on the number of trial centres and funding type, we created missing data response options (“Not Reported”) for these variables. We entered these response options in statistical models either as their own category (number of trial centres) when frequencies permitted, or as part of an ‘Else’ category where missing cases were grouped with other response options (funding type). Thus, our final analytical sample comprised n = 84 articles.