Domain | More pragmatic: examples from training | More explanatory: examples from training | Study criteria noted by raters that training did not specifically address | Rating commentary |
---|---|---|---|---|
Eligibility: who is selected to participate in the trial? | Participants essentially identical to those who would receive the intervention if it was in usual care: both health care systems and patients | Lots of exclusions (for example, those who do not comply, respond to treatment, or are not at high risk for primary outcome, are children or elderly), or use many selection tests not used in usual care. | Cognitive impairment; clinic size; clinic willingness to participate; patient willingness to consent (more explanatory) | Differentiating site vs. patient eligibility |
Especially for complex or novel interventions, are exclusions based on comorbidities any different for intervention than what would occur in usual care? | ||||
Recruitment: how are participants recruited into the trial? | Usual appointment or clinic | Targeted invitation letters, advertising in newspapers, radio plus incentives and other routes that would not be used in usual care | Informed consent procedures; extra screening procedures in EHR (more explanatory) | None noted |
Setting: where is the trial being done? | Setting of trial identical to usual care setting | A very explanatory approach would occur only a single center, or only specialized trial or academic centers | Diversity in terms of number, location, and organization type considered in very pragmatic ratings; regional studies, studies conducted in one type of health care system; consent requirements considered in more explanatory ratings | There are many dimensions to consider when comparing the setting to usual care. For example, was the goal to rate compared to a particular type of setting such as safety net clinics or to include multiple types of health systems? Additionally, participating institutions tended to have good quality improvement capacity; it is difficult to rate how this compares to “typical” settings. |
Organization: what expertise and resources are needed to deliver the intervention? | Resources, provider expertise and the organization of care delivery in the intervention arm identical to usual care | Trial increases staff levels, gives additional training, requires more than usual experience or certification and increases resources | Space, intervention delivery, outreach (more explanatory). A train the trainer approach is one way to make training procedures more pragmatic. | It was noted that many of these trials take place in innovative health systems that may not be typical of the country, potentially conflating setting and organization. |
Difficult to determine weight or importance of organization at provider level versus organization at IT or administration level. | ||||
The amount of effort undertaken to maintain ‘adherence’ at the organizational level was not particularly well documented. | ||||
Flexibility of delivery: how should the intervention be delivered? | Flexibility in delivering intervention identical to usual care | If there is a strict protocol, monitoring and measures to improve compliance, with specific advice on allowed co-interventions and complications | An example noted was that guidance in place but provider can deviate from it or the guidance can be modified over time based on stakeholder feedback | What happens if there is a strict protocol in usual care as well? |
Flexibility of adherence: what measures are in place to make sure participants adhere to the intervention? | No more than usual encouragement to adhere to the intervention | Exclusion based on adherence, and measures to improve adherence if found wanting | Distinguishing adherence encouragement that would happen in usual care from the intervention. At the clinic/provider level, monitoring was interpreted to relate both to flexibility of delivery and adherence. | |
Follow-up: how closely are the participants followed-up? | No more than usual follow-up | More frequent, longer visits, unscheduled visits triggered by primary outcome event or intervening event, and more extensive data collection | Use of electronic records (more pragmatic); additional contacts, additional measures (more explanatory) | None noted |
Primary outcome: how relevant is it to participants? | Outcome measure very relevant to participants | Using a surrogate, physiological outcome, central adjudication or use assessment expertise that is not available in usual care, or the outcome is measured at an earlier time than in usual care | Implementation feasibility or other procedural details (more explanatory) | How to handle clinic- versus patient-relevant outcomes |
Primary analysis: to what extent are all data included? | intention to treat with all available data | Excludes ineligible post-randomization participants, includes only completers or those following the treatment protocol | Extensive analytic details were not available in the materials raters had available. |