Skip to main content

Table 2 Considerations in using different sets of eligibility criteria for defining the analytic sample

From: Addressing identification bias in the design and analysis of cluster-randomized pragmatic trials: a case study

Analytic sample

General considerations

Implications for effectiveness outcome

No. days of acute care utilization

(patient-level outcome)

Analytic samples not affected by identification biasa

Sample 1. Patients with documented OUD before randomization

• Misses potential improvements in outcomes attributable to the intervention among new patients with OUD who were attracted to receive care after randomization due to the PROUD intervention. These patients could comprise a substantial proportion of patients with OUD treated due to the PROUD intervention (70%–90%)

• Patients with a prior documented OUD may not reflect the general population of patients with true OUDb

• Given that OUD is underdiagnosed, restricting to patients with documented OUD before randomization reduces the sample size and therefore power for patient-level outcomes, relative to an open cohort design that includes those diagnosed after randomization (Sample 4 below)

Estimates of intervention effects within this select population of patients with documented OUD before randomization may not generalize to the broader population of individuals with true OUD who may be treated as part of the intervention (and therefore may not detect the true benefit).

Sample 2. All patients with primary care visits before randomization

• Same as bullet #1 for Sample 1 above

• Sample includes patients with undocumented OUDs before randomization who might benefit from the intervention

• Sample also includes many individuals without true OUD who would not be impacted by the intervention

• Since most individuals in the site population do not have OUD, the effect of the intervention on acute care utilization would be diluted, resulting in attenuation of the treatment effect toward the null

• Relative to Sample 1, power could either be increased due to the higher sample size of patients with OUD or decreased due to including patients without true OUD in the analysis

Sample 3. Patients with primary care visits before randomization who have documented OUD or are at “increased risk” of OUDc

• Same as bullet #1 for Sample 1 above

• Need to develop a definition of “increased risk” of OUD that seeks to include as many patients who truly have OUD (maximizing sensitivity) while limiting the number of patients included who do not truly have OUD (maximizing specificity). For example, this definition could be selected to target a high specificity (Sample 3a), a high sensitivity (Sample 3b), or a balanced sensitivity and specificity option (Sample 3c)

• Relative to Sample 1, results in a larger sample size of patients with true OUD (higher sensitivity). This could increase power

• At the same time could lead to attenuated intervention effect estimates relative to Sample 1, since more of the identified individuals would not have OUD (lower specificity)

Analytic samples potentially affected by identification biasa

Sample 4. Patients with documented OUD (before and/or after randomization)

• Patients diagnosed after randomization in the intervention arm may not be comparable to patients diagnosed after randomization in the control arm

• Diagnosis of newly recognized OUD is expected to continue over time. Consequently, including individuals diagnosed after randomization could increase the sample size (and therefore power) as compared to Sample 1

Individuals diagnosed with OUD after randomization in the intervention arm are likely to be different (either sicker or healthier) with respect to their propensity for acute care utilization than individuals diagnosed with OUD after randomization in the control arm. This could lead to bias (see Table 1).

Sample 5. All patients with primary care visits (before and/or after randomization)

• Patients new to intervention sites after randomization may not be comparable to patients new to the control sites after randomization

• As in Sample 2, sample includes many individuals without true OUD who would not be impacted by the intervention

• Captures outcomes of all patients with OUD who could be treated: patients seen previously in the clinic (including those with and without documented OUD before randomization) and those attracted to receive care as part of the PROUD intervention

As in Sample 2, the effect of the intervention would be diluted in the entire site population relative to Samples 1 or 4 and power could either be increased or decreased.

Sample 6. Patients with primary care visits who have documented OUD or are at “increased risk” of OUDc (before and/or after randomization)

• Patients identified as at “increased risk” of OUD after randomization in the intervention arm may not be comparable to patients identified as at “increased risk” of OUD after randomization in the control arm

• As in Sample 3, need to develop a definition of “increased risk” of OUD that seeks to include as many patients who truly have OUD while limiting the proportion of patients who do not truly have OUD who are included

As in Sample 3, results in a larger sample size of patients with true OUD, but also could include many patients without OUD; thus, power could either be increased or decreased relative to Sample 4.

  1. EHR electronic health record, OUD opioid use disorder
  2. a Identification bias is a form of selection bias that can occur in open-cohort cluster-randomized trials when the randomized intervention group assignment affects who is identified as eligible for a particular analysis. Identifying eligibility for inclusion in trial analyses before randomization (or using data collected pre-randomization) avoids this source of bias
  3. b Documented OUD refers to patients with an OUD diagnosis documented in the EHR; True OUD refers to patients with OUD regardless of its recognition by clinicians and/or documentation in the EHR
  4. c Planned definition of “increased risk” of OUD included individuals with any documented OUD diagnosis at baseline or anyone with both chronic opioid therapy (outside of end of life, palliative care, or active cancer treatment) and at least one of the following risk factors associated with increased risk of OUD: high morphine equivalent dose, alcohol or other substance use disorders, mental health disorders, concurrent sedative use, or pain in two or more body regions (e.g. headache and back pain)