Systematizing the analysis of effect heterogeneity requires rethinking some fundamentals
James Scanlan, James P. Scanlan, Attorney at Law
1 June 2011
The article by Gabler et al.[1] questions the soundness of epidemiological literature’s reporting and analysis of heterogeneity of treatment effects (HTE) and calls for greater attention to HTE issues and more systematic analysis of such issues.
But the goals the authors seek cannot be achieved without reconsideration of certain fundamentals of subgroup analysis. Standard approaches to such analyses are based on an assumption that absent HTE all subgroups will experience equal proportionate changes in outcome rates (i.e., the same rate ratio across different baseline rates) and that HTE is observed in those cases where equal proportionate changes are not found. That assumption, however, is demonstrably unsound for the simple reason that it is not possible for a factor to cause equal proportionate changes in an outcome while causing equal proportionate changes in the opposite outcome.
That is, for example, if Group A has a baseline rate of 5% and Group B has a baseline rate of 10%, a factor that reduces the two rates by equal proportionate amounts, say 20% (from 5% to 4% and from 10% to 8%) would necessarily increase the opposite outcome by different proportionate amounts (95% increased to 96%, a 1.05% increase; 90% to 92%, a 2.2% increase). And since there is no more reason to expect that two group would undergo equal proportionate changes in one outcome than there is to expect they would undergo equal proportionate changes in the opposite outcome, there is no reason to expect that the two groups would undergo equal proportionate changes in either outcome.
For reasons inherent in the shapes of normal distributions of factors associated with experiencing or avoiding an outcome, it is more reasonable to expect that a treatment that reduces an outcome rate will tend to cause a larger proportionate decrease in that outcome for groups with the lower base rates while causing a larger proportionate increase in the opposite outcome for other groups.[2-6]
These considerations raise several issues for subgroup analyses. The first involves identifying benchmarks across baseline rates the departure from which would be deemed a subgroup effect. See Table 6 of the presentation in reference 4 for the application of the approach to identifying subgroup effects discussed in reference 4 to data in reference 49 of Gabler et al. A second, and more important, involves using information derived from observed treatment effects for particular groups to estimate absolute risk changes in groups for which treatment effect information is not available or not reliable.[5,6]
References:
1. Gabler NB, Naihua D, Liao D, et al. Dealing with heterogeneity treatments: is the literature up to the challenge. Trials 2009,10:43: http://www.trialsjournal.com/content/10/1/43 (Accessed May 29, 2011.)
2. Scanlan JP. Race and mortality. Society 2000;37(2):19-35: http://www.jpscanlan.com/images/Race_and_Mortality.pdf (Accessed May 29, 2011.)
4. Scanlan JP. Interpreting Differential Effects in Light of Fundamental Statistical Tendencies, presented at 2009 Joint Statistical Meetings of the American Statistical Association, International Biometric Society, Institute for Mathematical Statistics, and Canadian Statistical Society, Washington, DC, Aug. 1-6, 2009: PowerPointPresentation: http://www.jpscanlan.com/images/Scanlan_JSM_2009.ppt; Oral Presentation: http://www.jpscanlan.com/images/JSM_2009_ORAL.pdf (Accessed May 29, 2011.)
5. Scanlan’s Rule page of jpscanlan.com: http://jpscanlan.com/scanlansrule.html (Accessed May 29, 2011.)
6. Scanlan JP. Assessing heterogeneity of treatment effects in light fundamental statistical tendencies. Trials May 26, 2011(responding to Kent DM, Rothwell PM, Ionnadis JPA, et al. Assessing and reporting heterogeneity in treatment effects in clinical trials: a proposal. Trials 2010,11:85): http://www.trialsjournal.com/content/11/1/85/comments#498686 (Accessed May 29, 2011.)
Trials
Systematizing the analysis of effect heterogeneity requires rethinking some fundamentals
1 June 2011
The article by Gabler et al.[1] questions the soundness of epidemiological literature’s reporting and analysis of heterogeneity of treatment effects (HTE) and calls for greater attention to HTE issues and more systematic analysis of such issues.
But the goals the authors seek cannot be achieved without reconsideration of certain fundamentals of subgroup analysis. Standard approaches to such analyses are based on an assumption that absent HTE all subgroups will experience equal proportionate changes in outcome rates (i.e., the same rate ratio across different baseline rates) and that HTE is observed in those cases where equal proportionate changes are not found. That assumption, however, is demonstrably unsound for the simple reason that it is not possible for a factor to cause equal proportionate changes in an outcome while causing equal proportionate changes in the opposite outcome.
That is, for example, if Group A has a baseline rate of 5% and Group B has a baseline rate of 10%, a factor that reduces the two rates by equal proportionate amounts, say 20% (from 5% to 4% and from 10% to 8%) would necessarily increase the opposite outcome by different proportionate amounts (95% increased to 96%, a 1.05% increase; 90% to 92%, a 2.2% increase). And since there is no more reason to expect that two group would undergo equal proportionate changes in one outcome than there is to expect they would undergo equal proportionate changes in the opposite outcome, there is no reason to expect that the two groups would undergo equal proportionate changes in either outcome.
For reasons inherent in the shapes of normal distributions of factors associated with experiencing or avoiding an outcome, it is more reasonable to expect that a treatment that reduces an outcome rate will tend to cause a larger proportionate decrease in that outcome for groups with the lower base rates while causing a larger proportionate increase in the opposite outcome for other groups.[2-6]
These considerations raise several issues for subgroup analyses. The first involves identifying benchmarks across baseline rates the departure from which would be deemed a subgroup effect. See Table 6 of the presentation in reference 4 for the application of the approach to identifying subgroup effects discussed in reference 4 to data in reference 49 of Gabler et al. A second, and more important, involves using information derived from observed treatment effects for particular groups to estimate absolute risk changes in groups for which treatment effect information is not available or not reliable.[5,6]
References:
1. Gabler NB, Naihua D, Liao D, et al. Dealing with heterogeneity treatments: is the literature up to the challenge. Trials 2009,10:43: http://www.trialsjournal.com/content/10/1/43 (Accessed May 29, 2011.)
2. Scanlan JP. Race and mortality. Society 2000;37(2):19-35: http://www.jpscanlan.com/images/Race_and_Mortality.pdf (Accessed May 29, 2011.)
3. Scanlan JP. Divining difference. Chance 1994;7(4):38-9,48: http://jpscanlan.com/images/Divining_Difference.pdf (Accessed May 29, 2011.)
4. Scanlan JP. Interpreting Differential Effects in Light of Fundamental Statistical Tendencies, presented at 2009 Joint Statistical Meetings of the American Statistical Association, International Biometric Society, Institute for Mathematical Statistics, and Canadian Statistical Society, Washington, DC, Aug. 1-6, 2009: PowerPointPresentation: http://www.jpscanlan.com/images/Scanlan_JSM_2009.ppt; Oral Presentation: http://www.jpscanlan.com/images/JSM_2009_ORAL.pdf (Accessed May 29, 2011.)
5. Scanlan’s Rule page of jpscanlan.com: http://jpscanlan.com/scanlansrule.html (Accessed May 29, 2011.)
6. Scanlan JP. Assessing heterogeneity of treatment effects in light fundamental statistical tendencies. Trials May 26, 2011(responding to Kent DM, Rothwell PM, Ionnadis JPA, et al. Assessing and reporting heterogeneity in treatment effects in clinical trials: a proposal. Trials 2010,11:85): http://www.trialsjournal.com/content/11/1/85/comments#498686 (Accessed May 29, 2011.)
Competing interests
None.