Assessing heterogeneity of treatment effects in light fundamental statistical tendencies
James Scanlan, James P. Scanlan, Attorney at Law
26 May 2011
The article by Kent et al.[1] provides useful guidance on the reporting of results by subgroup. But the article suffers from the common assumption that the absence of a subgroup effect (heterogeneity) is reflected by equivalent relative risk reductions across subgroups. It is not logical to regard equivalent relative risk reductions as somehow normal (i.e., as reflecting the absence of a subgroup effect) for the simple reason that it is not possible for a factor to cause equal relative reductions in an outcome for groups with different base rates while causing equal relative increases in the opposite outcome for those groups.
The point can be illustrated with figures in Table 1 of Kent et al., which shows the same 25% relative risk reduction of an adverse outcome for an average risk group (4% reduced to 3%) and a high risk group (20% reduced to 15%). Such figures would mean that the increase in the favorable outcome rate was 1% for the low risk group (96% increased to 97%) but 6.3% for the high risk group (80% increased to 85%). Since there is no more reason to regard it as somehow normal that there will be equal relative changes in one outcome than to regard it as somehow normal that there will be equal relative changes in the opposite outcome, there is no reason to regard it as somehow normal that there will be equal relative changes in either outcome.
In fact, for reasons related to the shapes of normal risk distributions there is reason to expect that a factor that reduces the risk of an outcome will tend to cause groups with lower base rates to experience larger relative reductions in those rates, while causing other groups to experience larger relative increases in the opposite outcome rates.[2-7] Thus, reliance on relative risk as a measure of effect size tends to yield opposite conclusions as to whether high or low risk groups derive the greater benefit from an intervention depending on whether one examines the favorable or the adverse outcome.
The only way to determine whether there exists a meaningful subgroup effect is to derive from the rates for controls and treated subjects in each subgroup the differences between means of the hypothesized underlying distributions.[2-4,7] Taking the data from Table 1 of Kent et al., the reduction from 4% to 3% for the average risk group reflects a differences between means of the underlying distributions of .13 standard deviations. An equivalent reduction in the risk for the high risk group would be reflected by a reduction from 20% to 17%. Determinations of meaningful interaction should be based on departures from patterns such as these, not departures from equivalent relative changes in either the adverse or the favorable outcome. Similarly, appraisals of the likely absolute risk reduction for each subgroup – as Kent et al. note, the key clinical consideration – should be based on benchmarks derived as just described rather than on benchmarks based on equivalent relative risk reductions.
References:
1. Kent DM, Rothwell PM, Ionnadis JPA, et al. Assessing and reporting heterogeneity in treatment effects in clinical trials: a proposal. Trials 2010,11:85: http://www.trialsjournal.com/content/11/1/85 (Accessed May 1, 2011.)
2. Scanlan JP. Interpreting Differential Effects in Light of Fundamental Statistical Tendencies, presented at 2009 Joint Statistical Meetings of the American Statistical Association, International Biometric Society, Institute for Mathematical Statistics, and Canadian Statistical Society, Washington, DC, Aug. 1-6, 2009: http://www.jpscanlan.com/images/JSM_2009_ORAL.pdf;http://www.jpscanlan.com/images/Scanlan_JSM_2009.ppt (Accessed May 1, 2011.)
3. Scanlan JP. Rethinking the premises of subgroup analyses. BMJ June 7, 2010 (responding to Sun X, Briel M. Walter SD, and Guyatt GH. Is as subgroup effect believable? Updating criteria to evaluated the credibility of subgroup analyses. BMJ 2010;340:850-854): http://www.bmj.com/cgi/eletters/340/mar30_3/c117 (Accessed May 1, 2011.)
4. Scanlan JP. Problems in identifying interaction where groups have different base rates. BMJ Sept. 21, 2010 (responding to Altman DG, Bland JM. Interaction revisited: the difference between two estimates. BMJ 2003;326:219): Altman DG, Bland JM. Interaction revisited: the difference between two estimates. BMJ 2003;326:219): http://www.bmj.com/content/326/7382/219/reply#bmj_el_241943 (Accessed May 1, 2011.)
5. Scanlan JP. Race and mortality. Society 2000;37(2):19-35: http://www.jpscanlan.com/images/Race_and_Mortality.pdf (Accessed May 1, 2011.)
7. Subgroup Effects sub-page of Scanlan’s Rule page of jpscanlan.com: http://www.jpscanlan.com/scanlansrule/subgroupeffects.html (Accessed May 1, 2011.)
Trials
Assessing heterogeneity of treatment effects in light fundamental statistical tendencies
26 May 2011
The article by Kent et al.[1] provides useful guidance on the reporting of results by subgroup. But the article suffers from the common assumption that the absence of a subgroup effect (heterogeneity) is reflected by equivalent relative risk reductions across subgroups. It is not logical to regard equivalent relative risk reductions as somehow normal (i.e., as reflecting the absence of a subgroup effect) for the simple reason that it is not possible for a factor to cause equal relative reductions in an outcome for groups with different base rates while causing equal relative increases in the opposite outcome for those groups.
The point can be illustrated with figures in Table 1 of Kent et al., which shows the same 25% relative risk reduction of an adverse outcome for an average risk group (4% reduced to 3%) and a high risk group (20% reduced to 15%). Such figures would mean that the increase in the favorable outcome rate was 1% for the low risk group (96% increased to 97%) but 6.3% for the high risk group (80% increased to 85%). Since there is no more reason to regard it as somehow normal that there will be equal relative changes in one outcome than to regard it as somehow normal that there will be equal relative changes in the opposite outcome, there is no reason to regard it as somehow normal that there will be equal relative changes in either outcome.
In fact, for reasons related to the shapes of normal risk distributions there is reason to expect that a factor that reduces the risk of an outcome will tend to cause groups with lower base rates to experience larger relative reductions in those rates, while causing other groups to experience larger relative increases in the opposite outcome rates.[2-7] Thus, reliance on relative risk as a measure of effect size tends to yield opposite conclusions as to whether high or low risk groups derive the greater benefit from an intervention depending on whether one examines the favorable or the adverse outcome.
The only way to determine whether there exists a meaningful subgroup effect is to derive from the rates for controls and treated subjects in each subgroup the differences between means of the hypothesized underlying distributions.[2-4,7] Taking the data from Table 1 of Kent et al., the reduction from 4% to 3% for the average risk group reflects a differences between means of the underlying distributions of .13 standard deviations. An equivalent reduction in the risk for the high risk group would be reflected by a reduction from 20% to 17%. Determinations of meaningful interaction should be based on departures from patterns such as these, not departures from equivalent relative changes in either the adverse or the favorable outcome. Similarly, appraisals of the likely absolute risk reduction for each subgroup – as Kent et al. note, the key clinical consideration – should be based on benchmarks derived as just described rather than on benchmarks based on equivalent relative risk reductions.
References:
1. Kent DM, Rothwell PM, Ionnadis JPA, et al. Assessing and reporting heterogeneity in treatment effects in clinical trials: a proposal. Trials 2010,11:85: http://www.trialsjournal.com/content/11/1/85 (Accessed May 1, 2011.)
2. Scanlan JP. Interpreting Differential Effects in Light of Fundamental Statistical Tendencies, presented at 2009 Joint Statistical Meetings of the American Statistical Association, International Biometric Society, Institute for Mathematical Statistics, and Canadian Statistical Society, Washington, DC, Aug. 1-6, 2009: http://www.jpscanlan.com/images/JSM_2009_ORAL.pdf;http://www.jpscanlan.com/images/Scanlan_JSM_2009.ppt (Accessed May 1, 2011.)
3. Scanlan JP. Rethinking the premises of subgroup analyses. BMJ June 7, 2010 (responding to Sun X, Briel M. Walter SD, and Guyatt GH. Is as subgroup effect believable? Updating criteria to evaluated the credibility of subgroup analyses. BMJ 2010;340:850-854): http://www.bmj.com/cgi/eletters/340/mar30_3/c117 (Accessed May 1, 2011.)
4. Scanlan JP. Problems in identifying interaction where groups have different base rates. BMJ Sept. 21, 2010 (responding to Altman DG, Bland JM. Interaction revisited: the difference between two estimates. BMJ 2003;326:219): Altman DG, Bland JM. Interaction revisited: the difference between two estimates. BMJ 2003;326:219): http://www.bmj.com/content/326/7382/219/reply#bmj_el_241943 (Accessed May 1, 2011.)
5. Scanlan JP. Race and mortality. Society 2000;37(2):19-35: http://www.jpscanlan.com/images/Race_and_Mortality.pdf (Accessed May 1, 2011.)
6. Scanlan JP. Divining difference. Chance 1994;7(4):38-9,48: http://jpscanlan.com/images/Divining_Difference.pdf (Accessed May 1, 2011.)
7. Subgroup Effects sub-page of Scanlan’s Rule page of jpscanlan.com: http://www.jpscanlan.com/scanlansrule/subgroupeffects.html (Accessed May 1, 2011.)
Competing interests
None