Heterogeneous treatment effects

What are heterogeneous treatment effects?

Heterogeneous treatment effects are differences in an intervention's causal effect across people or conditions. The average treatment effect summarizes a population; it does not mean everyone changes by that amount. Nor does variation in people's outcomes alone prove that treatment effects differ. People might have different outcomes even without the intervention.

One reminder, two different circumstances

Consider a hypothetical randomized attendance experiment. Before assignment, participants report whether the event time fits their schedule. Among those with a compatible schedule, attendance is 40% without a reminder and 48% with one: an eight-percentage-point difference. Among those with a schedule conflict, attendance is 5% in either group: a zero-point difference.

If the two schedule groups are equally common in each treatment arm, the overall rates are 22.5% without the reminder and 26.5% with it, giving an average difference of four percentage points. The average describes neither subgroup's observed difference. The proposed explanation is that a reminder might help someone who can attend but cannot create free time for someone who cannot. These invented numbers demonstrate the distinction; they are not evidence for that mechanism.

Compare effects, not separate significance labels

To test whether the effect differs, compare the treatment contrasts: eight points minus zero points. The uncertainty of that difference matters. “Significant for compatible schedules, nonsignificant for conflicts” would not establish an interaction, because the subgroups may differ in size and precision. Cochrane's discussion of subgroup comparisons explains this error.

Specify the scale too. A change from 10% to 20% and a change from 40% to 80% are both risk ratios of two, but their absolute increases are 10 and 40 percentage points. Treatment-effect heterogeneity can therefore depend on whether you mean an absolute difference, ratio or another measure.

What would justify different treatment?

A credible subgroup claim needs a defensible comparison within the subgroups, adequate precision and a reason to expect the difference. Characteristics recorded before treatment, a prespecified interaction and confirmation in new data are more persuasive than sorting through dozens of post hoc groups for an attractive pattern. Small subgroups and many searches can produce unstable recommendations.

Do not define groups by something the intervention itself changes, such as whether people opened the reminder, and then treat them as equivalent to groups defined before assignment. Opening the message can be part of its effect and can also reflect prior interest. That selection changes the causal question.

Finally, a subgroup average is still not an individual's causal effect. As Hernán and Robins explain using potential outcomes, we ordinarily cannot observe the same person's outcome both with and without treatment under the same conditions. A profile may help form a hypothesis about fit; it does not reveal a proven prescription. Before using a targeting rule, test whether it improves outcomes compared with a reasonable untargeted alternative.