Practical significance

What is practical significance?

Practical significance is the importance of a result for a real decision. It asks whether the difference is large enough, useful enough, and dependable enough to justify what it costs. Statistical significance answers a different question about the data under a statistical model; it does not tell you whether the result is worth acting on. The American Statistical Association makes this distinction explicit.

Is a two-point increase worth buying?

Consider a hypothetical event-reminder experiment. Random assignment creates two independent groups with complete attendance records. Of 5,000 people receiving the usual invitation, 1,000 attend. Of 5,000 assigned to receive an additional reminder, 1,100 attend. Attendance at that event is 20% versus 22%: a two-percentage-point increase, or a 10% relative increase. The reminder group has 100 more attendees than the comparison group.

Suppose the additional reminder costs $500 for those 5,000 people, including staff time and delivery. Using the observed difference, the cost is $500 ÷ 100 = $5 per additional attendee. If your organization can justify spending up to $20 for one additional attendance, this looks promising. If the reminder instead requires $3,000 of additional work, the same estimate implies $30 per additional attendee. The behavioral result has not changed. The decision has.

The $20 threshold is an assumption in this example, not a universal value for attendance. It could reflect a budget ceiling or an estimate of the benefit. Nor is attendance necessarily the final benefit: a full room is not proof that anyone learned something useful. If the goal is learning, measure that separately and decide which costs and outcomes belong in the comparison.

Uncertainty can change the decision

For the independent-group experiment above, a simple normal-approximation 95% confidence interval runs from about 0.4 to 3.6 percentage points. Applied to 5,000 people, those endpoints correspond to roughly 20 to 180 additional attendees. At a fixed cost of $500, that range translates to approximately $25 to $2.80 per additional attendance. These are scenario calculations from the effect interval, not a complete economic uncertainty analysis. They omit uncertainty about costs, benefits and transfer to the next event.

The point estimate clears the $20 ceiling, but the less favorable end does not. A large commitment may justify another test or a lower-cost delivery method. A small, reversible use may be reasonable with continued measurement. What matters is the consequence of being wrong, not a compulsory rule that every interval must clear a threshold before anything can be tried.

Define the smallest effect size of interest

The smallest effect size of interest is the smallest effect that would matter for the question you have chosen. Set it before the results, with a reason. In this example, $500 divided by a $20 ceiling requires at least 25 additional attendees among 5,000 people: an increase of 0.5 percentage points. That is the break-even attendance gain under these cost assumptions.

Use that threshold when planning statistical power and interpreting the estimate. Do not call a result unimportant merely because it misses p < .05; a wide interval may still include a worthwhile effect. Equally, a precisely estimated improvement can be too small to justify a costly program. Compare the intervention with a useful alternative—perhaps a more convenient event time—not only with doing nothing.