Glossary
Statistical power
What is statistical power?
Statistical power is the probability that a planned analysis will reject a specified null hypothesis when a particular alternative effect is true.
A design with 80% power for a specified effect has a 20% chance of a nonsignificant result when that effect and the model assumptions hold. Power is always about an effect and a design, not just a study in the abstract.
What affects power?
Effect size, sample size, variability, study design, analysis, and the significance threshold all matter. The number of participants alone is not enough to calculate it.
A calculated example
Suppose a study compares the means of two independent groups using a two-sided, equal-variance t-test at a significance threshold of 0.05. Assume normally distributed outcomes, a true mean difference of 5 points and a common standard deviation of 10 points, giving a standardized effect of 0.5. Allocate equal numbers to the groups, with no missing observations.
- 30 people per group: approximately 48 percent power.
- 100 people per group: approximately 94 percent power.
These values are calculated from the noncentral t distribution, not observed study results. Holding the effect and other assumptions fixed, the larger sample makes that effect easier to detect. A smaller target difference would reduce power at either sample size. Even the 48-percent-power design’s nonsignificant result would not demonstrate a zero effect.
Why it matters
Low power makes real effects easier to miss. Among results selected for statistical significance, estimated effects can also be inflated—the “winner's curse.” Button and colleagues discuss these problems in their review of neuroscience studies.
Before collecting data, calculate power for the effects the study needs to detect. Afterward, a nonsignificant result does not establish that the effect is zero.
Power for an effect worth detecting
The calculation above assumes that a five-point difference is the effect of interest. Suppose the same score has a standard deviation of 10, but a two-point difference would already justify an inexpensive program. The target standardized difference is now 0.20. With 100 people per group, the two-sided t-test has only about 29% power for that smaller effect, under the same assumptions.
That does not make the study useless. It means the study was not well equipped to detect the improvement the decision actually cares about. Plan around a smallest worthwhile effect, and examine several plausible effect sizes if the expected effect is uncertain. Choosing only the largest effect from an early favorable paper can leave the next study too small.
After the results, inspect precision
Once the data are collected, calculating “observed power” from the observed effect adds little to its significance test. Hoenig and Heisey explain the problem: the same result determines both quantities. Report the estimate and confidence interval instead. An interval spanning both negligible and worthwhile effects says the decision remains uncertain. An interval concentrated near zero tells a different story. Neither conclusion follows from the word “nonsignificant” alone.
Power calculations also need the actual design. Repeated observations from one person and employees within the same assigned office are not independent new participants. Expected dropout and imperfect measurement matter too. A sample-size target is credible only when those features are represented.