P value

What is a p-value?

A p-value is the probability, under a specified statistical model that includes the null hypothesis, of obtaining a test statistic at least as extreme as the one observed. It measures how surprising the data are in a particular direction or in both directions, depending on the test. It does not give the probability that the null hypothesis is true. The American Statistical Association's guidance also distinguishes p-values from effect size and practical importance.

A coin example with all the possibilities visible

Suppose you toss a coin ten times and observe nine heads. You planned a two-sided test of a fair-coin model before tossing. Under that model, tosses are independent and each has a 50% chance of heads. There are 2¹⁰ = 1,024 equally likely sequences.

For this symmetric exact test, results at least as far from five heads as the observed nine are zero, one, nine or ten heads. There is one sequence with zero heads, ten with one head, ten with nine heads and one with ten heads. Their total probability is (1 + 10 + 10 + 1) ÷ 1,024 = 0.021484375, so the two-sided p-value is about .0215.

This says those extreme results have about a 2.15% probability under the fair, independent-toss model. It does not say there is a 2.15% probability that the coin is fair, nor a 97.85% probability that a favored alternative explanation is correct. The calculation assumes the model; it does not assign probabilities to models.

What does p < .05 mean?

If a prespecified procedure rejects its null when p < .05, its long-run false-positive rate is controlled at no more than 5% under the null and the procedure's assumptions. Discrete tests, like the coin test, can be conservative. The threshold is a decision rule, not a boundary between a real effect and an imaginary one.

Imagine repeating the coin experiment until one batch looks unusual, then reporting only that batch as the planned test. The single-batch p-value would omit the search that produced it. This is why the stopping rule, tested outcomes and analytical choices matter. The p-hacking entry explains that problem in a research setting.

A small p-value can accompany a small benefit

In behavioral research, a very large sample can estimate a tiny attendance gain precisely enough to produce a small p-value. A smaller study can miss the threshold despite an estimate that would matter if accurate. Neither p-value tells you the size of the benefit. Read the effect size and its confidence interval, then ask whether the comparison supports a causal claim and whether the improvement is worth its cost.

Likewise, a large p-value does not show that two interventions are equivalent. It may reflect little information. Demonstrating that differences are sufficiently small requires an appropriate question, a meaningful equivalence margin and an analysis designed to evaluate it.