Glossary
P-hacking
What is P-hacking?
P-hacking is outcome-guided selection among data-collection or analysis choices that makes a reported p-value look more convincing than the full research process warrants. Examples include selectively reporting outcomes, using an unplanned significance-based stopping rule, or trying multiple tests and presenting only the favorable one.
How it works
Simmons, Nelson, and Simonsohn (2011) used simulations and demonstrations to show that undisclosed choices about outcomes, stopping, covariates, and conditions can raise false-positive rates well above the nominal threshold. The rate depends on the choices and their combination.
P-hacking can occur without deliberate deception: researchers explore their data looking for 'what the story is' without recognizing how outcome-guided selection changes the interpretation of the reported test.
Pre-registration, reporting the range of analyses considered, and transparent sensitivity analyses are safeguards; none removes every source of bias.
Applied example
A researcher collects data on five outcome measures, analyzes each with and without two potential covariates, and includes or excludes borderline outliers. These choices create multiple possible analyses, whose results are correlated. The chance of obtaining at least one significant result depends on which analyses are used and how their results are related; it cannot be inferred from the count alone.
Suppose one version produces p < .05 and the researcher reports only that model, describing it as though it had been chosen before seeing the data. The hidden selection step is the problem: readers cannot tell that the reported result was selected from several attempts. Reporting all the specifications would show whether the apparent finding depends on one particular outcome, covariate choice, or outlier rule. That disclosure allows a sensitivity assessment; it does not automatically turn the selected test into a preplanned confirmation.
Why it matters
The ASA statement on p-values emphasizes full reporting and transparency. Exploration is useful, but readers need to know which analyses were considered and how the reported analysis was selected.
Exploration can be useful without becoming confirmation
Imagine the five-outcome study reveals that a training exercise helps only on a task the researchers had not expected. That is a legitimate clue. Report how the pattern was discovered, show the planned results, and test the new prediction with fresh data. The problem is presenting an outcome-selected analysis as if it were the one test chosen in advance.
Likewise, looking at accumulating data is not automatically p-hacking. A planned sequential design can account for repeated examination of results. Repeatedly applying an ordinary fixed-sample test and stopping at the first attractive p-value changes the error properties of that procedure. The stopping rule belongs in the methods, alongside the outcomes, exclusions and models considered.
A result just below .05 does not diagnose a researcher's intent or prove misconduct. Evaluate the plan, data and reporting. Pre-registration can make the planned comparison visible; Registered Reports add review before the main results. Neither substitutes for a sensible measure or an appropriate analysis.