Glossary
Replication crisis
What is the replication crisis?
The replication crisis is the ongoing discovery that many published research findings in psychology, medicine, and other sciences cannot be reproduced when studies are conducted again by independent researchers. It has prompted scrutiny of how studies are designed, analyzed, and published.
Read the diagram
First, an original study reports a finding. Independent researchers then test it with new data. Compare the estimates, uncertainty and methods rather than looking only at statistical significance. Retraction is a separate change to a paper’s publication status.
Sources: Center for Open Science: Replication and new data · COPE: Retraction guidelines
How it works
Replication results
The Open Science Collaboration (2015) replicated 100 studies from three psychology journals. Thirty-six percent of replication results were statistically significant; other replication criteria gave different percentages. This is not a field-wide estimate of how much research is false.
Contributing factors and reforms
Contributing factors include p-hacking, HARKing, publication bias, small sample sizes, and inflated effect sizes. These practices can make weak or unreliable effects look more convincing than they are. Reforms include pre-registration, open data, larger samples, and registered reports.
Applied example
In the Open Science Collaboration project, replication effect sizes were about half the magnitude of the original effects. This illustrates why the size and uncertainty of an effect matter alongside whether a statistical threshold is crossed.
Ego depletion as a replication example
A preregistered study across 23 laboratories tested an ego-depletion procedure and estimated an effect close to zero, with a confidence interval spanning zero. That challenged the expected effect for the tested procedure. It does not, by itself, settle every account of self-control or every possible depletion task.
Why it matters
A behavioral intervention should not earn trust because an early study produced an exciting result. Replication, transparent methods, and realistic effect estimates help test whether the finding holds up. Reforms are useful when they improve that evidence; their existence alone does not settle whether a claim is reliable.
Replication and reproducibility ask different questions
Terminology varies across fields. A useful distinction, also used by the National Academies, is between checking whether the same data and analysis produce the same result—computational reproducibility—and collecting new data to test the finding—replication. Re-running a script can reveal an analytical error, but cannot show that an effect holds in a new sample.
A direct replication tries to repeat the important procedures closely. A conceptual replication changes how the idea is tested, such as using a different task intended to measure the same construct. A successful conceptual test can broaden the evidence. A failure is harder to interpret if several features changed at once: perhaps the claim is wrong, or perhaps the new measure tests something different. Specify what was repeated and what changed before treating either label as a verdict.
Some findings do hold up
The crisis is a reason to distinguish findings, not to declare all psychology useless. Zwaan and colleagues tested nine established cognitive effects across perception/action, memory and language and reported successful online replications. That is encouraging evidence about the effects they tested, not a reliability certificate for every claim in cognitive psychology. Jason discussed the work in Psychology studies that actually pass the test.
For an applied decision, ask what the replicated result actually concerns. A dependable effect on a short laboratory task may illuminate how people process information without proving that a particular training product works. Conversely, an uncertain headline finding should not be rescued by attaching it to a respected theory. Compare the effect estimates, methods and uncertainty, then look for evidence about the outcome you need to improve.