Glossary
Publication bias
What is Publication bias?
Publication bias is the selective availability of studies according to their results, often favoring positive or statistically significant findings. It can arise when authors do not write up or submit studies, as well as during editorial selection. This can create a distorted literature that overestimates favorable effect sizes and makes the evidence look more consistent than it is.
How it works
The 'file drawer problem' (Rosenthal, 1979) means that for every published positive study, there may be several unpublished null studies that were conducted but never reported. This can bias a meta-analysis if missing studies differ systematically from those included. Reviews can seek unpublished evidence and examine sensitivity to selection. Publication bias also distorts researchers' incentives, encouraging p-hacking and HARKing to produce publishable positive results. Registered reports, which obtain an initial publication decision before results are known, are one safeguard. Reporting completed studies and seeking unpublished evidence also address selective availability; no single procedure removes every source of bias.
Applied example
In a hypothetical example, a company conducts ten studies. Three show significant effects and are published in major journals. Seven do not reach the chosen significance threshold and are filed away. Readers see only favorable studies. Counting significant versus nonsignificant studies is still not a sufficient estimate of the treatment effect; magnitudes, uncertainty, study quality, and missing evidence also matter.
Why it matters
Franco and colleagues’ study of 221 completed survey experiments directly compared published and unpublished work. It identified authors’ decisions not to write up null findings as an important source of missing evidence in that program.
How selective visibility changes a decision
In the ten-study example, the three favorable reports might persuade a team to buy the intervention. The hidden studies matter because readers are choosing from a selected set of results, not the full set of attempts. But “seven nonsignificant studies” is not the same as seven measured effects of exactly zero. Recover their estimates and uncertainty before deciding how much the overall conclusion changes.
This distinction also separates publication bias from selective outcome reporting. An entire study can remain unpublished. Or its paper can appear while leaving out the disappointing outcome or follow-up. A review of short-term sign-ups, for example, may look favorable even though the same trials measured later attendance and did not report it. The review needs the outcome relevant to the decision, including missing results within otherwise available papers.
Can publication bias be corrected?
Researchers can search study registers, compare protocols with reports and request missing results. Funnel plots can reveal patterns in effect estimates and precision, but asymmetry can also arise from genuine differences between studies or other biases. A symmetrical plot does not prove that all studies were reported. Cochrane's missing-evidence guidance treats these as evidence to interpret together.
Selection models and other sensitivity analyses estimate what a synthesis might look like under specified assumptions about missing evidence. They do not recover the unseen studies as observed facts. If conclusions change sharply under plausible assumptions, report that dependence. If they stay similar, explain which assumptions were tested. Registered Reports address an earlier step by arranging review and a conditional publication decision before the main results are known.
The review of The Psychology of Money applies this distinction to a dispute about the effects of financial education.