How well do nudges work at scale? | The Behavioral Scientist

A short message, a different default or a simpler form can sound like an inexpensive solution to a large problem. The appeal is easy to understand. If a small design change could reliably move behavior, organizations could avoid much harder changes to the service, the incentives or the activity itself.

The evidence is less convenient. Some interventions produce useful changes, but results from published academic studies can create inflated expectations. A large comparison of government nudge trials found substantially smaller average effects than a comparison sample from academic journals. That finding supports skepticism about generic promises; it does not establish that every nudge fails. DellaVigna and Linos, 2022.

What the large comparison found

Stefano DellaVigna and Elizabeth Linos assembled 126 randomized controlled trials, involving more than 23 million people, from two US nudge units. Unlike a collection of journal articles, the units' records included trials whose results had not necessarily been published.

The average effect in the nudge-unit portfolio was an increase of 1.4 percentage points, compared with 8.7 percentage points in the academic-journal portfolio. These are absolute differences in measured behavior, not relative percentage increases. The researchers examined differences in statistical power, intervention characteristics and selective publication as explanations for the gap. They estimated that selective publication in the academic journals, made worse by the low statistical power of many academic trials, explains about 70 percent of the difference; differences in the kinds of nudges tested account for most of the rest. Study and comparison.

This was a comparison between portfolios. It did not follow the same interventions as they grew from small experiments into large programs. So the result cannot show that making any particular intervention larger will reduce its effect by a fixed amount.

The practical warning still stands. An organization planning from the academic average of 8.7 percentage points would expect far more than one planning from the government-trial average of 1.4. The most visible results in a literature are not automatically a realistic forecast.

A percentage point is not a percent

Suppose, as a hypothetical example, that 20 percent of a group completes a form. An increase of 1.4 percentage points would bring completion to 21.4 percent. The relative increase would be 7 percent: 1.4 divided by 20.

For 10,000 people, that difference would correspond to 140 additional completions, if the same effect applied. Whether that is valuable depends on the cost of the intervention, the value of completion, possible harms and the alternative use of the resources. The calculation is an illustration, not a prediction from the study for a new program.

This is why “small” does not mean “worthless.” It is also why a small improvement cannot justify any cost. The relevant comparison is with what the organization could otherwise do, not with an imagined world in which the intervention costs nothing and has no alternatives. The lesson Read the evidence before the promise works through the same distinction with an invented example: a reminder that raises attendance from 20 to 22 percent.

Why the broader debate remains unsettled

A separate meta-analysis by Stephanie Mertens and colleagues combined research on choice-architecture interventions across several domains. The paper received a formal correction after problems were identified in its included observations and coding. The revised estimates did not change the pattern of results or the authors' conclusions. The correction is part of the paper's record, so any description of its findings should mention it. Original synthesis; 2022 correction.

Maximilian Maier and colleagues challenged the evidence for a pooled effect after adjusting for publication bias. The original authors' response acknowledged publication bias and unexplained variation while arguing for more differentiated research. These papers address a separate dataset and analysis from the nudge-unit comparison. Maier and colleagues; Mertens and colleagues' reply.

Publication bias means that results are not equally likely to appear in the visible literature. If striking positive findings are easier to publish than small or null findings, an average calculated from published papers can be too optimistic. Statistical adjustments try to address that problem, but they depend on assumptions. One adjusted average is not an experiment testing every intervention in every setting.

A 2025 second-order meta-analysis by Bo Hu and colleagues pooled 14 earlier meta-analyses, covering 1,638 studies. The pooled estimate was d=0.27, a small effect expressed in standard-deviation units. One bias-adjustment method, PET-PEESE, reduced it to d=0.004, essentially zero. Another, a robust Bayesian model-averaging approach, left it at about d=0.29. The authors also rated 12 of the 13 source reviews as low or critically low in quality under a standard appraisal checklist. Hu and colleagues, 2025.

That paper sharpens the problem rather than settling it. The included meta-analyses define “nudge” differently, their results vary enormously, and two of the 14 inputs are the samples described above: DellaVigna and Linos's academic-journal sample and their nudge-unit sample. Pooling weak reviews cannot produce a strong answer, and an average across defaults, reminders, social comparisons and simplified forms says little about any one of them. What it does show is how much the headline estimate depends on assumptions about the studies that never reached publication.

The site's article Do Nudges Work? What the Evidence Can Actually Tell Us takes the wider question. It adds a flu-vaccination text megastudy and a review of social norms messages, and sets out how to decide whether a nudge is worth testing.

Ask what actually changed

The word “nudge” covers interventions with different demands and purposes. Prompting someone to finish a nearly completed task is a different challenge from keeping them at an activity they dislike or cannot fit into their life. A rise in initial enrollment does not, by itself, show lasting participation.

Before adopting an intervention, state the outcome that would make it worthwhile. If the purpose is attendance, count attendance rather than clicks on a reminder. If the purpose is sustained participation, include enough follow-up to observe it. If more activity at one location could simply displace activity from another, measure the broader outcome as well.

Those are design questions, not additional findings from the portfolio average. They guard against a common mistake: treating a change in the measured response as proof that the underlying problem was solved.

A 2025 working paper by Silvia Saccardo, Hengchen Dai and colleagues, not yet peer reviewed, shows that mistake in a real trial. It reanalyzes the full sample of a UCLA Health trial of text reminders for COVID-19 vaccination, which Dai and colleagues first reported from early data in 2021 with larger effects. Dai and colleagues, 2021.

In the full sample, the first reminder raised vaccinations at UCLA Health within four weeks by 2.8 percentage points. Vaccinations anywhere in California over the same four weeks, measured through the state registry, rose by only 1.0 point. The authors read that gap as spatial substitution: some people who would have been vaccinated elsewhere went to UCLA Health instead.

A separate comparison points to timing. Over eight weeks, the statewide effect shrank by half and was no longer statistically distinguishable from zero, which suggests the reminders partly moved vaccinations earlier rather than adding new ones. The same paper reports smaller effects among people with low baseline motivation to act. Saccardo, Dai and colleagues, 2025 working paper.

The site's Bad News for Nudges explains the broader concern about generic behavioral fixes. The productive response is to examine the person, behavior and setting closely enough to choose a plausible intervention, then test the outcome that matters. A familiar theory does not excuse a proposed solution from that work.

For a focused case in which both replication results and a retraction changed the evidence, see signing first and honesty.

What the large comparison found

A percentage point is not a percent

Why the broader debate remains unsettled

Ask what actually changed

Key sources