“It works” leaves the important question unanswered
Once you have chosen a goal, you will find plenty of advice about how to reach it. A reminder will get people to follow through. A different message will make an offer more appealing. A small change to a form will increase sign-ups. These claims may come with research, but you still need to ask what the researchers actually found.
Consider a study showing that reminders increased registrations for an event. That result concerns registration. It does not, by itself, tell you whether more people attended, enjoyed the event, or came back the following month. Each is a different outcome, and a finding about one cannot simply stand in for the others.
This is the distinction from the first lesson applied to research: the behavior someone counted may be several steps away from the benefit you want. Before evaluating how impressive a result is, identify the people studied, the action they took, the comparison used, and the period over which researchers observed them.
Translate the effect into something you can picture
An effect size describes the magnitude of a difference or relationship. For an outcome such as event attendance, the most useful starting point is often the difference in attendance rates between the groups. Statistical significance concerns a different question. Researchers may begin with a model in which the reminder makes no difference. A p-value describes how unusual the observed difference, or a more extreme one, would be under that model and its assumptions. It does not tell you whether the difference is large enough to matter.
Here is an invented example. Suppose 20 out of every 100 invited people attend without a reminder, compared with 22 out of every 100 with a reminder. The absolute increase is 2 percentage points: 22% minus 20%. The relative increase is 10%: the additional 2 attendees divided by the original 20. Both descriptions are correct, but “10% more attendance” can create a very different impression from “two extra attendees per 100 invitations.”
That translation helps you make a decision. If the reminder takes a minute to set up and the additional attendance matters, the change might be worthwhile. If it requires a new system, staff training, and ongoing administration, the same increase may be a poor use of resources. The effect size gives you something to weigh against the cost; the word “significant” does not.
The American Statistical Association makes the same point in its guidance on p-values. A p-value is neither a measure of practical importance nor the probability that a claim is true. Read it alongside the estimated effect, the study design, and the uncertainty around the estimate.
American Statistical Association: interpreting p-values
If you have two rates to compare, the practical-effect calculator shows what their difference would mean for a group of any size. For real studies, the behavioral economics examples show why counting sign-ups answers a different question from tracking what people do next.
Ask how much uncertainty remains
A study observes a particular set of people, not every person who could ever encounter an intervention. Its estimate can therefore differ from the result in another sample. A confidence interval is one way researchers express uncertainty around an estimate, under the assumptions of their analysis. A wide interval means the data are compatible with quite different effect sizes. If it extends from a small decline to a useful improvement, you still have considerable uncertainty about whether the intervention helps. A narrow interval around a small improvement gives you a clearer picture, though you still need to decide whether that improvement is worthwhile.
Small samples often leave a decision unresolved because a few unusual observations can make a large difference. Imagine judging a reminder after inviting ten people. One additional attendee changes the attendance rate by ten percentage points. You would need to be careful about treating that change as a dependable estimate of what the reminder does.
A large sample helps with precision, but it does not repair a misleading comparison. If one group consists of people who chose to sign up early and another consists of people who ignored the invitation, their later attendance could differ for reasons that existed before any reminder. Random assignment can help separate the intervention from those pre-existing differences. You still need to check what was measured and whether the comparison answers your question.
How much do nudges help in practice?
A concrete reason for skepticism comes from Stefano DellaVigna and Elizabeth Linos’s study of 126 randomized trials run by two US nudge units, involving more than 23 million people. In the sample they analyzed, which excluded interventions that changed defaults or offered monetary incentives, the average increase in take-up of the targeted action was 1.4 percentage points. In their comparison sample of academic studies, the average was 8.7 percentage points.
These were different collections of interventions, not identical interventions repeated at a larger scale. The comparison therefore does not mean that every academic result shrinks by a fixed amount when an organization uses it. It does show why reading only the most promising published studies can leave you expecting larger improvements than these government nudge units achieved in their trials.
A 1.4-percentage-point gain can matter when an inexpensive intervention reaches a large population. It is a less compelling answer to a problem that needs a substantial change. For an organization with limited time and money, the decision is not merely whether an effect exists: it is whether pursuing it is better than improving the underlying service, removing a real obstacle, or choosing a different behavior to support.
Look beyond the first successful study
An independent replication asks whether researchers can obtain a finding again with new data. It matters because an original result can reflect a real effect, ordinary sampling variation, choices in analysis, or some combination of these. You have more reason to trust a result when other researchers find it too, using careful methods.
In 2015, the Open Science Collaboration reported replications of 100 studies from three psychology journals. On average, the replication effects were about half the magnitude of the originals. This was not a random sample of all psychological research, so it cannot provide a failure rate for the entire field. It is nevertheless a strong reason to question confident promises based on an isolated result.
Look for preregistration as well: a record of the study’s planned methods and analyses made before researchers know the results. This makes it easier to distinguish a prediction tested as planned from an interesting pattern discovered afterward. Neither preregistration nor replication makes a study perfect, but both can make its claims easier to evaluate.
A failed replication is different from a correction or a retraction. A correction changes part of the published record; a retraction withdraws a paper, with the notice explaining why. Check the replication report or the correction or retraction notice to see what happened and which conclusions it calls into question.
Open Science Collaboration (2015): Estimating the reproducibility of psychological science
Bring the finding back to your decision
Evidence becomes useful when you connect it to the problem in front of you. A study of event registrations might give our friends a reason to send a reminder if people are forgetting to reply to their dinner invitation. If their obstacle is incompatible schedules, the same finding offers little help: remembering an invitation does not create a free evening.
If you’re wondering whether a regular walk would help you spend more time with a friend, you can invite them and see how it goes. It costs little to try, and you can do something else if it doesn’t suit you. Before spending a large budget on a program for hundreds of people, though, you should ask for stronger evidence that it will help. In either case, be clear about what you know from research and what you still need to find out.
Would this reminder help?
Suppose you read that a reminder led to “10% more attendance.” That sounds promising. But in our invented example, attendance rose from 20% to 22%: two additional attendees per 100 invitations, not ten. Knowing the actual rates gives you a clearer picture of how much the reminder helped.
Before sending your friends another reminder, ask why the dinner hasn’t happened. If people want to come but keep forgetting to reply, a reminder might help. If they’re busy on Friday evenings, another message won’t make them free. You would have more reason to suggest a different time or a different way to catch up. The study tells you how much a reminder helped in one setting; you still need to work out whether forgetting is the problem in yours.
Check your understanding
A reminder raises attendance from 20% to 22%. Does that mean ten additional people attend for every hundred invited?
No. It means two additional attendees per hundred invitations: an increase of two percentage points. Ten percent is the increase relative to the original twenty attendees. You still need to consider how reliable the estimate is and whether that improvement would justify the cost.
What actually changed?
“The program worked” can mean that more people enrolled, more people did something, or people benefited. Those are different claims. As you read each case, ask what the study measured and which conclusions the results support.
These cases are fictional. They let you practice reading a result; they are not evidence about how often behavioral economics works.
1. More people open the first lesson
A training provider tests two welcome emails. The new one is shorter and has a prominent Start button; the other is its usual email. It randomly assigns 2,000 recent signups to one email or the other, then tracks everyone for seven days.
In the shorter-email group, 46% open the first lesson; in the usual-email group, 40% do. That is a difference of 6 percentage points. The study reports a 95% confidence interval from 2 to 10 points: a range of effects consistent with the data and the assumptions used to analyze them. The estimate is uncertain, but the whole interval is above zero.
The provider does not check whether anyone finishes a lesson, learns the material, or returns later. Its announcement says the email “builds a lasting learning habit.”
2. When the study really measures repeated action
An employer offers a six-month skills program to 4,000 employees who agree to be in a study. It randomly assigns them to two groups. One group gets recurring practice sessions booked automatically and can cancel them; employees in the other group book their own sessions.
The study records both enrollment and completed practice sessions for every employee. When the study checks enrollment, 80% of those offered automatic booking are enrolled, compared with 50% of those asked to book their own sessions. More people also complete at least one session in each of the six months: 38% versus 32%. That is a difference of 6 percentage points, with a 95% confidence interval from 3 to 9 points.
When analyzing the results, the researchers count everyone in the group they were originally assigned to, even if they canceled. The study does not test skills or track what happens after the program ends.
3. A real measure, an uncertain difference
A small nonprofit randomly assigns volunteers to receive weekly practice reminders or none at all. It tracks everyone for 12 weeks and counts how many people complete a practice session in at least 10 of those weeks.
The rates are 33% with reminders and 31% without them. That is a difference of 2 percentage points, with a 95% confidence interval from −4 to +8 points.
Before the trial, the nonprofit decided it would need an increase of at least 5 points to make the reminders worth the staff time. It has complete records for all volunteers.
Try it yourself
Find a behavior-change claim that includes a study. Write down who participated, what was compared, what outcome was measured, the size of the difference, and how long people were followed. Mark information you cannot find as unknown. Then explain, in ordinary language, what you would expect to improve if you tried it yourself, and what the study still cannot tell you.
Read further
- Effect size
- Replication crisis
- Publication bias
- Behavior-change models compared
- How well do nudges work at scale?
- Evidence Library: which behavioral science findings hold up?
A study can help you judge a possible action. To choose well, you also need to understand the particular person who would carry it out and the circumstances in which it would happen.