Behavioral Economics Examples: What Actually Changed?

Say your company wants employees to exercise more. One colleague brings a study showing that a reminder increased gym attendance. Another suggests signing everyone up for a wellness program automatically. Both are pitched as ways to change behavior.

But being enrolled in a program and working out are different things. Automatic enrollment can change the first without anyone lifting a weight. A reminder might get someone to the gym once without making the next visit any easier. So before you choose a technique, decide which problem you are trying to solve.

That is my concern with applied behavioral economics: a plausible story about why a technique should work can pass for evidence that it will solve your problem. The four cases below check that story against what people actually did, how much changed, and whether the change served the goal. They cover automatic enrollment in retirement plans, text reminders for COVID-19 vaccination, 53 programs to increase gym visits, and 126 trials that two US nudge units ran with federal agencies and local governments. These four cases are a selection, not a representative sample, so they cannot tell you how often behavioral interventions succeed.

1. Retirement defaults: joining a plan is only the beginning

A default is the option that takes effect when you make no active choice. With automatic retirement enrollment, part of each paycheck goes into a retirement account unless the employee chooses to stay out.

Suppose you want to help employees prepare for retirement. Automatic enrollment looks promising because it removes a step: nobody has to fill out a form before contributions begin. But joining is only the start. How much goes into the account, and how the money is invested, also shape the balance.

James Choi, David Laibson, Brigitte Madrian and Andrew Metrick (2004) studied this at three US employers that switched to automatic 401(k) enrollment. They used company records to compare employees hired under the different enrollment rules; nobody was assigned to a rule at random. Under automatic enrollment, more than 85% of employees at all three companies were in the plan, however long they had worked there. Before the switch, 26% to 43% had joined after six months on the job, and 57% to 69% after three years.

The default also shaped how people saved once they were in. Most new participants kept both default settings: 65% to 87% saved at the default rate (2% or 3% of pay, depending on the employer) and invested only in the fund their employer had selected.

The researchers looked past enrollment too, at contributions and account balances. Here the picture was less dramatic. Counting employees outside the plan as contributing nothing, average contribution rates rose only modestly. The authors point to the default itself: it brought people into the plan, but it also held many of them, at least for a while, at a low contribution rate.

Balances were harder to compare. Employees hired before the switch had enjoyed a much stronger stock market than most of those hired after, so a difference in their accounts could not simply be credited to the enrollment rule.

The contributions are still real. Automatic enrollment can put money into an account month after month without a new decision each time. That is useful. But it does not show that employees have learned to make better financial decisions, or that a household is wealthier once its other savings and debts are counted.

So if you are choosing a retirement plan for employees, ask more than how many will join. Will they put away enough? What happens to the money when they leave the employer? Are they saving more overall, or moving money they would have saved somewhere else? Enrollment figures alone cannot answer those questions.

Later research has tested a related question: do people borrow more elsewhere to make up for the money going into the plan? When the US Army began automatically enrolling newly hired civilian employees in its retirement savings plan, John Beshears and colleagues (2022) used the change to compare employees’ credit records. Four years after hire, automatic enrollment had raised total contributions to employees’ plan accounts, including the employer’s, by 4.1% of first-year salary; the employees’ own contributions rose by 1.4%.

The researchers found no significant change in credit scores, in debts other than car loans and first mortgages, or in signs of financial distress such as late payments and balances sent to collection. The one possible exception was higher first-mortgage balances in foreclosure. Car-loan and mortgage balances may have risen, but the evidence was less clear. For credit scores and debts such as credit cards, that is a reassuring answer at one large employer. It does not tell you whether those employees saved enough for retirement.

2. Vaccination reminders: a booking is not a dose

In a trial at UCLA Health, Hengchen Dai, Silvia Saccardo and colleagues (2021) randomly assigned eligible patients to receive either a text reminder with a link to book a COVID-19 vaccination appointment or no additional reminder. They analyzed results for 93,354 people, mostly older adults. Within six days, the share who had booked an appointment was an estimated 6.07 percentage points higher in the reminder group.

But a booking is not a dose. So the researchers also counted actual first doses over four weeks, in two ways:

• Vaccinations at UCLA Health: the reminder group’s vaccination rate was an estimated 3.57 percentage points higher than the no-reminder group’s, or about 36 additional people vaccinated at UCLA per 1,000 participants.
• Recorded vaccinations anywhere, including UCLA: the difference between the groups was 2.1 percentage points, or about 21 additional people vaccinated per 1,000 participants. The reported standard error, 0.4 percentage points, measures how precisely that difference was estimated. It is a reminder that 2.1 is an estimate, not an exact answer.

Why would counting more vaccinations make the difference smaller? A UCLA-only count misses patients in both groups who got vaccinated somewhere else. Adding those records raises the vaccination rate in both groups. If it raises the rate more in the no-reminder group, the gap narrows. That is what happened here: the gap was smaller when every recorded vaccination counted.

One possible explanation is that the reminder partly changed where people got vaccinated. Picture someone who would have gone to a local pharmacy but instead followed the link and booked at UCLA. UCLA counts one more vaccination, while the number of vaccinated people stays the same. This illustrates the idea; the study does not show which individual patients switched. And the broader count still showed more people vaccinated overall.

Both estimates cover the same four weeks, so the effect did not fall from 3.57 to 2.1 points over time. Both count doses people actually received, not intentions or bookings. What separates them, as the study defines them, is place: a dose at UCLA, or a recorded dose anywhere. The UCLA difference was still there after eight weeks; the count of vaccinations anywhere covered only the first four.

So the reminder got more people a first dose, which is useful in this setting. It does not show a general solution to vaccine hesitancy, a lasting change in health habits, or the number of illnesses prevented.

Could a more persuasive message beat a simple reminder? Saccardo and colleagues (2024) tested that question in later booster trials at UCLA Health. Some messages explained what the booster added to earlier vaccines or who could benefit. Others praised patients for completing their initial vaccinations, then encouraged them to get the booster. The idea was to make the next vaccination feel consistent with a choice they had already made.

Across three trials, reminders increased the share receiving a booster within four weeks by an average of 1.13 percentage points. The 95% confidence interval, from 0.83 to 1.43 points, shows the uncertainty around that estimate: the data supported a range of possible increases, not one exact value.

In a separate online experiment, people rated the extra information and praise as more persuasive than a basic reminder. But when the researchers sent those messages to patients and checked vaccination records, they did not find clear evidence that the added wording got more people vaccinated than the basic reminder. A message that sounds more convincing need not get more people to act.

For a team writing messages, this argues for testing a simple reminder first, then checking whether a more elaborate message actually gets more people vaccinated.

3. Gym attendance: a few more visits are not a new way of life

A vaccine dose is one action. Exercise has to be repeated, so the question becomes whether a program can keep people coming back to the gym. Katy Milkman, Angela Duckworth and colleagues (2021) tested 53 four-week programs with 61,293 members of the 24 Hour Fitness gym chain. Each program was compared with a control group: members who received a reward for registering in the study but none of the ongoing help. The basic program combined planning, reminders and small rewards.

The basic program increased attendance by an estimated 0.14 visits per person per week, with a 95% confidence interval of 0.04–0.23. Members of the control group averaged 1.48 visits a week during those four weeks, so that is an increase of about 9%. To picture it, imagine 100 members: about 14 additional visits between them each week, not 14 extra visits for every member. Some people might account for several of those visits while others make no extra visits.

One of the best-performing programs added a bonus for returning after a missed workout: 125 points, worth about 9 US cents. Compared with the control group, that whole program produced about 0.40 extra visits per person per week (about 40 additional visits across 100 members), with a confidence interval of 0.21–0.59. That does not make the bonus worth 0.40 visits. The 0.40 figure compares the full program with the control group, so it also includes the planning, reminders and other rewards. To find out what the bonus added, you would compare the program with the bonus against the basic program without it.

What happened when the programs stopped? During the four weeks of the programs, 45% of them produced an increase in gym visits that met the researchers’ test for statistical significance. In the four weeks after the programs ended, only 8% did. That test asks whether a difference is strong enough to count against an explanation based on chance variation alone. It does not tell you whether the increase is large enough to matter.

So most programs showed no clear continuing increase once they ended. That makes lasting change hard to promise. It does not mean the other programs had no effect: some estimates were too uncertain to rule out modest gains. And when the researchers combined results across programs, they still found positive effects during ten weeks of follow-up.

The records show that people checked in at the gym. They do not tell us how much exercise people did once inside or whether going to the gym became automatic. Those questions need their own evidence.

A few extra visits can be valuable. But if the goal is regular exercise, you need to know whether people keep going long enough for it to matter. You also have to ask whether the gym is something these members can and want to do, or whether another kind of exercise would fit them better. A program can be good at getting people through the gym door without being the best way to help them stay active.

4. Government nudges: how big is the effect at scale?

Suppose you are considering a reminder campaign to help people complete an application or pay a bill. Several published studies report large gains. Is that what you should expect from your own campaign?

Stefano DellaVigna and Elizabeth Linos (2022) offer one way to check. They gathered 126 randomized trials run by two US nudge units, teams that test changes such as reminders and simpler letters in government services. One, the Office of Evaluation Sciences, works with federal agencies. The other, the Behavioral Insights Team’s North America office, worked mainly with city and other local governments. In a randomized trial, chance decides who receives each version, so a difference between the groups is less likely to come from who was in them. The authors compared these government trials with 26 trials from academic journals.

The trials measured different actions, such as enrolling in a program or making a payment, and did not all use the same time period. The authors excluded interventions that changed defaults or offered financial incentives, such as payments for taking an action. So the comparison does not cover every approach on this page; the retirement defaults above fall outside it.

The gap was large. On average, the government interventions raised the share of people who took the measured action by 1.4 percentage points. The academic trials averaged 8.7 points, more than six times as much. Had you planned a program around the academic average, you would have expected a much larger improvement than the government units achieved on average.

Why was the academic average so much larger? The authors traced most of the gap to which trials get published. By their estimate, a trial without a statistically significant result was about one-tenth as likely to be written up and published as a trial with one.

The academic trials were also much smaller: the typical treatment group had 484 participants, compared with about 10,000 in the nudge units’ trials. In a small trial, only a large estimate is likely to reach significance, so filtering for significance also filters for large estimates. Together, selective publication and low statistical power explain about 70% of the gap, by the authors’ estimate. Differences between the interventions, such as their topics and the kind of behavioral change they attempted, account for most of the rest.

These were different interventions tested with different people. The study did not follow any single program from an academic trial into government use, so it cannot tell you how much that program’s effect would change. For a new campaign, the government average is the better starting point, because the nudge units’ trials were counted whether or not their results were impressive.

Neither average is exact. The authors reported a standard error of 0.3 percentage points for the government average and 2.5 for the academic average. A standard error tells you how precisely an average was estimated; the larger number means more uncertainty. It does not tell you how much the individual interventions differed from one another. And even a precisely estimated average gives you a starting expectation, not a forecast of what a particular new project will achieve.

What would a 1.4-point improvement mean in practice? Take a hypothetical program in which 20% of eligible people complete an application. The 20% is an illustration, not the study’s average application rate. Raising that rate by 1.4 percentage points brings it to 21.4%. Among 10,000 eligible people, that would mean 140 additional completed applications, if the same improvement held for that group.

You could describe the same change as a 7% relative increase, because 1.4 divided by the starting rate of 20 is 0.07. That sounds more impressive, but the number of additional applications is still 140. Counting applications rather than percentages makes it easier to ask whether the result is worth paying for. The practical-effect calculator does this arithmetic with your own numbers.

The answer depends on what those extra applications achieve. If a low-cost change to an existing message produces 140 useful applications, it might be worth making; the paper’s authors make that argument for inexpensive communications. If the same gain requires substantial staff time or a costly new system, the decision could be different. And if the real goal is for people to receive a benefit or find work, you still need to know whether those extra applications lead to that result.

My concern is the expectation being sold. A small improvement can be useful. It is a poor basis for promising that a minor psychological adjustment will solve a large problem. Before choosing a reminder campaign, compare its likely benefit and cost with the other things you could do: simplify the application, improve the service, or remove an obstacle that a better-worded message cannot fix.

Which result do you actually need?

Before you borrow a technique from a study, decide which of these results you need, then check that the study measured it:

If your goal requires…	Ask for evidence of…
Enrollment in a program	Whether people are enrolled, what happens after they join, and which decisions are made for them
An action completed once	Whether people do it, rather than just click a link, say they intend to do it, or book an appointment
An activity people keep doing	Whether they continue long enough to meet the goal, especially after a short-term reminder or reward ends
A benefit such as better health or greater savings	Whether the study measured that benefit itself or only an earlier step, such as joining a plan or checking in at a gym

The standard is not that every intervention must transform a life. If a cheap reminder produces a worthwhile increase in a one-time action, it has done that job. The problem starts when that result is sold as evidence for a much larger promise.

Before using an example to justify a project, write one sentence: “Compared with ___, this changed ___ by ___ over ___.” If you cannot fill it in, you do not yet know enough about the result to judge whether it would help with your problem. Then state how large an improvement would make the work worthwhile, given its cost and the other ways you could pursue the goal. No single percentage-point cutoff can tell you what is worth doing in every situation.

Try the distinction with a fictional case. A new reminder increases signups for a walking group, but the organizers do not record attendance. Can they claim more people are walking? They can claim more signups. They need attendance or activity data to support the stronger statement. That same gap appears whenever a study measures the first step and calls it the final result.

For a closer look at comparisons and uncertainty, continue with how to read behavioral evidence. For my broader argument about the practical limits of nudges, see Do Nudges Work?. For the field’s history and core ideas, read what behavioral economics is. The behavioral economics books guide compares 13 books and the evidence behind their claims.

References