Guide

How to Run a Sludge Audit of a Public Service

Published 22 min read

Your program has money left in the budget and a list of people who started an application and never finished. Some of them were probably ineligible. Some had their problem solved another way. But some were eligible, needed the help, and gave up somewhere between the eligibility page and the decision letter.

Which steps pushed them out? And which of those steps could you change without weakening the checks the program needs?

A sludge audit is a structured way to answer those questions. It is a kind of behavioral audit focused on burden. You follow the service the way an applicant experiences it and measure what each step costs them. Then you ask what each step is for, and test a better version of the steps that fail that test.

None of the method here is new. The steps build on the NSW Government Sludge Audit Method, developed by the New South Wales Behavioural Insights Unit. The 2024 OECD and NSW report Fixing frictions: ‘sludge audits’ around the world describes it, after 16 government organizations from 14 countries tested it in 2023. Impact Canada publishes a similar five-phase approach that ends with testing changes before scaling them. The Behavioural Insights Team offers sludge audit training for teams, and Australia’s BETA publishes a guide to improving government forms.

Below, one journey is audited step by step, with the reasoning shown.

The running example (hypothetical)

Everything about this program is invented, and its figures are hypothetical audit results, not findings from a real service.

A county runs a rent relief program. Tenants who are behind on rent and earn below an income limit can apply for a one-time payment, which the county sends directly to the landlord. Dana manages the program. Her team has noticed three things: many online applications are started and abandoned, many submitted applications are denied as “incomplete,” and people often call to ask where their application is.

The journey has six stages:

  1. Finding the requirements. The tenant learns the program exists and works out whether they qualify.
  2. Gathering documents. Photo ID, the lease, a landlord ledger showing the unpaid rent, a landlord tax form, and 30 days of pay stubs or benefit letters.
  3. Applying. An online form, or help at one walk-in office open weekdays from 9 to 3.
  4. Correcting an error. If anything is missing, the county mails a request for information with a 10-day deadline, set by program policy. No response means a denial for an incomplete application.
  5. Receiving a decision. A letter, sent by mail.
  6. Appealing. A written appeal within 15 days, with all documents submitted again.

What a sludge audit is for

Administrative burden is the cost people experience when they deal with a public program. Donald Moynihan, Pamela Herd and Hope Harvey divided it into three kinds. Learning costs come from finding out about a program, whether you qualify and how to apply. Compliance costs come from completing forms, producing documents and meeting requirements. Psychological costs include stigma, stress and a sense of lost autonomy.

Sludge is narrower. Richard Thaler popularized the term in a 2018 Science editorial. Cass Sunstein, in his paper Sludge Audits, defines it as excessive or unjustified friction. He is explicit that administrative burdens can be excessive, insufficient or about right; only the excessive ones are sludge. He expanded the argument for regular audits in his 2021 book, Sludge.

That distinction is the point of the audit. In Dana’s program, the ID check and the landlord tax form both impose compliance costs, so both are administrative burdens. Whether either is sludge depends on what it protects, whether it actually protects it, and whether a lighter version would protect it just as well. An inconvenient step is not automatically a needless one.

Step 1: Decide whether an audit fits, then set the scope

Before committing staff time, ask the screening questions the OECD report sets out, plus one point it makes about legal limits:

  • Is there a clear outcome people are trying to reach through the service?
  • Is there evidence that friction is causing bad outcomes, or only a hunch?
  • Can you get data and talk to the people who use the service?
  • Can you change what you find? Some requirements are set in law, and changing them is a political process, not a design decision.

Then define the outcome the service exists to produce. For Dana, it is not “submitted applications” but “eligible tenants receive a payment before they lose their home.” That wording tells you what to count later: a change that doubles submissions but pays the same number of eligible tenants has not helped. The lesson Start with the outcome makes the same distinction between a result and a route to it.

Set the scope wide enough. The OECD report points out that a standard process map often starts at the first step of the application process, while a sludge audit starts when the person begins looking for help. Dana’s audit runs from “a tenant hears about the program” to “the appeal is decided.”

Step 2: Map the journey as applicants live it

The NSW method, as the OECD report describes it, calls this a behavioral journey map. Write down every small action a person takes. Include the ones where they are not dealing with the agency at all: waiting, asking a landlord for a document, finding a printer, taking a bus to the office.

For Dana’s program, stage 2 alone contains six actions:

  • work out which documents count as proof of income;
  • ask the landlord for a ledger;
  • ask the landlord for a tax form;
  • wait for the landlord to reply;
  • photograph or scan each document;
  • convert the photos into an accepted file type.

Each of these can fail, and none of them appears on the county’s own process map.

Real journeys branch and loop: online, walk-in, with help from a tenant advocate, or through the error loop twice. The OECD report recommends mapping these branches rather than forcing one straight line, and some teams in the 2023 International Sludge Academy ran separate audits for different groups of users.

Map assisted access deliberately. In Dana’s program, that means at least three journeys:

  • a tenant with a smartphone and reliable internet;
  • a tenant who prefers a language other than English;
  • a tenant who can only apply in person or by phone, perhaps because of a disability, no internet access or a job that allows no calls during office hours.

A step that takes five minutes on one journey can be impossible on another.

Step 3: Collect evidence about what people actually do

For each step on the map, the NSW method asks four questions. How many people complete it? How long does it take? How do people experience it: what is easy, confusing or stressful? And what do people do when they get stuck?

Use the data you already have first: timestamps, the point where people abandon the online form, call logs, denial reasons and complaints. Then fill the gaps: observe a few people completing the process, interview recent applicants and talk to frontline staff.

Ask about what people did, not why they think they did it. “Walk me through the last time you tried to get the landlord ledger” produces more reliable information than “Why didn’t you finish the application?” People are better at describing what they did than explaining why, and records are better still: ask to see the letter they received. I made this argument about product research in Why Surveys & User Interviews Are (Almost) Always A Waste Of Time, and it applies just as much to applicants for a public program.

Talk to people who did not finish. Drop-off data tells you where people stopped, not why. In the OECD report, a French team expected the worst friction at the start of an aid application; feedback from applicants showed it was concentrated in the final steps.

Step 4: Measure the burden at each step

Now record, for every step, what it costs the person and who it loses. The burden audit worksheet has one card per step. Start with these fields:

  • What the person must do. The action in plain words.
  • Time. Active time plus waiting time. Record a range if it varies widely.
  • Documents. What they must find, request or produce.
  • Learning cost. What they must find out or understand.
  • Compliance cost. What they must complete, submit or attend.
  • Psychological cost. Stress, stigma, confusion or loss of control.
  • Who drops out. Completion at this step, overall and by group.

Together, the first six fields describe the step’s friction costs. The seventh tells you who those costs exclude. Summed across a whole journey, the time and paperwork are what journalist Annie Lowrey calls the time tax.

The NSW method rates effort and experience for each step on a five-point scale, from easy to very difficult, using published sludge scales; the worksheet has a field for that rating too. NSW also publishes free versions of its audit template for people outside NSW, a spreadsheet called Sludge Finder Lite and a one-page Sludge Finder Mini, on its Sludge Toolkit page. This guide’s worksheet follows the same steps and adds the safeguard questions and a test plan.

The method deliberately avoids a single sludge score, so that service owners face the tradeoffs between time, experience and inclusion directly.

Here is part of what Dana’s team finds. These are hypothetical audit results.

  • In the audit month, 1,000 people start the online or walk-in application. 640 submit it. You cannot count people who read the eligibility page and never started, so the first number already understates the loss.
  • Among the 880 who start online, 556 submit (63 percent). Among the 120 who start at the walk-in office with staff help, 84 submit (70 percent).
  • Gathering documents takes a median of six days. The landlord ledger and landlord tax form are the slowest items, and interviews describe asking a landlord for paperwork while facing eviction as stressful.
  • Of the 640 submitted applications, 288 (45 percent) receive a request for missing information. The mailed letter lists form codes rather than naming the documents.
  • Of those 288, 86 miss the 10-day deadline and are denied as incomplete. Half of the applicants who listed a language other than English are denied this way (30 of 60), against about a quarter of everyone else (56 of 228).
  • The median wait from submission to decision is 38 days. Status calls make up a large share of the phone line’s volume.

Pay most attention to who drops out, not just how many. A study by Manasi Deshpande and Yue Li shows why. When Social Security Administration field offices closed, the number of disability recipients in surrounding areas fell by 16 percent, while applications fell by only 10 percent. The largest effects were among applicants with moderately severe conditions and low levels of education. Judged by the program’s own eligibility standards, the extra burden screened out people the program was designed to serve.

Moving a service online is not automatically a fix; a 2026 systematic review by Maja Hegemann and Adrian Ritz notes that digital services can create burdens of their own. In Dana’s figures, the walk-in office keeps more of its applicants than the online form. That gap could reflect the staff help or the people who choose each route; the audit shows where to look, not the cause.

Step 5: Separate safeguards from sludge

Some friction does a job, so this step decides whether your audit improves the service or simply strips out its checks. Sunstein lists the legitimate reasons for burdens: preventing mistakes and impulsive decisions, protecting privacy and security, collecting data the program needs, and checking eligibility. The most obvious explanation, he notes, is program integrity: making sure benefits go only to people entitled to them.

Every eligibility check trades one error against another: paying people who are not eligible against failing to pay people who are. Sunstein’s point is that the right balance is a judgment on which reasonable people will disagree.

Herd and Moynihan describe three situations worth separating. A burden can deter fraud without stopping eligible people, which is easy to justify. It can deter some fraud while also stopping many eligible people, which forces a tradeoff. Or it can be justified by a goal it does nothing to achieve.

For each step, ask five questions:

  1. What is this step for? Name the specific purpose. “Compliance” is not a purpose. “Confirm the payment goes to a real landlord” is.
  2. Is there evidence it achieves that purpose? Pull last year’s records: how many problems did it catch?
  3. Who does it stop? Mainly ineligible applicants, or eligible ones too?
  4. Is there a lighter way to achieve the same purpose, and what would it miss? The second of Sunstein’s three proposed reforms is to choose the least burdensome method that works. Herd and Moynihan add a specific option: shift the burden from the applicant to the agency, for example by checking records the government already holds.
  5. Does the applicant understand why the step exists? The NSW guidance, as the OECD reports it, notes that a wait may not be sludge if people understand why it is needed or are kept informed while they wait.

Then classify the step: keep it as it is, keep its purpose but change the method, or remove it. Three of Dana’s steps show how the classification works; the worksheet carries the rest.

  • Landlord ledger and landlord tax form: keep the purpose, change the method. The purpose is to confirm the debt exists and the landlord is real. Suppose last year’s records show the check stopped 52 applications: 7 because the named landlord did not own the property, and 45 because the tenant could not get the forms back. It catches real fraud, and it also stops tenants whose landlords do not reply. The county can confirm ownership through property records and request the ledger and tax form from the landlord itself. Property records alone would miss whether the rent is owed, and the tax form serves the county’s own reporting of payments, so both documents stay. What changes is who collects them.
  • The request for information: keep the purpose and the policy deadline, change the method. A complete file before a decision is a real safeguard. A letter that lists form codes instead of documents, arrives in English only and depends on the post is avoidable friction. Some of the 86 applicants denied this way may not have been eligible; the test in Step 7 is built to show how many were.
  • Resubmitting every document to appeal: remove. The county already holds the documents. No purpose was found for asking again.

The audit’s conclusion was not that the checks should go. Most of the checks are needed, and several are carried out in a more burdensome way than they need to be.

Two mistakes are common at this stage. The first is treating drop-off as proof of sludge: a tenant whose landlord agreed to a payment plan has a good reason to stop applying. The second is counting success by the wrong measure. If a change raises completion but also raises payments to ineligible applicants, the service may be worse, not better. The behavioral friction entry makes the same point about shortcuts that increase completion but produce incorrect applications.

Step 6: Propose changes and choose one to test

List possible changes for each step that failed the safeguard questions. The OECD report describes three kinds: remove a step, improve it, or add support. Then prioritize by how much burden a change would remove and how feasible it is.

Simplification will probably dominate your list. When I asked a former member of the Obama administration’s behavioral science team about their work, every project they described involved making a bloated process smaller, quicker and easier. I wrote about it in The #1 behavior-change strategy used by Obama’s Behavioral Science Team.

Be wary of the cheapest option: a reminder or friendlier message bolted onto an unchanged process. In Stefano DellaVigna and Elizabeth Linos’s comparison of 126 trials run by two US nudge units, the average effect on the targeted behavior was 1.4 percentage points. If the friction is in the process itself, a message about the process is unlikely to fix it.

Dana’s team picks the request for missing information as its first test, for specific reasons. It is where 86 applications a month are denied without a decision on eligibility. It falls hardest on one identifiable group. It can be changed without altering any eligibility rule. And the safeguard it serves, a complete file before a decision, stays in place.

The team makes one fix standard before testing anything: the request goes out in the language the applicant chose. Withholding a translated notice from the group the audit found most harmed would not be an acceptable comparison, and language-access rules may already require it. The test covers the rest of the new version: a plain-language checklist naming the exact documents, sent by text and email as well as mail, with a call on day three. The 10-day deadline stays the same, because it is set by policy.

This is not a message added to a working process: the current letter fails to say what is required, so fixing it changes the step itself. Dana should still set expectations from the modest average above, not from hope.

Step 7: Test the change

The obvious test is to launch the new notice and compare this month with last month. That comparison is weak. Rent levels, staffing, the backlog and the mix of applicants all change from month to month, and any of them could explain a difference.

A stronger design is a field experiment: randomly assign applicants who receive a request for information to the current notice or the new one, during the same weeks, and compare the results. Random assignment makes the two groups comparable in expectation, so a difference in outcomes can be attributed to the notice. The research methods guide explains how to choose and plan this kind of design.

Write the plan down before the test starts:

  • Primary outcome. The share of applicants receiving a request for information whose application reaches an eligibility decision (approved, or denied on eligibility) rather than a denial for being incomplete. Measure it 60 days after the request, well past the program’s 38-day median, so slow reviews do not cut cases off. This outcome also shows how many of the rescued applications were eligible.
  • Secondary outcomes. Time from request to decision; time to payment; errors found in quality review; staff minutes per case; complaints.
  • Groups. Results by preferred language and by application channel.
  • A safeguard check. The share of approved cases flagged in the program’s existing post-payment review. If it rises, the change has weakened a check, and you need to know.
  • Unit of assignment. Assign individual applicants, and give the day-three calls to a small call team so caseworkers do not start calling the comparison group too. If you assign caseworker teams or weeks instead, the number of teams or weeks sets most of your statistical power; with only a handful, plan a much longer test.
  • Sample and duration. With about 290 requests a month assigned individually, two months of enrollment can detect an improvement of roughly 11 percentage points, but would likely miss a smaller one. Results arrive 60 days after the last request. Decide the smallest improvement worth acting on before you start, and plan the sample around it.

Check legal and ethical requirements, including consent for text and email contact. When a new process is untested, a time-limited comparison is often the responsible way to find out whether it works.

After the test, the NSW method, as the OECD report describes it, encourages a follow-up audit of the affected steps or the whole process. Measure the changed step again, the same way, and compare.

Try it: is this step sludge?

Dana’s appeal form asks the applicant to upload their photo ID again, even though the county verified it at the application stage. Is this sludge?

Most likely, yes. The purpose of the original ID check, confirming the applicant is who they say they are, has already been served, and the document is in the file. The burden falls on people who have just been denied and have 15 days to respond, which makes it costly at the worst moment. A lighter version achieves the same purpose: the caseworker confirms identity against the file, or the applicant signs in to their existing account.

Now change one condition. Suppose the appeal arrives on paper from a new address, with no account and nothing to match it to the verified file. Then a fresh identity check does protect something: it stops someone else from redirecting the payment. The step is the same; what it protects has changed. The question to ask is the same in both cases: what is this step for, and is this the lightest way to achieve it?

The next step

Most of the time, the answer an audit gives is not “remove the checks.” It is “keep the purpose, move the burden, and test the fix.”

Pick one service journey. Use the worksheet to map it with data you already hold and five conversations with recent applicants, including at least one who gave up. Then choose the single step that loses the most people without protecting anything, and test a better version of it.

Worksheet

Use the burden audit worksheet below for one service journey. It has blank fields for your own audit and Dana’s filled rows as an example. You can print the worksheet straight from this page.

Burden audit worksheet

Use one worksheet for one service journey. It accompanies the guide How to Run a Sludge Audit of a Public Service. The steps build on the NSW Government Sludge Audit Method as described in the 2024 OECD and NSW report, Fixing frictions: ‘Sludge audits’ around the world. NSW also publishes free templates for people outside NSW (Sludge Finder Lite, a spreadsheet, and the one-page Sludge Finder Mini) on its Sludge Toolkit page. This worksheet follows the same steps and adds Herd and Moynihan’s three costs, the safeguard questions and a test plan.

Fill in Part A once. Fill in one Part B card for every step on your journey map, including waiting and the steps people take away from your agency. Then use Part C to choose one change and plan its test. A blank field is useful: it tells you what you still need to find out.

Part A: Scope

Field Your answer
Service audited
Outcome the service exists to produce (a result, not “applications submitted”)
Where the journey starts (when people begin looking for help)
Where the journey ends
Journeys mapped separately (for example: online; assisted or in person; another language; disability access)
Requirements set in law or policy that the team cannot change directly
Existing data you will use (timestamps, drop-off points, call logs, denial reasons, complaints)
New evidence you will collect (observations, interviews with recent applicants including people who stopped, frontline staff)

Part B: One card per step

Copy this card for each step.

Field Your answer
Stage and step
What the person must do
Time (active time and waiting time; a range if it varies widely)
Documents (what they must find, request or produce)
Learning cost (what they must find out or understand)
Compliance cost (what they must complete, submit or attend)
Psychological cost (stress, stigma, confusion, loss of control)
Effort and experience rating (1 = easy, 5 = very difficult)
Who drops out (completion at this step, overall and by group)
Is the step a needed safeguard? (a) What is it for? (b) Is there evidence it achieves that? How many problems did it catch last year? (c) Who does it stop: ineligible people, eligible people or both? (d) Is there a lighter way, including moving the work to the agency, and what would the lighter way miss? (e) Does the applicant understand why it exists?
Classification: keep as is / keep the purpose, change the method / remove
Proposed change
How to test it (comparison, primary outcome, safeguard check)

Part C: Choose one change and plan the test

Field Your answer
Change chosen first
Why this one (burden removed, group affected, feasibility, safeguard kept)
Comparison (for example: random assignment to the current and new process during the same weeks)
Unit of assignment (person, caseworker team, office or week) and why
Primary outcome, tied to the service’s purpose
Secondary outcomes (time, errors, staff time, complaints)
Results by group
Safeguard check (a measure that would rise if the change weakened a check)
Smallest improvement worth acting on, sample and duration
Legal, ethical and privacy checks completed
Date of the follow-up audit of this step (encouraged, not required, by the NSW method)

Filled example: a county rent relief program (hypothetical)

This example is invented to show how the worksheet works. The program, the people and every figure are hypothetical. They are not findings from a real service.

Part A: Scope (example)
Field Example answer
Service audited County rent relief: a one-time payment, sent to the landlord, for tenants behind on rent who earn below an income limit.
Outcome the service exists to produce Eligible tenants receive a payment before they lose their home.
Where the journey starts A tenant hears about the program and tries to work out whether they qualify.
Where the journey ends The appeal is decided.
Journeys mapped separately Online with a smartphone; walk-in office with staff help (weekdays 9 to 3); a tenant who prefers a language other than English; a tenant who can apply only in person or by phone.
Requirements set in law or policy The income limit; the 10-day deadline to answer a request for information; the 15-day appeal window.
Existing data Online form timestamps and abandonment points; call center logs; denial reasons; complaints.
New evidence Observation of five applicants completing the form; interviews with recent applicants, including three who stopped; a session with walk-in office staff.
Part B: Cards (example)

In the audit month, 1,000 people started an application (880 online, 120 at the walk-in office) and 640 submitted it. People who read the eligibility page and never started cannot be counted.

Stage 1. Finding the requirements

Field Example answer
What the person must do Find the program page, read the eligibility rules, decide whether they qualify.
Time 10 to 40 minutes; longer for people who call the help line, which is open weekdays 9 to 3.
Documents None.
Learning cost High. The page does not say whether partial-month arrears count, and the income limit is shown only as an annual figure.
Compliance cost Low.
Psychological cost Uncertainty about whether applying is worth it.
Effort and experience rating 3
Who drops out Not measurable directly. Many help-line calls ask “do I qualify?”
Is the step a needed safeguard? (a) It informs people; it is not a check. (b) Not applicable. (c) Unclear rules may deter eligible people. (d) A short eligibility checker and a monthly income figure. (e) Not applicable.
Classification Keep the purpose, change the method.
Proposed change A plain-language eligibility checker with examples of partial arrears.
How to test it Later test: compare the share of submitted applications that are eligible, and help-line calls about eligibility, between page versions.

Stage 2. Gathering documents

Field Example answer
What the person must do Work out what counts as proof of income; ask the landlord for a ledger and a tax form; wait for the landlord; photograph or scan each document; convert files to an accepted type.
Time Median of six days from starting the application to the last upload.
Documents Photo ID; lease; landlord ledger; landlord tax form; 30 days of pay stubs or benefit letters.
Learning cost Medium. Which income documents count is unclear.
Compliance cost High. Two documents depend on the landlord.
Psychological cost High. Interviewees describe asking a landlord for paperwork while facing eviction as stressful.
Effort and experience rating 4
Who drops out Most of the 360 people who start and never submit stop before uploading the landlord documents. Interviews point to those two documents.
Is the step a needed safeguard? ID: (a) stop impostor applications; (b) last year it stopped 9 applications, 2 of them confirmed attempts to apply in someone else’s name; (c) most applicants can meet it; (d) accept a phone photo and more ID types, then check whether the wider list would have let those 2 through. Landlord ledger and tax form: (a) confirm the debt is real and the landlord exists, so the county does not pay a fake landlord; the tax form also serves the county’s own reporting of payments to landlords; (b) last year the check stopped 52 applications, 7 because the named landlord did not own the property and 45 because the tenant could not get the forms back; (c) it stops real fraud and also eligible tenants whose landlords do not reply; (d) the county checks property records and requests both documents from the landlord itself; property records alone would miss whether the rent is owed, so the ledger stays. Pay stubs: (a) confirm income; (d) accept a current benefit award letter where the rules allow, and check how often award letters are out of date.
Classification ID: keep, widen accepted forms. Landlord documents: keep the purpose and both documents, move the collecting to the county. Income: keep, widen accepted proof.
Proposed change County-led landlord verification; wider accepted ID and income documents.
How to test it Later test: compare submission rates and post-payment review flags between the current and county-led verification.

Stage 3. Applying

Field Example answer
What the person must do Complete the online form, or visit the walk-in office.
Time Online form: about 45 minutes, with no way to save and return. Walk-in: travel plus waiting time.
Documents Uploads from stage 2.
Learning cost Low to medium.
Compliance cost Medium to high. Uploads often fail on phones.
Psychological cost Medium. Losing progress means starting again.
Effort and experience rating 4 online; 3 at the walk-in office
Who drops out Cumulative submission by channel, including losses at stage 2: online, 556 of 880 submit (63 percent); walk-in with staff help, 84 of 120 submit (70 percent).
Is the step a needed safeguard? (a) Collects the information needed to decide eligibility. (b) The questions are needed; the lack of a save function serves no purpose. (d) Save and return; phone-friendly uploads; longer or phone-based assisted hours.
Classification Keep the purpose, change the method.
Proposed change Save and return; fix phone uploads; add a phone application option.
How to test it Later test: compare completion by channel before and after a staged rollout across offices, with a comparison office.

Stage 4. Correcting an error

Field Example answer
What the person must do Read a mailed request for information, work out what is missing, supply it within 10 days.
Time Up to 10 days, minus mail time.
Documents Whatever is missing. The letter lists form codes, not document names.
Learning cost High. Applicants must decode which document is meant.
Compliance cost Medium.
Psychological cost High. A deadline that ends in denial.
Effort and experience rating 5 for applicants who prefer another language; 4 for others
Who drops out 288 of 640 submitted applications (45 percent) receive a request. 86 of the 288 miss the deadline and are denied as incomplete: 30 of 60 applicants who prefer another language (50 percent) and 56 of the other 228 (about 25 percent). Some of the 86 may not have been eligible; the test’s primary outcome will show how many were.
Is the step a needed safeguard? (a) A complete file before a decision. (b) Yes, the purpose is real. (c) The current letter stops people unevenly by language; how many of them were eligible is not yet known. (d) A clearer, faster, translated notice serves the same purpose and asks for the same documents, so it should miss nothing the current letter catches. (e) Applicants do not know what is missing.
Classification Keep the purpose and the policy deadline; change the method.
Proposed change Make the applicant’s chosen language standard for every request now. Then test a plain-language checklist naming the exact documents, sent by text and email as well as mail, with a call on day three.
How to test it Chosen first. See Part C.

Stage 5. Receiving a decision

Field Example answer
What the person must do Wait for a mailed decision letter.
Time Median of 38 days from submission to decision.
Documents None.
Learning cost Medium. No way to check status.
Compliance cost Low, apart from status calls.
Psychological cost High. An eviction date may be approaching.
Effort and experience rating 3
Who drops out Not a drop-out point, but status calls are a large share of help-line volume.
Is the step a needed safeguard? (a) Review time protects the program. (b) Needed. (d) The silence is not needed; status updates would serve the same review. (e) People do not know what stage they are at.
Classification Keep the review; change the communication.
Proposed change Text or email status updates at each review stage.
How to test it Later test: compare status-call volume and complaints between applicants randomly assigned to updates and to current practice.

Stage 6. Appealing

Field Example answer
What the person must do File a written appeal within 15 days and submit every document again.
Time Several hours, inside a 15-day window.
Documents All documents already held by the county.
Learning cost Medium. The letter explains appeal rights in legal language.
Compliance cost High.
Psychological cost High. The applicant has just been denied.
Effort and experience rating 5
Who drops out Few appeals relative to denials. Interviews suggest many denied applicants did not know they could appeal or thought it meant starting again.
Is the step a needed safeguard? Written appeal and window: set by policy. Resubmitting documents: (a) none identified; the county already holds them. (d) The caseworker attaches the existing file. Photo ID again: its purpose was served at application.
Classification Remove the resubmission; keep the appeal right and window.
Proposed change Appeals reuse the existing file; a plain-language notice of appeal rights; phone appeals for people who cannot write one.
How to test it Later test: compare appeal filing rates among denied applicants before and after, alongside a comparison county or office if one is available.
Part C: Test plan (example)
Field Example answer
Change chosen first The new request-for-information notice (stage 4).
Why this one It is where 86 applications a month are denied without a decision on eligibility, it falls hardest on one group, it needs no change to eligibility rules, and it keeps the safeguard of a complete file.
Comparison Random assignment of applicants who receive a request for information to the current notice or the new one, during the same weeks. Both versions go out in the applicant’s chosen language, so translation is not part of the contrast.
Unit of assignment Individual applicant. A small call team makes the day-three calls and sees only assigned cases, so caseworkers do not start calling the comparison group. (Assigning by caseworker team or week would make the number of teams or weeks, not applicants, set most of the power.)
Primary outcome Share of request recipients whose application reaches an eligibility decision (approved, or denied on eligibility) rather than a denial for being incomplete, measured 60 days after the request (the program’s median from submission to decision is 38 days).
Secondary outcomes Time from request to decision; time to payment; errors found in quality review; staff minutes per case; complaints.
Results by group Preferred language; application channel.
Safeguard check Share of approved cases flagged in the existing post-payment review.
Smallest improvement worth acting on, sample and duration Decided before launch. With about 290 requests a month assigned individually, two months of enrollment can detect an improvement of roughly 11 percentage points, but would likely miss a smaller one. Results arrive 60 days after the last request. Extend the test if the smallest worthwhile improvement is smaller.
Legal, ethical and privacy checks Confirm that both versions meet notice and language-access requirements; translation is standard in both arms; text and email contact only with the applicant’s consent.
Date of the follow-up audit of this step (encouraged, not required, by the NSW method) Two months after the new notice becomes standard.

References