Illusion of Validity

What is the illusion of validity?

The illusion of validity is unwarranted confidence in a prediction because the available information forms a convincing match with the predicted outcome, even when that information has limited predictive value.

Tversky and Kahneman’s original discussion distinguishes confidence based on a coherent impression from evidence that a prediction is accurate. Repeated or highly correlated cues can appear consistent without providing much independent information.

Example in hiring

A coherent candidate profile, for example, might inspire confidence in future performance. The relevant question is whether the assessment predicts later outcomes, not only whether its details tell a persuasive story. A polished story about a candidate and an accurate prediction about that candidate are two different things.

Imagine an interviewer describes a candidate as articulate, confident, and persuasive after one polished conversation. Those three favorable impressions may arise from the same performance; counting them separately does not create three independent tests of job ability. An independently scored work sample could test a specific task, such as finding errors in a spreadsheet. To establish its predictive value, compare work-sample scores with later accuracy on that job task. The sample provides a different kind of evidence, but its usefulness must still be demonstrated.

How to check a confident judgment

Write down the prediction and compare it with what happens. Ask whether each new detail adds independent information or merely repeats the same impression. Several people agreeing on a persuasive story does not establish that it predicts anything. The useful test is how well the prediction matches later outcomes, not how coherent the explanation sounds.

For a related question about confidence and competence, see the Dunning–Kruger effect.