Glossary
Reliability
What is reliability?
Reliability is the consistency or reproducibility of measurement scores under specified conditions, for a particular population and purpose.
Types of reliability
Researchers use different approaches to examine consistency:
- Test–retest reliability: Consistency of scores over time.
- Inter-rater reliability: Consistency across people who score the same behavior or performance. For example, do two observers give similar ratings to the same interview?
- Internal consistency: How scores on a questionnaire’s individual questions or statements—its items—vary together across respondents. Do people who score higher on one item also tend to score higher on the other items intended to measure the same characteristic?
- Parallel-forms reliability: Consistency of scores from equivalent versions of a test.
Reliability vs validity
Measurement error can limit the conclusions drawn from a score, but reliability alone does not establish that a measure captures its intended construct—the characteristic it is meant to assess. Questions can produce consistent answers while measuring something narrower or different. High internal consistency also does not prove that all the questions measure just one characteristic.
For example, a questionnaire can consistently record whether people keep a tidy desk while providing little evidence about their ability to plan a difficult project. Repeating a narrow observation accurately does not make it a broad measure of conscientiousness.
| Question | Reliability | Validity |
|---|---|---|
| What are you evaluating? | Consistency and measurement error under stated conditions. | Evidence for the proposed score interpretation and use. |
| What would help? | Repeated scores, independent ratings or item relationships suited to the question. | Whether questions cover the intended characteristic, how people understand and answer them, how items relate to one another, and how scores relate to other measures or outcomes. |
| What would not be enough? | One person obtaining the same score twice. | A high numerical estimate of reliability or a convincing test name. |
See construct validity for how researchers test competing explanations of a score.
Applied example
If a screening score changes between Monday and Wednesday, the difference could reflect measurement error, actual change, or different circumstances. One person's two scores cannot establish a test's reliability; this requires appropriate data from a sample.
Checking repeated measurements in a sample
Suppose researchers administer a questionnaire twice to a group whose measured characteristic is expected to remain stable over the chosen interval. They use the same instructions and setting, record relevant changes between sessions, and examine how closely the paired scores agree across the sample. A consistent upward shift on the second occasion matters even if people keep the same rank order. If everyone scores higher the second time, the highest-scoring person may still be highest, but the scores themselves have changed.
A separate question is whether those scores represent the intended characteristic. Repeating a reading-comprehension test with consistent results would not by itself establish that it measures mathematical reasoning. Consistency supports interpretation, but it cannot supply the missing validity evidence.
Choosing the right reliability evidence
Match the estimate to the source of error that matters. For repeated administrations, report the test–retest interval and conditions. For scored observations, check agreement between raters. Internal consistency concerns relationships among items in one administration; it cannot substitute for either check.
Cronbach’s alpha is one internal-consistency statistic: a number calculated from the relationships among item scores. Its interpretation depends on assumptions about the items. McNeish’s methodological review explains why researchers should examine those assumptions and suitable alternatives rather than report alpha automatically. No single cutoff certifies that a measure is good enough for every purpose.
Why it matters
Reliability estimates help quantify uncertainty in measurement. They should be reported for the relevant population and interpreted alongside evidence of validity.