Psychometrics

What is psychometrics?

Psychometrics is the study of psychological measurement: how observations become scores, what those scores mean, and how much confidence to place in them. It includes the theories and statistical methods used to develop and evaluate measures of abilities, personality, attitudes and other characteristics.

A questionnaire can produce a number in seconds. Establishing what that number measures is the harder task. As the Psychometric Society’s introduction explains, the field extends beyond administering tests to the models and methods that connect observations with psychological attributes.

From an answer to a score

Imagine a researcher developing a short questionnaire about planning. This is a hypothetical example, not a validated assessment. One item asks whether you plan tomorrow’s tasks before finishing work. You choose a response from 1, “almost never,” to 5, “almost always.”

The response is an observation. Planning tendency is the proposed construct: the characteristic the researcher wants to measure. The two are not identical. You might make plans because your employer requires them, misunderstand the question, or answer as the person you would like to be.

Suppose four items have been scored so that a higher number always indicates more of the proposed planning tendency. Their scores are 4, 3, 5 and 2, giving a total of 14 on a possible range of 4–20. Adding them gives each item equal weight. Whether the questions belong together and adequately represent planning still requires evidence. Dividing 14 by the maximum of 20 produces 70%, but this does not mean the person is “70% organized”: the total comes from these four questions, not a scale running from no organization to complete organization.

What makes a score interpretable?

• Reliability and error: Would scores remain reasonably consistent under the conditions relevant to their use? A single administration cannot show stability over time. See reliability.
• Validity: Do the questions and the pattern of results support the claimed interpretation? Four nearly identical questions about making lists might miss other aspects of planning. See construct validity.
• Reference group: If a report gives a percentile, who supplied the comparison scores? A rank within a volunteer sample of students is not automatically a rank among all adults.
• Intended use: Describing a tendency, predicting a later outcome and deciding who gets a job are different claims. Evidence for one does not automatically support the others.

The Standards for Educational and Psychological Testing organize validity around proposed interpretations and uses. The relevant question is what the evidence permits you to conclude from this score.

Psychometric theory: different ways to study measurement

Classical test theory separates an observed score into a true-score component and error. “True score” has a technical meaning: the average score a person would receive across hypothetical repetitions under the conditions defined by the model. A single observed result can depart from that average because of measurement error. This is a way to model scores, not a person’s hidden, perfectly known essence.

Item response theory models how the chance of giving a particular answer relates to the characteristic being measured and the question itself. On an ability test, for example, it can model how the chance of answering a question correctly varies with ability and question difficulty. This helps researchers investigate which questions distinguish people at different points on the scale.

Factor analysis examines relationships among items or measures. If several planning questions tend to receive similar responses from the same people, researchers can examine whether a shared underlying tendency helps explain that pattern. This is one way to evaluate a questionnaire’s proposed structure. A model that fits the data is evidence to examine, not proof that the named attribute is the only possible explanation.

Using psychometrics in applied behavioral science

Before using a score to evaluate a project, define the outcome. If you want people to complete a task, completion records answer a different question from a questionnaire about confidence. Both can be useful, but improvement in one cannot stand in for improvement in the other.

For personal reflection, a well-supported measure can help you notice individual differences that matter when choosing activities. Compare the result with your interests, skills and repeated experience. The score helps frame a question; it does not choose your life for you.