Research and scientific reasoning
Psychometrics: reliability, validity and screening accuracy
The measurement ideas behind every test score: reliability, validity, norms, standard error, sensitivity, specificity and why screens are not diagnoses.
What this module draws on:
- Textbook concept Reliability and validity of assessment methods (Ch. 7–8).
- Original elaboration Worked screening example with fictional numbers, and the matching activity.
- Contemporary update Screening-accuracy terminology as used in current measurement practice.
Learning objectives
After this module you should be able to:
- Define and distinguish forms of reliability and validity.
- Explain norms, standard scores and the standard error of measurement.
- Calculate sensitivity, specificity and predictive values from a simple table.
- Explain why a positive screen is not a diagnosis and how base rates affect interpretation.
Introduction
Every assessment method — an interview, a checklist or an intelligence test — is a measurement. Psychometrics asks two basic questions: Is the measurement consistent (reliability)? Does it measure what it claims to measure (validity)? A test can be reliable without being valid, but it cannot be valid without being reasonably reliable.
Reliability Textbook concept
The textbook notes that two clinicians interviewing the same person can reach different conclusions — a problem of inter-rater reliability that structured interviews were designed to reduce.
| Type | Question | Example |
|---|---|---|
| Test–retest | Are scores stable over time? | A trait anxiety scale given two weeks apart |
| Internal consistency | Do items measure the same construct? | Items of a depression scale correlate with each other |
| Inter-rater | Do different raters agree? | Two clinicians rate the same interview |
| Parallel forms | Do equivalent versions give similar scores? | Two versions of a memory test |
Validity Textbook concept
- Content validity — do the items cover the construct adequately?
- Criterion validity — do scores relate to an external criterion now (concurrent) or later (predictive)?
- Construct validity — do scores behave as theory predicts, correlating with related measures (convergent) and not with unrelated ones (discriminant)?
- Face validity — does the test appear to measure what it claims? (Useful for acceptability, weak as evidence.)
Norms and measurement error Original elaboration
Many tests compare a person's score with a normative sample. Standard scores (such as a mean of 100 and standard deviation of 15 on many intelligence scales) and percentiles express where someone falls relative to that sample — so the sample must be appropriate.
No score is exact. The standard error of measurement lets us place a confidence interval around a score. Reporting 'a score of 92 (95% confidence interval roughly 87–97)' is more honest than '92'.
Screening accuracy Contemporary update
When a measure is used to screen, its results are compared with a reference standard, usually a structured diagnostic interview. Sensitivity is the proportion of people with the condition who screen positive. Specificity is the proportion without the condition who screen negative. Positive predictive value is the proportion of positive screens who actually have the condition — and it depends heavily on how common the condition is (the base rate).
Raising a cut-off usually increases specificity and decreases sensitivity, and vice versa. There is no 'correct' cut-off in the abstract — only one suited to a purpose, population and the costs of each kind of error.
Why a screen is not a diagnosis Original elaboration
- Screens are brief and symptom-focused; diagnosis considers duration, impairment, context, exclusions and differential explanations.
- Even an accurate screen produces false positives, especially where the condition is uncommon.
- Cultural expressions of distress, physical illness and life circumstances can raise scores without a disorder.
- Diagnosis is a professional judgement, typically supported by a structured or semi-structured interview.
Worked example: A thousand students, a fictional screen Original, fictional
Imagine 1,000 students, 100 of whom (10%) meet criteria for depression on a careful interview. A fictional screen has 80% sensitivity and 90% specificity. It correctly flags 80 of the 100 (true positives) and misses 20. Of the 900 without depression, it correctly clears 810 but wrongly flags 90 (false positives).
So 170 students screen positive, but only 80 of them — about 47% — actually meet criteria. That is the positive predictive value. If depression were rarer, the proportion of false alarms would be even higher. This is why positive screens call for follow-up assessment, not labels. (All numbers here are invented for teaching.)
Interactive activity
Match the psychometric concept
Match each fictional observation to the concept it illustrates.
Show the answer key
- Two psychologists independently rate the same video and largely agree → Inter-rater reliability
- A new anxiety scale correlates strongly with an established anxiety scale → Convergent validity
- Aptitude scores at admission predict later examination results → Predictive validity
- Of everyone with the condition, 85% screen positive → Sensitivity
- Of everyone without the condition, 92% screen negative → Specificity
- Of everyone who screens positive, only half have the condition → Positive predictive value
Knowledge check
5 questions. Every option has an explanation. Your answers are not sent anywhere when JavaScript is available; without it, they are checked by the server and immediately discarded.
Summary
- Reliability is consistency; validity is whether scores support the intended interpretation and use.
- Validity is specific to purpose, population and language.
- Every score has measurement error; report confidence intervals.
- Sensitivity and specificity describe a screen; predictive values also depend on base rates.
- A positive screen signals a need for assessment, never a diagnosis.
Questions for reflection
- What would you need to know before using an English-language screening questionnaire with Kashmiri-speaking clients?
Key terms
Further reading
- Textbook Plante, T. G. (2005). Contemporary clinical psychology (2nd ed.). John Wiley & Sons.
- Further reading Groth-Marnat, G., & Wright, A. J. Handbook of psychological assessment (latest edition). Wiley.
- Contemporary Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Houghton Mifflin.
Practise in the case lab: Aaliya: a first-year student who has stopped attending class · Imran: work stress, rising alcohol use and high blood pressure