Reliable does not mean accurate

Reliability designates the consistency of a measurement: a reliable instrument gives similar results when the same thing is measured twice. Validity designates something else: the fact that the result actually bears on what the instrument claims to measure. Everyday language readily confuses the two.

The classic image is that of a badly calibrated scale that shows two kilos too many every morning. It is perfectly reliable, since it repeats the same error, and it is not accurate. Sufficient reliability is necessary for validity; it never guarantees it.

The most widespread reliability index is coefficient alpha, proposed by Lee Cronbach in 1951. It estimates the extent to which the questions of a scale vary together. It is found in almost every publication, sometimes as the only argument for quality.

Yet it is often misread. A high alpha does not prove that a scale measures a single thing, and it rises mechanically with the number of questions. A scale made of almost identical wordings obtains an excellent coefficient, precisely because it repeats the same question.

Another form of reliability consists in administering the same questionnaire a few weeks apart. For a trait assumed to be stable, close scores are expected. For a state, the mood of the day for example, strong stability would on the contrary be suspect: the instrument would fail to capture what changes.

The question therefore has no general answer. The level of stability expected depends on what one claims to measure, and the same figure can be a quality for one scale and a flaw for another. This dependence does not appear in the coefficient itself.

No psychological measurement is free of error. Classical test theory draws a practical consequence from this: each individual score can be accompanied by an interval, calculated from the reliability, within which the value one is trying to estimate probably lies.

This interval is often wide. Two people whose scores differ by a few points may not differ at all, and the same person may change category from one administration to the next without anything having shifted in them. A result without a margin displays a precision it does not have.

Validity cannot be read off any single coefficient. It is established through a body of converging arguments: the content of the questions, the relationship of the score with other measures of the same object, and its relative independence from measures of neighbouring but different objects.

This last point weighs heavily in the emotional domain. A scale that correlates strongly with an existing personality questionnaire risks measuring, under a new name, something already known. Reliability can say nothing about this.

Reading a result therefore involves two separate questions: does the instrument give the same result under the same conditions, and does that result bear on what is claimed? The first comes down to a figure. The second calls for a whole file of evidence, and it is the one that decides what can be concluded.