Reading an effect size in education
A figure like zero point four circulates widely in educational syntheses. Knowing what it measures prevents comparing it to what it does not.
What the figure expresses
A standardised effect size expresses the gap between two groups in units of standard deviation. It allows results obtained with different instruments to be compared, which a raw gap in points does not.
The most cited interpretation thresholds, proposed by Jacob Cohen in 1988, call values of zero point two, five and eight small, medium and large. Cohen himself presented those markers as provisional conventions in the absence of a field reference.
An example helps fix ideas: an effect size of zero point two means that the average pupil in the treated group sits at roughly the fifty-eighth percentile of the control group. The gap is real; there is nothing spectacular about it.
Why those thresholds mislead in education
Synthesis work has shown that the real distribution of effects measured in educational trials is heavily concentrated at the low end. Values Cohen called medium are rare in rigorous large-scale trials.
Researchers, notably Matthew Kraft, have proposed field-specific markers, placing much lower the boundary of what matters for educational policy.
Kraft published these markers in 2020, also stressing the cost and scale of the intervention: a small effect obtained cheaply across many pupils does not carry the same weight as a comparable effect obtained through a heavy programme.
What inflates published effects
Three factors pull in the same direction. Small samples produce scattered estimates of which only the largest are published. Measures designed by the intervention's authors are more sensitive than standardised tests. And short trials measure an immediate effect that subsequently fades.
Comparing effects by type of measure is striking in several syntheses: moving from a bespoke test to a standardised one commonly halves the measured effect.
The case of syntheses of syntheses
Rankings of interventions by effect size, the best known being John Hattie's, aggregate heterogeneous meta-analyses. Methodological criticism has focused on mixing studies with incomparable designs and measures. That ranking, published in 2009 under the title Visible Learning, spread enormously through teacher training, which explains the attention paid to its methodological foundations.
The main objection is not that the figures are wrong but that they are not commensurable: placing in one ranking an intervention tested by randomised trial and a correlational association produces a misleading order.
How to read an isolated figure
Three questions usually suffice. On what measure, standardised or built for the occasion. On what sample size, and with what assignment. How long after the intervention the measurement was taken.
A modest effect obtained on an independent test, at scale and several months later, is more informative than a large effect obtained on a bespoke test, on fifty pupils, measured the next day.
Key points
Cohen's thresholds are general conventions, poorly fitted to educational trials.
Measures built by the intervention's authors produce far larger effects than standardised tests.
Effect size rankings aggregate non-comparable designs, which makes their ordering misleading.
This article describes documented mechanisms and disagreements. It recommends no position, no party and no vote, and commits to no reading other than that of the work it cites.
And you, where do you stand?
The Civix test situates your convictions on six dimensions and matches you with one of the twelve profiles, without ever telling you what to think.
Start my test