What the replication crisis changed

From 2011 onwards, several collective initiatives attempted to reproduce experiments published in the major psychology journals, with protocols announced in advance and sample sizes larger than the originals. A significant share of the results was not recovered, or was recovered with a markedly weaker effect.

The problem is not fraud, which remains marginal. It comes down to practices long considered ordinary: small samples, analyses decided after seeing the data, preferential publication of positive results.

The experiment with the pen held between the teeth, published in 1988 and supposed to show that facial position influences mood, was not reproduced in a coordinated attempt across seventeen laboratories in 2016. A later international collaborative study recovered a slight effect with other procedures: an effect probably real, very small, sensitive to details.

Power poses, presented in 2010 as altering hormones and risk-taking, did not withstand replication. In 2016, one of the authors of the original study publicly stated that she no longer believed in the effect.

Brain imaging had its own reckoning. A famous demonstration, presented in 2009, applied the then-standard statistical corrections to a scan of a dead salmon and obtained significant activation, to illustrate the problem of multiple comparisons.

An article published in 2022 added a heavier constraint: establishing a reliable association between an imaging measure and an individual trait would require samples of several thousand people, when the published literature often rests on a few dozen.

The collective responses are concrete: registration of protocols before data collection, publication of raw data, multi-laboratory replications, journals accepting an article on the quality of its protocol alone before knowing the results.

These practices change the nature of what gets published. They produce fewer spectacular results, more null effects, and make the field less convenient for anyone looking for a hook.

A few questions are enough to sort. How many participants? Was the protocol announced in advance? Has the result been reproduced by another team? Is the effect large or merely significant, which is not the same thing?

A single result, obtained on forty people, published twenty years ago and never reproduced, may be true. It simply does not provide a sufficient basis for asserting anything before an audience, still less for selling a training course.