Replication applied to childhood
Testing an experiment again is difficult everywhere. When the participants are twelve months old, it becomes a problem of organisation as much as of method.
The starting point
Replication work conducted in psychology from 2011 showed that a substantial share of published effects was not found, or was found weakened, when the protocol was repeated. The most cited project, published in 2015 by the Open Science Collaboration, repeated a hundred studies in social and cognitive psychology and recovered a significant result for only a minority of them.
Developmental psychology was affected later, for practical reasons: infants are hard to recruit, sessions are costly, and published sample sizes have historically been very small.
The small sample problem
A study of twenty infants can only detect a very large effect. If the real effect is small, it will not find it, except by chance, and it is precisely that chance finding which is most likely to be published.
The mechanism produces a literature whose published effects are systematically larger than real effects, without any author having to commit a fault.
Statistical power, that is the probability that a study detects an effect that exists, depends on sample size and on effect size. It was rarely calculated in the field's older publications, which made it impossible to gauge the scale of the problem.
The collaborative response
The ManyBabies consortium, launched in 2017, runs the same protocol in dozens of laboratories, which brings the sample to several thousand infants and allows the effect to be estimated precisely. The arrangement rests on decisions taken jointly before data collection, protocol, criteria for excluding sessions, planned analyses, which limits adjustments after the fact.
Its first results recovered certain classic effects, notably the preference for infant-directed speech, with smaller sizes than the original publications, and did not recover others.
What this work does not say
An effect not found is not necessarily non-existent. It may depend on conditions the shared protocol does not reproduce, notably characteristics of the population or of the materials used. The language of the stimuli, the children's exact age or their familiarity with the laboratory setting are examples of such conditions.
Consortia try to test that hypothesis by analysing variation between sites, which has become one of the most interesting contributions of these arrangements, independently of the verdict on the average effect.
Practices that are changing
Preregistration of analyses, publication of data and power calculation before collection have spread in the field, more slowly than in other areas for want of resources. They demand additional time and skills, which small teams find harder to bring together.
Those practices do not guarantee the truth of a result; they make visible what was decided before seeing the data, a distinction that is enough to change the value of a publication.
Key points
Historically small samples produce a literature with overstated effects.
Multi-laboratory consortia bring samples to several thousand and find smaller effects.
An effect not found may depend on local conditions, which analysis of between-site variation can test.
This article describes documented mechanisms and disagreements. It recommends no position, no party and no vote, and commits to no reading other than that of the work it cites.
And you, where do you stand?
The Civix test situates your convictions on six dimensions and matches you with one of the twelve profiles, without ever telling you what to think.
Start my test