Reading a survey commissioned by a company

The decisive distinction is not sample size but how it was built. A probability sample drawn from a sampling frame does not have the same properties as a panel of volunteers signed up online.

The classic example remains the Literary Digest, which in 1936 polled several million Americans and got the presidential election wrong, while far smaller but better constructed samples had correctly anticipated it. A large number of responses does not make up for poor recruitment.

Online panels are usable and widely used, including by serious institutes, provided the weighting method is described. The absence of that description is the first warning sign.

A survey commissioned by an organisation with an interest in a result is not necessarily distorted, but the choice of questions asked and of those not asked belongs to the commissioner. A remote work software publisher and an office property company will not choose the same questions on the same subject.

The most frequent bias is not manipulation of answers but selection of published results: favourable items are highlighted, the others appear in the full report, when it is published.

Methodological work on wording effects is old and robust: the same fact asked in different terms produces gaps of tens of points. As early as 1941, Donald Rugg showed that "forbid" and "not allow" do not elicit the same answers, and Howard Schuman and Stanley Presser systematised these experiments in 1981.

A figure cited without the exact question that produced it is therefore uninterpretable, and the presence of the full questionnaire in the publication is a good indicator of seriousness. Question order matters too: a question preceded by items on the difficulties of work does not get the same answers as when it is asked first.

A low response rate does not invalidate a survey but exposes it to selection bias, respondents differing from non-respondents on the dimensions measured. The work of Robert Groves, in the mid-2000s, shows that this bias depends less on the rate itself than on the link between the decision to respond and the subject covered.

Serious publications state that rate and the weighting applied. Most commissioned studies do not, which prevents assessing the size of the problem. In France, the polling commission, the Commission des sondages, oversees only electoral surveys, and studies on work are subject to no comparable oversight.

On almost every work-related subject, a public source exists: labour force surveys, working conditions surveys, administrative payroll and labour movement data.

They are slower to appear and less spectacular, which explains their lower uptake. They do, however, supply series comparable over time, which one-off studies never allow.