first of all, thanks to michael woodley, jan te nijenhuis, and raegan murphy for their response to my (here and here) and scott alexander’s comments on their recent iq paper (Were the Victorians cleverer than us?). thanks, too, to dr. james thompson for hosting their response!
just to refresh everybody’s memory, woodley et al., concluded from their recent research that 1) iqs have decreased in the western world 14.1 points since the victorian period (1889), and 2) that this decrease is due to dysgenics. scott alexander and i both questioned the sampling techniques used in various of the studies looked at by woodley and co. — both in the victorian period and in the modern (see our previous posts linked to above for details).
what is at issue here is whether or not woodley et al. have looked at the same sort of population at both ends of their research. if they’re trying to figure out whether or not iqs have decreased over time in the western world, then the subjects sampled in the past and in the present need to be representative of the two populations, and the two populations need to be of the same sort. unless i’m very much mistaken, sampling issues are considered to be of extreme importance in scientific endeavors (i.e. to get your sampling right). it also simply makes logical sense.
so, we need to know a few things: 1) were the samples taken in the victorian period representative of victorians, 2) were the samples taken in modern times representative of modern populations, and 3) are all of these samples from like populations?
1) were the samples taken in the victorian period representative of victorians?
the victorian samples came from two sources: galton and thompson. (see also here about some other victorian rts data sets.)
thompson’s samples came from university of chicago students in 1898-1900, so they’re unlikely to be representative of victorian americans on the whole. these were probably mostly pretty smart individuals primarily drawn from the elite classes. (see previous post.)
galton’s sample consisted of museum visitors who had paid to take galton’s test (and, in some amount of cases, had probably paid to enter the museum as well). scott alexander superbly analyzed this sample further by combing through the info in johnson et al.’s “Galton’s data a century later.” scott found that galton’s sample was not representative of victorian brits — in fact, it leans heavily towards the middle- and upper-classes. this is not surprising when you think about who, in those days, would’ve been able to afford the costs of taking the test and possibly the museum fee as well.
here’s what scott had to say:
“Tables 10 and 11 turn out to be a gold mine – I worried the records of exactly who took the tests would be lost, but as you might expect of someone who basically invented statistics single-handedly and then beat Darwin in a debate about evolution as an encore, Galton was *very good* at keeping careful data.
“This site tells me that about 3% of Victorians were “professionals” of one sort or another. But about 16% of Galton’s non-student visitors identified as that group. These students themselves (Galton calls them ‘students and scholars’, I don’t know what the distinction is) made up 44% of the sample – because the data was limited to those 16+, I believe these were mostly college students – aka once again the top few percent of society. Unskilled laborers, who made up 75% of Victorian society, made up less than four percent of Galton’s sample!”
a very kind person very kindly sent met the johnson et al. paper (thank you very kind person!), so i’ve cut-and-pasted tables 10 and 11 here for you to see for yourselves [click on tables for LARGER views]:
so galton’s sample is not representative of the victorian british population — it was unbalanced in that it did not include enough subjects from the lower classes.
furthermore, the subjects in galton’s study self-selected themselves. these were individuals who, first of all, chose to go to the south kensington museum (now the victoria and albert museum) to start with (except maybe for the “students and scholars”), AND then they further chose to take galton’s funny little test. this is NOT a good sampling technique. if gallup or pew were to use such a technique, they’d be laughed out of the polling business.
modern studies, of course, try to make sure that data from a representative sample of a population are gathered, otherwise your data might be skewed. which is exactly what scott found with regard to galton’s data. i mean, imagine what sort of person from the lower classes in victorian england first bothered to go to the south kensington museum AND then was interested in taking galton’s test. knowing what i know about the working classes (and i come right out of that class — more like the peasant class, in fact), those that want to go to a boring museum are the exceptions to the rule.
in woodley et al.’s response to our questions about these sampling issues, they point to a study by silverman — Simple reaction time: It is not what it used to be — which i haven’t seen, because i don’t have access to it (and i’m too cheap to pay for it (~_^) ). they say that silverman compensated for these victorian sampling errors by looking at reaction time (rt) data from similar socio-economic sub-groups from more modern eras. sounds like a good idea:
“One advantage of Silverman’s care and meticulous attention to detail is that it permits us to make like for like comparisons with specific socioeconomic and occupational groups in Galton’s data, thus we can directly test the claims of Alexander (2013). Concerning the post-Galton studies Silverman included five student samples, two of which date from the 1940s (Seashore et al. 1941), and the remaining three of which date from the 1970s to the 2000s (mean testing year = 1993; Brice & Smith, 2002; Lefcourt & Siegel, 1970; Reed et al., 2004)…. The difference between the 19th century and the ‘modern’ male students is very similar to the meta-regression-weighted increase in RT latency between 1889 and 2004, estimated on the basis of all samples included in the meta-analysis (81.41 ms). Silverman also included data from other socioeconomic groups. For example the study of Anger et al. (1993) included a combined male + female sample of 220 postal, hospital and insurance workers from three different US cities. These occupations clearly fall into the Clerical/Semiskilled and Semiprofessional groups identified in Galton’s study.”
i’m not going to look through all of these studies to check for sampling irregularities because, again, it’s not my research, so it’s not my job. but let’s take a quick look at the first one i highlighted above: seashore et al. 1941. i don’t have access to this paper either, and, no, i’m not gonna pay for it, so i’ll have to assume going by another of r.h. seashore’s papers that these rt data were drawn from american students (possibly northwestern university where seashore worked — correct me if i’m wrong), and going by the publication date (1941) that the samples were collected in the 1930s or 1940-41.
woodley et al. say that, in comparing the nineteenth century vs. 1940s-era students, there is a +16.8ms (183.2-200 ms respectively) difference between the two groups, and thus a decrease in average iq from the victorian period to the 1940s.
my question is: what makes silverman, or woodley et al., confident that the samples from an american university in the 1930s-40s are comparable to thompson’s university of chicago samples or galton’s samples from the 1800s? as shown above, both of those victorian studies were done on elite victorian groups, while, on the other hand, it is well know that already by the 1940s at least, the “college bubble” had started in the u.s. in which nearly anybody is admitted to university. when the average college grad shifts over time from having been drawn from the upper classes to the middle and even the lower, it’s not surprising that their average iqs drop!
even within galton’s self-selected group of male museum goers (see table 10 above), rts (iq) varied between the classes, with the upper classes having lower rts (indicative of higher iqs): ages 14-26: gentleman=.170, professional=.173, semiprofessional=.182, merchant/tradesperson=.190, clerical/semiskilled=.187, unskilled=.195. why, then, shouldn’t rts/iqs vary between college students of the late-1800s vs. the 1940s when more middle- and lower-class individuals attended college in the 1940s than in the late-1800s?
as for the second study i’ve highlighted above — anger et al. (1993) — which included “a combined male + female sample of 220 postal, hospital and insurance workers from three different US cities….” postal workers? seriously? comparing 1990s american postal workers to galton’s middle- and upper-class museum goers? really?
2) were the samples taken in modern times representative of modern populations?
yes and no.
some of the studies used by woodley et al. are probably better (i.e. more representative of modern populations) than others. as i pointed out in my first post on this study, i would imagine, although i haven’t actually checked it, that the sampling in the 1984–85 U.K. Health and Lifestyle Survey is probably fine, and so the rts representative of the whole of britiain. on the other hand, the 2002 study from the university of bristol may have included at least 10% minority individuals if not more in the study. since only ca. 8% of the u.k.’s population in 2001 was non-white, this bristol study may not have been representative of the nation’s population. hard to know without knowing the demographics of who was included in the study.
3) are all of these samples from like populations?
the victorian samples were drawn from mainly the middle- and upper-classes, not to mention a large group of self-selected individuals who were specifically interested in taking part in such a study, while the more modern studies have included greater numbers of middle- and lower-class individuals as well as a certain percentage of non-whites.
about that latter point, woodley et al. responded:
“Substantial changes in terms of the ethnic composition of test-takers would however be needed in order for the magnitude of change to be *solely* or even *substantially* a consequence of this process.”
i never said so. i only said that it’s not possible to compare apples to apples+oranges when looking for changes in just apples over time. apples vs. apples+oranges are likely to produce different results since two such groups are different. i never said, or even suggested, anything about the extent of those differences. (i wish people would read more closely. *sigh*)
so, again, i think that there are quite a few sampling issues in this study, and that the presence of these means that the researchers’ findings are not as reliable as they think that they are.
could there have been a decline in western iqs since the victorian period due to dysgenic factors? of course.
does the fact that there are sampling issues in the woodley et al. (and silverman, for that matter) paper prove that there hasn’t been a drop in iq since the victorian period? no. obviously not.
most importantly, though, the population geneticists say that such a severe drop in genotypic iqs could not have happened in such a relatively short space of time without some really severe selection pressures (see here, too). i believe ’em. perhaps if all of the sampling issues were cleared away from this study, woodley et al. would be left with a decline that was more realistic/believable. if such a decline happened at all, that is. (which it could have! or maybe it didn’t…. (~_^) )
again, like i said before, this is a really neat study from woodley et al.! i hope they continue investigating along these lines, because it would obviously be important to know if average iqs are declining in the western world.
p.s. – before bruce charlton shows up and gets all inquisitional on me (repent, or be damned to hellfire for eternity!), yes, i ALWAYS get all picky about sampling (see here and here and here and here for just a few examples). if you don’t get the fundamentals right, everything else will inevitably be wrong!
previously: we’re dumber than the victorians and btw, about those victorians…
(note: comments do not require an email. i never expected the spanish inquisition!)