on the add health interviewers

add health made a statement about kanazawa’s use of their data over @scientific american (scroll down, it’s at the bottom of the blog post), which includes a bit about the interviewers and their evaluations of subjects’ attractiveness:

“Interviewer ratings of respondent attractiveness represent a subjective ‘societal’ perception of the respondent’s attractiveness. We included these items because there is a long line of research evidence that indicates that perceived attractiveness is related to important health and social outcomes, including access to health care, health education and instruction, job search, promotions, academic achievement, and social success in friendship and marriage. For example, males who are rated more highly attractive tend to have higher wages, shorter periods of unemployment, and greater success in the job market*. In Add Health, we measure respondents’ self-perceptions and in the case of interviewer ratings, others’ perceptions. Despite one’s own perception of one’s intelligence, identity and appearance, often societal perceptions matter as well, and matter in ways that research needs to understand to inform policies to prevent discrimination, unequal access to resources, and social inequality.

“Because the interviewer’s perception is subjective, researchers need to account for the characteristics and life experiences of the interviewer in interpreting their ratings. A wealth of research on perceived attractiveness (that is, as perceived by others, not oneself) has shown that such ratings vary according to the characteristics of the rater. For example, a male interviewer might rate a female’s attractiveness according to different criteria than a female interviewer rating the same female’s attractiveness. Other interviewer characteristics that are important to take into account are age, race, ethnicity, education, geographic location, and life experiences, in general. Notably, several characteristics of the interviewers are available in the restricted use Add Health dataset at Waves 3 and 4. It is these data (e.g., interviewer age, sex, race, ethnicity, education) that might more usefully inform an analysis undertaken to investigate the role of other-perceived versus self-perceived attractiveness on some outcome of interest (employment, health, etc)….

so, to settle the question of “who were the interviewers,” somebody just needs to go get the data from add health and blog it. since it’s “restricted use” data, presumably a non-accredited nobody like yours truly prolly wouldn’t get access. but maybe some actual scientist** (perhaps one who is also a blogger!) will step up to the plate.

the neuroskeptic looked at the possible bias of the interviewers from another, very creative if i might say, angle. he tried to find out if any other researchers had found an anti-black bias on the part of the add health interviewers:

“The obvious problem is that maybe the interviewers were biased against black women, and rated them lower for that reason. Kanazawa didn’t consider this in his post, which is unquestionably an oversight, but he did go on to speculate as to the biological reasons why they might be less attractive.

“However, looking at the original Add Health data, can we check whether this bias was at play or not?

“Short answer: I found no evidence either way.

“Long answer: I first looked over the Add Health website but it doesn’t seem to mention anything about who the interviewers were. It doesn’t mention their own ethnicity, which would be helpful, although even if they were all black themselves, they might have internalized racism, so that wouldn’t be conclusive. They were trained, but then, you can’t train someone to not be a racist.

“Then I decided to look at the publications. I searched Google Scholar for ‘Add Health’ + attractiveness. This reveals a number of articles, including a 2007 one by Kanazawa ironically, but only one seemed really relevant: Weight Preoccupation as a Function of Observed Physical Attractiveness. (There are other hits, but I skimmed the most likely looking ones and they didn’t address bias.)

“The details are unimportant, but it involved race and attractiveness, so the authors had to deal with the question of potential rater bias. Unlike Kanazawa they didn’t just brush this under the carpet:

‘Although the interviewers were different races and ethnicities, there is no information about the race or ethnicity of the interviewer for any one respondent to examine systematic bias. [altho the add health people above seem to say otherwise for waves iii and iv. – hbdchick]

‘However, post hoc cluster analyses that controlled for an interviewer effect yielded similar results; thus, it is unlikely that interviewers had any substantial biases against any one ethnic group or that they rated attractiveness significantly differently from each other.’

“The point about ‘post-hoc cluster analysis’ is the key here. To try to control for rater effects (not just racial ones) they analyzed the data covarying for which interviewer rated each girl. They didn’t know what races the interviewers were, but they did know which girls got rated by the same interviewer. They found that controlling for the rater did not affect their results.

“So does that mean there was no bias? No. Because – this only applies to their results, which were not about attractiveness per se, but about the interaction of attractiveness with other factors to predict an outcome variable (dieting and concern about weight) within a given race….

“So in my judgement, we just can’t tell. Unless I’ve missed something, in which case, please tell us about it in the comments.”

previously: silly refutations of kanazawa’s blog post and the offensive mr. kanazawa

**i am, emphatically, NOT a scientist. i don’t even play one here on the innerwebs. i’m just a lay person interested in science-y stuff.

(note: comments do not require an email. or anti-matter. wait. wha?)


silly refutations of kanazawa’s blog post

there’ve been a lot — a LOT — of refutations of kanazawa’s post on the attractiveness of black women. i haven’t read them all — in fact i’ve read hardly any of them ’cause most of them just scream and yell WAAAAAAYCIST!! BURN HIM, BURN HIM!!


some of them, however, appear to be more scientific refutations. bering in mind links to another psychology today blogger (scott barry kaufman) who (along with someone named jelte wicherts in the netherlands) has supposedly (according to bering in mind) “failed to replicate” kanazawa’s findings.

i took at look at kaufman’s post — and the technical summary of their analysis — ’cause i thought, well gee — not able to replicate the findings — that’d be interesting.


here’s the evidence that kaufman and wicherts present (in the blog post) to show that kanazawa’s analysis was incorrect:

looks like black women were rated nearly as attractive as white women, right? and they were. in wave iv.

here’s how they fudged the data.

the add health thingie (from whence all the data comes) involved four waves of surveys over the course of several years (a couple of decades?). kaufman and wicherts decided that the only wave that should be included in any analysis on the attractiveness of women is wave iv, because in that wave the females were of legal age and, therefore, women.

no, i’m not kidding. they really said that!

now, i would agree with them if the subjects in the previous waves had been pre-pubescent children. but the ages in the waves were:

wave i = mean age 15.9 years
wave ii = average age 16.5 years
wave iii = mean age 22.1 years
wave iv = mean age 28.9 years

now come on! ok, so in waves i-iii most or all of the subjects were not of legal age, but probably the vast majority (except maybe for some late bloomers in wave i) were “reproductively of age” — meaning they could make babies. which is what the whole discussion is about! attractiveness, after all, is about attracting a mate.

*cough, cough* cherry-picking *cough, cough*

kanazawa didn’t include wave iv in his analysis, which is the wave when the attractiveness of whites and blacks was rated the most similar. don’t know why he didn’t use the data from that wave. kaufman mentions that the data from that wave has been available “for over a month.” well, maybe kanazawa didn’t realize the newest data had been published when he ran his analysis. i really dunno, but it’s definitely possible.

in any case, waves i-iii show that black women were rated as less attractive, altho i think the numbers in wave iii are not statistically significant. in wave iv, as i’ve said above, the rating for whites and blacks were pretty similar:

i still wanna know who the interviewers|evaluators were. were they all white folks? all asians? all illegal mexican workers picked up outside home-depot? their characteristics might’ve influenced the results.

actually, now that i mention it, the fieldwork for waves iii and iv was contracted out to a different company than waves i and ii. wonder if that made any difference in the evaluations?

btw, some real word data from okcupid maybe lends more support to kanazawa’s findings:

“Men don’t write black women back. Or rather, they write them back far less often than they should. Black women reply the most, yet get by far the fewest replies. Essentially every race—including other blacks—singles them out for the cold shoulder.”

that’s too bad.

previously: the offensive mr. kanazawa and african-american porn stars

update 06/07: see also on the add health interviewers

(note: comments do not require an email. or an eharmony.com subscription.)

the offensive mr. kanazawa

you prolly heard by now that satoshi kanazawa says black women are ugly.

of course, he didn’t really, but who cares about silly little ol’ details like that.

what happened was kanazawa got some “attractiveness” data from add health. the evaluations were made “three times by three different interviewers over seven years” (during waves i – iii). the data showed that, consistently, black females scored lower in attractiveness than white, asian, or native american women; this did not happen in the case of black males.

andrew over @evolvify makes a good point about how attractiveness was rated:

“At this point, we have no idea who the interviewers who rated the students were. The attractiveness ratings would have been altered by varying degrees of prior familiarity between the individuals… whether the interviewers were of a certain age… the same sex or opposite sex breakdown… ingroup/outgroup… interviewer race… et cetera. There are simply a lot of variables that bring the reliability of attractiveness data into question. Perhaps this information is available, but it wasn’t in Kanazawa’s article, and I couldn’t find it on the study’s website.”

well, add health (a university of south north carolina entity) apparently outsourced its fieldwork positions to both the national opinion research center (norc) of the university of chicago (waves i and ii) and rti international (wave iii).

fieldworkers seem to be part-time, contract workers, afaics. with the connection to the unversity of south north carolina and the university of chicago, i was guessing that the fieldworkers were likely college students — maybe grad students interested in the field — sociology or whatever the heck it is (i’m talking about the interviewers here — i realize that the add health people are in the medical field). and, who winds up in sociology? mostly white women, with perhaps a few asian women thrown in. so, i was thinking that a lot of the interviewers might’ve been young white college women — and they might not find black women to be very attractive.

however, on the norc website, there are some videos of field interviewers explaining why they love their job, etc., etc., and they’re all older folks, i.e. not college students. of course, i’m sure this group is a pc-selected group — they’ve got almost all the races included there. but, still, three out of the five are white folks. if that is at all representative, then, yeah — there could, again, very well be some bias introduced here.

it’s still interesting that black women were consistently evaluated as the least attractive, but who were the evaluators? if they had been all black men, perhaps the results would’ve been different.

what i think is even more interesting is that blacks — both men and women — consistently rated themselves as attractive or highly attractive, more than members of the other races. black and proud! good for them!:

update 05/19 – charts with y-axes starting at zero:

update – see also ANOTHER watsoning in the air?!

update 05/27 – wicherts and kaufman’s criticism of kanazawa’s number crunching: An independent analysis of race differences in ratings of attractiveness in the Add Health Study. “Kanazawa interpreted his results incorrectly as having a bearing on attractiveness of women because the ratings were taken mostly when Add Health participants were teenagers.” update 06/07: see silly refutations of kanazawa’s blog post

update 06/07: see also on the add health interviewers

(note: comments do not require an email.)