In general it’s probably best not to anonymize applications. Field studies generally show no effect on interview selection, and sometimes even show a negativeeffect (which has also been seen in the lab). Blinding may work for musicians, randomly generated resumes, and identical expressions of interest, but in reality there seem to be subtle cues of an applicant’s background that evaluators may pick up on, and the risk of anonymization backfiring is higher for recruiting groups which are actively interested in DEI. This may be because they are unable to proactively check their biases when blind, or to proactively accommodate disadvantaged candidates at this recruitment stage, or because their staff is already more diverse and people may favor candidates they identify with demographically.
I think you are mis-describing these studies. Essentially, they found that when reviewers knew the race and sex of the applicants, they were biased in favour of women and non-whites, and against white males.
I admit I only read two of the studies you linked two, but I think these quotes from them are quite clear that about the conclusions:
We find that participating firms become less likely to interview and hire minority candidates when receiving anonymous resumes.
The public servants reviewing the job applicants engaged in discrimination that favoured female applicants and disadvantaged male candidates
Affirmative action towards the Indigenous female candidate is the largest, being 22.2% more likely to be short listed on average when identified compared to the de-identified condition. On the other hand, the identified Indigenous male CV is 9.4% more likely to be shortlisted on average compared to when it is de-identified. In absolute terms most minority candidates are on average more likely to be shortlisted when named compared to the de-identified condition, but the difference for the Indigenous female candidate is the only one that is statistically significant at the 95% confidence level.
This is also supported by other papers on the subject. For example, you might enjoy reading Williams and Ceci (2015):
The underrepresentation of women in academic science is typically attributed, both in scientific literature and in the media, to sexist hiring. Here we report five hiring experiments in which faculty evaluated hypothetical female and male applicants, using systematically varied profiles disguising identical scholarship, for assistant professorships in biology, engineering, economics, and psychology. Contrary to prevailing assumptions, men and women faculty members from all four fields preferred female applicants 2:1 over identically qualified males with matching lifestyles (single, married, divorced), with the exception of male economists, who showed no gender preference. Comparing different lifestyles revealed that women preferred divorced mothers to married fathers and that men preferred mothers who took parental leaves to mothers who did not. Our findings, supported by real-world academic hiring data, suggest advantages for women launching academic science careers.
This doesn’t mean that anonymizing applications is a bad idea—it appears to have successfully reduced unfair bias—rather that the bias was in the opposite direction than the authors expected to find it.
Here is a recent study on the topic that I think is very relevant:
Gender, Race, and Entrepreneurship: A Randomized Field Experiment on Venture Capitalists and Angels (Gornall and Strebulaev)
We sent out 80,000 pitch emails introducing promising but fictitious start-ups to 28,000 venture capitalists and business angels. Each email was sent by a fictitious entrepreneur with a randomly selected gender (male or female) and race (Asian or White). Female entrepreneurs received an 8% higher rate of interested replies than male entrepreneurs pitching identical projects. Asian entrepreneurs received a 6% higher rate than White entrepreneurs. Our results are not consistent with discrimination against females or Asians at the initial contact stage of the investment process.
However, it does seem pretty applicable to EA. The EA community is in many ways similar to the VC community:
Similar geographies: the Bay Area, London, New York etc.
Similar education backgrounds.
Both involve evaluating speculative projects with a lot of uncertainty.
Similarly to the studies discussed above, this finds that people are biased against white men.
(I have some qualms about this type of study, because they involve wasting people’s time without their consent, but this doesn’t affect the conclusions.)
The link you shared does not work, but I assume it was meant to be pointing at the classic study on orchestral interviews from 1997/2000. However, recent re-analysis of the paper (here, here, here, here) shows if anything it supports the opposite conclusion:
This table unambiguously shows that men are doing comparatively better in blind auditions than in non-blind auditions. The −0.022 number is the proportion of women that are successful in the audition process minus the proportion of men that are successful. Thus a larger proportion of men than women are successful in blind auditions, the exact opposite of what is claimed.
The ‘fact’ that blinded auditions help women overcome bias in non-blinded auditions came from some dubious pre-replication-crisis analysis, where the authors picked a small subset (often less than three orchestras!) of the data to try to find the effect they are looking for:
The impact of the screen is positive and large in magnitude, but only when there is no semifinal round. Women are about 5 percentage points more likely to be hired than are men in a completely blind audition, although the effect is not statistically significant. The effect is nil, however, when there is a semifinal round, perhaps as a result of the unusual effects of the semifinal round.
… but even with this p-hacking the authors failed to achieve statistical significance.
So on the whole this suggests that musician interviews is another case where the process was originally biased against men, and blinding helped reduce this bias.
Thanks very much for linking that Williams and Ceci article. That was really interesting and quite heartening. I say heartening because I don’t think the bias being shown in that article is unfair. I think the gender of the candidate is a relevant factor in this instance, and in this scenario preferring women when all else is equal will ultimately lead to better outcomes for society.
Those decisions are being made in a context of women being underrepresented in the fields* and I think science is a field where equality in gender representation carries instrumental value. I think this instrumental value comes from provision of new perspectives and minimising blind spots, creating an environment conducive to all people contributing their best, and working towards a stronger applicant pool in the future, one where talented women aren’t discouraged from pursuing careers in these fields. So at this point in time, to me it seems that, all else being equal, being female makes you a more valuable candidate in those fields. This may change in the future if parity in representation is reached; in that case I think it could be unfair and potentially damaging for science and society if there was a persistent bias in favour of females.
To take a different example, I think gender equality is also valuable in school teaching. If I were a school principal and the vast majority of my teaching faculty were female I think I have good reason to prefer a male candidate for a new position if all else was equal in applications.
I think Kelly’s recommendations are aimed at someone who has decided that they want to improve diversity in their organisation/field, so it seems fine to be explicit about when tactics are or aren’t helpful for this particular aim. She’s given some reasons why diversity might be valuable in general but of course the value of diversity will vary depending on the field and context. If you don’t agree that gender equality carries value in science I’d be interested to hear why you hold that view.
*The article notes that in two of the fields (engineering and economics) “women are substantially underrepresented” and in two (biology and psychology) “women are well represented”. Unfortunately I can’t access the cited paper that describes what they mean by “well represented” – some quick googling suggests that women are still under-represented in higher positions in those fields, but feel free to correct that if you have better sources.
I think you are mis-describing these studies. Essentially, they found that when reviewers knew the race and sex of the applicants, they were biased in favour of women and non-whites, and against white males.
I admit I only read two of the studies you linked two, but I think these quotes from them are quite clear that about the conclusions:
This is also supported by other papers on the subject. For example, you might enjoy reading Williams and Ceci (2015):
This doesn’t mean that anonymizing applications is a bad idea—it appears to have successfully reduced unfair bias—rather that the bias was in the opposite direction than the authors expected to find it.
Here is a recent study on the topic that I think is very relevant:
However, it does seem pretty applicable to EA. The EA community is in many ways similar to the VC community:
Similar geographies: the Bay Area, London, New York etc.
Similar education backgrounds.
Both involve evaluating speculative projects with a lot of uncertainty.
Similarly to the studies discussed above, this finds that people are biased against white men.
(I have some qualms about this type of study, because they involve wasting people’s time without their consent, but this doesn’t affect the conclusions.)
The link you shared does not work, but I assume it was meant to be pointing at the classic study on orchestral interviews from 1997/2000. However, recent re-analysis of the paper (here, here, here, here) shows if anything it supports the opposite conclusion:
The ‘fact’ that blinded auditions help women overcome bias in non-blinded auditions came from some dubious pre-replication-crisis analysis, where the authors picked a small subset (often less than three orchestras!) of the data to try to find the effect they are looking for:
… but even with this p-hacking the authors failed to achieve statistical significance.
So on the whole this suggests that musician interviews is another case where the process was originally biased against men, and blinding helped reduce this bias.
Hi Larks,
Thanks very much for linking that Williams and Ceci article. That was really interesting and quite heartening. I say heartening because I don’t think the bias being shown in that article is unfair. I think the gender of the candidate is a relevant factor in this instance, and in this scenario preferring women when all else is equal will ultimately lead to better outcomes for society.
Those decisions are being made in a context of women being underrepresented in the fields* and I think science is a field where equality in gender representation carries instrumental value. I think this instrumental value comes from provision of new perspectives and minimising blind spots, creating an environment conducive to all people contributing their best, and working towards a stronger applicant pool in the future, one where talented women aren’t discouraged from pursuing careers in these fields. So at this point in time, to me it seems that, all else being equal, being female makes you a more valuable candidate in those fields. This may change in the future if parity in representation is reached; in that case I think it could be unfair and potentially damaging for science and society if there was a persistent bias in favour of females.
To take a different example, I think gender equality is also valuable in school teaching. If I were a school principal and the vast majority of my teaching faculty were female I think I have good reason to prefer a male candidate for a new position if all else was equal in applications.
I think Kelly’s recommendations are aimed at someone who has decided that they want to improve diversity in their organisation/field, so it seems fine to be explicit about when tactics are or aren’t helpful for this particular aim. She’s given some reasons why diversity might be valuable in general but of course the value of diversity will vary depending on the field and context. If you don’t agree that gender equality carries value in science I’d be interested to hear why you hold that view.
*The article notes that in two of the fields (engineering and economics) “women are substantially underrepresented” and in two (biology and psychology) “women are well represented”. Unfortunately I can’t access the cited paper that describes what they mean by “well represented” – some quick googling suggests that women are still under-represented in higher positions in those fields, but feel free to correct that if you have better sources.