Thanks for sharing these studies explaining why you are doing this. Unfortunately, in general I am very skeptical of the sort of studies you are referencing. The researchers typically have a clear agenda—they know what conclusions they want to come to ahead of time, and what conclusions will most advantageous to their career—and the statistical rigour is often lacking, with small sample sizes, lack of pre-registration, p-hacking, and other issues. I took a closer look at the four sources you referenced to see if these issues applied.
When more women participate in traditionally male-dominated fields like the sciences, the breadth of knowledge in that area usually grows, a surge in female involvement directly correlates with advancements in understanding[1]. [emphasis added]
The link you provide here, to a 2014 article in National Geographic, has a lot of examples of cases where male researchers supposedly overlooked the needs of women (e.g. not adequately studying how women’s biology affects how drugs and seat belts should work, or the importance of cleaning houses), and suggests that increasing number of female scientists helped address this. But female scientists being better at understanding women seems less relevant to AI technical alignment work, because AIs are not female or male. Maybe it is useful for understanding what distinctly female values we want AIs to promote, but it doesn’t seem particularly relevant for things like Interpretability or most other current research agendas. The article also suggests that women are more communal and emotionally aware, vs men who are more agentic. But it doesn’t really make any claims about overall levels of understanding ‘directly correlating’ with female involvement, especially in more abstract, less biological fields, and the word ‘correlate’ literally does not appear in the text.
Cox & Fisher (2008) found that women in a single-sex environment in a software engineering course reported higher levels of enjoyment, fairness, motivation, support, and comfort and allowed them to perform at a level that exceeded that of the all-male groups in the class [1].
The first paper describes a n=7 study of a female group project, which apparently scored more highly than other group projects run by men. The study was not pre-registered, blinded or randomised, the researcher was an active participant, and there was no control. The author also obliquely references the need to avoid ″rigid marking schemes’ if these might reveal the all-female group performing worse, which suggests a bias to me.
Kahveci (2008) explored a program for women in science, mathematics, and engineering and found that it helped marginalized women move towards legitimate participation in these fields and enhanced a sense of community and mutual engagement [2].
The second paper describes a n=74 study of a women-in-science program, where the positive result is basically that the participants gave positive reviews to the program and said it made them more likely to do science. The study was not pre-registered, blinded or randomised, the researcher was an active participant, and there was no control. The only concrete example provided of a student switching major was from Biology to Exercise Physiology, which seems like a move away from core science.
“It is not about men against women, but there is evidence to show through research that when you have more women in public decision-making, you get policies that benefit women, children and families in general. When women are in sufficient numbers in parliaments they promote women’s rights legislation, children’s rights and they tend to speak up more for the interests of communities, local communities, because of their close involvement in community life. [2]
The link here goes to a web page with a quote from Oxfam. There are no links to the evidence or research that supposedly backs up the claim.
Overall, my opinion of the linked research is it has very little scientific merit. They provide some interesting anecdotes, and the authors have some theories that someone else could test. But to the extent you are highlighting them because they are cruxes for your theory of change, they seem very weak. If your ‘Why We Are Doing This’ had been premised on ‘well some women just like sex-segregated programs, so proving this option will help with recruitment’ then I would have said fair enough. But if, as this post suggests, your theory of change is based on these sorts of dubious studies then that makes me significantly less optimistic about the project.
I upvoted this comment, since I think it’s a correct critique of poor quality studies and adds important context, but I also wanted to flag that I also broadly think Athena is a worthwhile initiative and I’m glad it’s happening! (In line with Lewis’ argument below). I think it can create bad vibes for the highest voted comment on a post about promoting diversity to be critical
Usually, if someone proposes something and then cites loads of weak literature supporting it, criticism is warranted. I think it is a good norm for people promoting anything to make good arguments for it and provide good evidence.
Maybe it is useful for understanding what distinctly female values we want AIs to promote, but it doesn’t seem particularly relevant for things like Interpretability or most other current research agendas.
Could it be that perhaps the research agendas themselves could benefit from a more diverse set of perspectives? I have not thought this through as carefully as you have but the seatbelt analogy seem perhaps appropriate—perhaps the issue there was exactly that the research agenda on seat belts did not include the impact on e.g. pregnant women (speculation from my side). Half the people affected by AI will be women so maybe mostly-men teams could possibly overlook considerations that apply less to men and more to women?
Thanks for sharing these studies explaining why you are doing this. Unfortunately, in general I am very skeptical of the sort of studies you are referencing. The researchers typically have a clear agenda—they know what conclusions they want to come to ahead of time, and what conclusions will most advantageous to their career—and the statistical rigour is often lacking, with small sample sizes, lack of pre-registration, p-hacking, and other issues. I took a closer look at the four sources you referenced to see if these issues applied.
The link you provide here, to a 2014 article in National Geographic, has a lot of examples of cases where male researchers supposedly overlooked the needs of women (e.g. not adequately studying how women’s biology affects how drugs and seat belts should work, or the importance of cleaning houses), and suggests that increasing number of female scientists helped address this. But female scientists being better at understanding women seems less relevant to AI technical alignment work, because AIs are not female or male. Maybe it is useful for understanding what distinctly female values we want AIs to promote, but it doesn’t seem particularly relevant for things like Interpretability or most other current research agendas. The article also suggests that women are more communal and emotionally aware, vs men who are more agentic. But it doesn’t really make any claims about overall levels of understanding ‘directly correlating’ with female involvement, especially in more abstract, less biological fields, and the word ‘correlate’ literally does not appear in the text.
The first paper describes a n=7 study of a female group project, which apparently scored more highly than other group projects run by men. The study was not pre-registered, blinded or randomised, the researcher was an active participant, and there was no control. The author also obliquely references the need to avoid ″rigid marking schemes’ if these might reveal the all-female group performing worse, which suggests a bias to me.
The second paper describes a n=74 study of a women-in-science program, where the positive result is basically that the participants gave positive reviews to the program and said it made them more likely to do science. The study was not pre-registered, blinded or randomised, the researcher was an active participant, and there was no control. The only concrete example provided of a student switching major was from Biology to Exercise Physiology, which seems like a move away from core science.
The link here goes to a web page with a quote from Oxfam. There are no links to the evidence or research that supposedly backs up the claim.
Overall, my opinion of the linked research is it has very little scientific merit. They provide some interesting anecdotes, and the authors have some theories that someone else could test. But to the extent you are highlighting them because they are cruxes for your theory of change, they seem very weak. If your ‘Why We Are Doing This’ had been premised on ‘well some women just like sex-segregated programs, so proving this option will help with recruitment’ then I would have said fair enough. But if, as this post suggests, your theory of change is based on these sorts of dubious studies then that makes me significantly less optimistic about the project.
I upvoted this comment, since I think it’s a correct critique of poor quality studies and adds important context, but I also wanted to flag that I also broadly think Athena is a worthwhile initiative and I’m glad it’s happening! (In line with Lewis’ argument below). I think it can create bad vibes for the highest voted comment on a post about promoting diversity to be critical
Usually, if someone proposes something and then cites loads of weak literature supporting it, criticism is warranted. I think it is a good norm for people promoting anything to make good arguments for it and provide good evidence.
Agreed!
+1, I appreciate you for upvoting the parent comment and then leaving this reply :)
(Edit: for what it’s worth, I am also excited Athena is happening)
Could it be that perhaps the research agendas themselves could benefit from a more diverse set of perspectives? I have not thought this through as carefully as you have but the seatbelt analogy seem perhaps appropriate—perhaps the issue there was exactly that the research agenda on seat belts did not include the impact on e.g. pregnant women (speculation from my side). Half the people affected by AI will be women so maybe mostly-men teams could possibly overlook considerations that apply less to men and more to women?