Any competent outside firm would gather input from stakeholders before releasing a survey. But I hear the broader concern, and note that some sort of internal-external hybrid is possible. The minimal level of outside involvement, to me, would involve serving as a data guardian, data pre-processor, and auditor-of-sorts. This is related to the two reasons I think outside involvement is important: external credibility, and respondent assurance.
As far as external validity, I think media reports like this have the capacity to do significant harm to EA’s objectives. Longtermist EA remains, on the whole, more talent-constrained and influence-constrained than funding-constrained. The adverse effect on talent joining EA could be considerable. Social influence is underrated; for example, technically solving AI safety might not actually accomplish much without the ability to socially pressure corporations to adopt effective (but profit-reducing) safety methods or convince governments to compel them to do so.
When the next article comes out down the road, here’s what I think EA would be best served by being able to say if possible:
(A) According to a study overseen by a respected independent investigator, the EA community’s rate of sexual misconduct is at most no greater than the base rate.
(B) We have best-in-class systems in place for preventing sexual misconduct and supporting survivors, designed in connection with outside experts. We recognize that sexual misconduct does occur, and we have robust systems for responding to reports and taking the steps we can to protect the community. There is independent oversight over the response system.
(C) Unfortunately, there isn’t that much we can do about problematic individuals who run in EA-adjacent circles but are unaffiliated with institutional EA.
(A) isn’t externally credible without some independent organization vouching for the analysis in some fashion. In my view, (B) requires at some degree of external oversight to be externally credible after the Owen situation, but that’s another story. Interestingly, I think a lot of the potential responses are appropriate either as defensive measures under the “this is overblown reporting by hostile media outlets” hypothesis or “there is a significant problem here” hypothesis. I’d like to see at least funding and policy commitments on some of those initatives in the near term, which would reduce the time pressure on other initiatives for which there is a good chance that further datagathering would substantially change the desirability, scope, layout, etc.
I think one has to balance the goal of external credibility against other goals. But moving the research to (say) RP as opposed to CEA wouldn’t move the external-credibility needle in any appreciable fashion.
The other element here is respondent assurance. Some respondents, especially those no longer associated with EA, may be more comfortable giving responses if the initial data collection itself and any necessary de-identification is done by an outside organization. (It’s plausible to me that the combination of responses in a raw survey response could be uniquely identifying.)
Ideally, you would want to maximize the number of survivors who would be willing to confidentally name the person who committed misconduct. This would allow the outside organization to do a few things that would address methodological concerns in the Time article. First, it could identify perpetrators who had committed misconduct against multiple survivors, avoiding the incorrect impression that perpetrators were more numerous than they were. Second, it could use pre-defined criteria to determine if the perpetrator was actually an EA, again addressing one of the issues with the Time article. Otherwise, you end up with a numerator covering all instances in which someone reports misconduct by someone they identified as an EA . . . but use narrower criteria to develop the denominator, leading to an inflated figure. It would likely be legally safer for CEA to turn over its event-ban list to the outside organization under an NDA for very limited purposes than it would be to turn it over to RP. That would help another criticism of the Time article, that it failed to address CEA’s response to various incidents.
Contingent on budget and maybe early datagathering, I would consider polling men too about things like attitudes associated with rape culture. Surveying or focusing-grouping people about deviant beliefs and behaviors (I’m using “deviant” here as sociologists do), not to mention their own harassment or misconduct, is extremely challenging to start with. You need an independent investigator with ironclad promises of confidentiality to have a chance at that kind of research. But then again, it’s been almost 20 years since my somewhat limited graduate training in social science research methods, so I could be wrong on this.
Any competent outside firm would gather input from stakeholders before releasing a survey. But I hear the broader concern, and note that some sort of internal-external hybrid is possible. The minimal level of outside involvement, to me, would involve serving as a data guardian, data pre-processor, and auditor-of-sorts. This is related to the two reasons I think outside involvement is important: external credibility, and respondent assurance.
As far as external validity, I think media reports like this have the capacity to do significant harm to EA’s objectives. Longtermist EA remains, on the whole, more talent-constrained and influence-constrained than funding-constrained. The adverse effect on talent joining EA could be considerable. Social influence is underrated; for example, technically solving AI safety might not actually accomplish much without the ability to socially pressure corporations to adopt effective (but profit-reducing) safety methods or convince governments to compel them to do so.
When the next article comes out down the road, here’s what I think EA would be best served by being able to say if possible:
(A) According to a study overseen by a respected independent investigator, the EA community’s rate of sexual misconduct is at most no greater than the base rate.
(B) We have best-in-class systems in place for preventing sexual misconduct and supporting survivors, designed in connection with outside experts. We recognize that sexual misconduct does occur, and we have robust systems for responding to reports and taking the steps we can to protect the community. There is independent oversight over the response system.
(C) Unfortunately, there isn’t that much we can do about problematic individuals who run in EA-adjacent circles but are unaffiliated with institutional EA.
(A) isn’t externally credible without some independent organization vouching for the analysis in some fashion. In my view, (B) requires at some degree of external oversight to be externally credible after the Owen situation, but that’s another story. Interestingly, I think a lot of the potential responses are appropriate either as defensive measures under the “this is overblown reporting by hostile media outlets” hypothesis or “there is a significant problem here” hypothesis. I’d like to see at least funding and policy commitments on some of those initatives in the near term, which would reduce the time pressure on other initiatives for which there is a good chance that further datagathering would substantially change the desirability, scope, layout, etc.
I think one has to balance the goal of external credibility against other goals. But moving the research to (say) RP as opposed to CEA wouldn’t move the external-credibility needle in any appreciable fashion.
The other element here is respondent assurance. Some respondents, especially those no longer associated with EA, may be more comfortable giving responses if the initial data collection itself and any necessary de-identification is done by an outside organization. (It’s plausible to me that the combination of responses in a raw survey response could be uniquely identifying.)
Ideally, you would want to maximize the number of survivors who would be willing to confidentally name the person who committed misconduct. This would allow the outside organization to do a few things that would address methodological concerns in the Time article. First, it could identify perpetrators who had committed misconduct against multiple survivors, avoiding the incorrect impression that perpetrators were more numerous than they were. Second, it could use pre-defined criteria to determine if the perpetrator was actually an EA, again addressing one of the issues with the Time article. Otherwise, you end up with a numerator covering all instances in which someone reports misconduct by someone they identified as an EA . . . but use narrower criteria to develop the denominator, leading to an inflated figure. It would likely be legally safer for CEA to turn over its event-ban list to the outside organization under an NDA for very limited purposes than it would be to turn it over to RP. That would help another criticism of the Time article, that it failed to address CEA’s response to various incidents.
Contingent on budget and maybe early datagathering, I would consider polling men too about things like attitudes associated with rape culture. Surveying or focusing-grouping people about deviant beliefs and behaviors (I’m using “deviant” here as sociologists do), not to mention their own harassment or misconduct, is extremely challenging to start with. You need an independent investigator with ironclad promises of confidentiality to have a chance at that kind of research. But then again, it’s been almost 20 years since my somewhat limited graduate training in social science research methods, so I could be wrong on this.