I think a very relevant question is to ask is how come none of the showy self-criticism contests and red-teaming exercises came up with this? A good amount of time and money and energy were put into such things and if the exercises are not in fact uncovering the big problems lurking in the movement then that suggests some issues
If this comment is more about “how could this have been foreseen”, then this comment thread may be relevant. I should note that hindsight bias means that it’s much easier to look back and assess problems as obvious and predictable ex post, when powerful investment firms and individuals who also had skin in the game also missed this.
TL;DR: 1) There were entries that were relevant (this one also touches on it briefly) 2) They were specifically mentioned 3) There were commentsrelevant to this. (notably one of these was apparently deleted because it received a lot of downvotes when initially posted) 4) There has been at least two other posts on the forum prior to the contest that engaged with this specifically
My tentative take is that these issues were in fact identified by various members of the community, but there isn’t a good way of turning identified issues into constructive actions—the status quo is we just have to trust that organisations have good systems in place for this, and that EA leaders are sufficiently careful and willing to make changes or consider them seriously, such that all the community needs to do is “raise the issue”. And I think looking at the systems within the relevant EA orgs or leadership is what investigations or accountability questions going forward should focus on—all individuals are fallible, and we should be looking at how we can build systems in place such that the community doesn’t have to just trust that people who have power and who are steering the EA movement will get it right, and that there are ways for the community to hold them accountable to their ideals or stated goals if it appears to, or risks not playing out in practice.
i.e. if there are good processes and systems in place and documentation of these processes and decisions, it’s more acceptable (because other organisations that probably have a very good due diligence process also missed it). But if there weren’t good processes, or if these decisions weren’t a careful + intentional decision, then that’s comparatively more concerning, especially in context of specific criticisms that have been raised,[1] or previous precedent. For example, I’d be especially curious about the events surrounding Ben Delo,[2] and processes that were implemented in response. I’d be curious about whether there are people in EA orgs involved in steering who keep track of potential risks and early warning signs to the EA movement, in the same way the EA community advocates for in the case of pandemics, AI, or even general ways of finding opportunities for impact. For example, SBF, who is listed as a EtG success story on 80k hours, has publicly stated he’s willing to go 5x over the Kelly bet, and described yield farming in a way that Matt Levine interpreted as a Ponzi. Again, I’m personally less interested in the object level decision (e.g. whether or not we agree with SBF’s Kelly bet comments as serious, or whether Levine’s interpretation as appropriate), but more about what the process was, how this was considered at the time with the information they had etc. I’d also be curious about the documentation of any SBF related concerns that were raised by the community, if any, and how these concerns were managed and considered (as opposed to critiquing the final outcome).
Outside of due diligence and ways to facilitate whistleblowers, decision-making processes around the steering of the EA movement is crucial as well. When decisions are made by orgs that bring clear benefits to one part of the EA community while bringing clear risks that are shared across wider parts of the EA community,[3] it would probably be of value to look at how these decisions were made and what tradeoffs were considered at the time of the decision. Going forward, thinking about how to either diversify those risks, or make decision-making more inclusive of a wider range stakeholders[4], keeping in mind the best interests of the EA movement as a whole.
(this is something I’m considering working on in a personal capacity along with the OP of this post, as well as some others—details to come, but feel free to DM me if you have any thoughts on this. It appears that CEA is also already considering this)
If this comment is about “are these red-teaming contests in fact valuable for the money and time put into it, if it misses problems like this”
I think my view here (speaking only for the red-teaming contest) is that even if this specific contest was framed in a way that it missed these classes of issues, the value of the very top submissions[5] may still have made the efforts worthwhile. The potential value of a different framing was mentioned by another panelist. If it’s the case that red-teaming contests are systematically missing this class of issues regardless of framing, then I agree that would be pretty useful to know, but I don’t have a good sense of how we would try to investigate this.
This tweet seems to have aged particularly well. Despite supportive comments from high-profile EAs on the original forum post, the author seemed disappointed that nothing came of it in that direction. Again, without getting into the object level discussion of the claims of the original paper, it’s still worth asking questions around the processes. If there was were actions planned, what did these look like? If not, was that because of a disagreement over the suggested changes, or the extent that it was an issue at all? How were these decisions made, and what was considered?
Apparently a previous EA-aligned billionaire ?donor who got rich by starting a crypto trading firm, who pleaded guilty to violating the bank secrecy act
Even before this, I had heard from a primary source in a major mainstream global health organisation that there were staff who wanted to distance themselves from EA because of misunderstandings around longtermism.
This doesn’t have to be a lengthy deliberative consensus-building project, but it should at least include internal comms across different EA stakeholders to allow discussions of risks and potential mitigation strategies.
...I for one didn’t take the self-criticism stuff particularly seriously, or consider EA to have scored any points by running the contest? And I thought that seemed too obvious to mention, I guess, that of course scoring diligent self-criticism points is harder than that? Zvi had a longer writeup on a similar take.
I have my own sense of what EA is doing all wrong, but it didn’t particularly occur to me to try to write it up for the criticism contest. That was obviously going to be an in-frame thing, and was obviously not going to be able to reward anything out-of-frame; doing that is really hard if you don’t want to just completely throw the contest over to wild randos. Like, the reason MIRI doesn’t run a contest like that is not that our underlying reality contains nothing worth criticizing, but that a contest like that obviously does not work to highlight your real actual problems.
There’s nothing particularly unhealthy about a movement where a contest like that fails to turn up any real issues—the contest is obviously doomed in that sense from the start. The only failure is the naivety required to imagine that a contest like that could possibly work, against a semiefficient background where smart people have already been saying various things and there are already reasons why those critiques have not been accepted. The best you can say about a contest like that is that maybe it’s worth $100K to get better centralized writeups of in-frame criticisms that your community already knows how to accept but hasn’t finished acting on yet.
There’s a related problem with CEA. Their mission is to nurture the EA community, and their sizeable community health team’s is to “preserve the EA community’s ability to grow and produce value”. But the CH team has focused on public relations, diversity, “discussion norms”, and crisis-response, while the community became unhealthy for a different reason—FTX. It seems like this team, or some other, needs to take on a wider mandate, that includes monitoring of such issues.
I would like to know, for example, whether any large EA orgs (and EA thinkers, I suppose too) have formally engaged with those criticism contest entries. Like, was it somebody’s job to think about when anything should change as a result? Or something discussed by a governance board (or similar)?
I think a very relevant question is to ask is how come none of the showy self-criticism contests and red-teaming exercises came up with this? A good amount of time and money and energy were put into such things and if the exercises are not in fact uncovering the big problems lurking in the movement then that suggests some issues
If this comment is more about “how could this have been foreseen”, then this comment thread may be relevant. I should note that hindsight bias means that it’s much easier to look back and assess problems as obvious and predictable ex post, when powerful investment firms and individuals who also had skin in the game also missed this.
TL;DR:
1) There were entries that were relevant (this one also touches on it briefly)
2) They were specifically mentioned
3) There were comments relevant to this. (notably one of these was apparently deleted because it received a lot of downvotes when initially posted)
4) There has been at least two other posts on the forum prior to the contest that engaged with this specifically
My tentative take is that these issues were in fact identified by various members of the community, but there isn’t a good way of turning identified issues into constructive actions—the status quo is we just have to trust that organisations have good systems in place for this, and that EA leaders are sufficiently careful and willing to make changes or consider them seriously, such that all the community needs to do is “raise the issue”. And I think looking at the systems within the relevant EA orgs or leadership is what investigations or accountability questions going forward should focus on—all individuals are fallible, and we should be looking at how we can build systems in place such that the community doesn’t have to just trust that people who have power and who are steering the EA movement will get it right, and that there are ways for the community to hold them accountable to their ideals or stated goals if it appears to, or risks not playing out in practice.
i.e. if there are good processes and systems in place and documentation of these processes and decisions, it’s more acceptable (because other organisations that probably have a very good due diligence process also missed it). But if there weren’t good processes, or if these decisions weren’t a careful + intentional decision, then that’s comparatively more concerning, especially in context of specific criticisms that have been raised,[1] or previous precedent. For example, I’d be especially curious about the events surrounding Ben Delo,[2] and processes that were implemented in response. I’d be curious about whether there are people in EA orgs involved in steering who keep track of potential risks and early warning signs to the EA movement, in the same way the EA community advocates for in the case of pandemics, AI, or even general ways of finding opportunities for impact. For example, SBF, who is listed as a EtG success story on 80k hours, has publicly stated he’s willing to go 5x over the Kelly bet, and described yield farming in a way that Matt Levine interpreted as a Ponzi. Again, I’m personally less interested in the object level decision (e.g. whether or not we agree with SBF’s Kelly bet comments as serious, or whether Levine’s interpretation as appropriate), but more about what the process was, how this was considered at the time with the information they had etc. I’d also be curious about the documentation of any SBF related concerns that were raised by the community, if any, and how these concerns were managed and considered (as opposed to critiquing the final outcome).
Outside of due diligence and ways to facilitate whistleblowers, decision-making processes around the steering of the EA movement is crucial as well. When decisions are made by orgs that bring clear benefits to one part of the EA community while bringing clear risks that are shared across wider parts of the EA community,[3] it would probably be of value to look at how these decisions were made and what tradeoffs were considered at the time of the decision. Going forward, thinking about how to either diversify those risks, or make decision-making more inclusive of a wider range stakeholders[4], keeping in mind the best interests of the EA movement as a whole.
(this is something I’m considering working on in a personal capacity along with the OP of this post, as well as some others—details to come, but feel free to DM me if you have any thoughts on this. It appears that CEA is also already considering this)
If this comment is about “are these red-teaming contests in fact valuable for the money and time put into it, if it misses problems like this”
I think my view here (speaking only for the red-teaming contest) is that even if this specific contest was framed in a way that it missed these classes of issues, the value of the very top submissions[5] may still have made the efforts worthwhile. The potential value of a different framing was mentioned by another panelist. If it’s the case that red-teaming contests are systematically missing this class of issues regardless of framing, then I agree that would be pretty useful to know, but I don’t have a good sense of how we would try to investigate this.
This tweet seems to have aged particularly well. Despite supportive comments from high-profile EAs on the original forum post, the author seemed disappointed that nothing came of it in that direction. Again, without getting into the object level discussion of the claims of the original paper, it’s still worth asking questions around the processes. If there was were actions planned, what did these look like? If not, was that because of a disagreement over the suggested changes, or the extent that it was an issue at all? How were these decisions made, and what was considered?
Apparently a previous EA-aligned billionaire ?donor who got rich by starting a crypto trading firm, who pleaded guilty to violating the bank secrecy act
Even before this, I had heard from a primary source in a major mainstream global health organisation that there were staff who wanted to distance themselves from EA because of misunderstandings around longtermism.
This doesn’t have to be a lengthy deliberative consensus-building project, but it should at least include internal comms across different EA stakeholders to allow discussions of risks and potential mitigation strategies.
e.g. A critical review of GiveWell’s 2022 cost-effectiveness model, Methods for improving uncertainty analysis in EA cost-effectiveness models, and
Biological Anchors external review
...I for one didn’t take the self-criticism stuff particularly seriously, or consider EA to have scored any points by running the contest? And I thought that seemed too obvious to mention, I guess, that of course scoring diligent self-criticism points is harder than that? Zvi had a longer writeup on a similar take.
I have my own sense of what EA is doing all wrong, but it didn’t particularly occur to me to try to write it up for the criticism contest. That was obviously going to be an in-frame thing, and was obviously not going to be able to reward anything out-of-frame; doing that is really hard if you don’t want to just completely throw the contest over to wild randos. Like, the reason MIRI doesn’t run a contest like that is not that our underlying reality contains nothing worth criticizing, but that a contest like that obviously does not work to highlight your real actual problems.
There’s nothing particularly unhealthy about a movement where a contest like that fails to turn up any real issues—the contest is obviously doomed in that sense from the start. The only failure is the naivety required to imagine that a contest like that could possibly work, against a semiefficient background where smart people have already been saying various things and there are already reasons why those critiques have not been accepted. The best you can say about a contest like that is that maybe it’s worth $100K to get better centralized writeups of in-frame criticisms that your community already knows how to accept but hasn’t finished acting on yet.
Have you written that up anywhere? Would be interesting to read.
In hindsight they look a bit performative.
There’s a related problem with CEA. Their mission is to nurture the EA community, and their sizeable community health team’s is to “preserve the EA community’s ability to grow and produce value”. But the CH team has focused on public relations, diversity, “discussion norms”, and crisis-response, while the community became unhealthy for a different reason—FTX. It seems like this team, or some other, needs to take on a wider mandate, that includes monitoring of such issues.
I would like to know, for example, whether any large EA orgs (and EA thinkers, I suppose too) have formally engaged with those criticism contest entries. Like, was it somebody’s job to think about when anything should change as a result? Or something discussed by a governance board (or similar)?