The OpenPhil/OpenAI article was a good read, thanks, although I haven’t read the comments on either post or Ben’s latest thoughts, and I don’t really have an opinion either way on the value/harm of OpenPhil funding OpenAI if they did so “to buy a seat on OpenAI’s board for Open Philanthropy Project executive director Holden Karnofsky”. But of course, I wasn’t suggesting that centralised action is never harmful; I was suggesting that it’s better on average [edit: in UC-type scenarios, which I’m not sure your two examples were...man this stuff is confusing!]. It’s also ironic that part of the reason funding OpenAI might have been a bad idea seems to be that it creates more of a Unilateralist’s Curse scenario (although I did notice that the first comment claims this is not their current strategy): “OpenAI’s primary strategy is to hire top AI researchers to do cutting-edge AI capacity research and publish the results, in order to ensure widespread access.”
If we think that “individuals underestimate potential downsides relative to their estimations concerning potential upsides”, why do we expect funders to be immune to this problem?
Excellent question. No strong opinion as I’m still in anecdote territory here, but I reckon emotional attachment to one’s own grand ideas is what’s driving the underestimation of risk, and you’d expect funders to be able to assess ideas more dispassionately.
I’m not sure that EA is all that relevant to the answer I’d give in your thought experiment. If they didn’t have much power then I’d say go for it. If their project would have large consequences before anyone else could step in I’d say stop. As I said before, “I currently still think the EA Hotel has positive expected value—I don’t think it’s giving individuals enough power for the Unilateralist’s Curse to really apply.” I genuinely do expect the typical idea someone has for improving the status quo to be harmful, whether they’re an EA or a stranger in a bar. Most of the time it’s good to encourage innovation anyway, because there are feedback mechanisms/power structures in place to stop things getting out of hand if they start to really not look like good ideas. But in UC-type scenarios i.e. where those checks are not in place, we have a problem.
We might be talking past each other. Perhaps we agree that: In your typical real-life scenario i.e. where an individual does not have unilateral power, we should encourage them to pursue their altruistic ideas. Perhaps this was even what you were saying originally, and I just misinterpreted it.
[Edit: I’m pretty sure we’re talking past each other to at least some extent. I don’t think there should be “blanket discouragement”. I think the typical project that someone/an EA thinks is a good idea is in fact a bad idea, but that they should test it anyway. I do think there should be blanket discouragement of actions with large consequences that can be taken by a small minority without the endorsement of others (eg. relating to reputational risk or information hazards).]
The OpenPhil/OpenAI article was a good read, thanks, although I haven’t read the comments on either post or Ben’s latest thoughts, and I don’t really have an opinion either way on the value/harm of OpenPhil funding OpenAI if they did so “to buy a seat on OpenAI’s board for Open Philanthropy Project executive director Holden Karnofsky”. But of course, I wasn’t suggesting that centralised action is never harmful; I was suggesting that it’s better on average [edit: in UC-type scenarios, which I’m not sure your two examples were...man this stuff is confusing!]. It’s also ironic that part of the reason funding OpenAI might have been a bad idea seems to be that it creates more of a Unilateralist’s Curse scenario (although I did notice that the first comment claims this is not their current strategy): “OpenAI’s primary strategy is to hire top AI researchers to do cutting-edge AI capacity research and publish the results, in order to ensure widespread access.”
Excellent question. No strong opinion as I’m still in anecdote territory here, but I reckon emotional attachment to one’s own grand ideas is what’s driving the underestimation of risk, and you’d expect funders to be able to assess ideas more dispassionately.
I’m not sure that EA is all that relevant to the answer I’d give in your thought experiment. If they didn’t have much power then I’d say go for it. If their project would have large consequences before anyone else could step in I’d say stop. As I said before, “I currently still think the EA Hotel has positive expected value—I don’t think it’s giving individuals enough power for the Unilateralist’s Curse to really apply.” I genuinely do expect the typical idea someone has for improving the status quo to be harmful, whether they’re an EA or a stranger in a bar. Most of the time it’s good to encourage innovation anyway, because there are feedback mechanisms/power structures in place to stop things getting out of hand if they start to really not look like good ideas. But in UC-type scenarios i.e. where those checks are not in place, we have a problem.
We might be talking past each other. Perhaps we agree that: In your typical real-life scenario i.e. where an individual does not have unilateral power, we should encourage them to pursue their altruistic ideas. Perhaps this was even what you were saying originally, and I just misinterpreted it.
[Edit: I’m pretty sure we’re talking past each other to at least some extent. I don’t think there should be “blanket discouragement”. I think the typical project that someone/an EA thinks is a good idea is in fact a bad idea, but that they should test it anyway. I do think there should be blanket discouragement of actions with large consequences that can be taken by a small minority without the endorsement of others (eg. relating to reputational risk or information hazards).]