Sure, but an EA hotel seems like a weird way to address this inefficiency: only few people with worthwhile projects can move to Blackpool to benefit from it, the funding is not flexible, it’s hard to target this well, the project has some time lag, etc. The most reasonable approach to fixing this is simply to give more money to some of the projects that didn’t get funded.
Maybe CEA will accept 20-30% of EA Grants applications in the next round, or other donors will jump in to fill the gaps. (I’d expect that a lot of the grants applications (maybe half) might have been submitted by people not really familiar with EA, and some of the others weren’t worth funding.)
[Disclosure: I’m planning to move to Blackpool before the end of this month.]
only few people with worthwhile projects can move to Blackpool to benefit from it, the funding is not flexible
If you’re working on a project full-time, there’s a good chance you’re not location-constrained.
the project has some time lag
I’m not sure what you’re referring to.
Over 3 months passed between the EA grants announcement and disbursement. Does that count as “time lag”?
The disadvantages you cite don’t seem compelling to me alongside the advantages cited in this post: dramatically lower costs, supportive EA community, etc. Yes, it’s not a great fit for every project—but if you’re offered a bargain on supporting one project, it seems silly to avoid taking it just because you weren’t offered a bargain on supporting some other project.
I think maybe our core disagreement is that you believe the bottom 95% of EA projects are risky and we should discourage people from funding them. Does that sound like an accurate summary of your beliefs?
I’ve writtensome about why I want to see more discussion of downside risk (and I’m compiling notes for several longer posts—maybe if I was living in Blackpool I’d have enough time to write them). However, the position that we should discourage funding the bottom 95% of projects seems really extreme to me, and it also seems like a really crude way to address downside risk.
Even if there is some downside risk from any given project, the expected value is probably positive, solely based on the fact that some EA thinks it’s a good idea. Value of information is another consideration in favor of taking action, especially doing research (I’m guessing most people who move to Blackpool will want to do research of some sort).
As for a good way to decrease downside risk, I would like to see a lot more people do what Joey does in this post and ask people to provide the best arguments they can against their project (or maybe ask a question on Metaculus).
The issue with downside risk is not that the world is currently on a wonderful trajectory that must not be disturbed. Rather, any given disturbance is liable to have effects that are hard to predict—but if we spent more time thinking and did more research, maybe we could get a bit better at predicting these effects. Loss aversion will cause us to overweight the possibility of a downside, but if we’re trying to maximize expected value then we should weight losses and gains of the same size equally. I’m willing to believe highly experienced EAs are better at thinking about downside risk, but I don’t think the advantage is overwhelming, and I suspect being a highly experienced EA can create its own set of blind spots. I am definitely skeptical that CEA can reliably pick the 33 highest-impact projects from a list of 722. Even experienced investors miss winners, and CEA is an inexperienced charitable “investor”.
Your example “the first political protest for AI regulation might in expectation do more harm than a thoughtful AI policy project could prevent” is one I agree with, and maybe it illustrates a more general principle that public relations efforts should be done in coordination with major EA organizations. Perhaps it makes sense to think about downside risk differently depending on the kind of work someone is interested in doing.
BTW, if the opinion of experienced EAs is considered reliable, maybe it’s worth noting that 80k advocated something like this a few years ago, and ideas like it have been floating around EA for a long time.
I suspect Greg/the manager would not be able to filter projects particularly well based on personal interviews; since the point of the hotel is basically ‘hits-based giving’, I think a blanket ban on irreversible projects is more useful (and would satisfy most of the concerns in the fb comment vollmer linked)
Just to play devil’s advocate for a moment, aren’t personal interviews and hits-based giving essentially the process used by other EA funders? I believe it was OpenPhil who coined the term hits-based giving. It sounds like maybe your issue is with the way funding works in EA broadly speaking, not this project in particular.
The same seems to apply to vollmer’s point about adverse selection effects. Over time, the project pool will increasingly be made up of projects everyone has rejected. So this could almost be considered a fully general counterargument against funding any project. (Note that this thinking directly opposes replaceability: replaceability encourages you to fund projects no other funder is willing to fund; this line of reasoning says just the opposite.) Anyway, I think the EA Hotel could easily be less vulnerable to adverse selection effects, if it appeals to a different crowd. I’m the first long-term resident of the hotel, and I’ve never applied for funding from any other source. (I’m self-studying machine learning at the hotel, which I don’t think I would ever get a grant for.)
Sounds like you really want a broader rule like “no irreversible projects without community consensus” or something. In general, mitigating downside risk seems like an issue that’s fairly orthogonal to establishing low cost of living EA hubs.
Even if there is some downside risk from any given project, the expected value is probably positive, solely based on the fact that some EA thinks it’s a good idea.
The Unilateralist’s Curse refers to situations where many people oppose a project and only one person supports it. This is an important case to consider, but I think a randomly chosen EA project is unlikely to fall into this category.
Hmm. I think the Unilateralist’s Curse rests on the assumption that individuals underestimate potential downsides relative to their estimations concerning potential upsides, at least when it comes to the consequences for other people. (Anecdotally this assumption seems likely but I basically have no idea if it’s true.) Centralised coordination/control is a way to counteract that. But the situation is even worse in this case because not only are we potentially facilitating projects with no centralised filtering, we’re actually selecting for projects that are more likely to have been rejected by a central filter.
Thank you for prompting me to clarify my thinking here. I expect it’s wrong, I haven’t read Bostrom’s paper on it, but that’s where I’m up to.
(Also it might be worth me saying that I currently still think the EA Hotel has positive expected value—I don’t think it’s giving individuals enough power for the Unilateralist’s Curse to really apply. But it’s worth continuing to think about the UC and it doesn’t seem clear to me that “the fact that some EA thinks it’s a good idea” is sufficient grounds to attribute positive expected value to a project, given no other information, which seemed to be what you were saying.)
Centralised coordination/control is a way to counteract that.
OpenPhil funding OpenAI might be a case of a “central” organization taking unilateral action that’s harmful. vollmer also mentions that he thinks some of EAF’s subprojects were probably negative impact elsewhere in this thread—presumably the EAF is relatively “central”.
If we think that “individuals underestimate potential downsides relative to their estimations concerning potential upsides”, why do we expect funders to be immune to this problem? There seems to be an assumption that if you have a lot of money, you are unusually good at forecasting potential downsides. I’m not sure. People like Joey and Paul Christiano have offered prizes for the best arguments against their beliefs. I don’t believe OpenPhil has ever done this, despite having a lot more money.
In general, funding doesn’t do much to address the unilateralist’s curse because any single funder can act unilaterally to fund a project that all the other funders think is a bad idea. I once proposed an EA donor’s league to address this problem, but people weren’t too keen on it for some reason.
it doesn’t seem clear to me that “the fact that some EA thinks it’s a good idea” is sufficient grounds to attribute positive expected value to a project, given no other information
Here’s a thought experiment that might be helpful as a baseline scenario. Imagine you are explaining effective altruism to a stranger in a loud bar. After hearing your explanation, the stranger responds “That’s interesting. Funny thing, I gave no thought to EA considerations when choosing my current project. I just picked it because I thought it was cool.” Then they explain their project to you, but unfortunately, the bar is too loud for you to hear what they say, so you end up just nodding along pretending to understand. Now assume you have two options: you can tell the stranger to ditch their project, or you can stay silent. For the sake of argument, let’s assume that if you tell the stranger to ditch their project, they will ditch it, but they will also get soured on EA and be unreceptive to EA messages in the future. If you stay silent, the stranger will continue their project and remain receptive to EA messages. Which option do you choose?
My answer is, having no information about the stranger’s project, I have no particular reason to believe it will be either good or bad for the world. So I model the stranger’s project as a small random perturbation on humanity’s trajectory, of the sort that happen thousands of times per day. I see the impact of such perturbations as basically neutral on expectation. In the same way the stranger’s project could have an unexpected downside, it could also have an unexpected upside. And in the same way that the stranger’s actions could have some nasty unforeseen consequence, my action of discouraging the stranger could also have some nasty unforeseen consequence! (Nasty unforeseen consequences of my discouragement action probably won’t be as readily observable, but that doesn’t mean they won’t exist.) So I stay silent, because I gain nothing on expectation by objecting to the project, and I don’t want to pay the cost of souring the stranger on EA.
Suppose you agree with my argument above. If so, do you think that we should default to discouraging EAs from doing projects in the absence of further information? Why? It seems a bit counterintuitive/implausible that being part of the EA community would increase the odds that someone’s project creates a downside. If anything, it seems like being plugged into the community should increase a person’s awareness of how their project might pose a risk. (Consider the EA hotel in comparison to an alternative of having people live cheaply as individuals. Being part of a community of EAs = more peer eyeballs on your project = more external perspectives to spot unexpected downsides.) And in the same way giving strangers default discouragement will sour them on EA, giving EAs default discouragement on doing any kind of project seems like the kind of thing that will suck the life out of the movement.
I don’t want to be misinterpreted, so to clarify:
I am in favor of people discouraging projects if, after looking at the project, they actually think the project will be harmful.
I am in favor of bringing up considerations that suggest a project might be harmful with the people engaged in it, even if you aren’t sure about the project’s overall impact.
I’m in favor of people trying to make the possibility of downsides mentally available, so folks will remember to check for them.
I’m in favor of more people doing what Joey does and offering prizes for arguments that their projects are harmful.
I’m in favor of people publicly making themselves available to shoot holes in the project ideas of others.
I’m in favor of people in the EA community trying to coordinate more effectively, engage in moral trade, cooperate in epistemic prisoner’s dilemmas, etc.
In general, I think brainstorming potential downsides has high value of information and people should do it more. But a gamble can still be positive expected value without having purchased any information! (Also, in order to avoid bias, maybe you should try to spend an equal amount of time brainstorming unexpected upsides.)
I think it may be reasonable to focus on projects which have passed the acid test of trying to think of plausible downsides (since those projects are likely to be higher expected value).
But I don’t really see what purpose blanket discouragement serves.
The OpenPhil/OpenAI article was a good read, thanks, although I haven’t read the comments on either post or Ben’s latest thoughts, and I don’t really have an opinion either way on the value/harm of OpenPhil funding OpenAI if they did so “to buy a seat on OpenAI’s board for Open Philanthropy Project executive director Holden Karnofsky”. But of course, I wasn’t suggesting that centralised action is never harmful; I was suggesting that it’s better on average [edit: in UC-type scenarios, which I’m not sure your two examples were...man this stuff is confusing!]. It’s also ironic that part of the reason funding OpenAI might have been a bad idea seems to be that it creates more of a Unilateralist’s Curse scenario (although I did notice that the first comment claims this is not their current strategy): “OpenAI’s primary strategy is to hire top AI researchers to do cutting-edge AI capacity research and publish the results, in order to ensure widespread access.”
If we think that “individuals underestimate potential downsides relative to their estimations concerning potential upsides”, why do we expect funders to be immune to this problem?
Excellent question. No strong opinion as I’m still in anecdote territory here, but I reckon emotional attachment to one’s own grand ideas is what’s driving the underestimation of risk, and you’d expect funders to be able to assess ideas more dispassionately.
I’m not sure that EA is all that relevant to the answer I’d give in your thought experiment. If they didn’t have much power then I’d say go for it. If their project would have large consequences before anyone else could step in I’d say stop. As I said before, “I currently still think the EA Hotel has positive expected value—I don’t think it’s giving individuals enough power for the Unilateralist’s Curse to really apply.” I genuinely do expect the typical idea someone has for improving the status quo to be harmful, whether they’re an EA or a stranger in a bar. Most of the time it’s good to encourage innovation anyway, because there are feedback mechanisms/power structures in place to stop things getting out of hand if they start to really not look like good ideas. But in UC-type scenarios i.e. where those checks are not in place, we have a problem.
We might be talking past each other. Perhaps we agree that: In your typical real-life scenario i.e. where an individual does not have unilateral power, we should encourage them to pursue their altruistic ideas. Perhaps this was even what you were saying originally, and I just misinterpreted it.
[Edit: I’m pretty sure we’re talking past each other to at least some extent. I don’t think there should be “blanket discouragement”. I think the typical project that someone/an EA thinks is a good idea is in fact a bad idea, but that they should test it anyway. I do think there should be blanket discouragement of actions with large consequences that can be taken by a small minority without the endorsement of others (eg. relating to reputational risk or information hazards).]
EA Grants rejected 95% of the applications they got.
Sure, but an EA hotel seems like a weird way to address this inefficiency: only few people with worthwhile projects can move to Blackpool to benefit from it, the funding is not flexible, it’s hard to target this well, the project has some time lag, etc. The most reasonable approach to fixing this is simply to give more money to some of the projects that didn’t get funded.
Maybe CEA will accept 20-30% of EA Grants applications in the next round, or other donors will jump in to fill the gaps. (I’d expect that a lot of the grants applications (maybe half) might have been submitted by people not really familiar with EA, and some of the others weren’t worth funding.)
[Disclosure: I’m planning to move to Blackpool before the end of this month.]
If you’re working on a project full-time, there’s a good chance you’re not location-constrained.
I’m not sure what you’re referring to.
Over 3 months passed between the EA grants announcement and disbursement. Does that count as “time lag”?
The disadvantages you cite don’t seem compelling to me alongside the advantages cited in this post: dramatically lower costs, supportive EA community, etc. Yes, it’s not a great fit for every project—but if you’re offered a bargain on supporting one project, it seems silly to avoid taking it just because you weren’t offered a bargain on supporting some other project.
I think maybe our core disagreement is that you believe the bottom 95% of EA projects are risky and we should discourage people from funding them. Does that sound like an accurate summary of your beliefs?
I’ve written some about why I want to see more discussion of downside risk (and I’m compiling notes for several longer posts—maybe if I was living in Blackpool I’d have enough time to write them). However, the position that we should discourage funding the bottom 95% of projects seems really extreme to me, and it also seems like a really crude way to address downside risk.
Even if there is some downside risk from any given project, the expected value is probably positive, solely based on the fact that some EA thinks it’s a good idea. Value of information is another consideration in favor of taking action, especially doing research (I’m guessing most people who move to Blackpool will want to do research of some sort).
As for a good way to decrease downside risk, I would like to see a lot more people do what Joey does in this post and ask people to provide the best arguments they can against their project (or maybe ask a question on Metaculus).
The issue with downside risk is not that the world is currently on a wonderful trajectory that must not be disturbed. Rather, any given disturbance is liable to have effects that are hard to predict—but if we spent more time thinking and did more research, maybe we could get a bit better at predicting these effects. Loss aversion will cause us to overweight the possibility of a downside, but if we’re trying to maximize expected value then we should weight losses and gains of the same size equally. I’m willing to believe highly experienced EAs are better at thinking about downside risk, but I don’t think the advantage is overwhelming, and I suspect being a highly experienced EA can create its own set of blind spots. I am definitely skeptical that CEA can reliably pick the 33 highest-impact projects from a list of 722. Even experienced investors miss winners, and CEA is an inexperienced charitable “investor”.
Your example “the first political protest for AI regulation might in expectation do more harm than a thoughtful AI policy project could prevent” is one I agree with, and maybe it illustrates a more general principle that public relations efforts should be done in coordination with major EA organizations. Perhaps it makes sense to think about downside risk differently depending on the kind of work someone is interested in doing.
BTW, if the opinion of experienced EAs is considered reliable, maybe it’s worth noting that 80k advocated something like this a few years ago, and ideas like it have been floating around EA for a long time.
I suspect Greg/the manager would not be able to filter projects particularly well based on personal interviews; since the point of the hotel is basically ‘hits-based giving’, I think a blanket ban on irreversible projects is more useful (and would satisfy most of the concerns in the fb comment vollmer linked)
Just to play devil’s advocate for a moment, aren’t personal interviews and hits-based giving essentially the process used by other EA funders? I believe it was OpenPhil who coined the term hits-based giving. It sounds like maybe your issue is with the way funding works in EA broadly speaking, not this project in particular.
The same seems to apply to vollmer’s point about adverse selection effects. Over time, the project pool will increasingly be made up of projects everyone has rejected. So this could almost be considered a fully general counterargument against funding any project. (Note that this thinking directly opposes replaceability: replaceability encourages you to fund projects no other funder is willing to fund; this line of reasoning says just the opposite.) Anyway, I think the EA Hotel could easily be less vulnerable to adverse selection effects, if it appeals to a different crowd. I’m the first long-term resident of the hotel, and I’ve never applied for funding from any other source. (I’m self-studying machine learning at the hotel, which I don’t think I would ever get a grant for.)
Sounds like you really want a broader rule like “no irreversible projects without community consensus” or something. In general, mitigating downside risk seems like an issue that’s fairly orthogonal to establishing low cost of living EA hubs.
I think the Unilateralist’s Curse is relevant here.
The Unilateralist’s Curse refers to situations where many people oppose a project and only one person supports it. This is an important case to consider, but I think a randomly chosen EA project is unlikely to fall into this category.
Hmm. I think the Unilateralist’s Curse rests on the assumption that individuals underestimate potential downsides relative to their estimations concerning potential upsides, at least when it comes to the consequences for other people. (Anecdotally this assumption seems likely but I basically have no idea if it’s true.) Centralised coordination/control is a way to counteract that. But the situation is even worse in this case because not only are we potentially facilitating projects with no centralised filtering, we’re actually selecting for projects that are more likely to have been rejected by a central filter.
Thank you for prompting me to clarify my thinking here. I expect it’s wrong, I haven’t read Bostrom’s paper on it, but that’s where I’m up to.
(Also it might be worth me saying that I currently still think the EA Hotel has positive expected value—I don’t think it’s giving individuals enough power for the Unilateralist’s Curse to really apply. But it’s worth continuing to think about the UC and it doesn’t seem clear to me that “the fact that some EA thinks it’s a good idea” is sufficient grounds to attribute positive expected value to a project, given no other information, which seemed to be what you were saying.)
OpenPhil funding OpenAI might be a case of a “central” organization taking unilateral action that’s harmful. vollmer also mentions that he thinks some of EAF’s subprojects were probably negative impact elsewhere in this thread—presumably the EAF is relatively “central”.
If we think that “individuals underestimate potential downsides relative to their estimations concerning potential upsides”, why do we expect funders to be immune to this problem? There seems to be an assumption that if you have a lot of money, you are unusually good at forecasting potential downsides. I’m not sure. People like Joey and Paul Christiano have offered prizes for the best arguments against their beliefs. I don’t believe OpenPhil has ever done this, despite having a lot more money.
In general, funding doesn’t do much to address the unilateralist’s curse because any single funder can act unilaterally to fund a project that all the other funders think is a bad idea. I once proposed an EA donor’s league to address this problem, but people weren’t too keen on it for some reason.
Here’s a thought experiment that might be helpful as a baseline scenario. Imagine you are explaining effective altruism to a stranger in a loud bar. After hearing your explanation, the stranger responds “That’s interesting. Funny thing, I gave no thought to EA considerations when choosing my current project. I just picked it because I thought it was cool.” Then they explain their project to you, but unfortunately, the bar is too loud for you to hear what they say, so you end up just nodding along pretending to understand. Now assume you have two options: you can tell the stranger to ditch their project, or you can stay silent. For the sake of argument, let’s assume that if you tell the stranger to ditch their project, they will ditch it, but they will also get soured on EA and be unreceptive to EA messages in the future. If you stay silent, the stranger will continue their project and remain receptive to EA messages. Which option do you choose?
My answer is, having no information about the stranger’s project, I have no particular reason to believe it will be either good or bad for the world. So I model the stranger’s project as a small random perturbation on humanity’s trajectory, of the sort that happen thousands of times per day. I see the impact of such perturbations as basically neutral on expectation. In the same way the stranger’s project could have an unexpected downside, it could also have an unexpected upside. And in the same way that the stranger’s actions could have some nasty unforeseen consequence, my action of discouraging the stranger could also have some nasty unforeseen consequence! (Nasty unforeseen consequences of my discouragement action probably won’t be as readily observable, but that doesn’t mean they won’t exist.) So I stay silent, because I gain nothing on expectation by objecting to the project, and I don’t want to pay the cost of souring the stranger on EA.
Suppose you agree with my argument above. If so, do you think that we should default to discouraging EAs from doing projects in the absence of further information? Why? It seems a bit counterintuitive/implausible that being part of the EA community would increase the odds that someone’s project creates a downside. If anything, it seems like being plugged into the community should increase a person’s awareness of how their project might pose a risk. (Consider the EA hotel in comparison to an alternative of having people live cheaply as individuals. Being part of a community of EAs = more peer eyeballs on your project = more external perspectives to spot unexpected downsides.) And in the same way giving strangers default discouragement will sour them on EA, giving EAs default discouragement on doing any kind of project seems like the kind of thing that will suck the life out of the movement.
I don’t want to be misinterpreted, so to clarify:
I am in favor of people discouraging projects if, after looking at the project, they actually think the project will be harmful.
I am in favor of bringing up considerations that suggest a project might be harmful with the people engaged in it, even if you aren’t sure about the project’s overall impact.
I’m in favor of people trying to make the possibility of downsides mentally available, so folks will remember to check for them.
I’m in favor of more people doing what Joey does and offering prizes for arguments that their projects are harmful.
I’m in favor of people publicly making themselves available to shoot holes in the project ideas of others.
I’m in favor of people in the EA community trying to coordinate more effectively, engage in moral trade, cooperate in epistemic prisoner’s dilemmas, etc.
In general, I think brainstorming potential downsides has high value of information and people should do it more. But a gamble can still be positive expected value without having purchased any information! (Also, in order to avoid bias, maybe you should try to spend an equal amount of time brainstorming unexpected upsides.)
I think it may be reasonable to focus on projects which have passed the acid test of trying to think of plausible downsides (since those projects are likely to be higher expected value).
But I don’t really see what purpose blanket discouragement serves.
The OpenPhil/OpenAI article was a good read, thanks, although I haven’t read the comments on either post or Ben’s latest thoughts, and I don’t really have an opinion either way on the value/harm of OpenPhil funding OpenAI if they did so “to buy a seat on OpenAI’s board for Open Philanthropy Project executive director Holden Karnofsky”. But of course, I wasn’t suggesting that centralised action is never harmful; I was suggesting that it’s better on average [edit: in UC-type scenarios, which I’m not sure your two examples were...man this stuff is confusing!]. It’s also ironic that part of the reason funding OpenAI might have been a bad idea seems to be that it creates more of a Unilateralist’s Curse scenario (although I did notice that the first comment claims this is not their current strategy): “OpenAI’s primary strategy is to hire top AI researchers to do cutting-edge AI capacity research and publish the results, in order to ensure widespread access.”
Excellent question. No strong opinion as I’m still in anecdote territory here, but I reckon emotional attachment to one’s own grand ideas is what’s driving the underestimation of risk, and you’d expect funders to be able to assess ideas more dispassionately.
I’m not sure that EA is all that relevant to the answer I’d give in your thought experiment. If they didn’t have much power then I’d say go for it. If their project would have large consequences before anyone else could step in I’d say stop. As I said before, “I currently still think the EA Hotel has positive expected value—I don’t think it’s giving individuals enough power for the Unilateralist’s Curse to really apply.” I genuinely do expect the typical idea someone has for improving the status quo to be harmful, whether they’re an EA or a stranger in a bar. Most of the time it’s good to encourage innovation anyway, because there are feedback mechanisms/power structures in place to stop things getting out of hand if they start to really not look like good ideas. But in UC-type scenarios i.e. where those checks are not in place, we have a problem.
We might be talking past each other. Perhaps we agree that: In your typical real-life scenario i.e. where an individual does not have unilateral power, we should encourage them to pursue their altruistic ideas. Perhaps this was even what you were saying originally, and I just misinterpreted it.
[Edit: I’m pretty sure we’re talking past each other to at least some extent. I don’t think there should be “blanket discouragement”. I think the typical project that someone/an EA thinks is a good idea is in fact a bad idea, but that they should test it anyway. I do think there should be blanket discouragement of actions with large consequences that can be taken by a small minority without the endorsement of others (eg. relating to reputational risk or information hazards).]