Note that we’ve only received a speculation grant from the SFF and haven’t received any s-process funding. This should be a downward update on the value of our work and an upward update on a marginal donation’s value for our work.
I’m waiting for feedback from SFF before actively fundraising elsewhere, but I’d be excited about getting in touch with potential funders and volunteers. Please message me if you want to chat! My email is ms@contact.ms, and you can find me everywhere else or send a DM on EA Forum.
On other organizations, I think:
MIRI’s work is very valuable. I’m optimistic about what I know about their comms and policy work. As Malo noted, they work with policymakers, too. Since 2021, I’ve donated over $60k to MIRI. I think they should be the default choice for donations unless they say otherwise.
OpenPhil risks increasing polarization and making it impossible to pass meaningful legislation. But while they make IMO obviously bad decisions, not everything they/Dustin fund is bad. E.g., Horizon might place people who actually care about others in places where they could have a huge positive impact on the world. I’m not sure, I would love to see Horizon fellows become more informed on AI x-risk than they currently are, but I’ve donated $2.5k to Horizon Institute for Public Service this year.
I’d be excited about the Center for AI Safety getting more funding. SB-1047 was the closest we got to a very good thing, AFAIK, and it was a coin toss on whether it would’ve been signed or not. They seem very competent. I think the occasional potential lack of rigor and other concerns don’t outweigh their results. I’ve donated $1k to them this year.
By default, I’m excited about the Center for AI Policy. A mistake they plausibly made makes me somewhat uncertain about how experienced they are with DC and whether they are capable of avoiding downside risks, but I think the people who run it are smart and have very reasonable models. I’d be excited about them having as much money as they can spend and hiring more experienced and competent people.
PauseAI is likely to be net-negative, especially PauseAI US. I wouldn’t recommend donating to them. Some of what they’re doing is exciting (and there are people who would be a good fit to join them and improve their overall impact), but they’re incapable of avoiding actions that might, at some point, badly backfire.
I’ve helped them where I could, but they don’t have good epistemics, and they’re fine with using deception to achieve their goals.
E.g., at some point, their website represented the view that it’s more likely than not that bad actors would use AI to hack everything, shut down the internet, and cause a societal collapse (but not extinction). If you talk to people with some exposure to cybersecurity and say this sort of thing, they’ll dismiss everything else you say, and it’ll be much harder to make a case for AI x-risk in the future. PauseAI Global’s leadership updated when I had a conversation with them and edited the claims, but I’m not sure they have mechanisms to avoid making confident wrong claims. I haven’t seen evidence that PauseAI is capable of presenting their case for AI x-risk competently (though it’s been a while since I’ve looked).
I think PauseAI US is especially incapable of avoiding actions with downside risks, including deception[1], and donations to them are net-negative. To Michael, I would recommend, at the very least, donating to PauseAI Global instead of PauseAI US; to everyone else, I’d recommend ideally donating somewhere else entirely.
Stop AI’s views include the idea that a CEV-aligned AGI would be just as bad as an unaligned AGI that causes human extinction. I wouldn’t be able to pass their ITT, but yep, people should not donate to Stop AI. The Stop AGI person participated in organizing the protest described in the footnote.
In February this year, PauseAI US organized a protest against OpenAI “working with the Pentagon”, while OpenAI only collaborated with DARPA on open-source cybersecurity tools and is in talks with the Pentagon about veteran suicide prevention. Most participants wanted to protest OpenAI because of AI x-risk and not because of Pentagon, but those I talked to have said they felt it was deceptive upon discovering the nature of OpenAI’s collaboration with the Pentagon. Also, Holly threatened me trying to prevent the publication of a post about this and then publicly lied about our conversations, in a way that can be easily falsified by looking at the messages we’ve exchanged.
Thanks for the comment! Disagreeing with my proposed donations is the most productive sort of disagreement. I also appreciate hearing your beliefs about a variety of orgs.
A few weeks ago, I read your back-and-forth with Holly Elmore about the “working with the Pentagon” issue. This is what I thought at the time (IIRC):
I agree that it’s not good to put misleading messages in your protests.
I think this particular instance of misleadingness isn’t that egregious, it does decrease my expectation of the value of PauseAI US’s future protests but not by a huge margin. If this was a recurring pattern, I’d be more concerned.
Upon my first reading, it was unclear to me what your actual objection was, so I’m not surprised that Holly also (apparently) misunderstood it. I had to read through twice to understand.
Being intentionally deceptive is close to a dealbreaker for me, but it doesn’t look to me like Holly was being intentionally deceptive.
I thought you both could’ve handled the exchange better. Holly included misleading messaging in the protest and didn’t seem to understand the problem, and you did not communicate clearly and then continued to believe that you had communicated well in spite of contrary evidence. Reading the exchange weakly decreased my evaluation of both your work and PauseAI US’s, but not by enough to change my org ranking. You both made the sorts of mistakes that I don’t think anyone can avoid 100% of the time. (I have certainly made similar mistakes.) Making a mistake once is evidence that you’ll make it more, but not very strong evidence.
I re-read your post and its comments just now and I didn’t have any new thoughts. I feel like I still don’t have great clarity on the implications of the situation, which troubles me, but by my reading, it’s just not as big a deal as you think it is.
General comments:
I think PauseAI US is less competent than some hypothetical alternative protest org that wouldn’t have made this mistake, but I also think it’s more competent than most protest orgs that could exist (or protest orgs in other cause areas).
I reviewed PauseAI’s other materials, although not deeply or comprehensively, and they seemed good to me. I listened to a podcast with Holly and my impression was that she had an unusually clear picture of the concerns around misaligned AI.
I think PauseAI US is less competent than some hypothetical alternative protest org that wouldn’t have made this mistake, but I also think it’s more competent than most protest orgs that could exist (or protest orgs in other cause areas).
Yes. In a short-timelines, high p(doom), world, we absolutely cannot let perfect be the enemy of the good. Being typical hyper-critical EAs might have lethal consequences[1]. We need many more people in advocacy if we are going to move the needle, so we shouldn’t be so discouraging of the people who are actually doing things. We should just accept that they won’t get everything right all the time.
In a short-timelines world, where inaction means very high p(doom), the bar for counterfactual net-negative[2] is actually pretty high. PauseAI is very far from reaching it.
This term is over-used in EA/LW spaces, to the point where I think people often don’t actually think through fully what they are actually saying by using it. Is it actually net negative, integrating over all expected future consequences in worlds where it both does and doesn’t happen? Or is it just negative?
Update: I’ve received feedback from the SFF round; we got positive evaluations from two recommenders (so my understanding is the funding allocated to us in the s-process was lower than the speculation grant) and one piece of negative feedback. The negative feedback mentioned that our project might lead to EA getting swamped by normies with high inferential distances, which can have negative consequences; and that because of that risk, “This initiative may be worthy of some support, but unfortunately other orgs in this rather impressive lineup must take priority”.
If you’re considering donating to AIGSI/AISGF, please reach out! My email is ms@contact.ms.
Note that we’ve only received a speculation grant from the SFF and haven’t received any s-process funding. This should be a downward update on the value of our work and an upward update on a marginal donation’s value for our work.
I’m waiting for feedback from SFF before actively fundraising elsewhere, but I’d be excited about getting in touch with potential funders and volunteers. Please message me if you want to chat! My email is ms@contact.ms, and you can find me everywhere else or send a DM on EA Forum.
On other organizations, I think:
MIRI’s work is very valuable. I’m optimistic about what I know about their comms and policy work. As Malo noted, they work with policymakers, too. Since 2021, I’ve donated over $60k to MIRI. I think they should be the default choice for donations unless they say otherwise.
OpenPhil risks increasing polarization and making it impossible to pass meaningful legislation. But while they make IMO obviously bad decisions, not everything they/Dustin fund is bad. E.g., Horizon might place people who actually care about others in places where they could have a huge positive impact on the world. I’m not sure, I would love to see Horizon fellows become more informed on AI x-risk than they currently are, but I’ve donated $2.5k to Horizon Institute for Public Service this year.
I’d be excited about the Center for AI Safety getting more funding. SB-1047 was the closest we got to a very good thing, AFAIK, and it was a coin toss on whether it would’ve been signed or not. They seem very competent. I think the occasional potential lack of rigor and other concerns don’t outweigh their results. I’ve donated $1k to them this year.
By default, I’m excited about the Center for AI Policy. A mistake they plausibly made makes me somewhat uncertain about how experienced they are with DC and whether they are capable of avoiding downside risks, but I think the people who run it are smart and have very reasonable models. I’d be excited about them having as much money as they can spend and hiring more experienced and competent people.
PauseAI is likely to be net-negative, especially PauseAI US. I wouldn’t recommend donating to them. Some of what they’re doing is exciting (and there are people who would be a good fit to join them and improve their overall impact), but they’re incapable of avoiding actions that might, at some point, badly backfire.
I’ve helped them where I could, but they don’t have good epistemics, and they’re fine with using deception to achieve their goals.
E.g., at some point, their website represented the view that it’s more likely than not that bad actors would use AI to hack everything, shut down the internet, and cause a societal collapse (but not extinction). If you talk to people with some exposure to cybersecurity and say this sort of thing, they’ll dismiss everything else you say, and it’ll be much harder to make a case for AI x-risk in the future. PauseAI Global’s leadership updated when I had a conversation with them and edited the claims, but I’m not sure they have mechanisms to avoid making confident wrong claims. I haven’t seen evidence that PauseAI is capable of presenting their case for AI x-risk competently (though it’s been a while since I’ve looked).
I think PauseAI US is especially incapable of avoiding actions with downside risks, including deception[1], and donations to them are net-negative. To Michael, I would recommend, at the very least, donating to PauseAI Global instead of PauseAI US; to everyone else, I’d recommend ideally donating somewhere else entirely.
Stop AI’s views include the idea that a CEV-aligned AGI would be just as bad as an unaligned AGI that causes human extinction. I wouldn’t be able to pass their ITT, but yep, people should not donate to Stop AI. The Stop AGI person participated in organizing the protest described in the footnote.
In February this year, PauseAI US organized a protest against OpenAI “working with the Pentagon”, while OpenAI only collaborated with DARPA on open-source cybersecurity tools and is in talks with the Pentagon about veteran suicide prevention. Most participants wanted to protest OpenAI because of AI x-risk and not because of Pentagon, but those I talked to have said they felt it was deceptive upon discovering the nature of OpenAI’s collaboration with the Pentagon. Also, Holly threatened me trying to prevent the publication of a post about this and then publicly lied about our conversations, in a way that can be easily falsified by looking at the messages we’ve exchanged.
Thanks for the comment! Disagreeing with my proposed donations is the most productive sort of disagreement. I also appreciate hearing your beliefs about a variety of orgs.
A few weeks ago, I read your back-and-forth with Holly Elmore about the “working with the Pentagon” issue. This is what I thought at the time (IIRC):
I agree that it’s not good to put misleading messages in your protests.
I think this particular instance of misleadingness isn’t that egregious, it does decrease my expectation of the value of PauseAI US’s future protests but not by a huge margin. If this was a recurring pattern, I’d be more concerned.
Upon my first reading, it was unclear to me what your actual objection was, so I’m not surprised that Holly also (apparently) misunderstood it. I had to read through twice to understand.
Being intentionally deceptive is close to a dealbreaker for me, but it doesn’t look to me like Holly was being intentionally deceptive.
I thought you both could’ve handled the exchange better. Holly included misleading messaging in the protest and didn’t seem to understand the problem, and you did not communicate clearly and then continued to believe that you had communicated well in spite of contrary evidence. Reading the exchange weakly decreased my evaluation of both your work and PauseAI US’s, but not by enough to change my org ranking. You both made the sorts of mistakes that I don’t think anyone can avoid 100% of the time. (I have certainly made similar mistakes.) Making a mistake once is evidence that you’ll make it more, but not very strong evidence.
I re-read your post and its comments just now and I didn’t have any new thoughts. I feel like I still don’t have great clarity on the implications of the situation, which troubles me, but by my reading, it’s just not as big a deal as you think it is.
General comments:
I think PauseAI US is less competent than some hypothetical alternative protest org that wouldn’t have made this mistake, but I also think it’s more competent than most protest orgs that could exist (or protest orgs in other cause areas).
I reviewed PauseAI’s other materials, although not deeply or comprehensively, and they seemed good to me. I listened to a podcast with Holly and my impression was that she had an unusually clear picture of the concerns around misaligned AI.
Yes. In a short-timelines, high p(doom), world, we absolutely cannot let perfect be the enemy of the good. Being typical hyper-critical EAs might have lethal consequences[1]. We need many more people in advocacy if we are going to move the needle, so we shouldn’t be so discouraging of the people who are actually doing things. We should just accept that they won’t get everything right all the time.
In a short-timelines world, where inaction means very high p(doom), the bar for counterfactual net-negative[2] is actually pretty high. PauseAI is very far from reaching it.
Or maybe I should say, “might actually be net-negative in and of itself”(!)
This term is over-used in EA/LW spaces, to the point where I think people often don’t actually think through fully what they are actually saying by using it. Is it actually net negative, integrating over all expected future consequences in worlds where it both does and doesn’t happen? Or is it just negative?
Update: I’ve received feedback from the SFF round; we got positive evaluations from two recommenders (so my understanding is the funding allocated to us in the s-process was lower than the speculation grant) and one piece of negative feedback. The negative feedback mentioned that our project might lead to EA getting swamped by normies with high inferential distances, which can have negative consequences; and that because of that risk, “This initiative may be worthy of some support, but unfortunately other orgs in this rather impressive lineup must take priority”.
If you’re considering donating to AIGSI/AISGF, please reach out! My email is ms@contact.ms.