Yale EA organizer and Projects & Research at the Center for AI Safety.
I like beets.
I think Is Power-Seeking AI An Existential Risk is probably the best introduction, though it’s probably too long as a first introduction if the person is yet that motivated. It’s also written as a list of propositions, with probabilities, and that might not appeal to many people.
I also listed some shorter examples in this post for the AI Safety Public Materials Bounty we’re running, that might be more suitable as a first introduction. Here are the ones most relevant to people not versed in machine learning:
AI risk executive summary (2014)
Robert Miles’ YouTube channel (2017-present)
AGI Safety From First Principles (2020)
The case for taking AI risk seriously as a threat to humanity (2020)
The competition is also trying to get more, because I think there is a lot more that can be done.
Thanks for all the work you are doing here, I think some really amazing groups could come out of this. I am cautiously excited about many different kinds of groups starting.
I found it a bit surprising that the list of criteria for group organizers (including “nice to have”) doesn’t seem to have anything like “really cares about the objectives of their group,” “really cares about improving the long term future,” “is altruistic to some degree”
Being truth-seeking and open-mindedHaving a strong understanding of whatever topic their group is about, and/or being self-aware about gaps in understandingBeing socially skilled enough that people won’t find them highly offputting (note that this is a much lower bar than being actively friendly, extroverted, etc.)Secondary “nice-to-have” desiderata include:Taking ideas seriouslyBeing conscientiousBeing ambitious / entrepreneurialBeing friendly / outgoingHaving good strategic judgment in what activities their group should be doingActively coming off as sharp in conversation, such that others find them fun to have object-level discussions with
Being truth-seeking and open-minded
Having a strong understanding of whatever topic their group is about, and/or being self-aware about gaps in understanding
Being socially skilled enough that people won’t find them highly offputting (note that this is a much lower bar than being actively friendly, extroverted, etc.)
Secondary “nice-to-have” desiderata include:
Taking ideas seriously
Being ambitious / entrepreneurial
Being friendly / outgoing
Having good strategic judgment in what activities their group should be doing
Actively coming off as sharp in conversation, such that others find them fun to have object-level discussions with
Maybe this is just implicit? But it seems useful to make it explicit. Otherwise, this list kind of makes me think of really nice, friendly, hardworking philosophers who do not actually behave any more ethically. Great understanding and leadership is not enough; group leaders need to actually care about whatever the thing is. Maybe this is what “taking ideas seriously” is supposed to mean (I never quite understand that phrase, people seem to use it in different ways)? If so, it seems like a must have, not a nice to have.
We don’t expect the work to be published anywhere when it’s submitted.
For certain pieces, we may work with authors to publish them somewhere, publish them on our website, or adapt them and publish an adapted version somewhere. But this is not guaranteed.
In general, we expect that the best pieces will be generally suited for an audience of either smart people who don’t know about ML, or ML researchers. Though there is a lot of room for pieces that are more optimized for particular audiences and venues, we think that more general pieces would serve as great inspiration for those later pieces.
I edited the title to say “$20k in bounties” to make it more clear.
From the original text:
Winners of the bounty will win $2,000 each, for a total of up to ten possible bounty recipients.
This doesn’t mean each person who submits an entry gets $2,000. We will award this to entries that meet a high bar for quality (roughly, material that we would actually be interested in using for outreach).
I missed that part of footnote 3, it does seem to address a lot of what I said. I appreciate your response.
I do think the vast majority of people will not read footnote 3, so it’s important for the main body of the text (and the visuals) to give the right impression. This means comparing averages to averages, or possible tail events to possible tail events. It sounds like this is your plan now, and if so that’s great!
Yes, that’s my mistake, sorry.
Posted too soon, didn’t realize he had changed his mind about crossposting, please ignore.
I linkposted this when it came out, and Devin Kalish sent this comment:
A quick note, this piece was already posted to the forum briefly, and then deleted. The author said in a comment that he would rather it not be crossposted to this forum:https://astralcodexten.substack.com/p/criticism-of-criticism-of-criticism/comment/7853073I don’t know if the two are related, but I might reach out to ask him if he’s alright with you posting it.
Which led to me taking down my post, since I don’t really like to crosspost things if people prefer that I not.
Just wanted to let you know!
This point is covered quite well by Derek Parfit in his seminal book Reasons and Persons, Chapter 1, Part 17. In my view the entire chapter is excellent and worth reading, but here is an excerpt from Part 17:
Consider, for example, theft. On some versions of C [Consequentialism], it is intrinsically bad if property is stolen. On other versions of C [such as hedonistic utilitarianism], this is not so. On these versions, theft is bad only when it makes the outcome worse. Avoiding theft is not part of our ultimate moral aim. But it might be true that it would make the outcome better if we were strongly disposed not to steal. And it might make the outcome better if we believed stealing to be intrinsically wrong, and would feel remorse when we do steal. Similar claims might be made about many other kinds of act.
This paragraph, I think, is especially relevant for EA:
This suggests that the most that could be true is that C is partly self-effacing. It might be better if most people caused themselves to believe some other theory, by some process of self-deception that, to succeed, must also be forgotten. But, as a precaution, a few people should continue to believe C, and should keep convincing evidence about this self-deception. These people need not live in Government House, or have any other special status. If things went well, the few would do nothing. But if the moral theory believed by most did become disastrous, the few could then produce their evidence. When most people learnt that their moral beliefs were the result of self-deception, this would undermine these beliefs, and prevent the disaster.
Edit: I also recommend the related When Utilitarians Should Be Virtue Theorists.
Didn’t actually know that about HIV, good to know!
This is great!
In the intro article, I don’t think I really like the comparison between pandemic prevention and counterterrorism.
A couple reasons:
First, counterterrorism might be construed to include counter bio terrorism. In which case, it’s not obvious to me that pandemic prevention and counterterrorism are even exclusive.
Second, both pandemics and counterterrorism are heavy-tailed and dominated by tail events. Tail events don’t happen...until they do. To give an example, here is the same graph but for 2009-2019:
Essentially no deaths from COVID-19! Looks like it’s unimportant!
Knowing almost nothing about terrorism, I would expect that a terrorism tail event, such as the detonation of a nuclear dirty bomb, could be similar: we wouldn’t see it in the statistics until it was too late.
When we broaden the scope, we can see that many more people died in global pandemics (other than COVID, since COVID barely existed) in that time period than terrorism:
However, this is extremely influenced by another tail event: HIV/AIDS. In a world without HIV/AIDS, it would look like this:
This would imply that in some counterfactual world where nothing was different except that AIDS did not exist, I should have thought in 2019 that global pandemics were about equal to terrorism in scale. This is not a conclusion that should be drawn from the data, because for tail-dominated phenomena, you can’t just consider historical average data (certainly not from a ten year period), you have to consider black swan events: events unlike any that have ever happened.
Comparing the most recent pandemic tail event to the average statistics on terrorism doesn’t make sense: it’s comparing apples to oranges. Either compare the average statistics over a long time period or the probability and severity of possible tail events. For newly emerging threats like engineered pandemics, average statistics doesn’t even make sense at all, since we’ve never had an engineered pandemic.
Thank you for writing this. A lot of this feels true for me.
A quick thought: some of what you wrote can also be generalized to “working really hard, all the time, on one thing.” A lot of EA community builders do this. So do a lot of student entrepreneurs, researchers, performing artists, debators, and athletes, and I think they can run into many of the same challenges. I also think some of the solutions you outlined are common for some of these communities (e.g. athletic teams often feel like friend groups). Maybe there are lessons that can be learned from people who fall into this more general category?
I would add, “people with a technical background who also have strong writing skills.” Maybe this is subsumed by communicators but I wanted to flag it specifically.
A lot of the best researchers either don’t like to write, are slow at writing well, or simply aren’t very good at writing well. But there is much that needs to be written. For this reason I’ve found recently that writing appears to be one of my comparative advantages.
You do need to be somewhat technical to understand the content you’re writing about, but you don’t have to be a top of the line researcher.
Agreed! I wrote a post about exactly this. Julia Wise also has a good one on similar topics.
For me, a big change happened when I had been around in EA long enough, done enough things, and spoken to enough people to be able to say, “if I say something disagreeable to somebody and it turns out they are one of those people who will judge me personally for disagreeing with the dominant paradigm on x thing, it’s their loss, not mine.” I also feel I can say something disagreeable to people and they will tend to hear me out rather than ignore me as a newbie who doesn’t know anything (in fairness, when I was just starting, I actually didn’t know much at all!).
For the newest people (I’ve only been in EA for 2 years, so I am still quite new) with few legible achievements and almost no connections, this is more difficult. If you constantly feel you have to think about how to get into x office or get in the good graces of y person or receive z grant, you feel far more pressure to fight against your own doubt for fear that people will judge you for your disagreements. This is unhealthy.
Obviously there is a continuum: it’s not that you can either disagree all the time or never disagree, but there are varying amounts of disagreement that people feel comfortable having.
Sometimes people talk about “f**k you money” to mean money that you can use to ride out unemployment if you decide you don’t like your job anymore and want to quit. In EA circles there is an analogous concept, something like “I respectfully disagree with your worldview social capital” or “I respectfully disagree with your worldview concrete achievements that you cannot ignore”. The more of that you have, especially the latter, the better. Luckily, the latter is also relatively correlated with how much you’ve been able to achieve, which is (hopefully) correlated with impact.
I wasn’t aware of that. In that case, I’ll delete this post.
This is similar to something I’ve thought about recently, which is that one option for a highly impactful person looks basically like having their head down and studying for many years, getting into a conventional position, and using the skills they’ve acquired and the leverage in that position for good. I think this is underemphasized and I wonder if that is just because it seems less exciting and different.
Anecdotally I’ve observed some people taking long leaves from college/talking about dropping out (edit: I took a leave from college and it was very beneficial for me! And dropping out might make sense for some people.). There is sometimes a mood of “classes don’t matter.” But I think they often do.
I also think optimizing too much too early can be bad. Some of my most useful classes weren’t what I would have expected them to be before I took them.