“returns that can later be deployed to greater altruistic effect as AI research progresses”
This is hiding an important premise, which is that you’ll actually be able to deploy those increased resources well enough to make up for the opportunities you forego now. E.g. Paul thinks that (as an operationalisation of slow takeoff) the economy will double in 4 years before the first 1 year doubling period starts. So after that 4 year period you might end up with twice as much money but only 1 or 2 years to spend it on AI safety.
One other thing that I just noticed: looking at the list of 80k’s 10 priority paths found here, the first 6 (and arguably also #8: China specialist) are all roles for which the majority of existing jobs are within an EA bubble. On one hand, this shows how well the EA community has done in creating important jobs, but it also highlights my concern about us steering people away from conventionally successful careers and engagement with non-EAs.
Hi Michelle, thanks for the thoughtful reply; I’ve responded below. Please don’t feel obliged to respond in detail to my specific points if that’s not a good use of your time; writing up a more general explanation of 80k’s position might be more useful?
You’re right that I’m positive about pretty broad capital building, but I’m not sure we disagree that much here. On a scale of breadth to narrowness of career capital, consulting is at one extreme because it’s so generalist, and the other extreme is working at EA organisations or directly on EA causes straight out of university. I’m arguing against the current skew towards the latter extreme, but I’m not arguing that the former extreme is ideal. I think something like working at a top think tank (your example above) is a great first career step. (As a side note, I mention consulting twice in my post, but both times just as an illustrative example. Since this seems to have been misleading, I’ll change one of those mentions to think tanks).
However, I do think that there are only a small number of jobs which are as good on so many axes as top think tanks, and it’s usually quite difficult to get them as a new grad. Most new grads therefore face harsher tradeoffs between generality and narrowness.
More importantly, in order to help others as much as we can, we really need to both work on the world’s most pressing problems and find what inputs are most needed in order to make progress on them. While this will describe a huge range of roles in a wide variety of areas, it will still be the minority of jobs.
I guess my core argument is that in the past, EA has overfit to the jobs we thought were important at the time, both because of explicit career advice and because of implicit social pressure. So how do we avoid doing so going forward? I argue that given the social pressure which pushes people towards wanting to have a few very specific careers, it’s better to have a community default which encourages people towards a broader range of jobs, for three reasons: to ameliorate the existing social bias, to allow a wider range of people to feel like they belong in EA, and to add a little bit of “epistemic modesty”-based deference towards existing non-EA career advice. I claim that if EA as a movement had been more epistemically modest about careers 5 years ago, we’d have a) more people with useful general career capital, b) more people in things which didn’t use to be priorities, but now are, like politics, c) fewer current grads who (mistakenly/unsuccessfully) prioritised their career search specifically towards EA orgs, and maybe d) more information about a broader range of careers from people pursuing those paths. There would also have been costs to adding this epistemic modesty, of course, and I don’t have a strong opinion on whether the costs outweight the benefits, but I do think it’s worth making a case for those benefits.
We’ve updated pretty substantially away from that in favour of taking a more directed approach to your career
Looking at this post on how you’ve changed your mind, I’m not strongly convinced by the reasons you cited. Summarised:
1. If you’re focused on our top problem areas, narrow career capital in those areas is usually more useful than flexible career capital.
Unless it turns out that there’s a better form of narrow career which it would be useful to be able to shift towards (e.g. shifts in EA ideas, or unexpected doors opening as you get more senior).
2. You can get good career capital in positions with high immediate impact
I’ve argued that immediate impact is usually a fairly unimportant metric which is outweighed by the impact later on in your career.
3. Discount rates on aligned-talent are quite high in some of the priority paths, and seem to have increased, making career capital less valuable.
I am personally not very convinced by this, but I appreciate that there’s a broad range of opinions and so it’s a reasonable concern.
It still seems to be the case that organisations like the Open Philanthropy Project and GiveWell are occasionally interested in hiring people 0-2 years out of university. And while there seem to be some people to whom working at EA organisations seems more appealing than it should, there are also many people for whom it seems less appealing or cognitively available than it should. For example, while the people on this forum are likely to be very inclined to apply for jobs at EA organisations, many of the people I talk to in coaching don’t know that much about various EA organisations and why they might be good places to work.
Re OpenPhil and GiveWell wanting to hire new grads: in general I don’t place much weight on evidence of the form “organisation x thinks their own work is unusually impactful and worth the counterfactual tradeoffs”.
I agree that you have a very difficult job in trying to convey key ideas to people who are are coming from totally different positions in terms of background knowledge and experience with EA. My advice is primarily aimed at people who are already committed EAs, and who are subject to the social dynamics I discuss above—hence why this is a “community” post. I think you do amazing work in introducing a wider audience to EA ideas, especially with nuance via the podcast as you mentioned.
Posts on the new Forum are split into two categories:
Frontpage posts are timeless content covering the ideas of effective altruism. They should be useful or interesting even to readers who only know the basic concepts of EA and aren’t very active within the community.
I’m a little confused about this description. I feel like intellectual progress often requires presupposition of fairly advanced ideas which build on each other, and which are therefore inaccessible to “readers who only know the basic concepts”. Suppose that I wrote a post outlining views on AI safety aimed at people who already know the basics of machine learning, or a post discussing a particular counter-argument to an unusual philosophical position. Would those not qualify as frontpage posts? If not, where would they go? And where do personal blogs fit into this taxonomy?
It crucially doesn’t ensure that the rewarded content will continue to be read by newcomers 5 years after it was written… New EAs on the Forum are not reading the best EA content of the past 10 years, just the most recent content.
This sentence deserves a strong upvote all by itself, it is exactly the key issue. There is so much good stuff out there, I’ve read pretty widely on EA topics but continue to find excellent material that I’ve never seen before, scattered across a range of blogs. Gathering that together seems vital as the movement gets older and it gets harder and harder to actually find and read everything.
I can imagine this being an automatic process based on voting, but I have an intuition that it’s good for humans to be in the loop. One reason is that when humans make decisions, you can ask why, but when 50 people vote, it’s hard to interrogate that system as to the reason behind its decision, and improve its reasoning the next time.
I think that’s true when there are moderators who are able to spend a lot of time and effort thinking about what to curate, like you do for Less Wrong. But right now it seems like the EA forum staff are very time-constrained, and in addition are worried about endorsing things. So in addition to the value of decentralising the work involved, there’s an additional benefit of voting in that it’s easier for CEA to disclaim endorsement.
Given that, I don’t have a strong opinion about whether it’s better for community members to be able to propose and vote on sequences, or whether it’s better for CEA to take a strong stance that they’re going to curate sequences with interesting content without necessarily endorsing it, and ensure that there’s enough staff time available to do that. The former currently seems more plausible (although I have no inside knowledge about what CEA are planning).
The thing I would like not to happen is for the EA forum to remain a news site because CEA is too worried about endorsing the wrong things to put up the really good content that already exists, or sets such a high bar for doing so that in practice you get only a couple of sequences. EA is a question, not a set of fixed endorsed beliefs, and I think the ability to move fast and engage with a variety of material is the lifeblood of an intellectual community.
Agreed that subforums are a good idea, but the way they’re done on facebook seems particularly bad for creating common knowledge, because (as you point out) they’re so scattered. Also the advantage of people checking facebook more is countered, for me, by the disadvantage of facebook being a massive time sink, so that I don’t want to encourage myself or others to go on it when I don’t have to. So it would be ideal if the solution could be a modification or improvement to the EA forum—especially given that the code for curation already exists!
The OpenAI and DeepMind posts you linked aren’t necessarily relevant, e.g. the Software Engineer, Science role is not for DeepMind’s safety team, and it’s pretty unclear to me whether the OpenAI ML engineer role is safety-relevant.
A few doubts:
It seems like MSR requires a multiverse large enough to have many well-correlated agents, but not large enough to run into the problems involved with infinite ethics. Most of my credence is on no multiverse or infinite multiverse, although I’m not particularly well-read on this issue.
My broad intuition is something like “Insofar as we can know about the values of other civilisations, they’re probably similar to our own. Insofar as we can’t, MSR isn’t relevant.” There are probably exceptions, though (e.g. we could guess the direction in which an r-selected civilisation’s values would vary from our own).
I worry that MSR is susceptible to self-mugging of some sort. I don’t have a particular example, but the general idea is that you’re correlated with other agents even if you’re being very irrational. And so you might end up doing things which seem arbitrarily irrational. But this is just a half-fledged thought, not a proper objection.
And lastly, I would have much more confidence in FDT and superrationality in general if there were a sensible metric of similarity between agents, apart from correlation (because if you always cooperate in prisoner’s dilemmas, then your choices are perfectly correlated with CooperateBot, but intuitively it’d still be more rational to defect against CooperateBot, because your decision algorithm isn’t similar to CooperateBot in the same way that it’s similar to your psychological twin). I guess this requires a solution to logical uncertainty, though.
Happy to discuss this more with you in person. Also, I suggest you cross-post to Less Wrong.
I also think there’s a lot of value to publishing a really good collection the first time around
The EA handbook already exists, so this could be the basis for the first sequence basically immediately. Also EA concepts.
More generally, I think I disagree with the broad framing you’re using, which feels like “we’re going to get the definitive collection of essays on each topic, which we endorse”. But even if CEA manages to put together a few such sequences, I predict that this will stagnate once people aren’t working on it as hard. By contrast, a more scalable type of sequence could be something like: ask Brian Tomasik, Paul Christiano, Scott Alexander, and other prolific writers, to assemble a reading list of the top 5-10 essays they’ve written relating to EA (as well as allowing community members to propose lists of essays related to a given theme). It seems quite likely that at least some of those points have been made better elsewhere, and also that many of them are controversial topics within EA, but people should be aware of this sort of thing, and right now there’s no good mechanism for that happening except vague word of mouth or spending lots of time scrolling through blogs.
I like “science-aligned” better than “secular”, since the former implies the latter as well as a bunch of other important concepts.
Also, it’s worth noting that “everyone’s welfare is to count equally” in Will’s account is approximately equivalent to “effective altruism values all people equally” in Ozymandias’ account, but neither of them imply the following paraphrase: “from the effective altruism perspective, saving the life of a baby in Africa is exactly as good as saving the life of a baby in America, which is exactly as good as saving the life of Ozy’s baby specifically.” I understand the intention of that phrase, but actually I’d save whichever baby would grow up to have the best life. Is there any better concrete description of what impartiality actually implies?
Note that your argument here is roughly Ben Pace’s position in this post which we co-wrote. I argued against Ben’s position in the post because I thought it was too extreme, but I agree with both of you that most EAs aren’t going far enough in that direction.
Excellent post, although I think about it using a slightly different framing. How vetting-constrained granters are depends a lot on how high their standards are. In the limit of arbitrarily high standards, all the vetting in the world might not be enough. In the limit of arbitrarily low standards, no vetting is required.
If we find that there’s not enough capability to vet, that suggests that either our standards are correct and we need more vetters, or that our standards are too high and we should lower them. I don’t have much inside information, so this is mostly based on my overall worldview, but I broadly think it’s more the latter: that standards are too high, and that worrying too much about protecting EA’s reputation makes it harder for us to innovate.
I think it would be very valuable to have more granters publicly explaining how they make tradeoffs between potential risks, clear benefits, and low-probability extreme successes; if these explanations exist and I’m just not aware of them, I’d appreciate pointers.
Another startup contacted at least 4 grantmaking organisations. Three of them deferred to the fourth.
One “easy fix” would simply be to encourage grantmakers to defer to each other less. Imagine that only one venture capital fund was allowed in Silicon Valley. I claim that’s one of the worst things you could do for entrepreneurship there.