AGB 🔸

Karma: 3,904

AGB 🔸Jan 12, 2021, 8:12 AM
35 points
0 ∶ 0
in reply to: Habryka [Deactivated]’s comment on: The Folly of “EAs Should”
The ‘any decent shot’ is doing a lot of work in that first sentence, given how hard the field is to get into. And even then you only say ‘probably stop’.

There’s a motte/bailey thing going on here, where the motte is something like ‘AI safety researchers probably do a lot more good than doctors’ and the bailey is ‘all doctors who come into contact with EA should be told to stop what they are doing and switch to becoming (e.g.) AI safety researchers, because that’s how bad being a doctor is’.

I don’t think we are making the world a better place by doing the second; where possible we should stick to ‘probably’ and communicate the first, nuance and all, as you did do here but as Khorton is noting people often don’t do in person.

AGB 🔸Jan 6, 2021, 9:10 AM
6 points
0 ∶ 0
in reply to: Paul_Christiano’s comment on: AGB’s Shortform
Thanks for the long comment, this gives me a much richer picture of how people might be thinking about this. On the first two bullets:
You say you aren’t anchoring, in a world where we defaulted to expressing probability in 1/10^6 units called Ms I’m just left feeling like you would write “you should be hesitant to assign 999,999M+ probabilities without a good argument. The burden of proof gets stronger and stronger as you move closer to 1, and 1,000,000 is getting to be a big number.”. So if it’s not anchoring, what calculation or intuition is leading you to specifically 99% (or at least, something in that ballpark), and would similarly lead you to roughly 990,000M with the alternate language?
My reply to Max and your first bullet both give examples of cases in the natural world where probabilities of real future events would go way outside the 0.01% − 99.99% range. Conjunctions force you to have extreme confidence somewhere, the only question is where. If I try to steelman your claim, I think I end up with an idea that we should apply our extreme confidence to the thing inside the product due to correlated cause, rather than the thing outside; does that sound fair?
The rest I see as an attempt to justify the extreme confidences inside the product, and I’ll have to think about more. The following are gut responses:
I’m not sure which step of this you get off the boat for
I’m much more baseline cynical than you seem to be about people’s willingness and ability to actually try, and try consistently, over a huge time period. To give some idea, I’d probably have assigned <50% probability to humanity surviving to the year 2150, and <10% for the year 3000, before I came across EA. Whether that’s correct or not, I don’t think its wildly unusual among people who take climate change seriously*, and yet we almost certainly aren’t doing enough to combat that as a society. This gives me little hope for dealing with <10% threats that will surely appear over the centuries, and as a result I found and continue to find the seemingly-baseline optimism of longtermist EA very jarring.
(Again, the above is a gut response as opposed to a reasoned claim.)
Applying the rule of thumb for estimating lifetimes to “the human species” rather than “intelligent life” seems like it’s doing a huge amount of work.
Yeah, Owen made a similar point, and actually I was using civilisation rather than ‘the human species’, which is 20x shorter still. I honestly hadn’t thought about intelligent life as a possible class before, and that probably is the thing from this conversation that has the most chance of changing how I think about this.

*”The survey from the Yale Program on Climate Change Communication found that 39 percent think the odds of global warming ending the human race are at least 50 percent. ”

AGB 🔸Jan 5, 2021, 8:21 PM
4 points
0 ∶ 0
in reply to: Max_Daniel’s comment on: AGB’s Shortform
Thanks for this. I won’t respond to your second/third bullets; as you say it’s not a defense of the claim itself, and while it’s plausible to me that many conclusions go through on much shorter timelines, I still want to understand the basis for the actual arguments made as best I can. Not least because if I can’t defend such arguments, then my personal pitches for longtermism (both to myself and to others) will not include them; they and I will focus on the next e.g. 10,000 years instead.
On your first bullet:
You are correct that within fixed models we can justifiably have extreme credences, e.g. for the probability of a specific result of 30 coin flips. However, I think the case for “modesty”—i.e. not ruling out very long futures—rests largely on model uncertainty...

...This insight that extremely low credences all-things-considered are often “forbidden” by model uncertainty is basically the point from Ord, Hillerbrand, & Sandberg (2008).
I’ll go and read the paper you mention, but flagging that my coinflip example is more general than you seem to think. Probability theory has conjunctions even outside of simple fixed models, and it’s the conjunction, not the fixed model, which is forcing you to have extreme credences. At best, we may be able to define a certain class of events where such credences are ‘forbidden’ (this could well be what the paper tries to do). We would then need to make sure that no such event can be expressed as a conjunction of a very large number of other such events.
Concretely, P(Humanity survives one billion years) is the product of one million probabilities of surviving each millenia, conditional on having survivied up to that point. As a result, we either need to set some of the intervening probabilities like P(Humanity survivies the next millenia | Humanity has survived to the year 500,000,000 AD) extremely high, or we need to set the overall product extremely low. Setting everything to the range 0.01% − 99.99% is not an option, without giving up on arithmetic or probability theory. And of course, I could break the product into a billion-fold conjunction where each component was ‘survive the next year’ if I wanted to make the requirements even more extreme.
Note I think it is plausible such extremes can be justified, since it seems like a version of humanity that has survived 500,000 millenia really should have excellent odds of surviving the next millenium. Indeed, I think that if you actually write out the model uncertainty argument mathematically, what ends up happening here is the fact that humanity has survivied 500,000 millenia is massive overwhelming Bayesian evidence that the ‘correct’ model is one of the ones that makes such a long life possible, allowing you to reach very extreme credences about the then-future. This is somewhat analagous to the intuitive extreme credence most people have that they won’t die in the next second.

AGB 🔸Jan 5, 2021, 10:52 AM
4 points
0 ∶ 0
in reply to: kierangreig🔸’s comment on: AGB’s Shortform
Thanks for the link. I did actually comment on that thread, and while I didn’t have it specifically in mind it was probably close to the start of me asking questions along these lines.

AGB 🔸Dec 31, 2020, 1:53 AM
7 points
0 ∶ 0
in reply to: Linch’s comment on: How modest should you be?
This comment is great, strong-upvoted.

There are enough individual and practical considerations here (in both directions) that in many situations the actual thing I would advocate for is something like “work out what you would do with both approaches, check against results ‘without fear or favour’, and move towards whatever method is working best for you”.
What links here?
- Linch's comment on My mistakes on the path to impact by Denise_Melchin (Jan 7, 2021, 10:01 AM; 4 points)

AGB 🔸Dec 28, 2020, 4:13 PM
39 points
0 ∶ 0
on: AGB’s Shortform
TL;DR: I’m curious what the most detailed or strongly-evidenced arguments are in favour of extinction risk eventually falling to extremely low levels.
An argument I often see goes to the effect of ‘we have a lot of uncertainty about the future, and given that it seems hard to be >99% confident that humanity will last <1 billion years”. As written, this seems like a case of getting anchored by percentages and failing to process just how long one billion years really is (weak supporting evidence for the latter is that I sometimes see eerily similar estimates for one million years...). Perhaps this is my finance background talking, but I can easily imagine a world where the dominant way to express probability is basis points and our go-to probability for ‘very unlikely thing’ was 1 bp rather than 1%, which is 100x smaller. Or we could have have a generic probability analogy to micromorts, which are 100x smaller still, etc. Yet such choices in language shouldn’t be affecting our decisions or beliefs about the best thing to do.
On the object level, one type of event I’m allowed to be extremely confident about is a large conjunction of events; if I flip a fair coin 30 times, the chance of getting 30 heads is approximately one in a billion.
Humanity surviving for a long time has a similar property; if you think that civilisation has a 50% chance of making it through the next 10,000 years, then conditional on that a 50% chance of making it through the next 20,000 years, then 50% for the next 40,000 years, etc. (applying a common rule-of-thumb for estimating uncertain lifetimes, starting with the observation that civilisation has been around for ~10,000 years so far), then the odds of surviving a billion years come out somewhere between 1 in 2^16 and 1 in 2^17, AKA roughly 0.001%.
We could also try to estimate current extinction risk directly based on known risks. Most attempts I’ve seen at this suggest that 50% to make it through the next 10,000 years, AKA roughly 0.007% per year, is very generous. As I see it, this is because an object-level anlysis of the risks suggests they have rising, not falling as the Lindy rule would imply.
When I’ve expressed this point to people in the past, I tend to get very handwavy (non-numeric) arguments about how a super-aligned-AI could dramatically cut existential risk to the levels required; another way of framing the above is that, to envisage a plausible future where we then have >one billion years in expectation, annualised risk needs to get to <0.0000001% in that future. Another thought is that space colonization could make humanity virtually invincible. So partly I’m wondering if there’s a better-developed version of these that accounts for the risks that would remain, or other routes to the same conclusion, since this assumption of a large-future-in-expectation seems critical to a lot of longtermist thought.
What links here?
- MichaelStJules's comment on Existential risk pessimism and the time of perils by David Thorstad (Aug 13, 2022, 2:57 PM; 24 points)

AGB’s Quick takes

AGB 🔸Dec 28, 2020, 4:13 PM

6 points

14 comments EA link

AGB 🔸Dec 28, 2020, 2:46 PM
37 points
0 ∶ 0
on: Strong Longtermism, Irrefutability, and Moral Progress
EA has yet to take irreversible action based on these ideas, but the philosophy is gaining traction and therefore deserves an equal amount of criticism. There have been millions donated to the cause of improving the long-term future: at the time of writing the Long-Term Future Fund has received just under $4.5 million USD in total, and the Open Philanthropy Project has dedicated a focus area to this cause in the form of “risks from advanced artificial intelligence.” While many millions more are still funneled through GiveWell, The Life You Can Save, and Animal Charity Evaluators
****
For what it’s worth, should the EA community abandon longtermism, I think many of its current priorities would remain unchanged; long-term causes do not yet dominate its portfolio.
I appreciate this tangential to the main point of the post, but these asides strike me as (unintentionally) likely to leave the reader with a common-but-inaccurate impression, and I think it’s worth correcting this impression as it arises in the name of general integrity and transparency.
Specifically, I think a reader of the above without further context would assume that longtermism is very new (say <2 years old, perhaps the age of the 2019 working paper), is basically just getting off the ground in EA, and it receives a small-but-significant amount of funding/time to date but has the potential to absorb much more.
The actual state of the world as I understand it is as follows:
- Open Philanthropy, as of Jan 2018, plan to donate more than half of their money to longtermist areas. In particular, I note that their grants to date in this area completely swamp the $4.5m given through the Long-Term Future Fund.
- The EA Survey suggests that ‘Long Term Future’ causes, broadly defined, are the top priority of 41% of the survey population.
- 80000 hours has a (commendably explicit) longtermist outlook and prioritises career paths largely on this basis.
- As of May 2018, CEA’s Current Thinking page stated they believed “the most effective opportunities to do good are aimed at helping the long-term future”. Note this page has a disclaimer saying this no longer reflects their current views; I do not know whether this section was part of what they no longer think. Still, the fact it was true at time of writing is relevant for assessing the age of this framework within the movement.
The overall picture this paints is of longtermism as the current dominant framework of the movement, at least within the core. I think it has been for at least three years. Questions of whether long-term causes dominate the portfolio, far from being settled, are very sensitive to whether we are counting only-money or money + time, and how broadly we define the EA community. The narrower the definition of ‘EA’ and the more you look at non-financial input, the more long-term causes dominate.

AGB 🔸Dec 19, 2020, 6:19 AM
4 points
0 ∶ 0
in reply to: Owen Cotton-Barratt’s comment on: A case against strong longtermism
Ah, gotcha. I agree that the process of scientific enquiry/discovery works best when people do as you said.

I think it’s worth distinguishing between that case where taking the less accurate path in the short-term has longer-term benefits, and more typical decisions like ‘what should I work on’, or even just truth-seeking that doesn’t have a decision directly attached but you want to get the right answer. There are definitely people who still believe what you wrote literally in those cases and ironically I think it’s a good example of an argument that sounds compelling but is largely incorrect, for reasons above.

AGB 🔸Dec 18, 2020, 2:00 PM
15 points
0 ∶ 0
in reply to: Owen Cotton-Barratt’s comment on: A case against strong longtermism
But if you’re engaged in the business of truth-seeking, why does it matter what the proponents do? You should ignore the bad arguments and pay attention to the best ones you can see
If someone who I have trusted with working out the answer to a complicated question makes an error that I can see and verify, I should also downgrade my assessment of all their work which might be much harder for me to see and verify.
Related: Gell-Mann Amnesia
(Edit: Also related, Epistemic Learned Helplessness)
Briefly stated, the Gell-Mann Amnesia effect is as follows. You open the newspaper to an article on some subject you know well. In Murray’s case, physics. In mine, show business. You read the article and see the journalist has absolutely no understanding of either the facts or the issues. Often, the article is so wrong it actually presents the story backward—reversing cause and effect. I call these the “wet streets cause rain” stories. Paper’s full of them.
In any case, you read with exasperation or amusement the multiple errors in a story, and then turn the page to national or international affairs, and read as if the rest of the newspaper was somehow more accurate about Palestine than the baloney you just read. You turn the page, and forget what you know.
The correct default response to this effect, in my view, mostly does not look like ‘ignoring the bad arguments and paying attention to the best ones’. That’s almost exactly the approach the above quote describes and (imo correctly) mocks; ignoring the show business article because your expertise lets you see the arguments are bad and taking the Palestine article seriously because the arguments appear to be good.
I think the correct default response is something closer to ‘focus on your areas of expertise, and see how the proponents conduct themselves within that area. Then use that as your starting point for guessing at their accurracy in areas which you know less well’.
Of course I’m expressing a super idealistic position here, and there are practical reasons not to be all the way there
I appreciate stuff like the above is part of why you wrote this. I still wanted to register that I think this framing is backwards; I don’t think you should evaluate the strength of arguments across all domains as they come and then adjust for trustworthiness of the person making them; in general I think it’s much better (measured by believing more true things) to assess the trustworthiness of the person in some domain you understand well and only then adjust to a limited extent based on the apparent strength of the arguments made in other domains.
It’s plausible that this boils down to a question of ‘how good are humans at assessing the strength of arguments in areas they know little about’. In the ideal, we are perfect. In reality, I think I am pretty terrible at it, in pretty much exactly the way the Gell-Mann quote describes, and so want to put minimal weight on those feelings of strength; they just don’t have enough predictive power to justify moving my priors all that much. YMMV.

AGB 🔸Dec 18, 2020, 1:21 PM
13 points
0 ∶ 0
in reply to: Owen Cotton-Barratt’s comment on: A case against strong longtermism
In their article vadmas writes:
Strong longtermism goes beyond its weaker counterpart in a significant way. While longtermism says we should be thinking primarily about the far-future consequences of our actions (which is generally taken to be on the scale of millions or billions of years), strong longtermism says this is the only thing we should think about.
Some of your comments, including this one, seem to me to be defending simple or weak longtermism (‘by far the most important effects are likely to be temporally distant’), rather than strong longtermism as defined above. I can imagine a few reasons for this:
1. You don’t actually agree with strong longtermism
2. You do agree with strong longtermism, but I (and presumably vadmas) am misunderstanding what you/MacAskill/Greaves mean by strong longtermism; the above quote is, presumably unintentionally, misunderstanding their views. In this case I think it would be good to hear what you think the ‘strong’ in ‘strong longermism’ actually means.
3. You think the above quote is compatible with what you’ve written above.
At the moment, I don’t have a great sense of which one is the case, and think clarity on this point would be useful. I could also have missed an another way to reconcile these.

AGB 🔸Dec 16, 2020, 7:50 PM
4 points
0 ∶ 0
in reply to: richard_ngo’s comment on: richard_ngo’s Shortform
Most of those people believe that there’s a consensus within EA that donations to Givewell’s top charities do more good than these systemic change donations, to a greater degree than there actually is.
Related to my other comment, but what would you guess is the split of donations from EAs to Givewell’s top charities versus ‘these systemic change donations’?
I ask because if it’s highly skewed, I would be strongly against pretending that we’re highly conflicted on this question while the reality of where we give says something very different; this question of how to represent ourselves accurately cuts both ways, and it is very tempting to try and be ‘all things to all people’.
All things considered, the limited data I have combined with anecdata from a large number of EAs suggests to me that it is in fact highly skewed.
A significant part of these confusions is due to EA simplifying its message in order to attract more people
I think this is backwards. The ‘systemic change’ objection, broadly defined, is by far the most common criticism of EA. Correspondingly, I think the movement would be much larger were it better-disposed to such interventions, largely neutralising this complaint and so appealing to a (much?) wider group of people.

AGB 🔸Dec 16, 2020, 6:49 PM
9 points
0 ∶ 0
in reply to: richard_ngo’s comment on: richard_ngo’s Shortform
I didn’t mean to ask whether no case has been made, or imply that they’ve “never been looked at”, but rather ask whether a compelling case has been made
I’m not quite sure what you’re trying to get at here. In some trivial sense we can see that many people were compelled, hence I didn’t bother to distinguish between ‘case’ and ‘compelling case’. I wonder whether by ‘compelling case’ you really mean ‘case I would find convincing’? In which case, I don’t know whether that case was ever made. I’d be happy to chat more offline and try to compel you :)
there’s a similarly powerful intuition going the other way: that wealthy countries are many times more influential than developing countries
I don’t think this intuition is similarly powerful at all, but more importantly I don’t think it ‘goes the other way’, or perhaps don’t understand what you mean by that phrase. Concretely, if we treat GDP-per-capita as a proxy for influentialness-per-person (not perfect, but seems like right ballpark), and how much we can influence people with $x also scales linearly with GDP-per-capita (i.e. it takes Y months’ wages to influence people Z amount), that would suggest that interventions aimed at influencing worldwide events have comparable impact anywhere, rather than actively favouring developed countries by anything like the 100x margin.
I suspect that our disagreement might be less about what research exists, and more about what standard to apply for justification.
I agree. I think the appropriate standard is basically the ‘do you buy your own bullshit’ standard. I.e. if I am donating to Givewell charities over climate change (CC) charities, that is very likely revealing that I truly think those opportunities are better all things considered, not just better according to some narrow criteria. At that point, I could be just plain wrong in expressing that opinion to others, but I’m not being dishonest. By contrast, if I give to CC charities over Givewell charities, I largely don’t think I should evangelise on behalf of Givewell charities, regardless of whether they score better on some specific criteria, unless I am very confident that the person I am talking to cares about those specific criteria (even then I’d want to add ‘I don’t support this personally’ caveats).
My impression is that EA broadly meets this standard, and I would be disappointed to hear of a case where an individual or group had pushed Givewell charities while having no interest in them for their personal or group-influenced donations.
the empirical claim should be evaluated against the donations made by people who want to do a lot of good, but aren’t familiar with EA. My guess is that climate change and politics are fairly central examples of such donations.
I’m happy to evaluate against these examples regardless, but (a) I doubt these are central, but not with high confidence, would be happy to see data and (b) I’m not sure evaluating against typical-for-that-group donations makes a whole lot of sense when for most people donations are a sideshow in their altruistic endeavours. The counterfactual where I don’t get involved with EA doesn’t look like me donating to climate change instead, it looks like me becoming a teacher rather than a trader and simply earning far less, or becoming a trader and retiring at 30 followed by doing volunteer work. On a quick scan of my relatively-altruistic non-EA friends (who skew economically-privileged and very highly educated, so YMMV) doing good in this kind of direct-but-local way looks like a far more typical approach than making large (say >5% of income) donations to favoured non-EA areas.
Givewell charities are still (I think) the main way that most outsiders perceive EA.
Communicating the fact that many core EA organisations have a firmly longtermist focus is something I am strongly in favour of. 80k has been doing a ton of work here to try and shift perceptions of what EA is about.
That said, in this venue I think it’s easy to overestimate the disconnect. 80k/CEA/EA forum/etc. are only one part of the movement, and heavily skew longtermist relative to the whole. Put plainly, in the event that outsiders perceive EA heavily through the lens of Givewell charities because most self-identified EAs are donating and their donations mostly go to Givewell charities, that seems fine, in the sense that perceptions match reality, regardless of what us oddballs are doing. In the event that outsiders perceive this because this used to be the case but is no longer, and there’s a lag, then I’m in favour of doing things to try and reduce the lag, example in previous paragraph.

AGB 🔸Dec 15, 2020, 11:59 PM
4 points
0 ∶ 0
in reply to: richard_ngo’s comment on: richard_ngo’s Shortform
You may also be interested in this piece from Open Phil:

https://www.openphilanthropy.org/blog/givewells-top-charities-are-increasingly-hard-beat

AGB 🔸Dec 15, 2020, 10:36 PM
38 points
0 ∶ 0
in reply to: richard_ngo’s comment on: richard_ngo’s Shortform
I think the strongest high-level argument for Givewell charities vs. most developed-world charity is the 100x multiplier.
That’s a strong reason to suspect the best opportunities to improve the lives of current humanity lie in the developing world, but not decisive, and so usually analyses have been done, particularly of ‘fan-favourite’ causes like the ones you mention.
I’d also note that both the examples you gave are not what I would consider ‘Mainstream charity’; both have prima facie plausible paths for high leverage (even if 100x feels a stretch), and if I had to guess right now my gut instinct is that both are in the top 25% for effectiveness. ‘Mainstream charity’ in my mind looks more like ‘your local church’, ‘the arts’, or ‘your local homeless shelter’. Some quantified insight into what people in the UK actually give to here.
At any rate, climate-change has had a few of these analyses over the years, off the top of my head here’s a recent one on the forum looking at the area in general, there’s also an old and more specific analysis of Cool Earth by GWWC, which after running through a bunch of numbers concludes:
Even with the most generous assumptions possible, this is still at least one order of magnitude greater than the cost of saving a life through donations to highly effective health charities such as the Against Malaria Foundation (at $3,461).
As for other areas, Givewell, (in?)famously, used to recommend charities in US education, but stopped after deciding their estimated effectiveness didn’t stack up to what they could achieve in the Global Health/Poverty space.
I don’t have anything to hand for the US Democratic party, but lots of people talked in various places about donations directed at helping the Clinton campaign in 2016 and then the Biden campaign in 2020, so I’d start there. 80k’s thoughts on the value of a vote would be a starting point.
If the case hasn’t been made
On a different note, I’m somewhere between bemused and disappointed that you could think this is a possibility, especially for causes which many EAs were very positively disposed to prior to their involvement with EA (and in the case of climate change, a large number remain so!). To be clear, I’m mostly disappointed in the movement’s ability to propagate information forward and in the fact that such analysis has apparently become so rare that you might think such common questions have never been looked at. Also to be clear, it could well be that the cases made are wrong, and I’d be happy to see refutations, but suggesting they haven’t been made is quite a bit stronger; suggests a wilful blind spot, and I’m reminded of this SSC post.
It’s plausible to me that there’s something near-term we simply haven’t done an analysis of, and should have done, and if we did it would look very strong, but if so I’d strongly expect to be in an area that does not naturally occur to EA’s core demographic, rather than in areas and ideas that naturally occur to that group.
What links here?
- EA Forum Prize: Winners for December 2020 by Aaron Gertler 🔸 (Feb 16, 2021, 8:46 AM; 26 points)

AGB 🔸Dec 14, 2020, 11:55 PM
10 points
0 ∶ 0
in reply to: richard_ngo’s comment on: Careers Questions Open Thread
That makes sense, thanks for the extra colour on PhDs.
Whereas I have more trouble picturing how a few years at a quant firm is helpful in switching to a different field, apart from the cash buffer.
I’ve heard variants on this a few times, so you aren’t alone. To give some extra colour on what I think you’re gaining from working at quant firms: Most of these firms still have a very start-up-like culture. That means that you get significant personal responsibility and significant personal choice about what you work on, within a generally supportive culture. In general this is valuable, but it means there isn’t one universal answer to this question. Still, some candidate skills I think you’ll get the opportunity to develop should you so choose.
- Project management
- People management
- Hiring
- Judgement (in the narrow 80k sense of the term)
(This list is illustrative based on my own experience, rather than exhaustive. Some of the above will apply to the PhD as well, it’s not intended as a comparison)

AGB 🔸Dec 14, 2020, 5:23 PM
2 points
0 ∶ 0
in reply to: richard_ngo’s comment on: Careers Questions Open Thread
In the world where your second paragraph is true, I’d expect the quant firms will start or have already started using AI heavily, and so by working as a software engineer at one of those firms you can expect to be able to build skills in that area. So then it’s a classic choice between ‘learning about something via a PhD’ versus ‘learning about something via working on a practical application’, which I generally think of as a YMMV question.
I’m curious if you expect the PhD to systematically have more optionality after accounting for that, if you weren’t already.

AGB 🔸Dec 14, 2020, 12:33 AM
25 points
0 ∶ 0
in reply to: asking_for_advice’s comment on: Careers Questions Open Thread
(Background: Have worked in trading since late 2013, with one ~18 month gap. Have also spoken to >5 people facing a decision similar to this one over the years. This is a set of points I often end up making. I’m moderately confident about each statement below but wouldn’t be surprised if one of them is wrong.)
I think both of these paths are very ‘spiky’, in the sense that I think the top 10% has many times more impact (either via donations or direct work) than the median. From a pure altruistic perspective, I think you mostly want to maximise the chance of spiking.
One of the best ways to maximise this is to be able to switch after you realise you aren’t in that category; in trading I think you’re likely to have a good sense of your appoximate path within 2-4 years, so in the likely even that you are not hitting the high end at that point you have an opportunity to switch, if your alternate allows you to switch (often but certainly not always the case). Similarly, I’d try to work out what flexibility you have on the AI PhD path; if you do in fact find the day-to-day frustrating and decide to quit in order to avoid burnout, what are your options? If you can switch either way, that’s great and this largely ceases to be an important consideration.
On a related note, and echoing some others, 2-4 years undoubtedly sounds like a long time now but it’s actually a pretty small fraction of your career. Don’t feel like a ‘bad’ choice now will forever condemn you to an inferior path! Any choice that does actually lock you in this way is probably a bad choice for that reason alone, but again usually this feels like it is the case more than it is actually the case.
Some people underestimate quant trading earnings at top-tier firms, because the firms are extremely cagey about them. FWIW, I think >$1m annual earnings within 5-10 years is more like the 40th-80th percentile case depending on firm, while the ‘spike’ case pushes into >$5m within their first decade. That’s for trading. Software engineer earnings at these firms I have a weaker sense of; they can either essentially match the traders or fall significantly behind (I don’t currently know of a case where it runs ahead); this depends heavily on the specific firm.
Predicting what you’re more likely to spike in advance is hard, and is going to be a question of personal fit, so difficult for us to answer, I just think that’s the right question. That said, there are some things in your post that make me think AI is more likely to be your life’s work:
- Writing code long-term (20+ years) sounds incredibly draining
- I know E2G is high impact but sometimes it doesn’t feel that way; definitely feel like a sell out
  ....
- Intellectually interesting; kind of a dream career
Sometimes people feel things like the above and assume everyone else feels the same way. As someone who has heard both pro-E2G and pro-AI versions of all of these points, the one thing I’m confident of is that this is not the case. As a community, there are worse ways we could coordinate this than having all the people who are more excited on a gut level about working at quant firms do that, and all the people who are more excited about working on AI do that. I do think there are more of the latter than the former, but that’s ok; one person E2G-ing at that level (especially if they spike) can fund many other people doing direct work.
The above is all written from a ‘For the Greater Good’ perspective. It sounds like a lot of what is pulling you towards quant work is that it’s the safe, already-available, almost-certainly-going-to-work-out-well-for-you-personally option. This is a reasonable thing to pay attention to! Just how much attention depends on things like your general level of financial security, do you have or expect to have dependents or other major family committments, how wide/deep is your career path if a particular opportunity goes away, what kinds of backup (e.g. parents you could move in with) do you have, etc. Try not to assume things will work out ok, but equally try not to assume one bad call will end it all if you have layers of protection. This is incredibly variable by individual, and most EA advice is calibrated for young people who are pretty secure, don’t have large external obligations, and have solid backup. This in turn strongly points to taking a higher-than-usual level of risk in your altruistically-motivated endeavours. If that’s you, then great. If not, it might be worth some reflection about to what extent you can afford to, e.g. aim for the PhD and fail.
What links here?
- AGB 🔸's comment on How much does performance differ between people? by Max_Daniel (Apr 3, 2021, 11:27 AM; 21 points)

AGB 🔸Dec 8, 2020, 2:28 PM
7 points
0 ∶ 0
in reply to: Michelle_Hutchinson’s comment on: My mistakes on the path to impact
I’m sad that you found the only way to feel ok about broadening your job search was to distance yourself from EA. It really seems like we’re doing things wrong as a community when we’re not only failing to support people but actively making their lives worse in that way. I’d be interested to hear ways in which you think we could do better, if you’d be up for sharing.

I think opinions on how to do better are rather sensitive to broader cause prioritisation and ‘what should the movement look like?’ questions, but some of Denise’s previous writing gives insight into her personal thoughts here, which I happen to agree with. I particularly note the following quote:

Participating in the EA community should make you feel more motivated about the amount of good you are able to do, not less. If it makes you feel less motivated on balance, then the EA community is doing something fundamentally wrong and everybody might be better off somewhere else until this is fixed.

If you have adapted to the belief that you can personally prevent lots of astronomical waste, it is time to go back to having more realistic expectations.

AGB 🔸Dec 7, 2020, 8:34 AM
55 points
1 ∶ 0
in reply to: Ben Kuhn’s comment on: My mistakes on the path to impact
That last paragraph is a good observation, and I don’t think it’s entirely coincidental. 80k has a few instances in their history of accidentally causing harm, which has led them (correctly) to be very conservative about it as an organisation.

The thing is, career advice and PR are two areas 80k is very involved in and which have particular likelihood of causing as much harm as good, due to bad advice or distorted messaging. Most decisions individual EAs make are not like this, and it’s a mistake if they treat 80k’s caution as a reflection of how cautious they should be. Or worse, act even more cautiously reasoning the combined intelligence of the 80k staff is greater than their own (likely true, but likely irrelevant).