Yeah I suppose we just disagree then. I think such a big error and hit to the community should downgrade any rational personâs belief in the output of what EA has to offer and also downgrade the trust they are getting it right.
Another side point: Many EAs like Cowen and think he is right most of the time. I think it is suspicious that when Cowen says something about EA that is negative he is labeled stuff like âdaftâ.
Hi Devon, FWIW I agree with John Halstead and Michael PJ re Johnâs point 1.
If youâre open to considering this question further, you may be interested in knowing my reasoning (note that I arrived at this opinion independently of John and Michael), which I share below.
Last November I commented on Tyler Cowenâs post to explain why I disagreed with his point:
I donât find Tylerâs point very persuasive: Despite the fact that the common sense interpretation of the phrase âexistential riskâ makes it applicable to the sudden downfall of FTX, in actuality I think forecasting existential risks (e.g. the probability of AI takeover this century) is a very different kind of forecasting question than forecasting whether FTX would suddenly collapse, so performance at one doesnât necessarily tell us much about performance on the other.
Additionally, and more importantly, the failure to anticipate the collapse of FTX seems to not so much be an example of making a bad forecast, but an example of failure to even consider the hypothesis. If an EA researcher had made it their job to try to forecast the probability that FTX collapses and assigned a very low probability to it after much effort, that probably would have been a bad forecast. But thatâs not what happened; in reality EAs just failed to even consider that forecasting question. EAs *have* very seriously considered forecasting questions on x-risk though.
So the better critique of EAs in the spirit of Tylerâs would not be to criticize EAâs existential risk forecasts, but rather to suggest that there may be an existential risk that destroys humanityâs potential that isnât even on our radar (similar to how the sudden end of FTX wasnât on our radar). Others have certainly talked about this possibility before though, so that wouldnât be a new critique. E.g. Toby Ord in The Precipice put âUnforeseen anthropogenic risksâ in the next century at ~1 in 30. (Source: https://ââforum.effectivealtruism.org/ââposts/ââZ5KZ2cui8WDjyF6gJ/ââsome-thoughts-on-toby-ord-s-existential-risk-estimates). Does Tyler think ~1 in 30 this century is too low? Or that people havenât spent enough effort thinking about these unknown existential risks?
You made a further point, Devon, that I want to respond to as well:
There is a certain hubris in claiming you are going to âbuild a flourishing futureâ and âsupport ambitious projects to improve humanityâs long-term prospectsâ (as the FFF did on its website) only to not exist 6 months later and for reasons of fraud to boot.
I agree with you here. However, I think the hubris was SBFâs hubris, not EAsâ or longtermists-in-generalâs hubris.
Iâd even go further to say that it wasnât the Future Fund teamâs hubris.
As John commented below, âEAs did a bad job on the governance and management of risks involved in working with SBF and FTX, which is very obvious and everyone already agrees.â
But thatâs a critique of the Future Fundâs (and othersâ) ability to think of all the right top priorities for their small team in their first 6 months (or however long it was), not a sign that the Future Fund had hubris.
Note, however, that I donât even consider the Future Fund teamâs failure to think of this to be a very big critique of them. Why? Because anyone (in the EA community or otherwise) could have entered in The Future Fundâs Project Ideas Competition and suggested the project of investigating the integrity of SBF and his businesses, and the risk that they may suddenly collapse, to ensure the stability of the funding source for the benefit of future Future Fund projects, and to protect EAâs and longtermistsâ reputation from risks arising from associating with SBF should SBF become involved in a scandal. (Even Tyler Cowen could have done so and won some easy money.) But no one did (as far as Iâm aware). So given that, I conclude that it was a hard risk to spot so early on, and consequently I donât fault the Future Fund team all that much for failing to spot this in their first 6 months.
There is a lesson to be learned from peoplesâ failure to spot the risk, but that lesson is not that longtermists lack the ability to forecast existential risks well, or even that they lack the ability to build a flourishing future.
I disagreed with the Scott analogy but after thinking it through it made me change my mind. Simply make the following modification:
âLeading UN climatologists are in serious condition after all being wounded in the hurricane Smithfield that further killed as many people as were harmed by the FTX scandal. These climatologists claim that their models can predict the temperature of the Earth from now until 2200 - but they couldnât even predict a hurricane in their own neighborhood. Why should we trust climatologists to protect us from some future catastrophe, when they canât even protect themselves or those nearby in the present?â
Now we are talking about a group rather than one person and also what they missed is much more directly in their domain expertise. I.e. it feels, like the FTX Future fund teamâs domain expertise on EA money, like something they shouldnât be able to miss.
Would you say any rational person should downgrade their opinion of the climatology community and any output they have to offer and downgrade the trust they are getting their 2200 climate change models right?
I shared the modification with an EA thatâlike meâat first agreed with Cowen. Their response was something like âOK, so the climatologists not seeing the existential neartermist threat to themselves appears to still be a serious failure (people they know died!) on their part that needs to be addressedâbut I agree it would be a mistake on my part to downgrade my confidence in their 2100 climate change model because if itâ
However, we conceded that there is a catch: if the climatology community persistently finds their top UN climatologists wounded in hurricanes to the point that they canât work on their models, then rationally we ought to update that their productive output should be lower than expected because they seem to have this neartermist blindspot to their own wellbeing and those nearby. This concession though comes with asterisks though. If we, for sake of argument, assume climatology research benefits greatly from climatologists getting close to hurricanes then we should expect climatologists, as a group, to see more hurricane wounds. In that case we should update, but not as strongly, if climatologists get hurricane wounds.
Ultimately I updated from agree with Cowen to disagree with Cowen after thinking this through. Iâd be curious if and where you disagree with this.
This feels wrong to me? Gell-Mann amnesia is more about general competency whereas I thought Cowen was referring to specficially the category of âexistential riskâ (which I think is a semantics game but others disagree)?
Imagine a forecaster that you havenât previously heard of told you that thereâs a high probability of a new novel pandemic (âpigeon fluâ) next month, and their technical arguments are too complicated for you to follow.[1]
Suppose you want to figure out how much you want to defer to them, and you dug through to find out the following facts:
a) The forecaster previously made consistently and egregiously bad forecasts about monkeypox, covid-19, Ebola, SARS, and 2009 H1N1.
b) The forecaster made several elementary mistakes in a theoretical paper on Bayesian statistics
c) The forecaster has a really bad record at videogames, like bronze tier at League of Legends.
I claim that the general competency argument technically goes through for a), b), and c). However, for a practical answer on deference, a) is much more damning than b) or especially c), as you might expect domain-specific ability on predicting pandemics to be much stronger evidence for whether the prediction of pigeon flu is reasonable than general competence as revealed by mathematical ability/âconscientiousness or videogame ability.
With a quote like
Hardly anyone associated with Future Fund saw the existential risk to⊠Future Fund, even though they were as close to it as one could possibly be.
I am thus skeptical about their ability to predict existential risk more generally, and for systems that are far more complex and also far more distant.
The natural interpretation to me is that Cowen (and by quoting him, by extension the authors of the post) is trying to say that FF not predicting the FTX fraud and thus âexistential risk to FFâ is akin to a). That is, a dispositive domain-specific bad forecast that should be indicative of their abilities to predict existential risk more generally. This is akin to how much you should trust someone predicting pigeon flu when theyâve been wrong on past pandemics and pandemic scares.
To me, however, this failure, while significant as evidence of general competency, is more similar to b). Itâs embarrassing and evidence of poor competence to make elementary errors in math. Similarly, itâs embarrassing and evidence of poor competence to not successfully consider all the risks to your organization. But using the phrase âexistential riskâ is just a semantics game tying them together (in the same way that âwhy would I trust the Bayesian updates in your pigeon flu forecasting when youâve made elementary math errors in a Bayesian statistics paperâ is a bit of a semantics game).
EAs do not to my knowledge claim to be experts on all existential risks, broadly and colloquially defined. Some subset of EAs do claim to be experts on global-scale existential risks like dangerous AI or engineered pandemics, which is a very different proposition.
[1] Or, alternatively, you think their arguments are inside-view correct but you donât have a good sense of the selection biases involved.
I agree that the focus on competency on existential risk research specifically is misplaced. But I still think the general competency argument goes through. And as I say elsewhere in the threadâtabooing âexistential riskâ and instead looking at Longtermism, it looks (and is) pretty bad that a flagship org branded as âlongtermistâ didnât last a year!
Funnily enough, the âpigeon fluâ example may cease to become a hypothetical. Pretty soon, we may need to look at the track record of various agencies and individuals to assess their predictions on H5N1.
Thank you! I remember hearing about Bayesian updates, but rationalizations can wipe those away quickly. From the perspective of Popper, EAs should try âtaking the hypothesis that EA...â and then try proving themselves wrong, instead of using a handful of data-points to reach their preferred, statistically irrelevant conclusion, all-the-while feeling confident.
Yeah I suppose we just disagree then. I think such a big error and hit to the community should downgrade any rational personâs belief in the output of what EA has to offer and also downgrade the trust they are getting it right.
Another side point: Many EAs like Cowen and think he is right most of the time. I think it is suspicious that when Cowen says something about EA that is negative he is labeled stuff like âdaftâ.
Hi Devon, FWIW I agree with John Halstead and Michael PJ re Johnâs point 1.
If youâre open to considering this question further, you may be interested in knowing my reasoning (note that I arrived at this opinion independently of John and Michael), which I share below.
Last November I commented on Tyler Cowenâs post to explain why I disagreed with his point:
You made a further point, Devon, that I want to respond to as well:
I agree with you here. However, I think the hubris was SBFâs hubris, not EAsâ or longtermists-in-generalâs hubris.
Iâd even go further to say that it wasnât the Future Fund teamâs hubris.
As John commented below, âEAs did a bad job on the governance and management of risks involved in working with SBF and FTX, which is very obvious and everyone already agrees.â
But thatâs a critique of the Future Fundâs (and othersâ) ability to think of all the right top priorities for their small team in their first 6 months (or however long it was), not a sign that the Future Fund had hubris.
Note, however, that I donât even consider the Future Fund teamâs failure to think of this to be a very big critique of them. Why? Because anyone (in the EA community or otherwise) could have entered in The Future Fundâs Project Ideas Competition and suggested the project of investigating the integrity of SBF and his businesses, and the risk that they may suddenly collapse, to ensure the stability of the funding source for the benefit of future Future Fund projects, and to protect EAâs and longtermistsâ reputation from risks arising from associating with SBF should SBF become involved in a scandal. (Even Tyler Cowen could have done so and won some easy money.) But no one did (as far as Iâm aware). So given that, I conclude that it was a hard risk to spot so early on, and consequently I donât fault the Future Fund team all that much for failing to spot this in their first 6 months.
There is a lesson to be learned from peoplesâ failure to spot the risk, but that lesson is not that longtermists lack the ability to forecast existential risks well, or even that they lack the ability to build a flourishing future.
I disagreed with the Scott analogy but after thinking it through it made me change my mind. Simply make the following modification:
âLeading UN climatologists are in serious condition after all being wounded in the hurricane Smithfield that further killed as many people as were harmed by the FTX scandal. These climatologists claim that their models can predict the temperature of the Earth from now until 2200 - but they couldnât even predict a hurricane in their own neighborhood. Why should we trust climatologists to protect us from some future catastrophe, when they canât even protect themselves or those nearby in the present?â
Now we are talking about a group rather than one person and also what they missed is much more directly in their domain expertise. I.e. it feels, like the FTX Future fund teamâs domain expertise on EA money, like something they shouldnât be able to miss.
Would you say any rational person should downgrade their opinion of the climatology community and any output they have to offer and downgrade the trust they are getting their 2200 climate change models right?
I shared the modification with an EA thatâlike meâat first agreed with Cowen. Their response was something like âOK, so the climatologists not seeing the existential neartermist threat to themselves appears to still be a serious failure (people they know died!) on their part that needs to be addressedâbut I agree it would be a mistake on my part to downgrade my confidence in their 2100 climate change model because if itâ
However, we conceded that there is a catch: if the climatology community persistently finds their top UN climatologists wounded in hurricanes to the point that they canât work on their models, then rationally we ought to update that their productive output should be lower than expected because they seem to have this neartermist blindspot to their own wellbeing and those nearby. This concession though comes with asterisks though. If we, for sake of argument, assume climatology research benefits greatly from climatologists getting close to hurricanes then we should expect climatologists, as a group, to see more hurricane wounds. In that case we should update, but not as strongly, if climatologists get hurricane wounds.
Ultimately I updated from agree with Cowen to disagree with Cowen after thinking this through. Iâd be curious if and where you disagree with this.
Tbh I took the Gell-Mann amnesia interpretation and just concluded that heâs probably being daft more often in areas I donât know so much about.
This is what Cowen was doing with his original remark.
This feels wrong to me? Gell-Mann amnesia is more about general competency whereas I thought Cowen was referring to specficially the category of âexistential riskâ (which I think is a semantics game but others disagree)?
Cowen is saying that he thinks EA is less generally competent because of not seeing the x-risk to the Future Fund.
Again if this was true he would not specifically phrase it as existential risk (unless maybe he was actively trying to mislead)
Fair enough. The implication is there though.
Imagine a forecaster that you havenât previously heard of told you that thereâs a high probability of a new novel pandemic (âpigeon fluâ) next month, and their technical arguments are too complicated for you to follow.[1]
Suppose you want to figure out how much you want to defer to them, and you dug through to find out the following facts:
a) The forecaster previously made consistently and egregiously bad forecasts about monkeypox, covid-19, Ebola, SARS, and 2009 H1N1.
b) The forecaster made several elementary mistakes in a theoretical paper on Bayesian statistics
c) The forecaster has a really bad record at videogames, like bronze tier at League of Legends.
I claim that the general competency argument technically goes through for a), b), and c). However, for a practical answer on deference, a) is much more damning than b) or especially c), as you might expect domain-specific ability on predicting pandemics to be much stronger evidence for whether the prediction of pigeon flu is reasonable than general competence as revealed by mathematical ability/âconscientiousness or videogame ability.
With a quote like
The natural interpretation to me is that Cowen (and by quoting him, by extension the authors of the post) is trying to say that FF not predicting the FTX fraud and thus âexistential risk to FFâ is akin to a). That is, a dispositive domain-specific bad forecast that should be indicative of their abilities to predict existential risk more generally. This is akin to how much you should trust someone predicting pigeon flu when theyâve been wrong on past pandemics and pandemic scares.
To me, however, this failure, while significant as evidence of general competency, is more similar to b). Itâs embarrassing and evidence of poor competence to make elementary errors in math. Similarly, itâs embarrassing and evidence of poor competence to not successfully consider all the risks to your organization. But using the phrase âexistential riskâ is just a semantics game tying them together (in the same way that âwhy would I trust the Bayesian updates in your pigeon flu forecasting when youâve made elementary math errors in a Bayesian statistics paperâ is a bit of a semantics game).
EAs do not to my knowledge claim to be experts on all existential risks, broadly and colloquially defined. Some subset of EAs do claim to be experts on global-scale existential risks like dangerous AI or engineered pandemics, which is a very different proposition.
[1] Or, alternatively, you think their arguments are inside-view correct but you donât have a good sense of the selection biases involved.
I agree that the focus on competency on existential risk research specifically is misplaced. But I still think the general competency argument goes through. And as I say elsewhere in the threadâtabooing âexistential riskâ and instead looking at Longtermism, it looks (and is) pretty bad that a flagship org branded as âlongtermistâ didnât last a year!
Funnily enough, the âpigeon fluâ example may cease to become a hypothetical. Pretty soon, we may need to look at the track record of various agencies and individuals to assess their predictions on H5N1.
I agree that is the other way out of the puzzle. I wonder whom to even trust if everyone is susceptible to this problem...
Thank you! I remember hearing about Bayesian updates, but rationalizations can wipe those away quickly. From the perspective of Popper, EAs should try âtaking the hypothesis that EA...â and then try proving themselves wrong, instead of using a handful of data-points to reach their preferred, statistically irrelevant conclusion, all-the-while feeling confident.