Seems like you and the other David T are talking past each other tbh.
Above you reasonably argue the [facetious] “time of carols” hypothesis is not remotely as credible as the time of perils hypothesis. But you also don’t assign a specific credence to it, or provide an argument that the “time of carols” is impossible or even <1%[1]
I don’t think it would be fair to conclude from this that you don’t understand how probability works, and I also don’t think that it is reasonable to assume that the probability of the ‘time of carols’ should be assumed sufficiently nontrivial to warrant action in the absence of any specific credence attached to it. Indeed if someone responded to you indirectly with an example which assigned a prior of “just 1%”, to the “time of carols”, you might feel justified in assuming it was them misunderstanding probability...
The rest of Thorstad’s post which doesn’t seem to be specifically targeted at you explicitly argues that in practice, specific claims involving navigating a ‘time of perils’ also fall into the “trivial” category,[2] in the absence of robust argument as to why of all the possible futures this one is less trivial than others. He’s not arguing for “many gods” which invert the stakes so much as “many gods/pantheons means the possibility of any specific god is trivial, in the absence of compelling evidence of said god’s relative likelihood”. He also doesn’t bring any evidence to the table (other than arguing that time of perils hypothesis involves claims about x-risk in different centuries which might be best understood as independent claims [3]) but his position is that this shouldn’t be the sceptic’s job...
(Personally I’m not sure what said evidence would even look like, but for related reasons I’m not writing papers on longtermism and am happy applying a very high discount rate to the far future)
I think everyone would agree that it is absurd (that’s a problem with facetious examples)[4] but if the default is that logical possibilities are considered nontrivial until proven otherwise...
he doesn’t state a personal threshold, but does imply many longtermist propositions dip below Monton’s 5 * 10^-16 once you start adding up the claims....
a more significant claim he fails to emphasize is that the relevant criteria for longtermist interventions isn’t so much that the baseline hypothesis about peril distribution is [incidentally] true but the impact of a specific intervention at the margin has a sustained positive influence on it.
I tend to dislike facetious examples, but hey, this is a literature in which people talk about paperclip maximisers and try to understand AI moral reasoning capacity by asking LLMs variations on trolley problems...
I am far from sure that Thorstad is wrong that time of perils should be assigned ultra-low probability. (I do suspect he is wrong, but this stuff is extremely hard to assess.) But in my view there are multiple pretty obvious reasons why “time of Carols” is a poor analogy to “time of perils”:
“Time of carols” is just way more specific, in a bad way than time of perils. I know that there are indefinitely many ways time of carols could happen if you get really fine-grained, but it nonetheless, intuitively, there is in some sense way more significantly different paths “X-risk could briefly be high then very low” than “everyone is physically tied up and made to listen to carols”. To me it’s like comparing “there will be cars on Mars in 2120″ to “there will be a humanoid crate-stacking robot on Mars in 2120 that is nicknamed Carol”.
Actually, longtermists argue for the “current X-risk is high” claim, making Thorstad’s point that lots of things should get ultra-low prior probability is not particularly relevant to that half of the time of perils hypothesis. In comparison, no one argues for time of carols.
(Most important disanalogy in my view.) The second half of time of perils, that x-risk will go very low for a long-time, is plausibly something that many people will consider desirable, and might therefore aim for. People are even more likely to aim for related goals like “not have massive disasters while I am alive.” This is plausibly a pretty stable feature of human motivation that has a fair chance of lasting millions of years; humans generally don’t want humans to die. In comparison there’s little reason to think decent numbers of people will always desire time of carols.
4. Maybe this isn’t an independent point from 1., but I actually do think it is relevant that “time of carols” just seems very silly to everyone as soon as they hear it, and time of perils does not. I think we should give some weight to people’s gut reactions here.
The meta-dispute here isn’t the most important thing in the world, but for clarity’s sake, I think it’s worth distinguishing the following questions:
Does a specific text—Thorstad (2022)—either actually or apparently commit a kind of “best model fallacy”, arguing as though establishing Time of Perils hypothesis as unlikely to be true thereby suffices to undermine longtermism?
Does another specific text—my ‘Rule High Stakes In, Not Out’—either actually or apparently have as its “primary argumentative move… to assign nontrivial probabilities without substantial new evidence”?
My linked post suggests that the answer to Q1 is “Yes”. I find it weird that others in the comments here are taking stands on this textual dispute a priori, rather than by engaging with the specifics of the text in question, the quotes I respond to, etc.
My primary complaint in this comment thread has simply been that the answer to Q2 is “No” (if you read my post, you’ll see that it’s instead warning against what I’m now calling the “best model fallacy”, and explaining how I think various other writings—including Thorstad’s—seem to go awry as a result of not attending to this subtle point about model uncertainty). The point of my post is not to try to assert or argue for any particular probability assignment. Hence Thorstad’s current blog post misrepresents mine.
***
There’s a more substantial issue in the background:
Q3. What is the most reasonable prior probability estimate to assign to the time of perils hypothesis? In case of disagreement, does one party bear a special “burden of proof” to convince the other, who should otherwise be regarded as better justified by default?
I have some general opinions about the probability being non-negligible—I think Carl Shulman makes a good case here—but it’s not something I’m trying to argue about with those who regard it as negligible. I don’t feel like I have anything distinctive to contribute on that question at this time, and prefer to focus my arguments on more tractable points (like the point I was making about the best model fallacy). I independently think Thorstad is wrong about how the burden of proof applies, but that’s an argument for another day.
So I agree that there is some “talking past” happening here. Specifically, Thorstad seems to have read my post as addressing a different question (and advancing a different argument) than what it actually does, and made unwarranted epistemic charges on that basis. If anyone thinks my ‘Rule High Stakes In’ post similarly misrepresents Thorstad (2022), they’re welcome to make the case in the comments to that post.
This comment deserves some kind of EA Forum award. My goodness, I envy and admire the breezy style with which you wrote this. I wish we had some analogue of Reddit gold, an award in limited supply, we could use to recognize exceptional contributions like this.
I agree with David Mathers that it’s simply psychologically implausible that David Thorstad, a sharp professional philosopher and an expert on bounded rationality, existential risk, longtermism, and effective altruism, doesn’t understand the concept of expected value. I think we need to jettison such accusations, which have more of personal insult about them than substantive argument. Such accusations, besides just being ridiculous on their face, are corrosive to productive, charitable discussion about substantive disagreements on important topics.
It’s not a psychological question. I wrote a blog post offering a philosophical critique of some published academic papers that, it seemed to me, involved an interesting and important error of reasoning. Anyone who thinks my critique goes awry is welcome to comment on it there. But whether my philosophical critique is ultimately correct or not, I don’t think that the attempt is aptly described as “personal insult”, “ridiculous on [its] face”, or “corrosive to productive, charitable discussion”. It’s literally just doing philosophy.
I’d like it if people read my linked post before passing judgment on it.
Seems like you and the other David T are talking past each other tbh.
Above you reasonably argue the [facetious] “time of carols” hypothesis is not remotely as credible as the time of perils hypothesis. But you also don’t assign a specific credence to it, or provide an argument that the “time of carols” is impossible or even <1%[1]
I don’t think it would be fair to conclude from this that you don’t understand how probability works, and I also don’t think that it is reasonable to assume that the probability of the ‘time of carols’ should be assumed sufficiently nontrivial to warrant action in the absence of any specific credence attached to it. Indeed if someone responded to you indirectly with an example which assigned a prior of “just 1%”, to the “time of carols”, you might feel justified in assuming it was them misunderstanding probability...
The rest of Thorstad’s post which doesn’t seem to be specifically targeted at you explicitly argues that in practice, specific claims involving navigating a ‘time of perils’ also fall into the “trivial” category,[2] in the absence of robust argument as to why of all the possible futures this one is less trivial than others. He’s not arguing for “many gods” which invert the stakes so much as “many gods/pantheons means the possibility of any specific god is trivial, in the absence of compelling evidence of said god’s relative likelihood”. He also doesn’t bring any evidence to the table (other than arguing that time of perils hypothesis involves claims about x-risk in different centuries which might be best understood as independent claims [3]) but his position is that this shouldn’t be the sceptic’s job...
(Personally I’m not sure what said evidence would even look like, but for related reasons I’m not writing papers on longtermism and am happy applying a very high discount rate to the far future)
I think everyone would agree that it is absurd (that’s a problem with facetious examples)[4] but if the default is that logical possibilities are considered nontrivial until proven otherwise...
he doesn’t state a personal threshold, but does imply many longtermist propositions dip below Monton’s 5 * 10^-16 once you start adding up the claims....
a more significant claim he fails to emphasize is that the relevant criteria for longtermist interventions isn’t so much that the baseline hypothesis about peril distribution is [incidentally] true but the impact of a specific intervention at the margin has a sustained positive influence on it.
I tend to dislike facetious examples, but hey, this is a literature in which people talk about paperclip maximisers and try to understand AI moral reasoning capacity by asking LLMs variations on trolley problems...
I am far from sure that Thorstad is wrong that time of perils should be assigned ultra-low probability. (I do suspect he is wrong, but this stuff is extremely hard to assess.) But in my view there are multiple pretty obvious reasons why “time of Carols” is a poor analogy to “time of perils”:
“Time of carols” is just way more specific, in a bad way than time of perils. I know that there are indefinitely many ways time of carols could happen if you get really fine-grained, but it nonetheless, intuitively, there is in some sense way more significantly different paths “X-risk could briefly be high then very low” than “everyone is physically tied up and made to listen to carols”. To me it’s like comparing “there will be cars on Mars in 2120″ to “there will be a humanoid crate-stacking robot on Mars in 2120 that is nicknamed Carol”.
Actually, longtermists argue for the “current X-risk is high” claim, making Thorstad’s point that lots of things should get ultra-low prior probability is not particularly relevant to that half of the time of perils hypothesis. In comparison, no one argues for time of carols.
(Most important disanalogy in my view.) The second half of time of perils, that x-risk will go very low for a long-time, is plausibly something that many people will consider desirable, and might therefore aim for. People are even more likely to aim for related goals like “not have massive disasters while I am alive.” This is plausibly a pretty stable feature of human motivation that has a fair chance of lasting millions of years; humans generally don’t want humans to die. In comparison there’s little reason to think decent numbers of people will always desire time of carols.
4. Maybe this isn’t an independent point from 1., but I actually do think it is relevant that “time of carols” just seems very silly to everyone as soon as they hear it, and time of perils does not. I think we should give some weight to people’s gut reactions here.
The meta-dispute here isn’t the most important thing in the world, but for clarity’s sake, I think it’s worth distinguishing the following questions:
Does a specific text—Thorstad (2022)—either actually or apparently commit a kind of “best model fallacy”, arguing as though establishing Time of Perils hypothesis as unlikely to be true thereby suffices to undermine longtermism?
Does another specific text—my ‘Rule High Stakes In, Not Out’—either actually or apparently have as its “primary argumentative move… to assign nontrivial probabilities without substantial new evidence”?
My linked post suggests that the answer to Q1 is “Yes”. I find it weird that others in the comments here are taking stands on this textual dispute a priori, rather than by engaging with the specifics of the text in question, the quotes I respond to, etc.
My primary complaint in this comment thread has simply been that the answer to Q2 is “No” (if you read my post, you’ll see that it’s instead warning against what I’m now calling the “best model fallacy”, and explaining how I think various other writings—including Thorstad’s—seem to go awry as a result of not attending to this subtle point about model uncertainty). The point of my post is not to try to assert or argue for any particular probability assignment. Hence Thorstad’s current blog post misrepresents mine.
***
There’s a more substantial issue in the background:
Q3. What is the most reasonable prior probability estimate to assign to the time of perils hypothesis? In case of disagreement, does one party bear a special “burden of proof” to convince the other, who should otherwise be regarded as better justified by default?
I have some general opinions about the probability being non-negligible—I think Carl Shulman makes a good case here—but it’s not something I’m trying to argue about with those who regard it as negligible. I don’t feel like I have anything distinctive to contribute on that question at this time, and prefer to focus my arguments on more tractable points (like the point I was making about the best model fallacy). I independently think Thorstad is wrong about how the burden of proof applies, but that’s an argument for another day.
So I agree that there is some “talking past” happening here. Specifically, Thorstad seems to have read my post as addressing a different question (and advancing a different argument) than what it actually does, and made unwarranted epistemic charges on that basis. If anyone thinks my ‘Rule High Stakes In’ post similarly misrepresents Thorstad (2022), they’re welcome to make the case in the comments to that post.
This comment deserves some kind of EA Forum award. My goodness, I envy and admire the breezy style with which you wrote this. I wish we had some analogue of Reddit gold, an award in limited supply, we could use to recognize exceptional contributions like this.
I agree with David Mathers that it’s simply psychologically implausible that David Thorstad, a sharp professional philosopher and an expert on bounded rationality, existential risk, longtermism, and effective altruism, doesn’t understand the concept of expected value. I think we need to jettison such accusations, which have more of personal insult about them than substantive argument. Such accusations, besides just being ridiculous on their face, are corrosive to productive, charitable discussion about substantive disagreements on important topics.
It’s not a psychological question. I wrote a blog post offering a philosophical critique of some published academic papers that, it seemed to me, involved an interesting and important error of reasoning. Anyone who thinks my critique goes awry is welcome to comment on it there. But whether my philosophical critique is ultimately correct or not, I don’t think that the attempt is aptly described as “personal insult”, “ridiculous on [its] face”, or “corrosive to productive, charitable discussion”. It’s literally just doing philosophy.
I’d like it if people read my linked post before passing judgment on it.