This sort of âmany godsâ-style response is precisely what I was referring to with my parenthetical: âunless one inverts the high stakes in a way that cancels out the other high-stakes possibility.â
I donât think that dystopian âtime of carolsâ scenarios are remotely as credible as the time of perils hypothesis. If someone disagrees, then certainly resolving that substantive disagreement would be important for making dialectical progress on the question of whether x-risk mitigation is worthwhile or not.
What makes both arguments instances of the nontrivial probability gambit is that they do not provide significant new evidence for the challenged claims. Their primary argumentative move is to assign nontrivial probabilities without substantial new evidence.
I donât think this is a good way to argue. I think that nontrivial probability assignments to strong and antecedently implausible claims should be supported by extensive argument rather than manufactured probabilities.
Iâd encourage Thorstad to read my post more carefully and pay attention to what I am arguing there. I was making an in principle point about how expected value works, highlighting a logical fallacy in Thorstadâs published work on this topic. (Nothing in the paper I responded to seemed to acknowledge that a 1% chance of the time of perils would suffice to support longtermism. He wrote about the hypothesis being âinconclusiveâ as if that sufficed to rule it out, and I think itâs important to recognize that this is bad reasoning on his part.)
Saying that my âprimary argumentative move is to assign nontrivial probabilities without substantial new evidenceâ is poor reading comprehension on Thorstadâs part. Actually, my primary argumentative move was explaining how expected value works. The numbers are illustrative, and suffice for anyone who happens to share my priors (or something close enough). Obviously, Iâm not in that post trying to persuade someone who instead thinks the correct probability to assign is negligible. Thorstad is just radically misreading what my post is arguing.
(What makes this especially strange is that, iirc, the published paper of Thorstadâs to which I was replying did not itself argue that the correct probability to assign to the ToP hypothesis is negligible, but just that the case for the hypothesis is âinconclusiveâ. So it sounds like heâs now accusing me of poor epistemics because I failed to respond to a different paper than the one he actually wrote? Geez.)
Seems like you and the other David T are talking past each other tbh.
Above you reasonably argue the [facetious] âtime of carolsâ hypothesis is not remotely as credible as the time of perils hypothesis. But you also donât assign a specific credence to it, or provide an argument that the âtime of carolsâ is impossible or even <1%[1]
I donât think it would be fair to conclude from this that you donât understand how probability works, and I also donât think that it is reasonable to assume that the probability of the âtime of carolsâ should be assumed sufficiently nontrivial to warrant action in the absence of any specific credence attached to it. Indeed if someone responded to you indirectly with an example which assigned a prior of âjust 1%â, to the âtime of carolsâ, you might feel justified in assuming it was them misunderstanding probability...
The rest of Thorstadâs post which doesnât seem to be specifically targeted at you explicitly argues that in practice, specific claims involving navigating a âtime of perilsâ also fall into the âtrivialâ category,[2] in the absence of robust argument as to why of all the possible futures this one is less trivial than others. Heâs not arguing for âmany godsâ which invert the stakes so much as âmany gods/âpantheons means the possibility of any specific god is trivial, in the absence of compelling evidence of said godâs relative likelihoodâ. He also doesnât bring any evidence to the table (other than arguing that time of perils hypothesis involves claims about x-risk in different centuries which might be best understood as independent claims [3]) but his position is that this shouldnât be the scepticâs job...
(Personally Iâm not sure what said evidence would even look like, but for related reasons Iâm not writing papers on longtermism and am happy applying a very high discount rate to the far future)
I think everyone would agree that it is absurd (thatâs a problem with facetious examples)[4] but if the default is that logical possibilities are considered nontrivial until proven otherwise...
he doesnât state a personal threshold, but does imply many longtermist propositions dip below Montonâs 5 * 10^-16 once you start adding up the claims....
a more significant claim he fails to emphasize is that the relevant criteria for longtermist interventions isnât so much that the baseline hypothesis about peril distribution is [incidentally] true but the impact of a specific intervention at the margin has a sustained positive influence on it.
I tend to dislike facetious examples, but hey, this is a literature in which people talk about paperclip maximisers and try to understand AI moral reasoning capacity by asking LLMs variations on trolley problems...
I am far from sure that Thorstad is wrong that time of perils should be assigned ultra-low probability. (I do suspect he is wrong, but this stuff is extremely hard to assess.) But in my view there are multiple pretty obvious reasons why âtime of Carolsâ is a poor analogy to âtime of perilsâ:
âTime of carolsâ is just way more specific, in a bad way than time of perils. I know that there are indefinitely many ways time of carols could happen if you get really fine-grained, but it nonetheless, intuitively, there is in some sense way more significantly different paths âX-risk could briefly be high then very lowâ than âeveryone is physically tied up and made to listen to carolsâ. To me itâs like comparing âthere will be cars on Mars in 2120âł to âthere will be a humanoid crate-stacking robot on Mars in 2120 that is nicknamed Carolâ.
Actually, longtermists argue for the âcurrent X-risk is highâ claim, making Thorstadâs point that lots of things should get ultra-low prior probability is not particularly relevant to that half of the time of perils hypothesis. In comparison, no one argues for time of carols.
(Most important disanalogy in my view.) The second half of time of perils, that x-risk will go very low for a long-time, is plausibly something that many people will consider desirable, and might therefore aim for. People are even more likely to aim for related goals like ânot have massive disasters while I am alive.â This is plausibly a pretty stable feature of human motivation that has a fair chance of lasting millions of years; humans generally donât want humans to die. In comparison thereâs little reason to think decent numbers of people will always desire time of carols.
4. Maybe this isnât an independent point from 1., but I actually do think it is relevant that âtime of carolsâ just seems very silly to everyone as soon as they hear it, and time of perils does not. I think we should give some weight to peopleâs gut reactions here.
The meta-dispute here isnât the most important thing in the world, but for clarityâs sake, I think itâs worth distinguishing the following questions:
Does a specific textâThorstad (2022)âeither actually or apparently commit a kind of âbest model fallacyâ, arguing as though establishing Time of Perils hypothesis as unlikely to be true thereby suffices to undermine longtermism?
Does another specific textâmy âRule High Stakes In, Not Outââeither actually or apparently have as its âprimary argumentative move⊠to assign nontrivial probabilities without substantial new evidenceâ?
My linked post suggests that the answer to Q1 is âYesâ. I find it weird that others in the comments here are taking stands on this textual dispute a priori, rather than by engaging with the specifics of the text in question, the quotes I respond to, etc.
My primary complaint in this comment thread has simply been that the answer to Q2 is âNoâ (if you read my post, youâll see that itâs instead warning against what Iâm now calling the âbest model fallacyâ, and explaining how I think various other writingsâincluding Thorstadâsâseem to go awry as a result of not attending to this subtle point about model uncertainty). The point of my post is not to try to assert or argue for any particular probability assignment. Hence Thorstadâs current blog post misrepresents mine.
***
Thereâs a more substantial issue in the background:
Q3. What is the most reasonable prior probability estimate to assign to the time of perils hypothesis? In case of disagreement, does one party bear a special âburden of proofâ to convince the other, who should otherwise be regarded as better justified by default?
I have some general opinions about the probability being non-negligibleâI think Carl Shulman makes a good case hereâbut itâs not something Iâm trying to argue about with those who regard it as negligible. I donât feel like I have anything distinctive to contribute on that question at this time, and prefer to focus my arguments on more tractable points (like the point I was making about the best model fallacy). I independently think Thorstad is wrong about how the burden of proof applies, but thatâs an argument for another day.
So I agree that there is some âtalking pastâ happening here. Specifically, Thorstad seems to have read my post as addressing a different question (and advancing a different argument) than what it actually does, and made unwarranted epistemic charges on that basis. If anyone thinks my âRule High Stakes Inâ post similarly misrepresents Thorstad (2022), theyâre welcome to make the case in the comments to that post.
This comment deserves some kind of EA Forum award. My goodness, I envy and admire the breezy style with which you wrote this. I wish we had some analogue of Reddit gold, an award in limited supply, we could use to recognize exceptional contributions like this.
I agree with David Mathers that itâs simply psychologically implausible that David Thorstad, a sharp professional philosopher and an expert on bounded rationality, existential risk, longtermism, and effective altruism, doesnât understand the concept of expected value. I think we need to jettison such accusations, which have more of personal insult about them than substantive argument. Such accusations, besides just being ridiculous on their face, are corrosive to productive, charitable discussion about substantive disagreements on important topics.
Itâs not a psychological question. I wrote a blog post offering a philosophical critique of some published academic papers that, it seemed to me, involved an interesting and important error of reasoning. Anyone who thinks my critique goes awry is welcome to comment on it there. But whether my philosophical critique is ultimately correct or not, I donât think that the attempt is aptly described as âpersonal insultâ, âridiculous on [its] faceâ, or âcorrosive to productive, charitable discussionâ. Itâs literally just doing philosophy.
Iâd like it if people read my linked post before passing judgment on it.
Writing is done for an audience. Effective altruists have a very particular practice of stating their personal credences in the hypotheses that they discuss. While this is not my practice, in writing for effective altruists I try to be as precise as I can about the relative plausibility that I assign to various hypotheses and the effect that this might have on their expected value.
When writing for academic audiences, I do not discuss uncertainty unless I have something to add which my audience will find to be novel and adequately supported.
I donât remind academic readers that uncertainty matters, because all of them know that on many moral theories uncertainty matters and many (but not all) accept such theories. I donât remind academic readers of how uncertainty matters on some popular approaches, such as expected value theory, because all of my readers know this and many (but fewer) accept such theories. The most likely result of invoking expected value theory would be to provoke protests that I am situating my argument within a framework which some of my readers do not accept, and that would be a distraction.
I donât state my personal probability assignments to claims such as the time of perils hypothesis because I donât take myself to have given adequate grounds for a probability assignment. Readers would rightly object that my subjective probability assignments had not been adequately supported by the arguments in the paper, and I would be forced to remove them by referees, if the paper were not rejected out of hand.
For the same reason, I donât use language forcing my personal probability assignments on readers. There are always more arguments to consider, and readers differ quite dramatically in their priors. For that reason, concluding a paper with the conclusion that a claim like the time of perils hypothesis has a probability on the order of 10^(-100) or 10^(-200) would again, rightly provoke the objection that this claim has not been adequately supported.
When I write, for example, that arguments for the time of perils hypothesis are inconclusive, my intention is to allow readers to make up their own minds as to precisely how poorly those arguments fare and what the resulting probability assignments should be. Academic readers very much dislike being told what to think, and they donât care a whit for what I think.
As a data point, almost all of my readers are substantially less confident in many of the claims that I criticize than I am. The most common reason why my papers criticizing effective altruism are rejected from leading journals is that referees or editors take the positions criticized to be so poor that they do not warrant comment. (For example, my paper on power-seeking theorems was rejected from BJPS by an editor who wrote, âThe arguments critically evaluated in the paper are just all over the place, verging from silly napkin-math, to speculative metaphysics, to formal explorations of reinforcement learning agents. A small minority of that could be considered philosophy of computer science, but the rest of it, in my view, is computer scientists verging into bad philosophy of mind and futurism ⊠The targets of this criticism definitely want to pretend theyâre doing science; I worry that publishing a critical takedown of these arguments could lend legitimacy to that appearance.â)
Against this background, there is not much pressure to remind readers that the positions in question could be highly improbable. Most think this already, and the only thing I am likely to do is to provoke quick rejections like the above, or to annoy the inevitable referee (an outlier among my readers) selected for their sympathies with the position being criticized.
To tell the truth, I often try to be even more noncommittal in the language of my papers than the published version would suggest. For example, the submitted draft of âMistakes in the moral mathematics of existential riskâ said in the introduction that âunder many assumptions, once these mistakes are corrected, the value of existential risk mitigation will be far from astronomical.â A referee complained that this was not strong enough, because (on their view) the only assumptions worth considering were those on which the value of existential risk mitigation is rendered extremely minimal. So I changed the wording to âUnder many assumptions, once these mistakes are corrected, short-termist interventions will be more valuable than long-termist interventions, even within models proposed by leading effective altruists.â Why did I discuss these assumptions, instead of a broader class of assumptions under which the value of existential risk mitigation is merely non-astronomical? Because thatâs what my audience wanted to talk about.
In general, I would encourage you to focus in your writing on the substantive descriptive and normative issues that divide you from your opponents. Anyone worth talking to understands how uncertainty works. The most interesting divisions are not elementary mistakes about multiplication, but substantive questions about probabilities, utilities, decision theories, and the like. You will make more significant contributions to the state of the discussion if you focus on identifying the most important claims that in fact divide you from your opponents and on giving extended arguments for those claims.
To invent and claim to resolve disagreements based on elementary fallacies is likely to have the effect of pushing away the few philosophers still genuinely willing to have substantive normative and descriptive conversations with effective altruists. We are not enthusiastic about trivialities.
To be fair to Richard, there is a difference between a) stating your own personal probability in time of perils and b) making clear that for long-termist arguments to fail solely because they rely on time of perils, you need it to have extremely low probability, not just low, at least if you accept the expected value theory and subjective probability estimates can legitimately be applied at all here, as you seemed to be doing for the sake of making an internal critique. I took it to be the latter that Richard was complaining your paper doesnât do.
How strong do you think your evidence is for most readers of philosophy papers think the claim that X-risk is currently high, but will go permanently very lowâ is extremely implausible? If you asked me to guess Iâd say most peopleâs reaction would be more like âIâve no idea how plausible this is, other than definitely quite unlikelyâ, which is very different, but I have no experience with reviewers here.
I am a bit -not necessarily entirely-skeptical of the âeveryone really knows EA work outside development and animal welfare is trashâ vibe of your post. I donât doubt a lot of people do think that in professional philosophy. But at the same time, Nick Bostrom is more highly cited than virtually any reviewer you will have encountered. Long-termist moral philosophy turns up in leading journals constantly. One of the people you critiqued in your very good paper attacking arguments for the singularity is Dave Chalmers, and you literally donât get more professionally distinguished in analytic philosophy than Dave. Your stuff criticizing long-termism seems to have made it into top journals too when I checked, which indicates there certainly are people who think it is not too silly to be worth refuting: https://ââwww.dthorstad.com/ââpapers
Hi David, Iâm afraid you might have gotten caught up in a tangent here! The main point of my comment was that your post criticizes me on the basis of a misrepresentation. You claim that my âprimary argumentative move is to assign nontrivial probabilities without substantial new evidence,â but actually thatâs false. Thatâs just not what my blog post was about.
In retrospect, I think my attempt to briefly summarize what my post was about was too breezy, and misled many into thinking that its point was trivial. But it really isnât. (In fact, Iâd say that my core point there about taking higher-order uncertainty into account is far more substantial and widely neglected than the ânaming gameâ fallacy that you discuss in the present post!) I mention in another comment how it applied to Schwitzgebelâs ânegligibility argumentâ against longtermism, for example, where he very explicitly relies on a single constant probability model in order to make his case. Failing to adequately take model uncertainty into account is a subtle and easily-overlooked mistake!
A lot of your comment here seems to misunderstand my criticism of your earlier paper. Iâm not objecting that you failed to share your personal probabilities. Iâm objecting that your paper gives the impression that longtermism is undermined so long as the time of perils hypothesis is judged to be likely false. But actually the key question is whether its probability is negligible. Your paper fails to make clear what the key question to assess is, and the point of my âRule High Stakes Inâ post is to explain why itâs really the question of negligibility that matters.
To keep discussions clean and clear, Iâd prefer to continue discussion of my other post over on that post rather than here. Again, my objection to this post is simply that it misrepresented me.
Obviously David, as a highly trained moral philosopher with years of engagement with EA understands how expected value works though. I think the dispute must really be about whether to assign time of perils very low credence. (A dispute where I would probably side with you if âvery lowâ is below say 1 in 10,000).
Thereâs âunderstandingâ in the weak sense of having the info tokened in a belief-box somewhere, and then thereâs understanding in the sense of never falling for tempting-but-fallacious inferences like those I discuss in my post.
Have you read the paper I was responding to? I really donât think itâs at all âobviousâ that all âhighly trained moral philosophersâ have internalized the point I make in my blog post (that was the whole point of my writing it!), and I offered textual support. For example, Thorstad wrote: âthe time of perils hypothesis is probably false. I conclude that existential risk pessimism may tell against the overwhelming importance of existential risk mitigation.â This is a strange thing to write if he recognized that merely being âprobably falseâ doesnât suffice to threaten the longtermist argument!
(Edited to add: the obvious reading is that heâs making precisely the sort of âbest model fallacyâ that I critique in my post: assessing which empirical model we should regard as true, and then determining expected value on the basis of that one model. Even very senior philosophers, like Eric Schwitzgebel, have made the same mistake.)
Going back to the OPâs claims about what is or isnât âa good way to argue,â I think itâs important to pay attention to the actual text of what someone wrote. Thatâs what my blog post did, and itâs annoying to be subject to criticism (and now downvoting) from people who arenât willing to extend the same basic courtesy to me.
Fair point, when I re-checked the paper, it doesnât clearly and explicitly display knowledge of the point you are making. I still highly doubt that Thorstad really misunderstands it though. I think he was probably just not being super-careful.
This sort of âmany godsâ-style response is precisely what I was referring to with my parenthetical: âunless one inverts the high stakes in a way that cancels out the other high-stakes possibility.â
I think you are making some unstated assumptions that it would be helpful to make explicit. You say your argument is basically just explaining how expected values work, but it doesnât seem like that is true to me, I think you need to make some assumptions unrelated to how expected values work for your argument to go through.
If I were to cast your argument in the language of âhow expected values workâ it would go like this:
An expected value is the the sum of a bunch of terms that involve multiplying an outcome by its probability, so of the form x * p, where x is the outcome (usually represented by some some number) and p is the probability associated with that outcome. To get the EV we take terms like that representing every possible outcome and add them up.
Because these terms have two parts, the term as a whole can be large even if the probability is small. So, the overall EV can be driven primarily by a small probability of a large positive outcome because it is dominated by this one large term, which is large even when the probability is small. We rule high stakes in, not out.
The problem is that this argument doesnât work without further assumptions. In my version I said âcan be drivenâ. I think your conclusion requires âis drivenâ, which doesnât follow. Because there are other terms in the EV calculation their sum could be negative and of sufficient magnitude that the overall EV is low or negative even if one term is large and positive. This doesnât require that any particular term in the sum has any particular relationship to the large positive term such that it is âinvertingâ that term, although that would be sufficient, it isnât the only way for the overall EV to be small/ânegative. Their could be a mix of moderate negative terms that adds up to enough to reduce the overall EV. Nothing about this seems weird or controversial to me. For example, a standard normal distribution has large positive values with small probabilities but has an expectation of zero.
I think you need to be more explicit about the assumptions you are making that result in your desired conclusion. In my view, part of the point of Thorstadâs âmany godsâ response is that it demonstrates that once we start picking apart these assumptions we essentially collapse back to having the model the entire space of possibilities. That is suggested by what you say here:
I donât think that dystopian âtime of carolsâ scenarios are remotely as credible as the time of perils hypothesis.
The issues isnât that the âtime of carolsâ is super plausible, its that if your response is to include it as a term in the EV and argue the sum is still positive, then it seems like your original argument kind of collapses. We are no longer âruling stakes inâ. We now also have to actually add in all those other terms as well before we can know the final result.
I could imagine there are assumptions that might make your argument go through, but I think you need to make them explicit and argument for them, rather than claiming your conclusion follows from âhow expected value worksâ.
The responses to my comment have provided a real object lesson to me about how a rough throwaway remark (in this case: my attempt to very briefly indicate what my other post was about) can badly distract readers from oneâs actual point! Perhaps I would have done better to entirely leave out any positive attempt to here describe the content of my other post, and merely offer the negative claim that it wasnât about asserting specific probabilities.
My brief characterization was not especially well optimized for conveying the complex dialectic in the other post. Nor was it asserting that my conclusion was logically unassailable. I keep saying that if anyone wants to engage with my old post, Iâd prefer that they did so in the comments to that postâensuring that they engage with the real post rather than the inadequate summary I gave here. My ultra-brief summary is not an adequate substitute, and was never intended to be engaged with as such.
On the substantive point: Of course, ideally one would like to be able to âmodel the entire space of possibilitiesâ. But as finite creatures, we need heuristics. If you think my other post was offering a bad heuristic for approximating EV, Iâm happy to discuss that more over there.
I think you have be underestimating to what extent the responses you are getting do speak to the core content of your post, but I will leave a comment there to go into it more.
This sort of âmany godsâ-style response is precisely what I was referring to with my parenthetical: âunless one inverts the high stakes in a way that cancels out the other high-stakes possibility.â
I donât think that dystopian âtime of carolsâ scenarios are remotely as credible as the time of perils hypothesis. If someone disagrees, then certainly resolving that substantive disagreement would be important for making dialectical progress on the question of whether x-risk mitigation is worthwhile or not.
Iâd encourage Thorstad to read my post more carefully and pay attention to what I am arguing there. I was making an in principle point about how expected value works, highlighting a logical fallacy in Thorstadâs published work on this topic. (Nothing in the paper I responded to seemed to acknowledge that a 1% chance of the time of perils would suffice to support longtermism. He wrote about the hypothesis being âinconclusiveâ as if that sufficed to rule it out, and I think itâs important to recognize that this is bad reasoning on his part.)
Saying that my âprimary argumentative move is to assign nontrivial probabilities without substantial new evidenceâ is poor reading comprehension on Thorstadâs part. Actually, my primary argumentative move was explaining how expected value works. The numbers are illustrative, and suffice for anyone who happens to share my priors (or something close enough). Obviously, Iâm not in that post trying to persuade someone who instead thinks the correct probability to assign is negligible. Thorstad is just radically misreading what my post is arguing.
(What makes this especially strange is that, iirc, the published paper of Thorstadâs to which I was replying did not itself argue that the correct probability to assign to the ToP hypothesis is negligible, but just that the case for the hypothesis is âinconclusiveâ. So it sounds like heâs now accusing me of poor epistemics because I failed to respond to a different paper than the one he actually wrote? Geez.)
Seems like you and the other David T are talking past each other tbh.
Above you reasonably argue the [facetious] âtime of carolsâ hypothesis is not remotely as credible as the time of perils hypothesis. But you also donât assign a specific credence to it, or provide an argument that the âtime of carolsâ is impossible or even <1%[1]
I donât think it would be fair to conclude from this that you donât understand how probability works, and I also donât think that it is reasonable to assume that the probability of the âtime of carolsâ should be assumed sufficiently nontrivial to warrant action in the absence of any specific credence attached to it. Indeed if someone responded to you indirectly with an example which assigned a prior of âjust 1%â, to the âtime of carolsâ, you might feel justified in assuming it was them misunderstanding probability...
The rest of Thorstadâs post which doesnât seem to be specifically targeted at you explicitly argues that in practice, specific claims involving navigating a âtime of perilsâ also fall into the âtrivialâ category,[2] in the absence of robust argument as to why of all the possible futures this one is less trivial than others. Heâs not arguing for âmany godsâ which invert the stakes so much as âmany gods/âpantheons means the possibility of any specific god is trivial, in the absence of compelling evidence of said godâs relative likelihoodâ. He also doesnât bring any evidence to the table (other than arguing that time of perils hypothesis involves claims about x-risk in different centuries which might be best understood as independent claims [3]) but his position is that this shouldnât be the scepticâs job...
(Personally Iâm not sure what said evidence would even look like, but for related reasons Iâm not writing papers on longtermism and am happy applying a very high discount rate to the far future)
I think everyone would agree that it is absurd (thatâs a problem with facetious examples)[4] but if the default is that logical possibilities are considered nontrivial until proven otherwise...
he doesnât state a personal threshold, but does imply many longtermist propositions dip below Montonâs 5 * 10^-16 once you start adding up the claims....
a more significant claim he fails to emphasize is that the relevant criteria for longtermist interventions isnât so much that the baseline hypothesis about peril distribution is [incidentally] true but the impact of a specific intervention at the margin has a sustained positive influence on it.
I tend to dislike facetious examples, but hey, this is a literature in which people talk about paperclip maximisers and try to understand AI moral reasoning capacity by asking LLMs variations on trolley problems...
I am far from sure that Thorstad is wrong that time of perils should be assigned ultra-low probability. (I do suspect he is wrong, but this stuff is extremely hard to assess.) But in my view there are multiple pretty obvious reasons why âtime of Carolsâ is a poor analogy to âtime of perilsâ:
âTime of carolsâ is just way more specific, in a bad way than time of perils. I know that there are indefinitely many ways time of carols could happen if you get really fine-grained, but it nonetheless, intuitively, there is in some sense way more significantly different paths âX-risk could briefly be high then very lowâ than âeveryone is physically tied up and made to listen to carolsâ. To me itâs like comparing âthere will be cars on Mars in 2120âł to âthere will be a humanoid crate-stacking robot on Mars in 2120 that is nicknamed Carolâ.
Actually, longtermists argue for the âcurrent X-risk is highâ claim, making Thorstadâs point that lots of things should get ultra-low prior probability is not particularly relevant to that half of the time of perils hypothesis. In comparison, no one argues for time of carols.
(Most important disanalogy in my view.) The second half of time of perils, that x-risk will go very low for a long-time, is plausibly something that many people will consider desirable, and might therefore aim for. People are even more likely to aim for related goals like ânot have massive disasters while I am alive.â This is plausibly a pretty stable feature of human motivation that has a fair chance of lasting millions of years; humans generally donât want humans to die. In comparison thereâs little reason to think decent numbers of people will always desire time of carols.
4. Maybe this isnât an independent point from 1., but I actually do think it is relevant that âtime of carolsâ just seems very silly to everyone as soon as they hear it, and time of perils does not. I think we should give some weight to peopleâs gut reactions here.
The meta-dispute here isnât the most important thing in the world, but for clarityâs sake, I think itâs worth distinguishing the following questions:
Does a specific textâThorstad (2022)âeither actually or apparently commit a kind of âbest model fallacyâ, arguing as though establishing Time of Perils hypothesis as unlikely to be true thereby suffices to undermine longtermism?
Does another specific textâmy âRule High Stakes In, Not Outââeither actually or apparently have as its âprimary argumentative move⊠to assign nontrivial probabilities without substantial new evidenceâ?
My linked post suggests that the answer to Q1 is âYesâ. I find it weird that others in the comments here are taking stands on this textual dispute a priori, rather than by engaging with the specifics of the text in question, the quotes I respond to, etc.
My primary complaint in this comment thread has simply been that the answer to Q2 is âNoâ (if you read my post, youâll see that itâs instead warning against what Iâm now calling the âbest model fallacyâ, and explaining how I think various other writingsâincluding Thorstadâsâseem to go awry as a result of not attending to this subtle point about model uncertainty). The point of my post is not to try to assert or argue for any particular probability assignment. Hence Thorstadâs current blog post misrepresents mine.
***
Thereâs a more substantial issue in the background:
Q3. What is the most reasonable prior probability estimate to assign to the time of perils hypothesis? In case of disagreement, does one party bear a special âburden of proofâ to convince the other, who should otherwise be regarded as better justified by default?
I have some general opinions about the probability being non-negligibleâI think Carl Shulman makes a good case hereâbut itâs not something Iâm trying to argue about with those who regard it as negligible. I donât feel like I have anything distinctive to contribute on that question at this time, and prefer to focus my arguments on more tractable points (like the point I was making about the best model fallacy). I independently think Thorstad is wrong about how the burden of proof applies, but thatâs an argument for another day.
So I agree that there is some âtalking pastâ happening here. Specifically, Thorstad seems to have read my post as addressing a different question (and advancing a different argument) than what it actually does, and made unwarranted epistemic charges on that basis. If anyone thinks my âRule High Stakes Inâ post similarly misrepresents Thorstad (2022), theyâre welcome to make the case in the comments to that post.
This comment deserves some kind of EA Forum award. My goodness, I envy and admire the breezy style with which you wrote this. I wish we had some analogue of Reddit gold, an award in limited supply, we could use to recognize exceptional contributions like this.
I agree with David Mathers that itâs simply psychologically implausible that David Thorstad, a sharp professional philosopher and an expert on bounded rationality, existential risk, longtermism, and effective altruism, doesnât understand the concept of expected value. I think we need to jettison such accusations, which have more of personal insult about them than substantive argument. Such accusations, besides just being ridiculous on their face, are corrosive to productive, charitable discussion about substantive disagreements on important topics.
Itâs not a psychological question. I wrote a blog post offering a philosophical critique of some published academic papers that, it seemed to me, involved an interesting and important error of reasoning. Anyone who thinks my critique goes awry is welcome to comment on it there. But whether my philosophical critique is ultimately correct or not, I donât think that the attempt is aptly described as âpersonal insultâ, âridiculous on [its] faceâ, or âcorrosive to productive, charitable discussionâ. Itâs literally just doing philosophy.
Iâd like it if people read my linked post before passing judgment on it.
Thanks Richard!
Writing is done for an audience. Effective altruists have a very particular practice of stating their personal credences in the hypotheses that they discuss. While this is not my practice, in writing for effective altruists I try to be as precise as I can about the relative plausibility that I assign to various hypotheses and the effect that this might have on their expected value.
When writing for academic audiences, I do not discuss uncertainty unless I have something to add which my audience will find to be novel and adequately supported.
I donât remind academic readers that uncertainty matters, because all of them know that on many moral theories uncertainty matters and many (but not all) accept such theories. I donât remind academic readers of how uncertainty matters on some popular approaches, such as expected value theory, because all of my readers know this and many (but fewer) accept such theories. The most likely result of invoking expected value theory would be to provoke protests that I am situating my argument within a framework which some of my readers do not accept, and that would be a distraction.
I donât state my personal probability assignments to claims such as the time of perils hypothesis because I donât take myself to have given adequate grounds for a probability assignment. Readers would rightly object that my subjective probability assignments had not been adequately supported by the arguments in the paper, and I would be forced to remove them by referees, if the paper were not rejected out of hand.
For the same reason, I donât use language forcing my personal probability assignments on readers. There are always more arguments to consider, and readers differ quite dramatically in their priors. For that reason, concluding a paper with the conclusion that a claim like the time of perils hypothesis has a probability on the order of 10^(-100) or 10^(-200) would again, rightly provoke the objection that this claim has not been adequately supported.
When I write, for example, that arguments for the time of perils hypothesis are inconclusive, my intention is to allow readers to make up their own minds as to precisely how poorly those arguments fare and what the resulting probability assignments should be. Academic readers very much dislike being told what to think, and they donât care a whit for what I think.
As a data point, almost all of my readers are substantially less confident in many of the claims that I criticize than I am. The most common reason why my papers criticizing effective altruism are rejected from leading journals is that referees or editors take the positions criticized to be so poor that they do not warrant comment. (For example, my paper on power-seeking theorems was rejected from BJPS by an editor who wrote, âThe arguments critically evaluated in the paper are just all over the place, verging from silly napkin-math, to speculative metaphysics, to formal explorations of reinforcement learning agents. A small minority of that could be considered philosophy of computer science, but the rest of it, in my view, is computer scientists verging into bad philosophy of mind and futurism ⊠The targets of this criticism definitely want to pretend theyâre doing science; I worry that publishing a critical takedown of these arguments could lend legitimacy to that appearance.â)
Against this background, there is not much pressure to remind readers that the positions in question could be highly improbable. Most think this already, and the only thing I am likely to do is to provoke quick rejections like the above, or to annoy the inevitable referee (an outlier among my readers) selected for their sympathies with the position being criticized.
To tell the truth, I often try to be even more noncommittal in the language of my papers than the published version would suggest. For example, the submitted draft of âMistakes in the moral mathematics of existential riskâ said in the introduction that âunder many assumptions, once these mistakes are corrected, the value of existential risk mitigation will be far from astronomical.â A referee complained that this was not strong enough, because (on their view) the only assumptions worth considering were those on which the value of existential risk mitigation is rendered extremely minimal. So I changed the wording to âUnder many assumptions, once these mistakes are corrected, short-termist interventions will be more valuable than long-termist interventions, even within models proposed by leading effective altruists.â Why did I discuss these assumptions, instead of a broader class of assumptions under which the value of existential risk mitigation is merely non-astronomical? Because thatâs what my audience wanted to talk about.
In general, I would encourage you to focus in your writing on the substantive descriptive and normative issues that divide you from your opponents. Anyone worth talking to understands how uncertainty works. The most interesting divisions are not elementary mistakes about multiplication, but substantive questions about probabilities, utilities, decision theories, and the like. You will make more significant contributions to the state of the discussion if you focus on identifying the most important claims that in fact divide you from your opponents and on giving extended arguments for those claims.
To invent and claim to resolve disagreements based on elementary fallacies is likely to have the effect of pushing away the few philosophers still genuinely willing to have substantive normative and descriptive conversations with effective altruists. We are not enthusiastic about trivialities.
To be fair to Richard, there is a difference between a) stating your own personal probability in time of perils and b) making clear that for long-termist arguments to fail solely because they rely on time of perils, you need it to have extremely low probability, not just low, at least if you accept the expected value theory and subjective probability estimates can legitimately be applied at all here, as you seemed to be doing for the sake of making an internal critique. I took it to be the latter that Richard was complaining your paper doesnât do.
How strong do you think your evidence is for most readers of philosophy papers think the claim that X-risk is currently high, but will go permanently very lowâ is extremely implausible? If you asked me to guess Iâd say most peopleâs reaction would be more like âIâve no idea how plausible this is, other than definitely quite unlikelyâ, which is very different, but I have no experience with reviewers here.
I am a bit -not necessarily entirely-skeptical of the âeveryone really knows EA work outside development and animal welfare is trashâ vibe of your post. I donât doubt a lot of people do think that in professional philosophy. But at the same time, Nick Bostrom is more highly cited than virtually any reviewer you will have encountered. Long-termist moral philosophy turns up in leading journals constantly. One of the people you critiqued in your very good paper attacking arguments for the singularity is Dave Chalmers, and you literally donât get more professionally distinguished in analytic philosophy than Dave. Your stuff criticizing long-termism seems to have made it into top journals too when I checked, which indicates there certainly are people who think it is not too silly to be worth refuting: https://ââwww.dthorstad.com/ââpapers
Hi David, Iâm afraid you might have gotten caught up in a tangent here! The main point of my comment was that your post criticizes me on the basis of a misrepresentation. You claim that my âprimary argumentative move is to assign nontrivial probabilities without substantial new evidence,â but actually thatâs false. Thatâs just not what my blog post was about.
In retrospect, I think my attempt to briefly summarize what my post was about was too breezy, and misled many into thinking that its point was trivial. But it really isnât. (In fact, Iâd say that my core point there about taking higher-order uncertainty into account is far more substantial and widely neglected than the ânaming gameâ fallacy that you discuss in the present post!) I mention in another comment how it applied to Schwitzgebelâs ânegligibility argumentâ against longtermism, for example, where he very explicitly relies on a single constant probability model in order to make his case. Failing to adequately take model uncertainty into account is a subtle and easily-overlooked mistake!
A lot of your comment here seems to misunderstand my criticism of your earlier paper. Iâm not objecting that you failed to share your personal probabilities. Iâm objecting that your paper gives the impression that longtermism is undermined so long as the time of perils hypothesis is judged to be likely false. But actually the key question is whether its probability is negligible. Your paper fails to make clear what the key question to assess is, and the point of my âRule High Stakes Inâ post is to explain why itâs really the question of negligibility that matters.
To keep discussions clean and clear, Iâd prefer to continue discussion of my other post over on that post rather than here. Again, my objection to this post is simply that it misrepresented me.
Obviously David, as a highly trained moral philosopher with years of engagement with EA understands how expected value works though. I think the dispute must really be about whether to assign time of perils very low credence. (A dispute where I would probably side with you if âvery lowâ is below say 1 in 10,000).
Thereâs âunderstandingâ in the weak sense of having the info tokened in a belief-box somewhere, and then thereâs understanding in the sense of never falling for tempting-but-fallacious inferences like those I discuss in my post.
Have you read the paper I was responding to? I really donât think itâs at all âobviousâ that all âhighly trained moral philosophersâ have internalized the point I make in my blog post (that was the whole point of my writing it!), and I offered textual support. For example, Thorstad wrote: âthe time of perils hypothesis is probably false. I conclude that existential risk pessimism may tell against the overwhelming importance of existential risk mitigation.â This is a strange thing to write if he recognized that merely being âprobably falseâ doesnât suffice to threaten the longtermist argument!
(Edited to add: the obvious reading is that heâs making precisely the sort of âbest model fallacyâ that I critique in my post: assessing which empirical model we should regard as true, and then determining expected value on the basis of that one model. Even very senior philosophers, like Eric Schwitzgebel, have made the same mistake.)
Going back to the OPâs claims about what is or isnât âa good way to argue,â I think itâs important to pay attention to the actual text of what someone wrote. Thatâs what my blog post did, and itâs annoying to be subject to criticism (and now downvoting) from people who arenât willing to extend the same basic courtesy to me.
Fair point, when I re-checked the paper, it doesnât clearly and explicitly display knowledge of the point you are making. I still highly doubt that Thorstad really misunderstands it though. I think he was probably just not being super-careful.
I think you are making some unstated assumptions that it would be helpful to make explicit. You say your argument is basically just explaining how expected values work, but it doesnât seem like that is true to me, I think you need to make some assumptions unrelated to how expected values work for your argument to go through.
If I were to cast your argument in the language of âhow expected values workâ it would go like this:
An expected value is the the sum of a bunch of terms that involve multiplying an outcome by its probability, so of the form x * p, where x is the outcome (usually represented by some some number) and p is the probability associated with that outcome. To get the EV we take terms like that representing every possible outcome and add them up.
Because these terms have two parts, the term as a whole can be large even if the probability is small. So, the overall EV can be driven primarily by a small probability of a large positive outcome because it is dominated by this one large term, which is large even when the probability is small. We rule high stakes in, not out.
The problem is that this argument doesnât work without further assumptions. In my version I said âcan be drivenâ. I think your conclusion requires âis drivenâ, which doesnât follow. Because there are other terms in the EV calculation their sum could be negative and of sufficient magnitude that the overall EV is low or negative even if one term is large and positive. This doesnât require that any particular term in the sum has any particular relationship to the large positive term such that it is âinvertingâ that term, although that would be sufficient, it isnât the only way for the overall EV to be small/ânegative. Their could be a mix of moderate negative terms that adds up to enough to reduce the overall EV. Nothing about this seems weird or controversial to me. For example, a standard normal distribution has large positive values with small probabilities but has an expectation of zero.
I think you need to be more explicit about the assumptions you are making that result in your desired conclusion. In my view, part of the point of Thorstadâs âmany godsâ response is that it demonstrates that once we start picking apart these assumptions we essentially collapse back to having the model the entire space of possibilities. That is suggested by what you say here:
The issues isnât that the âtime of carolsâ is super plausible, its that if your response is to include it as a term in the EV and argue the sum is still positive, then it seems like your original argument kind of collapses. We are no longer âruling stakes inâ. We now also have to actually add in all those other terms as well before we can know the final result.
I could imagine there are assumptions that might make your argument go through, but I think you need to make them explicit and argument for them, rather than claiming your conclusion follows from âhow expected value worksâ.
The responses to my comment have provided a real object lesson to me about how a rough throwaway remark (in this case: my attempt to very briefly indicate what my other post was about) can badly distract readers from oneâs actual point! Perhaps I would have done better to entirely leave out any positive attempt to here describe the content of my other post, and merely offer the negative claim that it wasnât about asserting specific probabilities.
My brief characterization was not especially well optimized for conveying the complex dialectic in the other post. Nor was it asserting that my conclusion was logically unassailable. I keep saying that if anyone wants to engage with my old post, Iâd prefer that they did so in the comments to that postâensuring that they engage with the real post rather than the inadequate summary I gave here. My ultra-brief summary is not an adequate substitute, and was never intended to be engaged with as such.
On the substantive point: Of course, ideally one would like to be able to âmodel the entire space of possibilitiesâ. But as finite creatures, we need heuristics. If you think my other post was offering a bad heuristic for approximating EV, Iâm happy to discuss that more over there.
I think you have be underestimating to what extent the responses you are getting do speak to the core content of your post, but I will leave a comment there to go into it more.