Hey Ben, I think these are pretty reasonable questions and do not make you look stupid.
On Pascal’s mugging in particular, I would consider this somewhat informal answer:
Though honestly, I don’t find this super satisfactory, and it is something and still bugs me.
Having said that, I don’t think this line of reasoning is necessary for answering your more practical questions 1-3.
Utilitarianism (and Effective Altruism) don’t require that there’s some specific metaphysical construct that is numerical and corresponds to human happiness. The utilitarian claim is just that some degree of quantification is, in principle, possible. The EA claim is that attempting to carry out this quantification leads to good outcomes, even if it’s not an exact science.
GiveWell painstakingly compiles cost-effectiveness estimates numerical, but goes on to state that they don’t view these as being “literally true”. These estimates still end up being useful for comparing one charity relative to another. You can read more about this thinking here:
In practice, GiveWell makes all sorts of tradeoffs to attempt to compare goods like “improving education”, “lives saved” or “increasing income”. Sometimes this involves directly asking the targeted populations about their preferences. You can read more about their approach here:
Finally, in the case of existential-risk, it’s often not necessary to make these kinds of specific calculations at all. By one estimate, the Earth alone could support something like 10^16 human lives, and the universe could support somewhere something like 10^34 human life-years, or up to 10^56 “cybernetic human life-years”. This is all very speculative, but the potential gains are so large that it doesn’t matter if we’re off by 40%, or 40x.
Returning to the original point, you might ask if work on x-risk is then a case of Pascal’s Mugging? Toby Ord gives the odds of human extinction in the next century at around 1⁄6. That’s a pretty huge chance. We’re much less confident what the odds are of EA preventing this risk, but it seems reasonable to think that it’s some normal number. I.e. much higher than 10^-10. In that case, EA has huge expected value. Of course that might all seem like fuzzy reasoning, but I think there’s a pretty good case to be made that our odds are not astronomically low. You can see one version of this argument here:
On Pascal’s mugging specifically, Robert Miles has an interesting youtube video arguing that AI Safety is not a Pascal mugging, which the OP might be interested in:
Thank you—this is all very interesting. I won’t try to reply to all of it, but just thought I would respond to agree on your last point. I think x-risk is worth caring about precisely because the probability seems to be in the “actually might happen” range. (I don’t believe at all that anyone knows it’s 1⁄6 vs. 1⁄10 or 1⁄2, but Toby Ord doesn’t claim to either does he?) It’s when you get to the “1 in a million but with a billion payoff” range I start to get skeptical, because then the thing in question actually just won’t happen, barring many plays of the game.