The problem (of worrying that you’re being silly and getting mugged) doesn’t arise when probabilities are tiny, it’s when probabilities are tiny and you’re highly uncertain. We have pretty good bounds in the three areas you listed, but I do not have good bounds on say, the odds that “spending the next year of my life on AI Safety research” will prevent x-risk.
In the former cases, we have base rates and many trials. In the latter case, I’m just doing a very rough fermi estimate. Say I have 5 parameters with an order of magnitude of uncertainty on each one, which when multiplied out, is just really horrendous.
Anyway, I mostly agree with what you’re saying, but it’s possible that you’re somewhat misunderstanding where the anxieties you’re responding to are coming from.
The problem (of worrying that you’re being silly and getting mugged) doesn’t arise when probabilities are tiny, it’s when probabilities are tiny and you’re highly uncertain. We have pretty good bounds in the three areas you listed, but I do not have good bounds on say, the odds that “spending the next year of my life on AI Safety research” will prevent x-risk.
In the former cases, we have base rates and many trials. In the latter case, I’m just doing a very rough fermi estimate. Say I have 5 parameters with an order of magnitude of uncertainty on each one, which when multiplied out, is just really horrendous.
Anyway, I mostly agree with what you’re saying, but it’s possible that you’re somewhat misunderstanding where the anxieties you’re responding to are coming from.