On the other hand, I can see even using the credences like I proposed to be way too upside-focused, i.e. focused on picking the best interventions, but little concern for avoiding the worst (badly net negative in EV) interventions. Consider an intervention that has a 55% chance of being the best and vastly net positive in expectation after further investigation, but a 45% chance of being the worst and vastly net negative in expectation (of similar magnitude), and your current overall belief is that it’s vastly net positive and highest in EV. It’s plausible some high-leverage interventions are sensitive in this way, because they involve tradeoffs for existential risks (tradeoffs between different x-risks, but also within x-risks, like differential progress), or, in the near-term, because of wild animal effects dominating and having uncertain sign. Would we really want to put the largest share of resources, let alone most resources, into such an intervention?
Alternatively, we may have multiple choices, among which three, A, B and C are such that, for some c>0, after further investigation, we expect that:
A is 40% likely to be the best, with EV = c, and 35% likely to be the worst, with EV=-c, and and the rest of the time EV=0.
B is 35% likely to be the best, with EV=c and 30% likely to be the worst, with EV=-c, and the rest of the time EV=0.
C is 5% likely to be the best, with EV=c, and otherwise has EV=0 and has probability 0 of being the worst.
How should we weight our resources between these three (ignoring other options)? Currently, they all have the same overall EV (=5%*c). What if we increase the probability that A is best slightly, without changing anything else? Or, what if we increase the probability that C is best slightly, without changing anything else?
On the other hand, I can see even using the credences like I proposed to be way too upside-focused, i.e. focused on picking the best interventions, but little concern for avoiding the worst (badly net negative in EV) interventions. Consider an intervention that has a 55% chance of being the best and vastly net positive in expectation after further investigation, but a 45% chance of being the worst and vastly net negative in expectation (of similar magnitude), and your current overall belief is that it’s vastly net positive and highest in EV. It’s plausible some high-leverage interventions are sensitive in this way, because they involve tradeoffs for existential risks (tradeoffs between different x-risks, but also within x-risks, like differential progress), or, in the near-term, because of wild animal effects dominating and having uncertain sign. Would we really want to put the largest share of resources, let alone most resources, into such an intervention?
Alternatively, we may have multiple choices, among which three, A, B and C are such that, for some c>0, after further investigation, we expect that:
A is 40% likely to be the best, with EV = c, and 35% likely to be the worst, with EV=-c, and and the rest of the time EV=0.
B is 35% likely to be the best, with EV=c and 30% likely to be the worst, with EV=-c, and the rest of the time EV=0.
C is 5% likely to be the best, with EV=c, and otherwise has EV=0 and has probability 0 of being the worst.
How should we weight our resources between these three (ignoring other options)? Currently, they all have the same overall EV (=5%*c). What if we increase the probability that A is best slightly, without changing anything else? Or, what if we increase the probability that C is best slightly, without changing anything else?