Are there any principled probability assignments we could use? E.g., the probability that this would be my top choice after N further hours of investigation into it and alternatives (including realistically collecting data or performing experiments), maybe allowing N to be unrealistic?
From my understanding, softmax is formally sensitive to parametrizations, so the specific outputs seem pretty unprincipled unless you actually have feedback and are doing some kind of optimization like minimizing some kind of softmax loss.
On the other hand, I can see even using the credences like I proposed to be way too upside-focused, i.e. focused on picking the best interventions, but little concern for avoiding the worst (badly net negative in EV) interventions. Consider an intervention that has a 55% chance of being the best and vastly net positive in expectation after further investigation, but a 45% chance of being the worst and vastly net negative in expectation (of similar magnitude), and your current overall belief is that itโs vastly net positive and highest in EV. Itโs plausible some high-leverage interventions are sensitive in this way, because they involve tradeoffs for existential risks (tradeoffs between different x-risks, but also within x-risks, like differential progress), or, in the near-term, because of wild animal effects dominating and having uncertain sign. Would we really want to put the largest share of resources, let alone most resources, into such an intervention?
Alternatively, we may have multiple choices, among which three, A, B and C are such that, for some c>0, after further investigation, we expect that:
A is 40% likely to be the best, with EV = c, and 35% likely to be the worst, with EV=-c, and and the rest of the time EV=0.
B is 35% likely to be the best, with EV=c and 30% likely to be the worst, with EV=-c, and the rest of the time EV=0.
C is 5% likely to be the best, with EV=c, and otherwise has EV=0 and has probability 0 of being the worst.
How should we weight our resources between these three (ignoring other options)? Currently, they all have the same overall EV (=5%*c). What if we increase the probability that A is best slightly, without changing anything else? Or, what if we increase the probability that C is best slightly, without changing anything else?
Are there any principled probability assignments we could use? E.g., the probability that this would be my top choice after N further hours of investigation into it and alternatives (including realistically collecting data or performing experiments), maybe allowing N to be unrealistic?
From my understanding, softmax is formally sensitive to parametrizations, so the specific outputs seem pretty unprincipled unless you actually have feedback and are doing some kind of optimization like minimizing some kind of softmax loss.
On the other hand, I can see even using the credences like I proposed to be way too upside-focused, i.e. focused on picking the best interventions, but little concern for avoiding the worst (badly net negative in EV) interventions. Consider an intervention that has a 55% chance of being the best and vastly net positive in expectation after further investigation, but a 45% chance of being the worst and vastly net negative in expectation (of similar magnitude), and your current overall belief is that itโs vastly net positive and highest in EV. Itโs plausible some high-leverage interventions are sensitive in this way, because they involve tradeoffs for existential risks (tradeoffs between different x-risks, but also within x-risks, like differential progress), or, in the near-term, because of wild animal effects dominating and having uncertain sign. Would we really want to put the largest share of resources, let alone most resources, into such an intervention?
Alternatively, we may have multiple choices, among which three, A, B and C are such that, for some c>0, after further investigation, we expect that:
A is 40% likely to be the best, with EV = c, and 35% likely to be the worst, with EV=-c, and and the rest of the time EV=0.
B is 35% likely to be the best, with EV=c and 30% likely to be the worst, with EV=-c, and the rest of the time EV=0.
C is 5% likely to be the best, with EV=c, and otherwise has EV=0 and has probability 0 of being the worst.
How should we weight our resources between these three (ignoring other options)? Currently, they all have the same overall EV (=5%*c). What if we increase the probability that A is best slightly, without changing anything else? Or, what if we increase the probability that C is best slightly, without changing anything else?