Anyone feel free to correct me, but I believe in such a scenario spreading your prior evenly over all possible outcomes wouldn’t arbitrarily just include spitting the difference between 10^-15 or 10^-50 but spreading your belief over all positive outcomes below some reasonable barrier and (potentially) above another* (and this isn’t taking into account the non-zero, even if unlikely, probability that despite caution AI research is indeed speeding up our doom).
It’s complicated, but I don’t think it makes sense to have a probability distribution over probability distributions, because it collapses. We should just have a probability distribution over outcomes. We choose our prior estimate for chance of success based on other cases of people attempting to make safer tech.
Despite what appeared to him to be this large uncertainty, he seemed to encounter many people who brushed aside, or seemingly belittled, all other possible cause areas and this rubbed him the wrong way.
In fairness, for people who adhere to expected value thinking to the fullest extent (some of whom would have turned out to the conference), arguments purely on the basis of scope of potential impact would be persuasive. But if it’s even annoying folks at EA Global, then probably people ought to stop using them.
It’s complicated, but I don’t think it makes sense to have a probability distribution over probability distributions, because it collapses. We should just have a probability distribution over outcomes.
I did mean over outcomes. I was referring to this:
If we’re uncertain about Matthews propositions, we ought to place our guesses somewhere closer to 50%. To do otherwise would be to mistake our deep uncertainty deep scepticism.
That seems mistaken to me but it could be because I’m misinterpreting it. I was reading it as saying we should split the difference between the two probabilities of success Matthews proposed. However, I thought he was suggesting, and believe it is correct, that we shouldn’t just pick the median between the two because the smaller number was just an example. His real point being that any tiny probability of success seems equally as reasonable from the vantage point of now. If true we’d then have to split our prior evenly over that range instead of picking the median between 10^-15 and 10^-50. And given it’s very difficult to put a lower bound on the reasonable range but a $1000 donation being a good investment depends on a specific lower bound higher than he believes can be justified with evidence, some people came across as unduly confident.
But if it’s even annoying folks at EA Global, then probably people ought to stop using them.
Let me be very clear, I was not annoyed by them, even if I disagree, but people definitely used this reasoning. However, as I often point out, extrapolating from me to other humans is not a good idea even within the EA community.
It’s complicated, but I don’t think it makes sense to have a probability distribution over probability distributions, because it collapses. We should just have a probability distribution over outcomes. We choose our prior estimate for chance of success based on other cases of people attempting to make safer tech.
In fairness, for people who adhere to expected value thinking to the fullest extent (some of whom would have turned out to the conference), arguments purely on the basis of scope of potential impact would be persuasive. But if it’s even annoying folks at EA Global, then probably people ought to stop using them.
I did mean over outcomes. I was referring to this:
That seems mistaken to me but it could be because I’m misinterpreting it. I was reading it as saying we should split the difference between the two probabilities of success Matthews proposed. However, I thought he was suggesting, and believe it is correct, that we shouldn’t just pick the median between the two because the smaller number was just an example. His real point being that any tiny probability of success seems equally as reasonable from the vantage point of now. If true we’d then have to split our prior evenly over that range instead of picking the median between 10^-15 and 10^-50. And given it’s very difficult to put a lower bound on the reasonable range but a $1000 donation being a good investment depends on a specific lower bound higher than he believes can be justified with evidence, some people came across as unduly confident.
Let me be very clear, I was not annoyed by them, even if I disagree, but people definitely used this reasoning. However, as I often point out, extrapolating from me to other humans is not a good idea even within the EA community.