Can you explain your expected far future population size? It looks like your upper bound is something like 10 orders of magnitude lower than Bostrom’s most conservative estimates.
That disagreement makes all the other uncertainty look extremely trivial in comparison.
colonisation of the Supercluster could have a very low probability.
What do you mean by very low probability? If you mean a one in a million chance, that’s not improbable enough to answer Bostrom. If you mean something that would actually answer Bostrom, then please respond to the SlateStarCodex post Stop adding zeroes.
I think Bostrom is on the right track, and that any analysis which follows your approach should use at least a 0.1% chance of more than 10^50 human life-years.
Most of the point estimates seem far too optimistic. Is it really feasible (probability greater than 1 in one billion) that MIRI could really create some kind of provably safe AI framework AND have it implemented before Google, Microsoft, Amazon, the US government, the Russian government, and the Chinese government (or by these entities) instead of using their own?
How strictly do you mean when you say “provably safe”? That seems like an area where all AI safety researchers are hesitant to say how high they’re aiming.
And by “have it implemented”, do you mean fully develop it own their own, or do you include scenarios where they convey keys insights to Google, and thereby cause Google to do something safer?
One in a billion? Of course. Have you thought about how tiny one in a billion is? I’m confident in saying that the average user of this forum has a greater than one-in-a-billion chance of personally designing and implementing safe AI.
I don’t see that number in the model, though. You might want to make a copy of the model with your own estimates—or find something specific to dispute. You seem to be working backwards from a conclusion; it’s better to consider the input probabilities one step at a time.
Estimate 3, acknowledged to be “pretty made up,” is said to be 1% but I don’t see any reason why it couldn’t take any other particular value like say 1 in 1 million.
The model doesn’t directly use the 1% per year figure; rather, it says the mean of the probability distribution for solving the agenda is 18% overall. That seems pretty reasonable to me. One in a million, on the other hand, would be very wrong (even per-year), because it is many orders of magnitude below the base rates of researchers solving the questions to which they set themselves to solving. And clearly Paul does not feel as though they are only meeting one-millionth of their agenda per year, nor do I from what I have seen of MIRI’s research so far.
I see, I double checked the model. But obviously there are an infinite number of distributions that can be set for that assumption, and I don’t have any particular reason to believe the chance of MIRI solving its research agenda (in such a time as it could actually be used) is as high as to be modeled with a Beta(10, 45) distribution. In fact the problems I see identified in MIRI’s research agenda are extremely difficult, probably far more difficult than almost any Millennium Prize problem.
Considering solving the Poincare Conjecture problem took more than a century of effort with intense attention from the academic mathematical community (and was finally solved by a mathematics professor, not a “lone wolf” type), what reason do we have to believe that a small team of researchers could solve a set of probably far more difficult problems working more or less in a silo, outside of the purview of traditional academia? Has MIRI even produced any peer-reviewed scientific papers? I’d expect solving some of these problems to be about on par with the difficulty of solving P=NP, something I’d expect a small team to have something on the order of a 1 in 1 million chance of solving before anyone else given a refusal to work within the existing dominant form of mass intellectual collaboration (the system of academia). I could fit any distribution to that I wanted with at least as much plausibility as the estimates that went into this post, and get a far reduced overall value of MIRI with a point estimate somewhere around 1 in 1 million.
In fact, P=NP very plausibly might even be a prerequisite for a provably safe or friendly AI to even exist, or an explosion of computational power allowing every solution to actually be checked by brute force. Within certain domains there could be relatively simple software checks ton the output that ensure it is “safe” but whether that’s even possible and how it manifests depends on the particular application.
Also, using a skewed (i.e. alpha =/= beta) beta distribution is suspect to me. Intuitively I’d have little reason not to initially expect a symmetric distribution like a uniform distribution around the estimates.
Have there been any models of other activities that might reduce existential risk.
E.g. convincing prospective AGI researchers that it is dangerous and should be handled carefully? It would seem that that might increase the pool of potential researchers and also give more time for a safe approach to be developed?
Can you explain your expected far future population size? It looks like your upper bound is something like 10 orders of magnitude lower than Bostrom’s most conservative estimates.
That disagreement makes all the other uncertainty look extremely trivial in comparison.
What do you mean by very low probability? If you mean a one in a million chance, that’s not improbable enough to answer Bostrom. If you mean something that would actually answer Bostrom, then please respond to the SlateStarCodex post Stop adding zeroes.
I think Bostrom is on the right track, and that any analysis which follows your approach should use at least a 0.1% chance of more than 10^50 human life-years.
Most of the point estimates seem far too optimistic. Is it really feasible (probability greater than 1 in one billion) that MIRI could really create some kind of provably safe AI framework AND have it implemented before Google, Microsoft, Amazon, the US government, the Russian government, and the Chinese government (or by these entities) instead of using their own?
I’m a little unclear on what you are asking.
How strictly do you mean when you say “provably safe”? That seems like an area where all AI safety researchers are hesitant to say how high they’re aiming.
And by “have it implemented”, do you mean fully develop it own their own, or do you include scenarios where they convey keys insights to Google, and thereby cause Google to do something safer?
One in a billion? Of course. Have you thought about how tiny one in a billion is? I’m confident in saying that the average user of this forum has a greater than one-in-a-billion chance of personally designing and implementing safe AI.
I don’t see that number in the model, though. You might want to make a copy of the model with your own estimates—or find something specific to dispute. You seem to be working backwards from a conclusion; it’s better to consider the input probabilities one step at a time.
Estimate 3, acknowledged to be “pretty made up,” is said to be 1% but I don’t see any reason why it couldn’t take any other particular value like say 1 in 1 million.
The model doesn’t directly use the 1% per year figure; rather, it says the mean of the probability distribution for solving the agenda is 18% overall. That seems pretty reasonable to me. One in a million, on the other hand, would be very wrong (even per-year), because it is many orders of magnitude below the base rates of researchers solving the questions to which they set themselves to solving. And clearly Paul does not feel as though they are only meeting one-millionth of their agenda per year, nor do I from what I have seen of MIRI’s research so far.
I see, I double checked the model. But obviously there are an infinite number of distributions that can be set for that assumption, and I don’t have any particular reason to believe the chance of MIRI solving its research agenda (in such a time as it could actually be used) is as high as to be modeled with a Beta(10, 45) distribution. In fact the problems I see identified in MIRI’s research agenda are extremely difficult, probably far more difficult than almost any Millennium Prize problem.
Considering solving the Poincare Conjecture problem took more than a century of effort with intense attention from the academic mathematical community (and was finally solved by a mathematics professor, not a “lone wolf” type), what reason do we have to believe that a small team of researchers could solve a set of probably far more difficult problems working more or less in a silo, outside of the purview of traditional academia? Has MIRI even produced any peer-reviewed scientific papers? I’d expect solving some of these problems to be about on par with the difficulty of solving P=NP, something I’d expect a small team to have something on the order of a 1 in 1 million chance of solving before anyone else given a refusal to work within the existing dominant form of mass intellectual collaboration (the system of academia). I could fit any distribution to that I wanted with at least as much plausibility as the estimates that went into this post, and get a far reduced overall value of MIRI with a point estimate somewhere around 1 in 1 million.
In fact, P=NP very plausibly might even be a prerequisite for a provably safe or friendly AI to even exist, or an explosion of computational power allowing every solution to actually be checked by brute force. Within certain domains there could be relatively simple software checks ton the output that ensure it is “safe” but whether that’s even possible and how it manifests depends on the particular application.
Also, using a skewed (i.e. alpha =/= beta) beta distribution is suspect to me. Intuitively I’d have little reason not to initially expect a symmetric distribution like a uniform distribution around the estimates.
Have there been any models of other activities that might reduce existential risk.
E.g. convincing prospective AGI researchers that it is dangerous and should be handled carefully? It would seem that that might increase the pool of potential researchers and also give more time for a safe approach to be developed?