The model doesn’t directly use the 1% per year figure; rather, it says the mean of the probability distribution for solving the agenda is 18% overall. That seems pretty reasonable to me. One in a million, on the other hand, would be very wrong (even per-year), because it is many orders of magnitude below the base rates of researchers solving the questions to which they set themselves to solving. And clearly Paul does not feel as though they are only meeting one-millionth of their agenda per year, nor do I from what I have seen of MIRI’s research so far.
I see, I double checked the model. But obviously there are an infinite number of distributions that can be set for that assumption, and I don’t have any particular reason to believe the chance of MIRI solving its research agenda (in such a time as it could actually be used) is as high as to be modeled with a Beta(10, 45) distribution. In fact the problems I see identified in MIRI’s research agenda are extremely difficult, probably far more difficult than almost any Millennium Prize problem.
Considering solving the Poincare Conjecture problem took more than a century of effort with intense attention from the academic mathematical community (and was finally solved by a mathematics professor, not a “lone wolf” type), what reason do we have to believe that a small team of researchers could solve a set of probably far more difficult problems working more or less in a silo, outside of the purview of traditional academia? Has MIRI even produced any peer-reviewed scientific papers? I’d expect solving some of these problems to be about on par with the difficulty of solving P=NP, something I’d expect a small team to have something on the order of a 1 in 1 million chance of solving before anyone else given a refusal to work within the existing dominant form of mass intellectual collaboration (the system of academia). I could fit any distribution to that I wanted with at least as much plausibility as the estimates that went into this post, and get a far reduced overall value of MIRI with a point estimate somewhere around 1 in 1 million.
In fact, P=NP very plausibly might even be a prerequisite for a provably safe or friendly AI to even exist, or an explosion of computational power allowing every solution to actually be checked by brute force. Within certain domains there could be relatively simple software checks ton the output that ensure it is “safe” but whether that’s even possible and how it manifests depends on the particular application.
Also, using a skewed (i.e. alpha =/= beta) beta distribution is suspect to me. Intuitively I’d have little reason not to initially expect a symmetric distribution like a uniform distribution around the estimates.
The model doesn’t directly use the 1% per year figure; rather, it says the mean of the probability distribution for solving the agenda is 18% overall. That seems pretty reasonable to me. One in a million, on the other hand, would be very wrong (even per-year), because it is many orders of magnitude below the base rates of researchers solving the questions to which they set themselves to solving. And clearly Paul does not feel as though they are only meeting one-millionth of their agenda per year, nor do I from what I have seen of MIRI’s research so far.
I see, I double checked the model. But obviously there are an infinite number of distributions that can be set for that assumption, and I don’t have any particular reason to believe the chance of MIRI solving its research agenda (in such a time as it could actually be used) is as high as to be modeled with a Beta(10, 45) distribution. In fact the problems I see identified in MIRI’s research agenda are extremely difficult, probably far more difficult than almost any Millennium Prize problem.
Considering solving the Poincare Conjecture problem took more than a century of effort with intense attention from the academic mathematical community (and was finally solved by a mathematics professor, not a “lone wolf” type), what reason do we have to believe that a small team of researchers could solve a set of probably far more difficult problems working more or less in a silo, outside of the purview of traditional academia? Has MIRI even produced any peer-reviewed scientific papers? I’d expect solving some of these problems to be about on par with the difficulty of solving P=NP, something I’d expect a small team to have something on the order of a 1 in 1 million chance of solving before anyone else given a refusal to work within the existing dominant form of mass intellectual collaboration (the system of academia). I could fit any distribution to that I wanted with at least as much plausibility as the estimates that went into this post, and get a far reduced overall value of MIRI with a point estimate somewhere around 1 in 1 million.
In fact, P=NP very plausibly might even be a prerequisite for a provably safe or friendly AI to even exist, or an explosion of computational power allowing every solution to actually be checked by brute force. Within certain domains there could be relatively simple software checks ton the output that ensure it is “safe” but whether that’s even possible and how it manifests depends on the particular application.
Also, using a skewed (i.e. alpha =/= beta) beta distribution is suspect to me. Intuitively I’d have little reason not to initially expect a symmetric distribution like a uniform distribution around the estimates.