My understanding of why MIRI’s expected returns didn’t come out on top is that you have a strong prior against any org being able to do that much good, and because MIRI’s expect impact was so high variance (i.e uncertain), it didn’t cause your model to update in any particular direction very much.
What confuses me is this: It feels like if I hadn’t thought of astronomical waste / xrisk, and found a great org like AMF, hearing those arguments should make me update strongly that I’m looking at the wrong areas. Yet, the argument that it’s high potential cancels out with your prior means I could’ve been right the whole time, even before I took into account far future considerations.
Which seems implausible. The whole point of astronomical waste is that you should update your probability of being able to have an outsized impact.
I’m not sure which part of your model I’m disagreeing with, but would appreciate knowing if you do?
My understanding of why MIRI’s expected returns didn’t come out on top is that you have a strong prior against any org being able to do that much good, and because MIRI’s expect impact was so high variance (i.e uncertain), it didn’t cause your model to update in any particular direction very much.
What confuses me is this: It feels like if I hadn’t thought of astronomical waste / xrisk, and found a great org like AMF, hearing those arguments should make me update strongly that I’m looking at the wrong areas. Yet, the argument that it’s high potential cancels out with your prior means I could’ve been right the whole time, even before I took into account far future considerations.
Which seems implausible. The whole point of astronomical waste is that you should update your probability of being able to have an outsized impact.
I’m not sure which part of your model I’m disagreeing with, but would appreciate knowing if you do?