I think your first priority is promising and seemingly neglected (though I’m not familiar with a lot of work done by governance folk, so I could be wrong here). I also get the impression that MIRI folk believe they have an unusually clear understanding of risks, would like to see risky development slow down and are pessimistic about their near-term prospects for solving technical problems of aligning very capable intelligent systems and generally don’t see any clearly good next steps. It appears to me that this combination of skills and views positions them relatively well for developing AI safety standards. I’d be shocked if you didn’t end up talking to MIRI about this issue, but I just wanted to point out that from my point of view there seems to be a substantial amount of fit here.
are pessimistic about their near-term prospects for solving technical problems of aligning very capable intelligent systems and generally don’t see any clearly good next steps
I don’t think they claim to have better longer-term prospects, though.
I’m not sold on how well calibrated their predictions of catastrophe are, but I think they have contributed a large number of novel & important ideas to the field.
I don’t think they would claim to have significantly better predictive models in a positive sense, they just have far stronger models of what isn’t possible and cannot work for ASI, and it constrains their expectations about the long term far more. (I’m not sure I agree with, say, Eliezer about his view of uselessness of governance, for example—but he has a very clear model, which is unusual.) I also don’t think their view about timelines or takeoff speeds is really a crux—they have claimed that even if ASI is decades away, we still can’t rely on current approaches to scale.
I think your first priority is promising and seemingly neglected (though I’m not familiar with a lot of work done by governance folk, so I could be wrong here). I also get the impression that MIRI folk believe they have an unusually clear understanding of risks, would like to see risky development slow down and are pessimistic about their near-term prospects for solving technical problems of aligning very capable intelligent systems and generally don’t see any clearly good next steps. It appears to me that this combination of skills and views positions them relatively well for developing AI safety standards. I’d be shocked if you didn’t end up talking to MIRI about this issue, but I just wanted to point out that from my point of view there seems to be a substantial amount of fit here.
I don’t think they claim to have better longer-term prospects, though.
I think they do? Nate at least says he’s optimistic about finding a solution given more time
“Believe” being the operative word here. I really don’t think they do.
I’m not sold on how well calibrated their predictions of catastrophe are, but I think they have contributed a large number of novel & important ideas to the field.
I don’t think they would claim to have significantly better predictive models in a positive sense, they just have far stronger models of what isn’t possible and cannot work for ASI, and it constrains their expectations about the long term far more. (I’m not sure I agree with, say, Eliezer about his view of uselessness of governance, for example—but he has a very clear model, which is unusual.) I also don’t think their view about timelines or takeoff speeds is really a crux—they have claimed that even if ASI is decades away, we still can’t rely on current approaches to scale.