This post is really approximate and lightly sketched, but at least it says so. Overall I think the numbers are wrong and the argument is sound.
Synthesising responses:
Industry is going to be a bigger player in safety, just as it’s a bigger player in capabilities.
My model could be extremely useful if anyone could replace the headcount with any proxy of productivity on the real problem. Any proxy at all.
Doing the bottom up model was one of the most informative parts for me. You can cradle the whole field in the palm of your mind. It is a small and precious thing. At the same time, many organisations you would assume are doing lots of AGI existential risk reduction, are not.
Half of the juice of this post is in the caveats and the caveat overflow gdoc.
I continue to be confused by how little attention people pay to the agendas, in particular the lovely friendly CHAI bibliography.
Todd notes that you could say the same about most causes, everything is in fact connected. If this degree of indirect effect was uniform, then the ranking of causes would be unchanged. But there’s something very important about the absolute level, and not just for timelines and replaceability. Safety people need gears, and “is a giant wave of thousands of smart people going to come?” is a big gear.
This post is really approximate and lightly sketched, but at least it says so. Overall I think the numbers are wrong and the argument is sound.
Synthesising responses:
Industry is going to be a bigger player in safety, just as it’s a bigger player in capabilities.
My model could be extremely useful if anyone could replace the headcount with any proxy of productivity on the real problem. Any proxy at all.
Doing the bottom up model was one of the most informative parts for me. You can cradle the whole field in the palm of your mind. It is a small and precious thing. At the same time, many organisations you would assume are doing lots of AGI existential risk reduction, are not.
Half of the juice of this post is in the caveats and the caveat overflow gdoc.
I continue to be confused by how little attention people pay to the agendas, in particular the lovely friendly CHAI bibliography.
Todd notes that you could say the same about most causes, everything is in fact connected. If this degree of indirect effect was uniform, then the ranking of causes would be unchanged. But there’s something very important about the absolute level, and not just for timelines and replaceability. Safety people need gears, and “is a giant wave of thousands of smart people going to come?” is a big gear.
A lot can change in 3 years, at MIRI.