This piece is...pretty amazing. I could see this being really useful for me as an AI governance researcher, possibly the most useful thing I’ve read this year. Thanks!
Do you have any advice for eliciting feedback from people when you’re doing rapid iteration? I generally find it valuable to share Google Docs with people as I’m working through ideas, but it can be hard to communicate the kind of feedback that’s most useful for rough documents. Maybe it’s good to flag “these are hot takes, I’m looking for strong arguments against them to refine my viewpoint, don’t bother with small details for now”?
In general, I’m skeptical of surveys like this—I participated in a similar one a few years ago that didn’t have super useful results, though I think it was kind of useful for clarifying my own thinking. But that’s pretty outside-viewy. Let me take a stab at making that general skepticism concrete—trying to elucidate why people might struggle to answer, slash why the questions you’re asking won’t yield super useful answers.
I expect that the ‘right’ answer depends on carefully enumerating and considering a bunch of different plausible scenarios, and what you’ll get instead is either uncertainty or vague intuitive guesses. If you mostly want vague intuitive guesses, great! I would guess you’d get more clarity from trying to elicit people’s particular models / expected trajectories.
My rough experience is that people working in AI governance mostly think about particular trajectories/dynamics of AI progress that they consider especially plausible/important/tractable, so they might only have insight into particular configurations of variables you consider. Or their insight might be at a more granular level, weighing e.g. the impact of AI development in particular corporate labs.
Skimming your survey, the answer that feels right to me is often that the effect depends a lot on circumstances. For example, fast takeoff worlds where fast takeoff is anticipated look extremely different from fast takeoff worlds where it comes as a surprise.