the capability program is an easier technical problem than the alignment program.
You don’t know that; nobody knows that
Do you concede that frontier AI research is intrinsically dangerous?
That it is among the handful of the most dangerous research programs ever pursued by our civilization?
If not, I hope you can see why those who do consider it intrinsically dangerous are not particularly mollified or reassured by “well, who knows? maybe it will turn out OK in the end!”
The distinction between capabilities and alignment is a useful concept when choosing research on an individual level; but it’s far from robust enough to be a good organizing principle on a societal level.
When I wrote “the alignment program” above, I meant something specific, which I believe you will agree is robust enough to organize society (if only we could get society to go along with it): namely, I meant thinking hard together about alignment without doing anything dangerous like training up models with billions of parameters till we have at least a rough design that most of the professional researchers agree is more likely to help us than to kill us even if it turns out to have super-human capabilities—even if our settling on that design takes us many decades. E.g., what MIRI has been doing the last 20 years.
I dislike the phrase “we all die”, nobody has justifiable confidence high enough to make that claim, even if ASI is misaligned enough to seize power there’s a pretty wide range of options for the future of humans
It makes me sad that you do not see that “we all die” is the default outcome that naturally happens unless a lot of correct optimization pressure is applied by the researchers to the design of the first sufficiently-capable AI before the AI is given computing resources. It would have been nice to have someone with your capacity for clear thinking working on the problem. Are you sure you’re not overly attached (e.g., for intrapersonal motivational reasons) to an optimistic vision in which AI research “feels like the early days of hacker culture” and “there are hackathons where people build fun demos”?
Do you concede that frontier AI research is intrinsically dangerous?
That it is among the handful of the most dangerous research programs ever pursued by our civilization?
If not, I hope you can see why those who do consider it intrinsically dangerous are not particularly mollified or reassured by “well, who knows? maybe it will turn out OK in the end!”
When I wrote “the alignment program” above, I meant something specific, which I believe you will agree is robust enough to organize society (if only we could get society to go along with it): namely, I meant thinking hard together about alignment without doing anything dangerous like training up models with billions of parameters till we have at least a rough design that most of the professional researchers agree is more likely to help us than to kill us even if it turns out to have super-human capabilities—even if our settling on that design takes us many decades. E.g., what MIRI has been doing the last 20 years.
It makes me sad that you do not see that “we all die” is the default outcome that naturally happens unless a lot of correct optimization pressure is applied by the researchers to the design of the first sufficiently-capable AI before the AI is given computing resources. It would have been nice to have someone with your capacity for clear thinking working on the problem. Are you sure you’re not overly attached (e.g., for intrapersonal motivational reasons) to an optimistic vision in which AI research “feels like the early days of hacker culture” and “there are hackathons where people build fun demos”?