40. “Geniuses” with nice legible accomplishments in fields with tight feedback loops where it’s easy to determine which results are good or bad right away, and so validate that this person is a genius, are (a) people who might not be able to do equally great work away from tight feedback loops, (b) people who chose a field where their genius would be nicely legible even if that maybe wasn’t the place where humanity most needed a genius, and (c) probably don’t have the mysterious gears simply because they’re rare.You cannot just pay $5 million apiece to a bunch of legible geniuses from other fields and expect to get great alignment work out of them. They probably do not know where the real difficulties are, they probably do not understand what needs to be done, they cannot tell the difference between good and bad work, and the funders also can’t tell without me standing over their shoulders evaluating everything, which I do not have the physical stamina to do.
This may well be true, but I think we are well past the stage where this should at least be established by significant empirical evidence. We should at least try, whilst we have the opportunity. I would feel a lot better to be in a world where a serious attempt was made at this project (to the point where I’m willing to personally contribute a significant fraction of my own net worth toward it). More.
Also, we can generalise “”Geniuses” with nice legible accomplishments” (e.g. Fields Medalists) to “people who a panel of top people in AGI Safety would have on their dream team”. Is there such a list? Who would you nominate to be on the dream team? A great first step would be a survey of top Alignment researchers on this question.
The way I see it is that we need to throw all we have at both AGI Alignment research and AGI governance. On the research front, getting the most capable people in the world working on the problem—a Manhattan Project for alignment as it were—seems like our best bet. (On the governance front, it would be something along the lines of global regulation of, or a moratorium on, AGI capabilities research. Or in the extreme, a full-on Butlerian Jihad. That seems harder.)
He’s just one person, so I wouldn’t say that’s significant empirical evidence. Unless a bunch of other people they approached turned them down (and if they did, it would be interesting to know why).
This may well be true, but I think we are well past the stage where this should at least be established by significant empirical evidence. We should at least try, whilst we have the opportunity. I would feel a lot better to be in a world where a serious attempt was made at this project (to the point where I’m willing to personally contribute a significant fraction of my own net worth toward it). More.
Sounds right to me! I think we should try lots of things.
Also, we can generalise “”Geniuses” with nice legible accomplishments” (e.g. Fields Medalists) to “people who a panel of top people in AGI Safety would have on their dream team”. Is there such a list? Who would you nominate to be on the dream team? A great first step would be a survey of top Alignment researchers on this question.
The way I see it is that we need to throw all we have at both AGI Alignment research and AGI governance. On the research front, getting the most capable people in the world working on the problem—a Manhattan Project for alignment as it were—seems like our best bet. (On the governance front, it would be something along the lines of global regulation of, or a moratorium on, AGI capabilities research. Or in the extreme, a full-on Butlerian Jihad. That seems harder.)
Didn’t they try this? e.g. Kmett
He’s just one person, so I wouldn’t say that’s significant empirical evidence. Unless a bunch of other people they approached turned them down (and if they did, it would be interesting to know why).
Seems likely to me
I don’t think MIRI has tried this much; we were unusually excited about Edward Kmett.