I think we need to get good at predicting what language models will be able to do — in the real world, not just on benchmarks.
Any ideas how to do this? It seems like one key difficulty is that we just don’t have good explicit understandings of many cognitive abilities, and don’t have much hope of achieving such understandings in the relevant time frame.
So I’m not sure what can be done aside from applying human intuition to whatever relevant info we have (like how LMs qualitatively progressed in the past, how hard various future capabilities seem). Maybe try to find people with particularly good intuitions and calibrations (as demonstrated by past records of predictions)? More/better usage of prediction markets?
Anyway, does anyone have any qualitative predictions of what AIs produced by $1B training runs will look like? What do you think they will be able to do that will be most interesting or useful or or dangerous or economically valuable?
Any ideas how to do this? It seems like one key difficulty is that we just don’t have good explicit understandings of many cognitive abilities, and don’t have much hope of achieving such understandings in the relevant time frame.
So I’m not sure what can be done aside from applying human intuition to whatever relevant info we have (like how LMs qualitatively progressed in the past, how hard various future capabilities seem). Maybe try to find people with particularly good intuitions and calibrations (as demonstrated by past records of predictions)? More/better usage of prediction markets?
Anyway, does anyone have any qualitative predictions of what AIs produced by $1B training runs will look like? What do you think they will be able to do that will be most interesting or useful or or dangerous or economically valuable?