Some papers you might like if you haven’t seen them yet:
This paper uses your two methods, as well as mapping verbal descriptions to probabilities
Anthropic investigation of language model calibration. Interesting techniques include asking the model whether its answer was correct, using temperature scaling to restore calibration after RLHF, and training models to be better calibrated.
Foundational overview of calibration in ML models, advocates temperature scaling
Paper showing calibration often suffers under distribution shift of the dataset
Nicely written, these make a lot of sense to me. My case for AI winter would focus on two tailwinds that will likely cease by the end of the decade: money and data.
Money can’t continue scaling like this. Spending on training runs has gone up by about an order of magnitude every two years over the last decade. By 2032 this trend would put us at $100B training runs, which would be 3x Google’s entire R&D budget. If TAI doesn’t emerge before then, spending growth will need to slow down, likely slowing AI progress as well.
Maybe data can’t either. Full analysis here, but basically high quality language data will run out well before 2030, possibly within the next year or two. But there are other kinds of data that could continue scaling. For example, the Epoch report discusses low quality language data like private texts and emails as one possibility. I would look more to the transition from language models to multimodal vision-and-language models as an important trend not only because vision is a useful modality, but because it would allow data scaling to continue.
I’d like to have a better view of questions about the continuation of Moore’s Law. Without a full writeup, the claims about Moore’s Law ending seem more credible now than in the past. I would be really interested in an aggregation of historical forecasts about Moore’s Law to see whether the current doomsaying is any different from the long run trend.