Full disclosure—I read a draft of this piece and provided titotal with some feedback on it
Some high level reflections on this piece and interpreting the surrounding Debate:
I think this is an excellent piece, and I think well worth reading, especially for EAs who are tempted to easily defer to ‘high status’ organisations/​individuals in the field of AI Safety. Having said that, I also think @elifland and @kokotajlod deserve a lot of credit for being open to criticism and working with titotal privately, and communicating in good faith here, on LW, titotal’s substack etc[1]
I think the piece shows that there are clear flaws with how the model is constructed, and that its design betrays the assumptions of those creating it. Of course this is a feature, not a bug, as it’s meant to be a formalisation of the beliefs of the AI2027 team (at least as I understand it). This is completely ok and even a useful exercise, but then I think nostalgebraist is accurate in saying that if you didn’t buy the priors/​arguments for a near-term intelligence explosion before the model, then you won’t afterwards. The arguments for our assumptions, and how to interpret the data we have are ~the whole ball game, and not the quantitative forecasts/​scenarios that they then produce.
I particularly want to draw attention to the ‘Six stories that fit the data’ section, because to my mind it demonstrates the core issue/​epistemological crux. The whole field of AI Safety is an intense case of the underdetermination of theory given evidence,[2] and choosing which explanation of the world to go with given our limited experience of it is the key question of epistemology. But as titotal points out, a similar exercise to AI2027 could pick any one of those curves (or infinitely many alternatives) - the key points here are the arguments and assumptions underlying the models and how they clash.
Why the AI debates seem so intractable is a combination of:
We have limited data with which to constrain hypotheses (METR’s curve has 11 data points!)
The assumptions that underly the differences are based on pretty fundamental worldview differences and/​or non-technical beliefs about how the world works—see AI as Normal Technology[3]or Ajeya Cotra’s appearance on the AI Summer Podcast
The lack of communication between the various ‘different camps’ involved in frontier AI and AI research contributes to misunderstandings, confrontational/​adversarial stances etc.
The increased capability and saliency of AI in the world starting to lead to political polarisation effects which might make the above worse
Linked to 4, what happens with AI seems to be very high stakes. It’s not just opinions differ, but the range of what could happen is massive. There’s a lot at risk if actions are taken which are later proben to be misguided.
Given this epistemological backdrop, and the downward spiral in AI discourse over the last 2 years,[4] I don’t know how to improve the current state of affairs apart from ‘let reality adjudicate the winners’ - which often leaves me frustrated and demotivated. I’m thinking perhaps of adversarial collaborations between different camps, boosting collaborative AI Safety strategies, and using AI to help develop high-trust institutions.[5] But I don’t think exercise like AI2027 push the field forward because of the ‘peak forecasting’ involved, but instead by surfacing the arguments which underline the forecasts for scrutiny and falsification.
Full disclosure—I read a draft of this piece and provided titotal with some feedback on it
Some high level reflections on this piece and interpreting the surrounding Debate:
I think this is an excellent piece, and I think well worth reading, especially for EAs who are tempted to easily defer to ‘high status’ organisations/​individuals in the field of AI Safety. Having said that, I also think @elifland and @kokotajlod deserve a lot of credit for being open to criticism and working with titotal privately, and communicating in good faith here, on LW, titotal’s substack etc[1]
I think the piece shows that there are clear flaws with how the model is constructed, and that its design betrays the assumptions of those creating it. Of course this is a feature, not a bug, as it’s meant to be a formalisation of the beliefs of the AI2027 team (at least as I understand it). This is completely ok and even a useful exercise, but then I think nostalgebraist is accurate in saying that if you didn’t buy the priors/​arguments for a near-term intelligence explosion before the model, then you won’t afterwards. The arguments for our assumptions, and how to interpret the data we have are ~the whole ball game, and not the quantitative forecasts/​scenarios that they then produce.
I particularly want to draw attention to the ‘Six stories that fit the data’ section, because to my mind it demonstrates the core issue/​epistemological crux. The whole field of AI Safety is an intense case of the underdetermination of theory given evidence,[2] and choosing which explanation of the world to go with given our limited experience of it is the key question of epistemology. But as titotal points out, a similar exercise to AI2027 could pick any one of those curves (or infinitely many alternatives) - the key points here are the arguments and assumptions underlying the models and how they clash.
Why the AI debates seem so intractable is a combination of:
We have limited data with which to constrain hypotheses (METR’s curve has 11 data points!)
The assumptions that underly the differences are based on pretty fundamental worldview differences and/​or non-technical beliefs about how the world works—see AI as Normal Technology[3] or Ajeya Cotra’s appearance on the AI Summer Podcast
The lack of communication between the various ‘different camps’ involved in frontier AI and AI research contributes to misunderstandings, confrontational/​adversarial stances etc.
The increased capability and saliency of AI in the world starting to lead to political polarisation effects which might make the above worse
Linked to 4, what happens with AI seems to be very high stakes. It’s not just opinions differ, but the range of what could happen is massive. There’s a lot at risk if actions are taken which are later proben to be misguided.
Given this epistemological backdrop, and the downward spiral in AI discourse over the last 2 years,[4] I don’t know how to improve the current state of affairs apart from ‘let reality adjudicate the winners’ - which often leaves me frustrated and demotivated. I’m thinking perhaps of adversarial collaborations between different camps, boosting collaborative AI Safety strategies, and using AI to help develop high-trust institutions.[5] But I don’t think exercise like AI2027 push the field forward because of the ‘peak forecasting’ involved, but instead by surfacing the arguments which underline the forecasts for scrutiny and falsification.
Though I think Alfredo Parra’s recent post is also worth bearing in mind
Though, tbf, all of human knowledge is. This problem isn’t unique to AI/​AI Safety
Particularly ‘The challenge of policy making under uncertainty’ section
A spiral of which no side is blameless
If you’re interested in collaborating on/​supporting any of the above, please reach out