To make outcome-based decisions, you have to decide on the period in which you’re considering them. Considering any given period costs non-0 resources (reductio ad absurdum: in practice, considering all possible future timelines would cost infinite resources, so we presumably agree on the principle that excluding some from consideration is not only reasonable but necessary).
I think it’s a reasonable position to believe that if something can’t be empirically validated then it at least needs exceptionally strong conceptual justifications to inform such decisions.
This cuts both ways, so if the argument of AI2027 is ‘we shouldn’t dismiss this outcome out of hand’ then it’s a reasonable position (although I find Titotal’s longer backcasting an interesting counterweight, and it prompted me to wonder about a good way to backcast still further). If the argument is that AI safety researchers should meaningfully update towards shorter timelines based on the original essay or that we should move a high proportion of the global or altruistic economy towards event planning for AGI in 2027 - which seems to be what the authors are de facto pushing for—that seems much less defensible.
And I worry that they’ll be fodder for views like Aschenbrenner’s, and used to justify further undermining US-China relations and increasing the risk of great power conflict or nuclear war, both of which seems to me like more probable events in the next decade than AGI takeover.
if the argument of AI2027 is ‘we shouldn’t dismiss this outcome out of hand’ then it’s a reasonable position
Yep, that is how Titotal summarizes the argument:
The scenario in the short story is not the median forecast for any AI futures author, and none of the AI2027 authors actually believe that 2027 is the median year for a singularity to happen. But the argument they make is that 2027 is a plausible year
And if titotal had ended their post with something like ”… and so I think 2027 is a bit less plausible than the authors do” I would have no confusion. But they ended with:
What I’m most against is people taking shoddy toy models seriously and basing life decisions on them, as I have seen happen for AI2027
And I therefore am left wondering what less shoddy toy models I should be basing my life decisions on.[1]
I think their answer is partly “naively extrapolating the METR time horizon numbers forward is better than AI 2027”? But I don’t want to put words in their mouth and also I interpret them to have much longer timelines than this naive extrapolation would imply.
I think less selective quotation makes the line of argument clear.
Continuing the first quote:
The scenario in the short story is not the median forecast for any AI futures author, and none of the AI2027 authors actually believe that 2027 is the median year for a singularity to happen. But the argument they make is that 2027 is a plausible year,and they back it up with images of sophisticated looking modelling like the following:
[img]
This combination of compelling short story and seemingly-rigorous research may have been the secret sauce that let the article to go viral and be treated as a serious project:
[quote]
Now, I was originally happy to dismiss this work and just wait for their predictions to fail, but this thing just keeps spreading, including a youtube video with millions of views. So I decided to actually dig into the model and the code, and try to understand what the authors were saying and what evidence they were using to back it up.
The article is huge, so I focussed on one section alone: their “timelines forecast” code and accompanying methodology section. Not to mince words, I think it’s pretty bad. It’s not just that I disagree with their parameter estimates, it’s that I think the fundamental structure of their model is highly questionable and at times barely justified, there is very little empirical validation of the model, and there are parts of the code that the write-up of the model straight up misrepresents.
So the summary of this would not be ”… and so I think AI 2027 is a bit less plausible than the authors do”, but something like: “I think the work motivating AI 2027 being a credible scenario is, in fact, not good, and should not persuade those who did not believe this already. It is regrettable this work is being publicised (and perhaps presented) as much stronger than it really is.”
Continuing the second quote:
What I’m most against is people taking shoddy toy models seriously and basing life decisions on them, as I have seen happen for AI2027. This is just a model for a tiny slice of the possibility space for how AI will go, and in my opinion it is implemented poorly even if you agree with the author’s general worldview.
The right account for decision making under (severe) uncertainty is up for grabs, but in the ‘make a less shoddy toy model’ approach the quote would urge having a wide ensemble of different ones (including, say, those which are sub-exponential, ‘hit the wall’ or whatever else), and further urge we should put very little weight on the AI2027 model in whatever ensemble we will be using for important decisions.
Titotal actually ended their post with an alternative prescription:
I think people are going to deal with the fact that it’s really difficult to predict how a technology like AI is going to turn out. The massive blobs of uncertainty shown in AI 2027 are still severe underestimates of the uncertainty involved. If your plans for the future rely on prognostication, and this is the standard of work you are using, I think your plans are doomed. I would advise looking into plans that are robust to extreme uncertainty in how AI actually goes, and avoid actions that could blow up in your face if you turn out to be badly wrong.
I would advise looking into plans that are robust to extreme uncertainty in how AI actually goes, and avoid actions that could blow up in your face if you turn out to be badly wrong.
Seeing you highlight this now it occurs to me that I basically agree with this w.r.t. AI timelines (at least on one plausible interpretation, my guess is that titotal could have a different meaning in mind). I mostly don’t think people should take actions that blow up in their face if timelines are long (there are some exceptions, but overall I think long timelines are plausible and actions should be taken with that in mind).
A key thing that titotal doesn’t mention is how much probability mass they put on short timelines like, say, AGI by 2030. This seems very important for weighing various actions, even though we both agree that we should also be prepared for longer timelines.
In general, I feel like executing plans that are robust to extreme uncertainty is a prescription that is hard to follow without having at least a vague idea of the distribution of likelihood of various possibilities.
Thanks! This is helpful, although I would still be interested to hear if they believe there are models with “have strong conceptual justifications or empirical validation with existing data”.
I was going to reply with something longer here, but I think Gregory Lewis’ excellent comment highlights most of what I wanted to, r.e. titotal does actually give an alternative suggestion in the piece.
So instead I’ll counter two claims I think you make (or imply) in your comments here:
1. A shoddy toy model is better than no model at all
I mean this seems clearly not true, if we take model to be referring to the sort of formalised, quantified exercise similar to AI-2027. Some examples here might be Samuelson’s infamous predictions of the Soviet Union inevitably overtaking the US in GNP.[1] This was a bad model of the world, and even if it was ‘better’ than the available alternatives or came from a more prestigious source, it was still bad and I think worse than no model (again, defined as formal exercise ala AI2027).
A second example I can think of is the infamous Growth in a Time of Debtpaper, which I remember being used to win arguments and justify austerity across Europe in the 2010s, being rendered much less convincing after an Excel error was corrected.[2]
This also seems clearly false, unless we’re stretching “model” to mean simply “a reason/argument/justification” or defining “life decisions” narrowly as only those with enormous consequences instead of any ‘decision about my life’.
Even in the more serious cases, the role of models is to support presenting arguments for or against some decision or not, or to frame some explanation about the world, and of course simplification and quantification can be useful and powerful, but they shouldn’t be the only game in town. Other schools of thought are available.[3]
The reproduction paper turned critique is here, feels crazy that I can’t see the original data but the ‘model’ here seemed just to be spreadsheet of ~20 countries where the average only counted 15
This also seems clearly false, unless we’re stretching “model” to mean simply “a reason/argument/justification”
Yep, this is what I meant, sorry for the confusion. Or to phrase it another way: “I’m going off my intuition” is not a type of model which has privileged epistemic status; it’s one which can be compared with something like AI 2027 (and, like you say, may be found better).
Besides the point that “shoddy toy models” might be emotionally charged, I just want to point out that accelerating progress majorly increases variance and unknown unknowns? The higher energy a system is and the more variables you have the more chaotic it becomes. So maybe an answer is that a agile short-range model is the best? Outside view it in moderation and plan with the next few years being quite difficult to predict?
You don’t really need another model to disprove an existing one, you might as well point out that we don’t know and that is okay too.
To make outcome-based decisions, you have to decide on the period in which you’re considering them. Considering any given period costs non-0 resources (reductio ad absurdum: in practice, considering all possible future timelines would cost infinite resources, so we presumably agree on the principle that excluding some from consideration is not only reasonable but necessary).
I think it’s a reasonable position to believe that if something can’t be empirically validated then it at least needs exceptionally strong conceptual justifications to inform such decisions.
This cuts both ways, so if the argument of AI2027 is ‘we shouldn’t dismiss this outcome out of hand’ then it’s a reasonable position (although I find Titotal’s longer backcasting an interesting counterweight, and it prompted me to wonder about a good way to backcast still further). If the argument is that AI safety researchers should meaningfully update towards shorter timelines based on the original essay or that we should move a high proportion of the global or altruistic economy towards event planning for AGI in 2027 - which seems to be what the authors are de facto pushing for—that seems much less defensible.
And I worry that they’ll be fodder for views like Aschenbrenner’s, and used to justify further undermining US-China relations and increasing the risk of great power conflict or nuclear war, both of which seems to me like more probable events in the next decade than AGI takeover.
Yep, that is how Titotal summarizes the argument:
And if titotal had ended their post with something like ”… and so I think 2027 is a bit less plausible than the authors do” I would have no confusion. But they ended with:
And I therefore am left wondering what less shoddy toy models I should be basing my life decisions on.[1]
I think their answer is partly “naively extrapolating the METR time horizon numbers forward is better than AI 2027”? But I don’t want to put words in their mouth and also I interpret them to have much longer timelines than this naive extrapolation would imply.
I think less selective quotation makes the line of argument clear.
Continuing the first quote:
So the summary of this would not be ”… and so I think AI 2027 is a bit less plausible than the authors do”, but something like: “I think the work motivating AI 2027 being a credible scenario is, in fact, not good, and should not persuade those who did not believe this already. It is regrettable this work is being publicised (and perhaps presented) as much stronger than it really is.”
Continuing the second quote:
The right account for decision making under (severe) uncertainty is up for grabs, but in the ‘make a less shoddy toy model’ approach the quote would urge having a wide ensemble of different ones (including, say, those which are sub-exponential, ‘hit the wall’ or whatever else), and further urge we should put very little weight on the AI2027 model in whatever ensemble we will be using for important decisions.
Titotal actually ended their post with an alternative prescription:
Seeing you highlight this now it occurs to me that I basically agree with this w.r.t. AI timelines (at least on one plausible interpretation, my guess is that titotal could have a different meaning in mind). I mostly don’t think people should take actions that blow up in their face if timelines are long (there are some exceptions, but overall I think long timelines are plausible and actions should be taken with that in mind).
A key thing that titotal doesn’t mention is how much probability mass they put on short timelines like, say, AGI by 2030. This seems very important for weighing various actions, even though we both agree that we should also be prepared for longer timelines.
In general, I feel like executing plans that are robust to extreme uncertainty is a prescription that is hard to follow without having at least a vague idea of the distribution of likelihood of various possibilities.
Thanks! This is helpful, although I would still be interested to hear if they believe there are models with “have strong conceptual justifications or empirical validation with existing data”.
I was going to reply with something longer here, but I think Gregory Lewis’ excellent comment highlights most of what I wanted to, r.e. titotal does actually give an alternative suggestion in the piece.
So instead I’ll counter two claims I think you make (or imply) in your comments here:
1. A shoddy toy model is better than no model at all
I mean this seems clearly not true, if we take model to be referring to the sort of formalised, quantified exercise similar to AI-2027. Some examples here might be Samuelson’s infamous predictions of the Soviet Union inevitably overtaking the US in GNP.[1] This was a bad model of the world, and even if it was ‘better’ than the available alternatives or came from a more prestigious source, it was still bad and I think worse than no model (again, defined as formal exercise ala AI2027).
A second example I can think of is the infamous Growth in a Time of Debt paper, which I remember being used to win arguments and justify austerity across Europe in the 2010s, being rendered much less convincing after an Excel error was corrected.[2]
TL;dr, as Thane said on LessWrong, we shouldn’t grade models on a curve
2. You need to base life decisions on a toy model
This also seems clearly false, unless we’re stretching “model” to mean simply “a reason/argument/justification” or defining “life decisions” narrowly as only those with enormous consequences instead of any ‘decision about my life’.
Even in the more serious cases, the role of models is to support presenting arguments for or against some decision or not, or to frame some explanation about the world, and of course simplification and quantification can be useful and powerful, but they shouldn’t be the only game in town. Other schools of thought are available.[3]
The reproduction paper turned critique is here, feels crazy that I can’t see the original data but the ‘model’ here seemed just to be spreadsheet of ~20 countries where the average only counted 15
Such as:
Decisionmaking under Deep Uncertainty
Do The Math, Then Burn The Math and Go With Your Gut
Make a decision based on the best explanation of the world
Go with common-sense heuristics since they likely encode knowledge gained from cultural evolution
Yep, this is what I meant, sorry for the confusion. Or to phrase it another way: “I’m going off my intuition” is not a type of model which has privileged epistemic status; it’s one which can be compared with something like AI 2027 (and, like you say, may be found better).
Besides the point that “shoddy toy models” might be emotionally charged, I just want to point out that accelerating progress majorly increases variance and unknown unknowns? The higher energy a system is and the more variables you have the more chaotic it becomes. So maybe an answer is that a agile short-range model is the best? Outside view it in moderation and plan with the next few years being quite difficult to predict?
You don’t really need another model to disprove an existing one, you might as well point out that we don’t know and that is okay too.