Speaking only for my own views, the “most important century” hypothesis seems to have survived all of this. Indeed, having examined the many angles and gotten more into the details, I believe it more strongly than before.
The footnote text reads, in part:
Reviews of Bio Anchors are here; reviews of Explosive Growth are here; reviews of Semi-informative Priors are here.
Many of these reviewers disagree strongly with the reports under review.
By my judgment, all three made strong negative assessments, in the sense (among others) that if one agreed with the review, one would not use the report’s reasoning to inform decision-making in the manner advocated by Karnofsky (and by Beckstead).
From Hajek and Strasser’s review:
His final probability of 7.3% is a nice summary of his conclusion, but its precision (including a decimal place!) belies the vagueness of the question, the imprecise and vague inputs, and the arbitrary/subjective choices Tom needs to make along the way—we discuss this more in our answers to question 8. We think a wider range is appropriate given the judgment calls involved. Or one might insist that an imprecise probability assignment is required here. Note that this is not the same as a range of permissible sharp probabilities. Following e.g. Joyce, one might think that no precise probability is permissible, given the nature of the evidence and the target proposition to which we are assigning a credence.
From Hanson’s review:
I fear that for this application, this framework abstracts too much from important details.
For example, if the actual distribution is some generic lump, but the model distribution is an exponential falling from an initial start, then the errors that result from this difference are probably worse regarding the lowest percentiles of either distribution, where the differences are most stark. So I’m more comfortable using such a simple model to estimate distribution medians, relative to low percentiles. Alas, the main products of this analysis are exactly these problematic low percentile estimates.
From Halpern’s review:
If our goal were a single estimate, then this is probably as reasonable as any other. I have problems with the goal (see below). [...]
As I said above, I have serious concerns about the way that dynamic issues are being handled. [...]
I am not comfortable with modeling uncertainty in this case using a single probability measure.
Two of the reviewers found little to disagree with. These were Leopold Aschenbrenner (a Future Fund researcher) and Ege Erdil (a Metaculus forecaster).
The other three reviewers were academic economists specializing in growth and/or automation. Two of them made strong negative assessments.
From Ben Jones’ review:
Nonetheless, while this report suggests that a rapid growth acceleration is substantially less likely than singularity-oriented commentators sometimes advocate, to my mind this report still sees 30% growth by 2100 as substantially likelier than my intuitions would suggest. Without picking numbers, and acknowledging that my views may prove wrong, I will just say that achieving 30% growth strikes me as very unlikely. Here I will articulate some reasons why, to provoke further discussion.
From Dietrich Vollrath’s review:
All that said, I think the probability of explosive growth in GWP is very low. Like 0% low. I think those issues I raised above regarding output and demand will bind and bite very hard if productivity grows that fast.
The third economist, Paul Gaggl, agreed with the report about the possibility of high GWP growth but raised doubts as to how long it could be sustained. (How much this matters depends on what question we’re asking; “a few decades” of 30% GWP growth is not a permanent new paradigm, but it is certainly a big “transformation.”)
I expect that some experts would be much more likely to spend time and effort on the contest if
They had clearer evidence that the Future Fund was amendable to persuasion at all.
E.g. examples of somewhat-analogous cases in which a critical review did change the opinion of someone currently at the Future Fund (perhaps before the Future Fund existed).
They were told why the specific critical reviews discussed above did not have significant impact on the Future Fund’s views.
This would help steer them toward critiques likely to make an impact, mitigate the sense that entrants are “shooting in the dark,” and move writing-for-the-contest outside of a reference class where all past attempts have failed.
These considerations seems especially relevant for the “dark matter” experts hypothesized in this post and Karnofsky’s, who “find the whole thing so silly that they’re not bothering to engage.” These people are unusually likely to have a low opinion of the Future Fund’s overall epistemics (point 1), and they are also likely to disagree with the Fund’s reasoning along a relatively large number of axes, so that locating a crux becomes more of a problem (point 2).
Finally: I, personally would be more likely to submit to the contest if I had a clearer sense where the cruxes were, and why past criticisms have failed to stick. (For clarity, I don’t consider myself an “expert” in any relevant sense.)
While I don’t “find the whole thing so silly I don’t bother to engage,” I have relatively strong methodological objections to some of the OpenPhil reports cited here. There is a large inferential gap between me and anyone who finds these reports prima facie convincing. Given the knowledge that someone does find them prima facie convincing, and little else, it’s hard to know where to begin in trying to close that gap.
Even if I had better guidance, the size of the gap increases the effort required and decreases my expected probability of success, and so it makes me less likely to contribute. This dynamic seems like a source of potential bias in the distribution of the responses, though I don’t have any great ideas for what to do about it.
By my judgment, all three made strong negative assessments, in the sense (among others) that if one agreed with the review, one would not use the report’s reasoning to inform decision-making in the manner advocated by Karnofsky (and by Beckstead).
For Hajek&Strasser’s and Halpern’s reviews, I don’t think “strong negative assessment” is supported by your quotes. The quotes focus on things like ‘the reported numbers are too precise’ and ‘we should use more than a single probability measure’ rather than whether the estimate is too high or too low overall or whether we should be worrying more vs less about TAI. I also think the reviews are more positive overall than you imply, e.g. Halpern’s review says “This seems to be the most serious attempt to estimate when AGI will be developed that I’ve seen”
I agree that these two reviewers assign much lower probabilities to explosive growth than I do (I explain why I continue to disagree with them in my responses to their reviews). Again though, I think these reviews are more positive overall than you imply, e.g. Jones states that the report “is balanced, engaging a wide set of viewpoints and acknowledging debates and uncertainties… is also admirably clear in its arguments and in digesting the literature… engages key ideas in a transparent way, integrating perspectives and developing its analysis clearly and coherently.” This is important as it helps us move from “maybe we’re completely missing a big consideration” to “some experts continue to disagree for certain reasons, but we have a solid understanding of the relevant considerations and can hold our own in a disagreement”.
I agree that finding the cruxes of disagreement are important, but I don’t think any of the critical quotes you present above are that strong. The reviews of semi-informative priors talk about error bars and precision (i.e. critique the model), but don’t actually give different answers. On explosive growth, Jones talks about the conclusion being contrary to his “intuitions”, and acknowledges that “[his] views may prove wrong”. Vollrath mentions “output and demand”, but then talks about human productivity when regarding outputs, and admits that AI could create new in-demand products. If these are the best existing sources for lowering the Future Fund’s probabilities, then I think someone should be able to do better.
On the other hand, I think that the real probabilities are higher, and am confused as to why the Future Fund haven’t already updated to higher probabilities, given some of the writing already out there. I give a speculative reason here.
Weakly downvoting due to over-strong claims and the evidence doesn’t fully support your view. This is weak evidence against AGI claims, but the claims in this comment are too strong.
Quoting Greg Colbourn:
I agree that finding the cruxes of disagreement are important, but I don’t think any of the critical quotes you present above are that strong. The reviews of semi-informative priors talk about error bars and precision (i.e. critique the model), but don’t actually give different answers. On explosive growth, Jones talks about the conclusion being contrary to his “intuitions”, and acknowledges that “[his] views may prove wrong”. Vollrath mentions “output and demand”, but then talks about human productivity when regarding outputs, and admits that AI could create new in-demand products. If these are the best existing sources for lowering the Future Fund’s probabilities, then I think someone should be able to do better.
...if they had explained why their views were not moved by the expert reviews OpenPhil has already solicited.
In “AI Timelines: Where the Arguments, and the ‘Experts,’ Stand,” Karnofsky writes:
The footnote text reads, in part:
Many of these reviewers disagree strongly with the reports under review.
Davidson 2021 on semi-informative priors received three reviews.
By my judgment, all three made strong negative assessments, in the sense (among others) that if one agreed with the review, one would not use the report’s reasoning to inform decision-making in the manner advocated by Karnofsky (and by Beckstead).
From Hajek and Strasser’s review:
From Hanson’s review:
From Halpern’s review:
Davidson 2021 on explosive growth received many reviews; I’ll focus on the five reviewers who read the final version.
Two of the reviewers found little to disagree with. These were Leopold Aschenbrenner (a Future Fund researcher) and Ege Erdil (a Metaculus forecaster).
The other three reviewers were academic economists specializing in growth and/or automation. Two of them made strong negative assessments.
From Ben Jones’ review:
From Dietrich Vollrath’s review:
The third economist, Paul Gaggl, agreed with the report about the possibility of high GWP growth but raised doubts as to how long it could be sustained. (How much this matters depends on what question we’re asking; “a few decades” of 30% GWP growth is not a permanent new paradigm, but it is certainly a big “transformation.”)
Reviews of Cotra (2020) on Biological Anchors were mostly less critical than the above.
I expect that some experts would be much more likely to spend time and effort on the contest if
They had clearer evidence that the Future Fund was amendable to persuasion at all.
E.g. examples of somewhat-analogous cases in which a critical review did change the opinion of someone currently at the Future Fund (perhaps before the Future Fund existed).
They were told why the specific critical reviews discussed above did not have significant impact on the Future Fund’s views.
This would help steer them toward critiques likely to make an impact, mitigate the sense that entrants are “shooting in the dark,” and move writing-for-the-contest outside of a reference class where all past attempts have failed.
These considerations seems especially relevant for the “dark matter” experts hypothesized in this post and Karnofsky’s, who “find the whole thing so silly that they’re not bothering to engage.” These people are unusually likely to have a low opinion of the Future Fund’s overall epistemics (point 1), and they are also likely to disagree with the Fund’s reasoning along a relatively large number of axes, so that locating a crux becomes more of a problem (point 2).
Finally: I, personally would be more likely to submit to the contest if I had a clearer sense where the cruxes were, and why past criticisms have failed to stick. (For clarity, I don’t consider myself an “expert” in any relevant sense.)
While I don’t “find the whole thing so silly I don’t bother to engage,” I have relatively strong methodological objections to some of the OpenPhil reports cited here. There is a large inferential gap between me and anyone who finds these reports prima facie convincing. Given the knowledge that someone does find them prima facie convincing, and little else, it’s hard to know where to begin in trying to close that gap.
Even if I had better guidance, the size of the gap increases the effort required and decreases my expected probability of success, and so it makes me less likely to contribute. This dynamic seems like a source of potential bias in the distribution of the responses, though I don’t have any great ideas for what to do about it.
I included responses to each review, explaining my reactions to it. What kind of additional explanation were you hoping for?
For Hajek&Strasser’s and Halpern’s reviews, I don’t think “strong negative assessment” is supported by your quotes. The quotes focus on things like ‘the reported numbers are too precise’ and ‘we should use more than a single probability measure’ rather than whether the estimate is too high or too low overall or whether we should be worrying more vs less about TAI. I also think the reviews are more positive overall than you imply, e.g. Halpern’s review says “This seems to be the most serious attempt to estimate when AGI will be developed that I’ve seen”
I agree that these two reviewers assign much lower probabilities to explosive growth than I do (I explain why I continue to disagree with them in my responses to their reviews). Again though, I think these reviews are more positive overall than you imply, e.g. Jones states that the report “is balanced, engaging a wide set of viewpoints and acknowledging debates and uncertainties… is also admirably clear in its arguments and in digesting the literature… engages key ideas in a transparent way, integrating perspectives and developing its analysis clearly and coherently.” This is important as it helps us move from “maybe we’re completely missing a big consideration” to “some experts continue to disagree for certain reasons, but we have a solid understanding of the relevant considerations and can hold our own in a disagreement”.
Wow, thanks for this well written summary of expert reviews that I didn’t know existed! Strongly upvoted.
I agree that finding the cruxes of disagreement are important, but I don’t think any of the critical quotes you present above are that strong. The reviews of semi-informative priors talk about error bars and precision (i.e. critique the model), but don’t actually give different answers. On explosive growth, Jones talks about the conclusion being contrary to his “intuitions”, and acknowledges that “[his] views may prove wrong”. Vollrath mentions “output and demand”, but then talks about human productivity when regarding outputs, and admits that AI could create new in-demand products. If these are the best existing sources for lowering the Future Fund’s probabilities, then I think someone should be able to do better.
On the other hand, I think that the real probabilities are higher, and am confused as to why the Future Fund haven’t already updated to higher probabilities, given some of the writing already out there. I give a speculative reason here.
Weakly downvoting due to over-strong claims and the evidence doesn’t fully support your view. This is weak evidence against AGI claims, but the claims in this comment are too strong.
Quoting Greg Colbourn: