This post is a bait-and-switch: It starts off with a discussion of the Good Judgement Project and what lessons it teaches us about forecasting superintelligence. However, starting with the section “What lessons should we learn?”, you switch from a general discussion of these techniques towards making a narrow point about which areas of expertise forecasters should rely on, an opinion which I suspect the author arrived at through means not strongly motivated from the Good Judgement Project.
While I also suspect the Good Judgement Project could have valuable lessons on superintelligence forecasting, I think that taking verbal descriptions of the how superforecasters make good predictions and citing them for arguments about loosely related specific policies is a poor way to do that. As a comparison, I don’t think that giving a forecaster this list of suggestions and asking them to make predictions with those suggestions in mind would lead to performance similar to that of a superforecaster. In my opinion, the best way to draw lessons from the Good Judgement Project is to directly rely on existing forecasting teams, or new forecasting teams trained and tested in the same manner, to give us their predictions on potential superintelligence, and to give the appropriate weight to their expertise.
Moreover, among the list of suggestions in the section “What they found to work”, you almost entirely focus on the second one, “Looking at a problem from multiple different view points and synthesising them?” to make your argument. You can also be said to be relying on the last suggestion to the extent they say essentially the same thing, that we should rely on multiple points of view. The only exception is that you rely on the fifth suggestion, “Striving to distinguish as many degrees of doubt as possible—be as precise in your estimates as you can”, when you argue their strategy documents should have more explicit probability estimates. In response to that, keep in mind that these forecasters are specifically tested on giving well-calibrated probabilistic predictions. Therefore I expect that this overestimates the importance of precise probability estimates in other contexts. My hunch is that giving numerically precise subjective probability estimates is useful in discussions among people already trained to have a good subjective impression of what these probabilities mean, but among people without such training the effect of using precise probabilities is neutral or harmful. However, I have no evidence for this hunch.
I disapprove of this bait-and-switch. I think it deceptively builds a case for diversity in intelligence forecasting, and adds confusion to both the topics it discusses.
Sorry if you felt I was being deceptive. The list of areas of expertise I mentioned in the 80K hours section was relatively broad and not meant to be exhaustive. I could add physics and economics off the top of my head. I’m sure there were many more. I was considering each AGI team as having to do small amounts of forecasting about the likely success and usefulness of their projects. I think building it in the superforecasting mindset at all levels of endeavours could be valuable, without having to rely on explicit superforecasters for every decision.
In my opinion, the best way to draw lessons from the Good Judgement Project is to directly rely on existing forecasting teams, or new forecasting teams trained and tested in the same manner, to give us their predictions on potential superintelligence, and to give the appropriate weight to their expertise.
It would be great to have a full team of forecasters working on intelligence in general (so they would have something to correlate their answers on Superintelligence). I was being moderate in my demands in how much Open Philanthropy Project should change how they make forecasts about what is good to do. I just wanted it to be directionally correct.
As a comparison, I don’t think that giving a forecaster this list of suggestions and asking them to make predictions with those suggestions in mind would lead to performance similar to that of a superforecaster
There was a simple thing people could do to improve their predictions.
From the book:
One result was particularly surprised me was the effect of a tutorial covering some basic concepts that we’ll explore in this book and are summarized in the Ten Commandments appendix. It took only sixty minutes to read and yet it improved accuracy by roughly 10% through the entire tournament year.
The ten commandment appendix is where I got the list of things to do. I figure if I managed to get Open Philosophy Project to try and follow them, things would improve. But I agree them getting good forecasters somehow would be a lot better.
This post is a bait-and-switch: It starts off with a discussion of the Good Judgement Project and what lessons it teaches us about forecasting superintelligence. However, starting with the section “What lessons should we learn?”, you switch from a general discussion of these techniques towards making a narrow point about which areas of expertise forecasters should rely on, an opinion which I suspect the author arrived at through means not strongly motivated from the Good Judgement Project.
While I also suspect the Good Judgement Project could have valuable lessons on superintelligence forecasting, I think that taking verbal descriptions of the how superforecasters make good predictions and citing them for arguments about loosely related specific policies is a poor way to do that. As a comparison, I don’t think that giving a forecaster this list of suggestions and asking them to make predictions with those suggestions in mind would lead to performance similar to that of a superforecaster. In my opinion, the best way to draw lessons from the Good Judgement Project is to directly rely on existing forecasting teams, or new forecasting teams trained and tested in the same manner, to give us their predictions on potential superintelligence, and to give the appropriate weight to their expertise.
Moreover, among the list of suggestions in the section “What they found to work”, you almost entirely focus on the second one, “Looking at a problem from multiple different view points and synthesising them?” to make your argument. You can also be said to be relying on the last suggestion to the extent they say essentially the same thing, that we should rely on multiple points of view. The only exception is that you rely on the fifth suggestion, “Striving to distinguish as many degrees of doubt as possible—be as precise in your estimates as you can”, when you argue their strategy documents should have more explicit probability estimates. In response to that, keep in mind that these forecasters are specifically tested on giving well-calibrated probabilistic predictions. Therefore I expect that this overestimates the importance of precise probability estimates in other contexts. My hunch is that giving numerically precise subjective probability estimates is useful in discussions among people already trained to have a good subjective impression of what these probabilities mean, but among people without such training the effect of using precise probabilities is neutral or harmful. However, I have no evidence for this hunch.
I disapprove of this bait-and-switch. I think it deceptively builds a case for diversity in intelligence forecasting, and adds confusion to both the topics it discusses.
Sorry if you felt I was being deceptive. The list of areas of expertise I mentioned in the 80K hours section was relatively broad and not meant to be exhaustive. I could add physics and economics off the top of my head. I’m sure there were many more. I was considering each AGI team as having to do small amounts of forecasting about the likely success and usefulness of their projects. I think building it in the superforecasting mindset at all levels of endeavours could be valuable, without having to rely on explicit superforecasters for every decision.
It would be great to have a full team of forecasters working on intelligence in general (so they would have something to correlate their answers on Superintelligence). I was being moderate in my demands in how much Open Philanthropy Project should change how they make forecasts about what is good to do. I just wanted it to be directionally correct.
There was a simple thing people could do to improve their predictions.
From the book:
The ten commandment appendix is where I got the list of things to do. I figure if I managed to get Open Philosophy Project to try and follow them, things would improve. But I agree them getting good forecasters somehow would be a lot better.
Does that clear up where I was coming from?