This leads to scepticism about the rationality of predictors which show a pattern of ‘steadily moving in one direction’ for a given question [...] Eliezer Yudkowsky has confidently asserted it is an indicator of sub-par Bayesian updating
I’m confused about your interpretation of Eliezer’s words. What he seems to be saying is that recent advances in ML shouldn’t have caused such a notable update in predictions on AI timelines on Metaculus, since a more rational predictor would have expected that such things would be happening more likely in this timeframe than the Metaculus crowd apparently did. Admittedly, he may be wrong about that, but I see what he wrote as a claim concerning one particular subject, not about steady updates in one direction in general.
The two quotes seem to be explicitly, specifically about the general process of “updating” (although with some stretching, the first quote below could also be rationalized by saying the forecasters are obtuse and incredibly bad—but if you think people are allowed to have different beliefs, then the trend indicated should occur).
To be a slightly better Bayesian is to spend your entire life watching others slowly update in excruciatingly predictable directions that you jumped ahead of 6 years earlier so that your remaining life could be a random epistemic walk like a sane person with self-respect.
I wonder if a Metaculus forecast of “what this forecast will look like in 3 more years” would be saner. Is Metaculus reflective, does it know what it’s doing wrong?
My current model is that seeing predictable updating is still bayesian evidence of people following subpar algorithm, though it’s definitely not definite evidence.
To formalize this, assume you have two hypotheses about how Metaculus users operate:
H1: They perform correct bayesian updates
H2: They update sluggishly to new evidence
First, let’s discuss priors between these two hypotheses. IIRC we have a decent amount of evidence that sluggish updating is a pretty common occurrence in forecasting contexts, so raising sluggish updating in this context doesn’t seem unreasonable to me. Also, anecdotally, I find it hard to avoid sluggish updating, even if I try to pay attention to it, and would expect that it’s a common epistemic mistake.
Now, predictable updating is clearly more likely under H2 than under H1. The exact odds ratio of course depends on the questions being asked, but my guess is that in the context of Metaculus, something in the range of 2:1 for the data observed seems kind of reasonable to me (though this is really fully made up). This means the sentence that if you don’t have the problem of sluggish updating, you frequently see people update slowly in your direction, seems correct and accurate.
I do think Eliezer is just wrong when he says that a proper bayesian would have beliefs that look like a “random epistemic walk”, unless that “epistemic” modifier there is really doing a lot of work that is not-intuitive. If I am not super confused, the only property that sequences of beliefs should fulfill based on conservation of expected evidence is the Martingale property, which is a much broader class than random walks.
I don’t have any knowledge that would add to this discussion.
With that personal limitation in mind, what you said you seems very informative and useful, including the composition of what fraction of people are poorly updating. The experience you express here about forecasting and other belief updating seems enormous.
I’m confused about your interpretation of Eliezer’s words. What he seems to be saying is that recent advances in ML shouldn’t have caused such a notable update in predictions on AI timelines on Metaculus, since a more rational predictor would have expected that such things would be happening more likely in this timeframe than the Metaculus crowd apparently did. Admittedly, he may be wrong about that, but I see what he wrote as a claim concerning one particular subject, not about steady updates in one direction in general.
The two quotes seem to be explicitly, specifically about the general process of “updating” (although with some stretching, the first quote below could also be rationalized by saying the forecasters are obtuse and incredibly bad—but if you think people are allowed to have different beliefs, then the trend indicated should occur).
My current model is that seeing predictable updating is still bayesian evidence of people following subpar algorithm, though it’s definitely not definite evidence.
To formalize this, assume you have two hypotheses about how Metaculus users operate:
H1: They perform correct bayesian updates
H2: They update sluggishly to new evidence
First, let’s discuss priors between these two hypotheses. IIRC we have a decent amount of evidence that sluggish updating is a pretty common occurrence in forecasting contexts, so raising sluggish updating in this context doesn’t seem unreasonable to me. Also, anecdotally, I find it hard to avoid sluggish updating, even if I try to pay attention to it, and would expect that it’s a common epistemic mistake.
Now, predictable updating is clearly more likely under H2 than under H1. The exact odds ratio of course depends on the questions being asked, but my guess is that in the context of Metaculus, something in the range of 2:1 for the data observed seems kind of reasonable to me (though this is really fully made up). This means the sentence that if you don’t have the problem of sluggish updating, you frequently see people update slowly in your direction, seems correct and accurate.
I do think Eliezer is just wrong when he says that a proper bayesian would have beliefs that look like a “random epistemic walk”, unless that “epistemic” modifier there is really doing a lot of work that is not-intuitive. If I am not super confused, the only property that sequences of beliefs should fulfill based on conservation of expected evidence is the Martingale property, which is a much broader class than random walks.
Thank you for this thoughtful reply.
I don’t have any knowledge that would add to this discussion.
With that personal limitation in mind, what you said you seems very informative and useful, including the composition of what fraction of people are poorly updating. The experience you express here about forecasting and other belief updating seems enormous.