Somewhat relatedly, do you know what the results would be if we model noise as increasing sublinearly (rather than linearly or superlinearly), in a situation where the signal isnāt suggesting the longtermist intervention has benefits for an infinite length of time?
Obviously this would increase the relative value of the longtermist intervention compared to the neartermist one, but I wonder if there are other implications as well, if the difference that that would make would be huge or moderate or small, what that depends on, etc.
One reason this seems interesting to me is that Iād currently guess that noise does indeed increase sublinearly. This is based on two things:
I think a small amount of data from Tetlock gives weak evidence of this, if Iām interpreting it and Muehlhauserās commentary on it (here) correctly.
See in particular footnote 17 from that link. Hereās the key quote (but without the useful graph, methodological info, etc.):
āFor our purposes here, the key results shown above are, roughly speaking, that (1) regular forecasters did approximately no better than chance on this metric at ~375 days before each question closed, (2) superforecasters did substantially better than chance on this metric at ~375 days before each question closed, (3) both regular forecasters and superforecasters were almost always āon the right side of maybeā immediately before each question closed, and (4) superforecasters were roughly as accurate on this metric at ~125 days before each question closed as they were at ~375 days before each question closed.
If GJP had involved questions with substantially longer time horizons, how quickly would superforecaster accuracy declined with longer time horizons? We canāt know, but an extrapolation of the results above is at least compatible with an answer of āfairly slowly.āā
I think that, a priori, Iād predict that noise would increase sublinearly. But I havenāt tried to work out whatās driving that intuition, and itās possible that itās just that Iāve now seen that data from Tetlock and Muehlhauserās commentary.
Somewhat relatedly, do you know what the results would be if we model noise as increasing sublinearly (rather than linearly or superlinearly), in a situation where the signal isnāt suggesting the longtermist intervention has benefits for an infinite length of time?
Obviously this would increase the relative value of the longtermist intervention compared to the neartermist one, but I wonder if there are other implications as well, if the difference that that would make would be huge or moderate or small, what that depends on, etc.
One reason this seems interesting to me is that Iād currently guess that noise does indeed increase sublinearly. This is based on two things:
I think a small amount of data from Tetlock gives weak evidence of this, if Iām interpreting it and Muehlhauserās commentary on it (here) correctly.
See in particular footnote 17 from that link. Hereās the key quote (but without the useful graph, methodological info, etc.):
āFor our purposes here, the key results shown above are, roughly speaking, that (1) regular forecasters did approximately no better than chance on this metric at ~375 days before each question closed, (2) superforecasters did substantially better than chance on this metric at ~375 days before each question closed, (3) both regular forecasters and superforecasters were almost always āon the right side of maybeā immediately before each question closed, and (4) superforecasters were roughly as accurate on this metric at ~125 days before each question closed as they were at ~375 days before each question closed.
If GJP had involved questions with substantially longer time horizons, how quickly would superforecaster accuracy declined with longer time horizons? We canāt know, but an extrapolation of the results above is at least compatible with an answer of āfairly slowly.āā
I think that, a priori, Iād predict that noise would increase sublinearly. But I havenāt tried to work out whatās driving that intuition, and itās possible that itās just that Iāve now seen that data from Tetlock and Muehlhauserās commentary.