Thanks that was a useful read.
Sam Abbott
Karma: 57
Thanks!
Our recent submission, “Training experts to be forecasters”, to the cause exploration prize may be of interest (I certainly found this post interesting as a justification for some of the ideas we experiment with).
https://forum.effectivealtruism.org/posts/WFbf2d4LHjgvWJCus/cause-exploration-prizes-training-experts-to-be-forecasters
Since we wrote this submission I have come across this nice forum piece looking at some of the evidence comparing generalist forecasters and domain experts. From my (biased) perspective it generally confirms some of the statements we make in this piece but also really highlights that more work is needed. I’d definitely suggest giving it a read.
https://forum.effectivealtruism.org/posts/qZqvBLvR5hX9sEkjR/comparing-top-forecasters-and-domain-experts
Some critical thoughts on this in this thread: https://twitter.com/nikosbosse/status/1562424476792672259?s=21&t=7SArpibug5ZqkEDhWi3AjA
It seems like there would be value in another pass at this kind of post with a more critical framing and perhaps wider more inclusive scope. It would be interesting to hear more on what EA as a community contributed and some critical reflections on those contributions especially contrasted to non-EA aligned work.
Something I’d be very interested in hearing more about is the choice to target Science and other legacy “high impact“ journals. In your other article on this you mention review delays and not being able to publish what you wished (note I think I was part of one of the review teams for one of these pieces of work (perhaps a later one—I don’t remember anything about CE but it had an entire second paper in the supplement so it was a mission to review) so perhaps I am biased to assume I did a great job).
I would have thought it would have been more effective to target a progressive open science journal given you relied so heavily on none traditional media channels to spread the findings. Obviously, the cynical reason many people go to “high impact” journals is prestige but there is also an argument that they help boost the legitimacy of findings. Given how much preprints were used during the pandemic it’s not clear to me that would be useful. Have you had a chance to do some analysis of altmetrics and the useful window of your findings to see what proportion of the impact came before and after publishing vs preprinting?
For 2 how important do you think forecasting is if those best suited to it (assuming experts are) shouldn’t be spending their time on it?
For ID settings early outbreak forecasts can be critical and the decisions made are often informed by local + international expert teams.