Can’t think of anything better than a t-test, but open for suggestions.
If a forecaster is consistently off by like 10 percentage points—I think that is a difference that matters. But even in that extreme scenario where the (simulated) difference between two forecasters is in fact quite large, we have a hard time picking that up using standard significance tests.
Interesting, thanks for sharing the paper. Yeah agree that using the Brier score / log score might change results and it would definitely be good to check that as well.
In principle yes. In practice also usually yes, but the specifics depend on whether the average user who predicted on a question gets a positive amount of points. So if you predicted very late and your points are close to zero, but the mean number of points forecasters on that question received is positive, then you will end up with a negative update to your reputation score.
Completely agree that a lot hinges on that reputation score. It seems to work decent for the Metaculus Prediction, but it would be good to see what results look like for a different metric of past performance.
Not sure how to quantify that (open for ideas). But intuitively I agree with you and would suspect it’s at least a sizable part
Yeah, definitely. The title was a bit tongue-in-cheek (it’s a movie quote)
And is the code to the MetaculusBot public somewhere? :)
It should be possible to fully automate the bot and just run a CRON job that regularly checks the Metaculus API for new questions, right?
I slightly tend towards yes, but that’s mere intuition. As someone on Twitter put it, “Metaculus has a more hardcore user base, because it’s less fun”—I find it plausible that the Metaculus user base and the Manifold user base differs. But higher trading volume I think would have helped.
For this particular analysis I’m not sure correcting for the number of forecasters would really be possible in a sound way. It would be great to get the MetaculusBot more active again to collect more data.
Is it possible to get rid of the question mode for this post?
For Metaculus there are lots of ways to drive engagement: prioritise making the platform easier to use, increase cash prizes, community building and outreach etc.
But as mentioned in the article the problem in practice is that the bootstrap answer is probably misleading, as increasing the number of forecasters likely changes forecaster composition.
However, one specific example where the analysis might be actually applicable is when you’re thinking about how many Pro Forecasters you hire for a job.
In principle yes, you’ll just still always have the problem that people are predicting at different time points. If the best and the 2nd best predict weeks or months apart then that changes results.
Ah snap! I forgot to remove that paragraph… I did subsampling initially, then switched to bootstrapipng. Resulsts remained virtually unchanged. Thanks for pointing that out, will update the text.
Hi Simon, I’m working on a follow-up to this post that uses individual-level data. Could you please give some detail on how you “sampled” k predictors? As in, did you have access to individual data and could actually do the sampling? I’m not entirely sure what the x-axis in your plot means and what the difference betwenn “>N predictors” and “k predictors” is. Thank you!
I acknowledge that transparency is complex, that there are trade-offs and that it isn’t clear what the correct amount of transparency is. I also acknowledge that it is normal that regular grants are published with a delay. So I’m not making a general claim or demand that everything needs to be public (I even explicitly say that). What I say is that
a) I’m in favour of valuing transparency highly by default.
b) I feel in this specific case more communication would have helped
My intuition is that Open Phil overall is quite transparent. I was less sure about EVF, given that there are a relevant number of comments and posts on the forum criticising them for lack of transparency. But my data point is mostly this one where I feel more openness would have been helpful plus reading other people’s comments.
Given that, I think it’s especially weird to say EVF hasn’t said anything while linking to an explanation by the board member who was most involved.
I feel that characterisation is a bit uncharitable. I said
So far CEA/EVF hasn’t come out explaining their rationale in detail (albeit some of those involved in the decision have commented under an EA Forum Post asking about the decision and shared their perspectives)
At the time when Émile posted on Twitter, this was definitely true. And even today I find it fair to say that Owen’s response was not a detailed explanation by CEA/EVF (maybe you’re right and it’s Open Phil that should do the explaining, but that is not the main point). Owen’s comment was a response to a question / criticism by someone else, was a comment under a post by someone else and felt mostly like a personal account to me. And he doesn’t give much information on the assumptions involved, from his post it wasn’t even clear where the money came from. To be clear: I’m glad he posted his thoughts. But I think it’s also fair to say that this isn’t the kind of official announcement or detailed explanation targeted to convince someone who isn’t already mostly convinced that the decision was reasonable.
I think your criticism of bikeshedding somewhat misses the point people are raising. Of course the amount of money spent on WA is tiny compared to other things. The reason it’s worth talking about it is that it tells you something about EA culture and how EA operates.
This is in large parts a discussion about what culture the movement should have, what EA wants to be and how it wants to communicate to the world. The reason you care about how someone builds a bike shed is because that carries information about what kind of person they are, how trustworthy they are and how they might approach building a reactor.
The discussion is 1-2 steps away from the actual decision precisely because
a) we don’t have the information necessary to have a discussion on the actual decision (i.e. the numbers) and
b) things like decision processes are important.
There is a PR element to my original post—I think CEA/EVF could have communicated smarter and it’s important to point that out. But what is at heart of all this is not at all about PR, spin and appearance. Not about appearing virtuous, but about being virtuous. It’s about processes, what culture EA should foster and the value of transparency.
$1000 per person per day just for the place seems pretty expensive for a 30 person conference…
I’m confused why you wouldn’t feel concerned about EA potentially wasting 15M pounds (talking about your hypothetical example, not the real purchase). I feel that would mean that EA is not living up to its own standards of using evidence and reasoning to help others in the best possible way.
Currently there are 4 people (including me) working on the project. I focus on coordination, the other three are professional forecasters and focus on the data collection. At the moment we’re aiming for wide feedback from anyone who would be interested in certain base rates, but we’re not actively crowd-sourcing the collection process.
Do you mean have the table be in a long-format rather than a wide-format?
Good comment, thank you!