A related issue is that people may be more comfortable making predictions about less important aspects of the project, since the consequences of being wrong are lower
I’m actually concerned about the same thing but for exactly the opposite reason, i.e. that because the consequences of being wrong (a hit to one’s Brier score) are the same regardless of the importance of the prediction people might allocate the same time and effort to any prediction, including the more important ones that should perhaps warrant closer examination.
We’re currently trialing some of the stuff you suggest about bringing in other people to suggest predictions. This might be an improvement, but it’s too early to say, and scaling it up wouldn’t be easy for a few reasons:
It’s hard to make good predictions about a grant without lots of context.
Grant investigators are very time-constrained, so they can’t afford to provide that context by having a lot of back and forth with the person suggesting the predictions.
Most of the information needed to gain context about the grant is confidential by default.
I’m actually concerned about the same thing but for exactly the opposite reason, i.e. that because the consequences of being wrong (a hit to one’s Brier score) are the same regardless of the importance of the prediction people might allocate the same time and effort to any prediction, including the more important ones that should perhaps warrant closer examination.
We’re currently trialing some of the stuff you suggest about bringing in other people to suggest predictions. This might be an improvement, but it’s too early to say, and scaling it up wouldn’t be easy for a few reasons:
It’s hard to make good predictions about a grant without lots of context.
Grant investigators are very time-constrained, so they can’t afford to provide that context by having a lot of back and forth with the person suggesting the predictions.
Most of the information needed to gain context about the grant is confidential by default.