I’m not advocating for “poorly done quantitative estimates.” I think anyone reasonable would admit that it’s possible to bungle them.
I’m definitely not happy with a local optimum of “not having estimates”. It’s possible that “having a few estimates” can be worse, but I imagine we’ll want to get to the point of “having lots of estimates, and becoming more mature to be able to handle them.” at some point, so that’s the direction to aim for.
I think the “local vs global optima” framing is an interesting way of looking at it.
That reminds me of some of my thinking when I was trying to work out whether it’d be net positive to make that database of existential risk estimates (vs it being net negative due to anchoring, reputational issues to EA/longtermists, etc.). In particular, a big part of my reasoning was something like:
It’s plausible that it’s worse for this database to exist than for there to be no public existential risk estimates. But what really matters is whether it’s better that this database exist than that there be a small handful of existential risk estimates, scattered in various different places, and with people often referring to only one set in a given instance (e.g., the 2008 FHI survey), sometimes as if it’s the ‘final word’ on the matter.
That situation seems probably even worse from an anchoring and reputational perspective than there being a database. This is because seeing a larger set of estimates side by side could help people see how much disagreement there is and thus have a more appropriate level of uncertainty and humility.
With your comment in mind, I’d now add:
But all of that is just about how good various different present-day situations would be. We should also consider what position we ultimately want to reach.
It seems plausible that we could end up with a larger set of more trustworthy and more independently-made existential risk estimates. And it seems likely that this would be better than the situation we’re in now.
Furthermore, it seems plausible that making this database moves us a step towards that destination. This could be a reason to make the database, even if doing so was slightly counterproductive in the short term.
Nice post. I think I agree with all of that.
I’m not advocating for “poorly done quantitative estimates.” I think anyone reasonable would admit that it’s possible to bungle them.
I’m definitely not happy with a local optimum of “not having estimates”. It’s possible that “having a few estimates” can be worse, but I imagine we’ll want to get to the point of “having lots of estimates, and becoming more mature to be able to handle them.” at some point, so that’s the direction to aim for.
I think the “local vs global optima” framing is an interesting way of looking at it.
That reminds me of some of my thinking when I was trying to work out whether it’d be net positive to make that database of existential risk estimates (vs it being net negative due to anchoring, reputational issues to EA/longtermists, etc.). In particular, a big part of my reasoning was something like:
With your comment in mind, I’d now add: