My credence could be that working on AI safety will reduce existential risk by 5% and yours could be 10^-19%, and there’s no way to discriminate between them.
We can look at their track record on other questions, and see how reliably (or otherwise) different people’s predictions track reality.
I agree that below a certain level (certainly by 10^-19, and possibly as high as 10^-3) direct calibration-in-practice becomes somewhat meaningless. But we should be pretty suspicious of people claiming extremely accurate probabilities at the 10^-10 mark if they aren’t even accurate at the 10^-1 mark.
In general I’m not a fan of this particular form of epistemic anarchy where people say that they can’t know anything with enough precision under uncertainty to give numbers, and then act as if their verbal non-numeric intuitions are enough to carry them through consistently making accurate decisions.
It’s easy to lie (including to yourself) with numbers, but it’s even easier to lie without them.
We can look at their track record on other questions, and see how reliably (or otherwise) different people’s predictions track reality.
I’d rather not rely on the authority of past performance to gauge whether someone’s arguments are good. I think we should evaluate the arguments directly. If they are, they’ll stand on their own regardless of someone’s prior luck/circumstance/personality.
In general I’m not a fan of this particular form of epistemic anarchy where people say that they can’t know anything with enough precision under uncertainty to give numbers, and then act as if their verbal non-numeric intuitions are enough to carry them through consistently making accurate decisions.
I would actually argue that it’s the opposite of epistemic anarchy. Admitting that we can’t know the unknowable changes our decision calculus: Instead of focusing on making the optimal decision, we recognize that all decisions will have unintended negative consequences which we’ll have to correct. Fostering an environment of criticism and error-correction becomes paramount.
It’s easy to lie (including to yourself) with numbers, but it’s even easier to lie without them.
I’d disagree. I think trying to place probabilities on inherently unknowable events lends us a false sense of security.
The potential downsides I cover include causing overconfidence, underestimating the value of information, and anchoring, among other things that are less directly related to your point. That said, I ultimately conclude that:
There are some real downsides that can occur in practice when actual humans use [explicit probabilities] (or [explicit probabilistic models], or maximising expected utility)
But some downsides that have been suggested (particularly causing overconfidence and understating the [value of information]) might actually be more pronounced for approaches other than using [explicit probabilities]
Some downsides (particularly relating to the optimizer’s curse, anchoring, and reputational issues) may be more pronounced when the probabilities one has (or could have) are less trustworthy
Other downsides (particularly excluding one’s intuitive knowledge) may be more pronounced when the probabilities one has (or could have) are more trustworthy
Only one downside (reputational issues) seems to provide any argument for even acting as if there’s a binary risk-uncertainty distinction
And even in that case the argument is quite unclear, and wouldn’t suggest we should use the idea of such a distinction inour own thinking
The above point, combined with arguments I made in an earlier post, makes me believe that we should abandon the concept of the risk-uncertainty distinction in our own thinking (and at least most communication), and that we should think instead in terms of:
a continuum of more to less trustworthy probabilities
the practical upsides and downsides of using [explicit probabilities], for actual humans. [emphasis added]
Relatedly, I think it’s not at all obvious that putting numbers on things, forecasting, etc. would tend to get in the way of “Fostering an environment of criticism and error-correction becomes paramount”. (It definitely could get in the way sometimes; it depends on the details.) There are various reasons why putting numbers on things and making forecasts can be actively helpful in fostering such an environment (some of which I discuss in my post).
[Disclaimer that I haven’t actually read your post yet—sorry! - though I may do so soon :)]
I’d rather not rely on the authority of past performance to gauge whether someone’s arguments are good. I think we should evaluate the arguments directly. If they are, they’ll stand on their own regardless of someone’s prior luck/circumstance/personality.
I agree that we should often/usually evaluate arguments directly. But:
We have nowhere near enough time to properly evaluate all arguments relevant to our decisions. And in some cases, we also lack the relevant capabilities. So in effect, it’s often necessary and/or wise to base certain beliefs mostly on what certain other people seem to believe.
For example, I don’t actually know that much about how climate science works, and my object-level understanding of the arguments for climate change being real, substantial, and anthropogenic are too shallow for me to be confident—on that basis alone—that those conclusions are correct. (I think a clever person could’ve made false claims about climate science sound similarly believable to me, if they’d been motivated to do so and I’d only looked into it to the extent that I have.)
The same is more strongly true for people with less education and intellectual curiosity than me.
But it’s good for us to default to being fairly confident that things most relevant scientists agree are true are indeed true.
The same basic point is even more clearly true when it comes to things like the big bang or the fact that dinosaurs existed and when they did so
We can both evaluate arguments directly and consider people’s track records
We could also evaluate the “meta argument” that “people who have been shown to be decent forecasters (or better forecasters than other people are) on relatively short time horizons will also be at least slightly ok forecasts (or at least slightly better forecasters than other people are) on relatively long time horizons”
Evaluating that argument directly, I think we should land on “This seems more likely to be true than not, though there’s still room for uncertainty”
Another way of making a perhaps similar point is that it very often makes sense to see past outcomes from some person/object/process or whatever as at least a weak indicator of what the future outcomes from that same thing will be
E.g., the more often a car has failed to start up properly in the past, the more often we should expect it to do so in future
E.g., the more a person has done well at a job in the past, the more we should expect them to do well at that job or similar jobs in future
It’s not clear why this would fail to be the case for forecasting
And indeed, there is empirical evidence that it is the case for forecasting
That said, there is the issue that we’re comparing forecasts over short time horizons to forecasts over long time horizons, and that does introduce some more room for doubt, as noted above
What Linch was talking about seems very unlikely to boil down to just “someone’s prior luck/circumstance/personality”.
Actual track records would definitely not be a result of personality except inasmuch as personality is actually relevant to better performance (e.g. via determination to work hard at forecasting).
They’re very likely partly due to luck, but the evidence shows that some forecasters tend to do better over a large enough set of questions that it can’t be just due to luck (I have in mind Tetlock’s work).
We can look at their track record on other questions, and see how reliably (or otherwise) different people’s predictions track reality.
I agree that below a certain level (certainly by 10^-19, and possibly as high as 10^-3) direct calibration-in-practice becomes somewhat meaningless. But we should be pretty suspicious of people claiming extremely accurate probabilities at the 10^-10 mark if they aren’t even accurate at the 10^-1 mark.
In general I’m not a fan of this particular form of epistemic anarchy where people say that they can’t know anything with enough precision under uncertainty to give numbers, and then act as if their verbal non-numeric intuitions are enough to carry them through consistently making accurate decisions.
It’s easy to lie (including to yourself) with numbers, but it’s even easier to lie without them.
Hi Linch!
I’d rather not rely on the authority of past performance to gauge whether someone’s arguments are good. I think we should evaluate the arguments directly. If they are, they’ll stand on their own regardless of someone’s prior luck/circumstance/personality.
I would actually argue that it’s the opposite of epistemic anarchy. Admitting that we can’t know the unknowable changes our decision calculus: Instead of focusing on making the optimal decision, we recognize that all decisions will have unintended negative consequences which we’ll have to correct. Fostering an environment of criticism and error-correction becomes paramount.
I’d disagree. I think trying to place probabilities on inherently unknowable events lends us a false sense of security.
(All said with a smile of course :) )
You or other readers might find this post of mine from last year of interest: Potential downsides of using explicit probabilities.
The potential downsides I cover include causing overconfidence, underestimating the value of information, and anchoring, among other things that are less directly related to your point. That said, I ultimately conclude that:
Relatedly, I think it’s not at all obvious that putting numbers on things, forecasting, etc. would tend to get in the way of “Fostering an environment of criticism and error-correction becomes paramount”. (It definitely could get in the way sometimes; it depends on the details.) There are various reasons why putting numbers on things and making forecasts can be actively helpful in fostering such an environment (some of which I discuss in my post).
[Disclaimer that I haven’t actually read your post yet—sorry! - though I may do so soon :)]
I agree that we should often/usually evaluate arguments directly. But:
We have nowhere near enough time to properly evaluate all arguments relevant to our decisions. And in some cases, we also lack the relevant capabilities. So in effect, it’s often necessary and/or wise to base certain beliefs mostly on what certain other people seem to believe.
For example, I don’t actually know that much about how climate science works, and my object-level understanding of the arguments for climate change being real, substantial, and anthropogenic are too shallow for me to be confident—on that basis alone—that those conclusions are correct. (I think a clever person could’ve made false claims about climate science sound similarly believable to me, if they’d been motivated to do so and I’d only looked into it to the extent that I have.)
The same is more strongly true for people with less education and intellectual curiosity than me.
But it’s good for us to default to being fairly confident that things most relevant scientists agree are true are indeed true.
The same basic point is even more clearly true when it comes to things like the big bang or the fact that dinosaurs existed and when they did so
See also epistemic humility
We can both evaluate arguments directly and consider people’s track records
We could also evaluate the “meta argument” that “people who have been shown to be decent forecasters (or better forecasters than other people are) on relatively short time horizons will also be at least slightly ok forecasts (or at least slightly better forecasters than other people are) on relatively long time horizons”
Evaluating that argument directly, I think we should land on “This seems more likely to be true than not, though there’s still room for uncertainty”
See also How Feasible Is Long-range Forecasting?, and particularly footnote 17
Another way of making a perhaps similar point is that it very often makes sense to see past outcomes from some person/object/process or whatever as at least a weak indicator of what the future outcomes from that same thing will be
E.g., the more often a car has failed to start up properly in the past, the more often we should expect it to do so in future
E.g., the more a person has done well at a job in the past, the more we should expect them to do well at that job or similar jobs in future
It’s not clear why this would fail to be the case for forecasting
And indeed, there is empirical evidence that it is the case for forecasting
That said, there is the issue that we’re comparing forecasts over short time horizons to forecasts over long time horizons, and that does introduce some more room for doubt, as noted above
What Linch was talking about seems very unlikely to boil down to just “someone’s prior luck/circumstance/personality”.
Actual track records would definitely not be a result of personality except inasmuch as personality is actually relevant to better performance (e.g. via determination to work hard at forecasting).
They’re very likely partly due to luck, but the evidence shows that some forecasters tend to do better over a large enough set of questions that it can’t be just due to luck (I have in mind Tetlock’s work).