My answer to your question depends on how you define “good for the long-term future”. When I think about evaluating the chance an action is good including of long-run effects, specifying a few more dimensions matters to me. It feels like several combinations of these could be reasonable and would often lead to fairly different probabilities.
Expected value vs. realized value
Does “good for the long-term future” mean: good in expectation, or actually having good observed effects?
What is the ground truth evaluation?
Is the ground truth evaluation one that would be performed by:
An oracle that has all knowledge of all events?
The best evaluation that is in some sense realizable, e.g.:
A large (1,000 people?), well-organized team of competent people evaluating the action for a long time (1,000 years?)?
The best evaluation AI 100 years after AI surpasses human-level.
I think usually people mean (1), but in practice it often feels useful to me to think about some version of (2).
Foresight vs. hindsight
Does the ground truth evaluation occur before the action occurs, or after it occurs and all (or some) effects can be observed?
(Note: Using this as an excuse to ask this clarifying question that I’ve thought about some recently, that could apply to many posts. Haven’t done a thorough lit review on this so apologies if this is already covered somewhere else)
I came here to say this—in particular that I think my prior probability for “good things are good for the long-term future” might be very different than my prior for “good things are good for the long-term future in expectation”, so it matters a lot which is being asked.
I think the former is probably much closer to 50% than the latter. These aren’t my actual estimates, but for illustrative purposes I think the numbers might be something like 55% and 90%.
I agree with Eli that my actual estimates would also depend on the other questions Eli raises.
Another factor that might affect my prior a lot is what the reference class of “good things” looks like. In particular, are we weighting good things based on how often these good things are done / how much money is spent on them, or weighting them once per unique thing as if someone were generating a list of good things? E.g. Does “donation to a GiveWell top charity” count a lot, or once? (Linch’s wording at the end of the post makes it seem like he means the latter.)
Perhaps it would be helpful to Linch’s question to generate a list of 10-20 “good things” and then actually think about each one carefully and estimate the probability that it is good for the future, and good for the future in expectation, and use these 10-20 data points to estimate what one’s prior should be. (Any thoughts on whether this would be a worthwhile research activity, Linch or others reading this?)
Perhaps it would be helpful to Linch’s question to generate a list of 10-20 “good things” and then actually think about each one carefully and estimate the probability that it is good for the future, and good for the future in expectation, and use these 10-20 data points to estimate what one’s prior should be. (Any thoughts on whether this would be a worthwhile research activity, Linch or others reading this?)
For what is it worth, I think it would be worthwhile.
My answer to your question depends on how you define “good for the long-term future”. When I think about evaluating the chance an action is good including of long-run effects, specifying a few more dimensions matters to me. It feels like several combinations of these could be reasonable and would often lead to fairly different probabilities.
Expected value vs. realized value
Does “good for the long-term future” mean: good in expectation, or actually having good observed effects?
What is the ground truth evaluation?
Is the ground truth evaluation one that would be performed by:
An oracle that has all knowledge of all events?
The best evaluation that is in some sense realizable, e.g.:
A large (1,000 people?), well-organized team of competent people evaluating the action for a long time (1,000 years?)?
The best evaluation AI 100 years after AI surpasses human-level.
I think usually people mean (1), but in practice it often feels useful to me to think about some version of (2).
Foresight vs. hindsight
Does the ground truth evaluation occur before the action occurs, or after it occurs and all (or some) effects can be observed?
(Note: Using this as an excuse to ask this clarifying question that I’ve thought about some recently, that could apply to many posts. Haven’t done a thorough lit review on this so apologies if this is already covered somewhere else)
I came here to say this—in particular that I think my prior probability for “good things are good for the long-term future” might be very different than my prior for “good things are good for the long-term future in expectation”, so it matters a lot which is being asked.
I think the former is probably much closer to 50% than the latter. These aren’t my actual estimates, but for illustrative purposes I think the numbers might be something like 55% and 90%.
I agree with Eli that my actual estimates would also depend on the other questions Eli raises.
Another factor that might affect my prior a lot is what the reference class of “good things” looks like. In particular, are we weighting good things based on how often these good things are done / how much money is spent on them, or weighting them once per unique thing as if someone were generating a list of good things? E.g. Does “donation to a GiveWell top charity” count a lot, or once? (Linch’s wording at the end of the post makes it seem like he means the latter.)
Perhaps it would be helpful to Linch’s question to generate a list of 10-20 “good things” and then actually think about each one carefully and estimate the probability that it is good for the future, and good for the future in expectation, and use these 10-20 data points to estimate what one’s prior should be. (Any thoughts on whether this would be a worthwhile research activity, Linch or others reading this?)
Hi William,
For what is it worth, I think it would be worthwhile.