Imagine we can divide up the global economy into natural clusters. We’ll refer to each cluster as a “Global Project.” Each Global Project consists of people and their ideas, material resources, institutional governance, money, incentive structures, and perhaps other factors.
Some Global Projects seem “bad” on the whole. They might have directly harmful goals, irresponsible risk management, poor governance, or many other failings. Others seem “good” on net. This is not in terms of expected value for the world, but in terms of the intrinsic properties of the GP that will produce that value.
It might be reasonable to assume that Global Project quality is normally distributed. One point of possible difference is the center of that distribution. Are most Global Projects of bad quality, neutral, or good quality?
We might make a further assumption that the expected value of a Global Project follows a power law, such that projects of extremely low or high quality produce exponentially more value (or more harm). Perhaps, if Q is project quality and V is value, V=QN. But we might disagree on the details of this power law.
One possibility is that in fact, it’s easier to destroy the world than to improve the world. We might model this with two power laws, one for Q > 0 and one for Q < 0, like so:
V=Q3, Q >= 0
V=Q7, Q < 0
In this case, whether or not progress is good will depend on the details of our assumptions about both the project quality distribution and the power law for expected value:
The size of N, and whether or not the power law is uniform or differs for projects of various qualities. Intuitively, “is it easier for a powerful project to improve or destroy the world, and how much easier?”
How many standard deviations away from zero the project quality distribution is centered, and in which direction. Intuitively, “are most projects good or bad, and how much?”
In this case, whether or not average expected value across many simulations of such a model is positive or negative can hinge on small alterations of the variables. For example, if we set N = 7 for bad projects and N = 3 for good projects, but we assume that the average project quality is +0.6 standard deviations from zero, then average expected value is mildly negative. At project quality +0.7 standard deviations from zero, the average expected value is mildly positive.
Here’s what an X-risk “we should slow down” perspective might look like. Each plotted point is a simulated “world.” In this case, the simulation produces negative average EV across simulated worlds.
And here is a Progress Studies “we should speed up” perspective might look like, with positive average EV.
The joke is that it’s really hard to tell these two simulations apart. In fact, I generated the second graph by altering the center point of the project quality distribution 0.01 standard deviations to the right relative to the first graph. In both case, a lot of the expected value is lost to a few worlds in which things go cataclysmically wrong.
One way to approach a double crux would be for adherents of the two sides to specify, in the spirit of “if it’s worth doing, it’s worth doing with made up statistics,” their assumptions about the power law and project quality distribution, then argue about that. Realistically, though, I think both sides understand that we don’t have any realistic way of saying what those numbers ought to be. Since the details matter on this question, it seems to me that it would be valuable to find common ground.
For example, I’m sure that PS advocates would agree that there are some targeted risk-reduction efforts that might be good investments, along with a larger class of progress-stimulating interventions. Likewise, I’m sure that XR advocates would agree that there are some targeted tech-stimulus projects that might be X-risk “security factors.” Maybe the conversation doesn’t need to be about whether “more progress” or “less progress” is desirable, but about the technical details of how we can manage risk while stimulating growth.
Imagine we can divide up the global economy into natural clusters. We’ll refer to each cluster as a “Global Project.” Each Global Project consists of people and their ideas, material resources, institutional governance, money, incentive structures, and perhaps other factors.
Some Global Projects seem “bad” on the whole. They might have directly harmful goals, irresponsible risk management, poor governance, or many other failings. Others seem “good” on net. This is not in terms of expected value for the world, but in terms of the intrinsic properties of the GP that will produce that value.
It might be reasonable to assume that Global Project quality is normally distributed. One point of possible difference is the center of that distribution. Are most Global Projects of bad quality, neutral, or good quality?
We might make a further assumption that the expected value of a Global Project follows a power law, such that projects of extremely low or high quality produce exponentially more value (or more harm). Perhaps, if Q is project quality and V is value, V=QN. But we might disagree on the details of this power law.
One possibility is that in fact, it’s easier to destroy the world than to improve the world. We might model this with two power laws, one for Q > 0 and one for Q < 0, like so:
V=Q3, Q >= 0
V=Q7, Q < 0
In this case, whether or not progress is good will depend on the details of our assumptions about both the project quality distribution and the power law for expected value:
The size of N, and whether or not the power law is uniform or differs for projects of various qualities. Intuitively, “is it easier for a powerful project to improve or destroy the world, and how much easier?”
How many standard deviations away from zero the project quality distribution is centered, and in which direction. Intuitively, “are most projects good or bad, and how much?”
In this case, whether or not average expected value across many simulations of such a model is positive or negative can hinge on small alterations of the variables. For example, if we set N = 7 for bad projects and N = 3 for good projects, but we assume that the average project quality is +0.6 standard deviations from zero, then average expected value is mildly negative. At project quality +0.7 standard deviations from zero, the average expected value is mildly positive.
Here’s what an X-risk “we should slow down” perspective might look like. Each plotted point is a simulated “world.” In this case, the simulation produces negative average EV across simulated worlds.
And here is a Progress Studies “we should speed up” perspective might look like, with positive average EV.
The joke is that it’s really hard to tell these two simulations apart. In fact, I generated the second graph by altering the center point of the project quality distribution 0.01 standard deviations to the right relative to the first graph. In both case, a lot of the expected value is lost to a few worlds in which things go cataclysmically wrong.
One way to approach a double crux would be for adherents of the two sides to specify, in the spirit of “if it’s worth doing, it’s worth doing with made up statistics,” their assumptions about the power law and project quality distribution, then argue about that. Realistically, though, I think both sides understand that we don’t have any realistic way of saying what those numbers ought to be. Since the details matter on this question, it seems to me that it would be valuable to find common ground.
For example, I’m sure that PS advocates would agree that there are some targeted risk-reduction efforts that might be good investments, along with a larger class of progress-stimulating interventions. Likewise, I’m sure that XR advocates would agree that there are some targeted tech-stimulus projects that might be X-risk “security factors.” Maybe the conversation doesn’t need to be about whether “more progress” or “less progress” is desirable, but about the technical details of how we can manage risk while stimulating growth.