I don’t know if I understand why tractability doesn’t vary much. It seems like it should be able to vary just as much as cost-effectiveness can vary.
For example, imagine two problems with the same cost-effectiveness, the same importance, but one problem has 1000x fewer resources invested in it. Then the tractability of that problem should be 1000x higher [ETA: so that the cost-effectiveness can still be the same, even given the difference in neglectedness.]
Another example: suppose an AI safety researcher solved AI alignment after 20 years of research. Then the two problems “solve the sub-problem which will have been solved by tomorrow” and “solve AI alignment” have the same local cost-effectiveness (since they are locally the same actions), the same amount of resources invested into each, but potentially massively different importances. This means the tractabilities must also be massively different.
These two examples lead me to believe that in as much as tractability doesn’t vary much, it’s because of a combination of two things:
The world isn’t dumb enough to massively underinvest in a really cost-effective and important problems
The things we tend to think of as problems are “similarly sized” or something like that
I’m still not fully convinced, though, and am confused for instance about what “similarly sized” might actually mean.
Problems vary on three axes: u’(R), R, u_total. You’re expressing this in the basis u’(R), R, u_total. The ITN framework uses the basis I, T, N = u_total, u’(R) * R * 1/ u_total, 1/R. The basis is arbitrary: we could just as easily use some crazy basis like X, Y, Z = u_total^2, R^5, sqrt(u’(R)). But we want to use a basis that’s useful in practice, which means variables with intuitive meanings, hence ITN.
But why is tractability roughly constant with neglectedness in practice? Equivalently, why are there logarithmic returns to many problems? I don’t think it’s related to your (1) or (2) because those are about comparing different problems, whereas the mystery is the relationship between u’(R) * R * 1/ u_total and 1/R for a given problem. One model that suggests log returns is if we have to surpass some unknown resource threshold r* (the “difficulty”) to solve the problem, and r* ranges over many orders of magnitude with an approximately log-uniform distribution [1]. Owen C-B has empirical evidence and some theoretical justification for why this might happen in practice. When it does, my post then derives that tractability doesn’t vary dramatically between (most) problems.
Note that sometimes we know r* or have a different prior, and then our problem stops being logarithmic, like in the second section of the post. This is exactly when tractability can vary dramatically between problems. In your AI alignment subproblem example, we know that alignment takes 20 years, which means a strong update away from the logarithmic prior.
[1]: log-uniform distributions over the reals don’t exist, so I mean something like “doesn’t vary by more than ~1.5x every doubling in the fat part of the distribution”.
The entire time I’ve been thinking about this, I’ve been thinking of utility curves as logarithmic, so you don’t have to sell me on that. I think my original comment here is another way of understanding why tractability perhaps doesn’t vary much between problems, not within a problem.
For example, imagine two problems with the same cost-effectiveness, the same importance, but one problem has 1000x fewer resources invested in it. Then the tractability of that problem should be 1000x higher.
In the ITN framework, this will be modeled under “neglectedness” rather than “tractability”
I don’t know if I understand why tractability doesn’t vary much. It seems like it should be able to vary just as much as cost-effectiveness can vary.
For example, imagine two problems with the same cost-effectiveness, the same importance, but one problem has 1000x fewer resources invested in it. Then the tractability of that problem should be 1000x higher [ETA: so that the cost-effectiveness can still be the same, even given the difference in neglectedness.]
Another example: suppose an AI safety researcher solved AI alignment after 20 years of research. Then the two problems “solve the sub-problem which will have been solved by tomorrow” and “solve AI alignment” have the same local cost-effectiveness (since they are locally the same actions), the same amount of resources invested into each, but potentially massively different importances. This means the tractabilities must also be massively different.
These two examples lead me to believe that in as much as tractability doesn’t vary much, it’s because of a combination of two things:
The world isn’t dumb enough to massively underinvest in a really cost-effective and important problems
The things we tend to think of as problems are “similarly sized” or something like that
I’m still not fully convinced, though, and am confused for instance about what “similarly sized” might actually mean.
Problems vary on three axes: u’(R), R, u_total. You’re expressing this in the basis u’(R), R, u_total. The ITN framework uses the basis I, T, N = u_total, u’(R) * R * 1/ u_total, 1/R. The basis is arbitrary: we could just as easily use some crazy basis like X, Y, Z = u_total^2, R^5, sqrt(u’(R)). But we want to use a basis that’s useful in practice, which means variables with intuitive meanings, hence ITN.
But why is tractability roughly constant with neglectedness in practice? Equivalently, why are there logarithmic returns to many problems? I don’t think it’s related to your (1) or (2) because those are about comparing different problems, whereas the mystery is the relationship between u’(R) * R * 1/ u_total and 1/R for a given problem. One model that suggests log returns is if we have to surpass some unknown resource threshold r* (the “difficulty”) to solve the problem, and r* ranges over many orders of magnitude with an approximately log-uniform distribution [1]. Owen C-B has empirical evidence and some theoretical justification for why this might happen in practice. When it does, my post then derives that tractability doesn’t vary dramatically between (most) problems.
Note that sometimes we know r* or have a different prior, and then our problem stops being logarithmic, like in the second section of the post. This is exactly when tractability can vary dramatically between problems. In your AI alignment subproblem example, we know that alignment takes 20 years, which means a strong update away from the logarithmic prior.
[1]: log-uniform distributions over the reals don’t exist, so I mean something like “doesn’t vary by more than ~1.5x every doubling in the fat part of the distribution”.
The entire time I’ve been thinking about this, I’ve been thinking of utility curves as logarithmic, so you don’t have to sell me on that. I think my original comment here is another way of understanding why tractability perhaps doesn’t vary much between problems, not within a problem.
I don’t see why logarithmic utility iff tractability doesn’t change with neglectedness.
In the ITN framework, this will be modeled under “neglectedness” rather than “tractability”
There was an inference there—you need tractability to balance with the neglectedness to add up to equal cost-effectiveness