I disagree with Roodman’s criticism quoted here. Cotra’s approach involves estimating that there’s an X% chance that the first achievable TAI will look like A, a Y% chance like B, and so on. Some anchors (e.g., short-horizon neural network and long-horizon neural network) are obviously incompatible; whatever the future looks like, they won’t both describe the first achievable TAI. Multiplying them is clearly not meaningful; Roodman’s proposed “restriction that the various frameworks agree” makes no sense. (Multiplying them would be correct if Cotra’s different anchors represented something like different information-sources on necessary-and-sufficient-conditions-for-TAI, but that’s not what her anchors represent.)
Roodman’s proposed “restriction that the various frameworks agree” makes no sense.
I’m with you. I think Roodman must disagree with the idea of giving probabilies to different (and necessarily conflicting) models of the world, but to me this seems like an odd position to hold. I might also be missing something.
Agree with what you’re saying. This part of the review in particular stood out to me:
In pure Bayesian reasoning, if one has several uncertain measurements of the same value, each represented by a probability distribution...
Since Cotra isn’t presenting the different anchors as all-things-considered estimates, but instead more like different hypotheses. Consider the evolutionary anchor – Cotra could have divided the compute requirements in this anchor by a scaling factor for how much more efficient she believes human-directed SGD (or similar) will be compared to how efficient evolution was at finding intelligence, yielding an all-things-considered estimate of how much compute will be necessary for TAI, but instead she leaves the value as is and considers it a soft upper bound.
I disagree with Roodman’s criticism quoted here. Cotra’s approach involves estimating that there’s an X% chance that the first achievable TAI will look like A, a Y% chance like B, and so on. Some anchors (e.g., short-horizon neural network and long-horizon neural network) are obviously incompatible; whatever the future looks like, they won’t both describe the first achievable TAI. Multiplying them is clearly not meaningful; Roodman’s proposed “restriction that the various frameworks agree” makes no sense. (Multiplying them would be correct if Cotra’s different anchors represented something like different information-sources on necessary-and-sufficient-conditions-for-TAI, but that’s not what her anchors represent.)
(I suspect I may be missing something.)
I’m with you. I think Roodman must disagree with the idea of giving probabilies to different (and necessarily conflicting) models of the world, but to me this seems like an odd position to hold. I might also be missing something.
Agree with what you’re saying. This part of the review in particular stood out to me:
Since Cotra isn’t presenting the different anchors as all-things-considered estimates, but instead more like different hypotheses. Consider the evolutionary anchor – Cotra could have divided the compute requirements in this anchor by a scaling factor for how much more efficient she believes human-directed SGD (or similar) will be compared to how efficient evolution was at finding intelligence, yielding an all-things-considered estimate of how much compute will be necessary for TAI, but instead she leaves the value as is and considers it a soft upper bound.