> do you think that, if we had a theory of sociopolitics that was about as good as 20th-century economics, then we wouldnât be clueless about how to do sociopolitical interventions (like founding AI safety movements) effectively?
No, because I think âfounding AI safety movements that succeed at making the far future go betterâ is a pretty out-of-distribution kind of sociopolitical intervention.
Suppose instead we had a comparably good theory of the right reference class, e.g. âmovements trying to shape transformative technologies.â Would we still be clueless about AI safety movement-building?
More generally: you list various considerations across your posts and I have a hard time understanding which is load-bearing for your answer here. Some possibilities:
Weâre clueless because we havenât yet developed the relevant theory (Richardâs reading IIUC, on which cluelessness is contingent and reducible)
No such theory could be validated even in principle, because we never observe the target variable (far-future value) and calibration on near-term proxies doesnât transfer
Even a validated theory wouldnât help, because impact is dominated by considerations inaccessible to any theory (e.g. unconceived hypothesis classes)
I think itâs mostly (1), but Iâm open to something like (2) or (3) as well.
(Following (1):) There is in principle some (a) amount of information that non-ideal agents could attain about the cosmos with non-Pascalian probability,[1] + (b) a priori modeling and induction we could apply to that information, such that we wouldnât be clueless. So I donât think we need to observe the target variable, or empirically âvalidateâ the theory, to be non-clueless.
But the bar to achieve such an (a)+(b) seems very high, because:
If we do try to empirically validate the theory by appealing to calibration on near-term proxies:
I indeed donât see why we should expect such calibration to transfer, up to the degree of precision we need to escape cluelessness (sec. 2.3.1.1). This bites even if, say, we use AI to get much more calibrated on ~years-long time horizons.
If we donât, and instead try to argue conceptually that the theory captures enough of the relevant considerations in fine-grained enough detail:
The web of factors this theory would have to capture seems ludicrously complex (the rest of sec. 2.3). Of course, good theories can compress complexity, but getting that amount of compression while keeping things computationally tractable[2] sounds rough.
So my suspicion is that yeah, weâd still be clueless given the kind of theory you mention. But I find it hard to say, because I canât imagine exactly what âcomparably goodâ looks like, concretely. I appreciate that thatâs hard to spell out on your end.
Maybe sufficiently advanced AI could get around this. Maybe not, e.g. if âthe universal priorâ is irreducibly imprecise, or if (following (3)) information about simulators or causally disconnected worlds is fundamentally inaccessible.
(Iâm happy to unpack any of this more if useful, not sure if I answered your question properly!)
You respond to Richard Ngo here:
Suppose instead we had a comparably good theory of the right reference class, e.g. âmovements trying to shape transformative technologies.â Would we still be clueless about AI safety movement-building?
More generally: you list various considerations across your posts and I have a hard time understanding which is load-bearing for your answer here. Some possibilities:
Weâre clueless because we havenât yet developed the relevant theory (Richardâs reading IIUC, on which cluelessness is contingent and reducible)
No such theory could be validated even in principle, because we never observe the target variable (far-future value) and calibration on near-term proxies doesnât transfer
Even a validated theory wouldnât help, because impact is dominated by considerations inaccessible to any theory (e.g. unconceived hypothesis classes)
I think itâs mostly (1), but Iâm open to something like (2) or (3) as well.
(Following (1):) There is in principle some (a) amount of information that non-ideal agents could attain about the cosmos with non-Pascalian probability,[1] + (b) a priori modeling and induction we could apply to that information, such that we wouldnât be clueless. So I donât think we need to observe the target variable, or empirically âvalidateâ the theory, to be non-clueless.
But the bar to achieve such an (a)+(b) seems very high, because:
If we do try to empirically validate the theory by appealing to calibration on near-term proxies:
I indeed donât see why we should expect such calibration to transfer, up to the degree of precision we need to escape cluelessness (sec. 2.3.1.1). This bites even if, say, we use AI to get much more calibrated on ~years-long time horizons.
If we donât, and instead try to argue conceptually that the theory captures enough of the relevant considerations in fine-grained enough detail:
The web of factors this theory would have to capture seems ludicrously complex (the rest of sec. 2.3). Of course, good theories can compress complexity, but getting that amount of compression while keeping things computationally tractable[2] sounds rough.
So my suspicion is that yeah, weâd still be clueless given the kind of theory you mention. But I find it hard to say, because I canât imagine exactly what âcomparably goodâ looks like, concretely. I appreciate that thatâs hard to spell out on your end.
Maybe sufficiently advanced AI could get around this. Maybe not, e.g. if âthe universal priorâ is irreducibly imprecise, or if (following (3)) information about simulators or causally disconnected worlds is fundamentally inaccessible.
(Iâm happy to unpack any of this more if useful, not sure if I answered your question properly!)
As in, if I were to represent this probability numerically, the interval wouldnât all be less than the Pascalian threshold.
Like, something analogous to the SchrĂśdinger equation doesnât count. :)