I think I agree maybe ~80%. My main reservation (although quite possibly we agree here) is that if Forethought hired e.g. the âAI as a normal technologyâ people, or anyone with equivalently different baseline assumptions and ways of thinking to most of Forethought, I think that would be pretty frustrating and unproductive. (That said, I think brining people like that in for a week or so might be great, to drill down into cruxes and download each othersâ world models more.) I think there is something great about having lots of foundational things in common with the people you work closely with.
But I agree that having more people who share some basic prerequisites of thinking ASI is possible, likely to come this century, being somewhat longtermist and cosmopolitan and altruistic, etc, but disagree a lot on particular topics like AI timelines and threat models and research approaches and so forth can be pretty useful.
Yeah, I guess I donât want to say that itâd be better if the team had people who are (already) strongly attached to various specific perspectives (like the âAI as a normal technologyâ worldviewâmaybe especially that one?[1]). And I agree that having shared foundations is useful /â constantly relitigating foundational issues would be frustrating. I also really do think the points I listed under âwho I think would be a good fitâ â willingness to try on and ditch conceptual models, high openness without losing track of taste, & flexibility â matter, and probably clash somewhat with central examples of âperson attached to a specific perspective.â
= rambly comment, written quickly, sorry! =
But in my opinion we should not just all (always) be going off of some central AI-safety-style worldviews. And I think that some of the divergence I would like to see more of could go pretty deepâe.g. possibly somewhere in the grey area between what you listed as âbasic prerequisitesâ and âparticular topics like AI timelines...â. (As one example, I think accepting terminology or the way people in this space normally talk about stuff like âalignmentâ or âan AIâ might basically bake in a bunch of assumptions that I would like Forethoughtâs work to not always rely on.)
One way to get closer to that might be to just defer less or more carefully, maybe. And another is to have a team that includes people who better understand rarer-in-this-space perspectives, which diverge earlier on (or people who are by default inclined to thinking about this stuff in ways that are different from othersâ defaults), as this could help us start noticing assumptions we didnât even realize we were making, translate between frames, etc.
So maybe my view is that (1) there were more ~independent worldview formation/â exploration going on, and that (2) the (soft) deferral that is happening (because some deferral feels basically inevitable) were less overlapping.
(I expect we donât really disagree, but still hope this helps to clarify things. And also, people at Forethought might still disagree with me.)
If this perspective involves a strong belief that AI will not change the world much, then IMO thatâs just one of the (few?) things that are ~fully out of scope for Forethought. I.e. my guess is that projects with that as a foundational assumption wouldnât really make much sense to do here. (Although IMO even if, say, I believed that this conclusion was likely right, I might nevertheless be a good fit for Forethought if I were willing to view my work as a bet on the worlds in which AI is transformative.)
But I donât really remember what the âAI as normal..â position is, and could imagine that itâs somewhat different â e.g. more in the direction of âautomation is the wrong frame for understanding the most likely scenariosâ /â something like this. In that case my take would be that someone exploring this at Forethought could make sense (havenât thought about this one much), and generally being willing to consider this perspective at least seems good, but Iâd still be less excited about people whoâd come with the explicit goal of pursuing that worldview & no intention of updating or whatever.
--
(Obviously if the âAI will not be a big dealâ view is correct, Iâd want us to be able to come to that conclusionâand change Forethughtâs mission or something. So I wouldnât e.g. avoid interacting with this view or its proponents, and agree that e.g. inviting people with this POV as visitors could be great.)
If this perspective involves a strong belief that AI will not change the world much, then IMO thatâs just one of the (few?) things that are ~fully out of scope for Forethought
I disagree with this. There would need to be some other reason for why they should work at Forethought rather than elsewhere, but there are plausible answers to that â e.g. they work on space governance, or they want to write up why they think AI wonât change the world much and engage with the counterarguments.
On the âAI as normal technologyâ perspectiveâI donât think it involves a strong belief that AI wonât change the world much. The authors restate their thesis in a later post:
There is a long causal chain between AI capability increases and societal impact. Benefits and risks are realized when AI is deployed, not when it is developed. This gives us (individuals, organizations, institutions, policymakers) many points of leverage for shaping those impacts. So we donât have to fret as much about the speed of capability development; our efforts should focus more on the deployment stage both from the perspective of realizing AIâs benefits and responding to risks. All this is not just true of todayâs AI, but even in the face of hypothetical developments such as self-improvement in AI capabilities. Many of the limits to the power of AI systems are (and should be) external to those systems, so that they cannot be overcome simply by having AI go off and improve its own technical design.
The idea of focusing more on the deployment stage seems pretty consistent with Will MacAskillâs latest forum post about making the transition to a post-AGI society go well. There are other aspects of the âAI as normal technologyâ worldview that I expect will conflict more with Forethoughtâs, but Iâm not sure that conflict would necessarily be frustrating and unproductiveâas you say, it might depend on the personâs characteristics like openness and willingness to update, etc.
Nice, yes I think we roughly agree! (Though maybe you are nobler than me in terms of finding a broader range of views provocatively plausible and productive to engage with.)
I canât speak to the âAI as a normal technologyâ people in particular, but a shortlist I created of people Iâd be very excited about includes someone who just doesnât buy at all that AI will drive an intelligence explosion or explosive growth.
I think there are lots of types of people where it wouldnât be a great fit, though. E.g. continental philosophers; at least some of the âsociotechnicalâ AI folks; more mainstream academics who are focused on academic publishing. And if youâre just focused on AI alignment, probably youâll get more at a different org than you would at Forethought.
More generally, Iâm particularly keen on situations where V(X, Forethought team) is much greater than than V(X) + V(Forethought team), either because there are synergies between X and the team, or because X is currently unable to do the most valuable work they could in any of the other jobs they could be in.
I think I agree maybe ~80%. My main reservation (although quite possibly we agree here) is that if Forethought hired e.g. the âAI as a normal technologyâ people, or anyone with equivalently different baseline assumptions and ways of thinking to most of Forethought, I think that would be pretty frustrating and unproductive. (That said, I think brining people like that in for a week or so might be great, to drill down into cruxes and download each othersâ world models more.) I think there is something great about having lots of foundational things in common with the people you work closely with.
But I agree that having more people who share some basic prerequisites of thinking ASI is possible, likely to come this century, being somewhat longtermist and cosmopolitan and altruistic, etc, but disagree a lot on particular topics like AI timelines and threat models and research approaches and so forth can be pretty useful.
Yeah, I guess I donât want to say that itâd be better if the team had people who are (already) strongly attached to various specific perspectives (like the âAI as a normal technologyâ worldviewâmaybe especially that one?[1]). And I agree that having shared foundations is useful /â constantly relitigating foundational issues would be frustrating. I also really do think the points I listed under âwho I think would be a good fitâ â willingness to try on and ditch conceptual models, high openness without losing track of taste, & flexibility â matter, and probably clash somewhat with central examples of âperson attached to a specific perspective.â
= rambly comment, written quickly, sorry! =
But in my opinion we should not just all (always) be going off of some central AI-safety-style worldviews. And I think that some of the divergence I would like to see more of could go pretty deepâe.g. possibly somewhere in the grey area between what you listed as âbasic prerequisitesâ and âparticular topics like AI timelines...â. (As one example, I think accepting terminology or the way people in this space normally talk about stuff like âalignmentâ or âan AIâ might basically bake in a bunch of assumptions that I would like Forethoughtâs work to not always rely on.)
One way to get closer to that might be to just defer less or more carefully, maybe. And another is to have a team that includes people who better understand rarer-in-this-space perspectives, which diverge earlier on (or people who are by default inclined to thinking about this stuff in ways that are different from othersâ defaults), as this could help us start noticing assumptions we didnât even realize we were making, translate between frames, etc.
So maybe my view is that (1) there were more ~independent worldview formation/â exploration going on, and that (2) the (soft) deferral that is happening (because some deferral feels basically inevitable) were less overlapping.
(I expect we donât really disagree, but still hope this helps to clarify things. And also, people at Forethought might still disagree with me.)
In particular:
If this perspective involves a strong belief that AI will not change the world much, then IMO thatâs just one of the (few?) things that are ~fully out of scope for Forethought. I.e. my guess is that projects with that as a foundational assumption wouldnât really make much sense to do here. (Although IMO even if, say, I believed that this conclusion was likely right, I might nevertheless be a good fit for Forethought if I were willing to view my work as a bet on the worlds in which AI is transformative.)
But I donât really remember what the âAI as normal..â position is, and could imagine that itâs somewhat different â e.g. more in the direction of âautomation is the wrong frame for understanding the most likely scenariosâ /â something like this. In that case my take would be that someone exploring this at Forethought could make sense (havenât thought about this one much), and generally being willing to consider this perspective at least seems good, but Iâd still be less excited about people whoâd come with the explicit goal of pursuing that worldview & no intention of updating or whatever.
--
(Obviously if the âAI will not be a big dealâ view is correct, Iâd want us to be able to come to that conclusionâand change Forethughtâs mission or something. So I wouldnât e.g. avoid interacting with this view or its proponents, and agree that e.g. inviting people with this POV as visitors could be great.)
I disagree with this. There would need to be some other reason for why they should work at Forethought rather than elsewhere, but there are plausible answers to that â e.g. they work on space governance, or they want to write up why they think AI wonât change the world much and engage with the counterarguments.
On the âAI as normal technologyâ perspectiveâI donât think it involves a strong belief that AI wonât change the world much. The authors restate their thesis in a later post:
The idea of focusing more on the deployment stage seems pretty consistent with Will MacAskillâs latest forum post about making the transition to a post-AGI society go well. There are other aspects of the âAI as normal technologyâ worldview that I expect will conflict more with Forethoughtâs, but Iâm not sure that conflict would necessarily be frustrating and unproductiveâas you say, it might depend on the personâs characteristics like openness and willingness to update, etc.
Nice, yes I think we roughly agree! (Though maybe you are nobler than me in terms of finding a broader range of views provocatively plausible and productive to engage with.)
I canât speak to the âAI as a normal technologyâ people in particular, but a shortlist I created of people Iâd be very excited about includes someone who just doesnât buy at all that AI will drive an intelligence explosion or explosive growth.
I think there are lots of types of people where it wouldnât be a great fit, though. E.g. continental philosophers; at least some of the âsociotechnicalâ AI folks; more mainstream academics who are focused on academic publishing. And if youâre just focused on AI alignment, probably youâll get more at a different org than you would at Forethought.
More generally, Iâm particularly keen on situations where V(X, Forethought team) is much greater than than V(X) + V(Forethought team), either because there are synergies between X and the team, or because X is currently unable to do the most valuable work they could in any of the other jobs they could be in.