I think we could try to build AGI, but I am skeptical it could be anything useful or helpful (a broad alignment problem) because of vague or inapt success criteria, and because of the lack of embodiment of AGI (so it wonât get beat up on by the world generally or have emotional/âaffective learning). Because of these problems, I think we shouldnât try (1).
Hmm, I guess I donât think lack of emotional/âaffective states is a problem for making useful AGIs. Obviously those are parts of how humans learn, but seems like a machine can learn with any reward functionâit just needs some way of mapping a world state to value.
Re success criteria, you could for example train an AI to improve a companyâs profit in a simulated environment. That task requires a broad set of capacities, including high-level ones like planning/âstrategising. If you do this for many things humans care about, youâll get a more general system, as with DeepMindâs GATO. But of course Iâm speculating.
Further, I am trying this line of argument out to see if it will encourage (3) (not building AGI), because these concerns cast doubt on the value of AGI to us (and thus the incentives to build it).
I suppose if you donât think thereâs any value for us in AGI, and if you donât think there are sufficient incentives for us to build it, thereâs no need to encourage not building it? Or is your concern more that weâre wasting energy and resources trying to build it, or even thinking about it?
This takes on additional potency if we embrace the shift to thinking about âshouldâ and not just âcanâ in scientific and technological development generally. So that brings us to the questions I think we should be asking, which is how to encourage a properly responsible approach to AI, rather than shifting credences on the Future Fundsâ propositions about.
The first propositionââConditional on AGI being developed by 2070, humanity will go extinct or drastically curtail its future potential due to loss of control of AGIââseems directly linked to whether or not we should build AGI. If AGI carries a serious risk of catastrophe, we obviously shouldnât build it. So to me it looks like the Future Fund is already thinking about the âshouldâ question?
Hmm, I guess I donât think lack of emotional/âaffective states is a problem for making useful AGIs. Obviously those are parts of how humans learn, but seems like a machine can learn with any reward functionâit just needs some way of mapping a world state to value.
Re success criteria, you could for example train an AI to improve a companyâs profit in a simulated environment. That task requires a broad set of capacities, including high-level ones like planning/âstrategising. If you do this for many things humans care about, youâll get a more general system, as with DeepMindâs GATO. But of course Iâm speculating.
I suppose if you donât think thereâs any value for us in AGI, and if you donât think there are sufficient incentives for us to build it, thereâs no need to encourage not building it? Or is your concern more that weâre wasting energy and resources trying to build it, or even thinking about it?
The first propositionââConditional on AGI being developed by 2070, humanity will go extinct or drastically curtail its future potential due to loss of control of AGIââseems directly linked to whether or not we should build AGI. If AGI carries a serious risk of catastrophe, we obviously shouldnât build it. So to me it looks like the Future Fund is already thinking about the âshouldâ question?