They don’t know, esp. about what-it-is-likeness of any sentient experience (although, once again, this may be orthogonal to the risk, at least in theory with unlimited computational power)
Yes, and to to the orthogonality, but I don’t think it needs that much computational power (certainly not unlimited). Good enough generalisations could allow it to accomplish a lot (e.g. convincing a lab tech to mix together some mail order proteins/DNA in order to bootstrap nanotech).
or at least to be accurate enough to mislead all of us into a paperclip “hell”
How accurate does it need to be? I think human behaviour could be simulated enough to be manipulated with feasible levels of compute. There’s no need for consciousness/empathy. Arguably, social media algorithms are already having large effects on human behaviour.
Hi nil :)
Yes, and to to the orthogonality, but I don’t think it needs that much computational power (certainly not unlimited). Good enough generalisations could allow it to accomplish a lot (e.g. convincing a lab tech to mix together some mail order proteins/DNA in order to bootstrap nanotech).
How accurate does it need to be? I think human behaviour could be simulated enough to be manipulated with feasible levels of compute. There’s no need for consciousness/empathy. Arguably, social media algorithms are already having large effects on human behaviour.