I’m not sure I follow your examples and logic, perhaps you could explain because drunk driving is in itself a serious crime in every country I know of. Are you suggesting it be criminal to merely develop an AI model regardless of whether it’s commercialized or released?
Regarding pharmaceuticals, yes, they certainly do need to pass several phases of clinical research and development to prove sufficient levels of safety and efficacy because by definition, FDA approves drugs to treat specific diseases. If those drugs don’t do what they claim, people die. The many reasons for regulating drugs should be obvious. However, there is no such similar regulation on software. Developing a drug discovery platform or even the drug itself is not a crime (as long as it’s not released.)
You could just as easily extrapolate to individuals. We cannot legitimately litigate (sue) or prosecute someone for a crime they haven’t committed. This is why we have due process and basic legal rights.( Technically anything can be litigated with enough money thrown at it but you cant sue for damages unless damages actually occurred)
Haven’t yet read the paper but based on your post, I completely agree that the “C” words tends to be polarizing. Also agree that the relevant conversations around consciousness, sentience, sapience, self-awareness, meta-cognition, subjective experience etc. etc. are overly entrenched.
However, to be honest making statements like “These forms of consciousness, if they are there at all, will be of a profoundly alien, radically un-human-like kind.” lack the humility we need in the area and effectively perpetuate these sorts of polarized positions. In truth, there is little evidence either way and we simply do not know. Yes, it is entirely feasible that this is true, It is also feasible that it is not. It seems to me it will largely depend on the developmental route that leads to AGI (and there will certainly be more than one).
There are reasons to think that human-like ‘consciousness’ within AGI/SI may be highly beneficial to long-term safety (not to trivialize the profound moral, ethical and legal implications). IMHO, Limited ‘forms’ or aspects of consciousness are clearly already here, particularly within some multi-modal systems that have integrated modes of visual and auditory perception but these existing forms are so limited and rudimentary that they do not deserve much moral consideration.
Again, just my $0.02 but embodied systems with more broadly integrated modes of perception, memory, attention etc will continue to push us further up the spectrum that is colloquially known as ‘Consciousness’. We have to keep in mind that consciousness, for lack of a better word, is not a single thing but rather a highly multi-faceted mutli-dimensional collective phenomena.