Founder of CEEALAR (née the EA Hotel; ceealar.org)
Hi nil :)
They don’t know, esp. about what-it-is-likeness of any sentient experience (although, once again, this may be orthogonal to the risk, at least in theory with unlimited computational power)
Yes, and to to the orthogonality, but I don’t think it needs that much computational power (certainly not unlimited). Good enough generalisations could allow it to accomplish a lot (e.g. convincing a lab tech to mix together some mail order proteins/DNA in order to bootstrap nanotech).
or at least to be accurate enough to mislead all of us into a paperclip “hell”
How accurate does it need to be? I think human behaviour could be simulated enough to be manipulated with feasible levels of compute. There’s no need for consciousness/empathy. Arguably, social media algorithms are already having large effects on human behaviour.
Thanks for the detailed reply, that makes sense. What do you make of Google’s Pathways?
Interesting. Yes I guess such “full-spectrum superintelligence” might well be good by default, but the main worry from the perspective of the Yudkowsky/Bostrom paradigm is not this—perhaps it’s better described as super-optimisation, or super-capability (i.e. a blind optimisation process that has no subjective internal experience, and no inclination to gain one, given it’s likely initial goals).Regarding feasibility of conscious AGI / Pearce’s full-spectrum superintelligence, maybe it would be possible with biology involved somewhere. But the getting from here to there seems very fraught ethically (e.g. the already-terrifying experiments with mini-brains). Or maybe quantum computers would be enough?
I will also say that I like Vinding’s other work, especially You Are Them. A problem for Alignment is that the AGI isn’t Us though (as it’s default non-conscious). Perhaps it’s possible that an AGI could independently work out Valence Realism and Open/Empty Individualism, and even solve the phenomenal binding problem so as to become conscious itself. But I think these are unlikely possibilities a priori. Although perhaps they should be deliberately aimed for? (Is anyone working on this?)
I’ve not read the whole book, but reading the linked article Consciousness – Orthogonal or Crucial? I feel like Vinding’s case is not very convincing. It was written before GPT-3, and this shows. In GPT-3 we already have a (narrow) AI that can convincingly past the Turing Test in writing. Including writing displaying “social skills” and “general wisdom”. And very few people are arguing that GPT-3 is conscious.In general, if you consider that the range of human behaviour is finite, what’s to say that it couldn’t be recreated simply with a large enough (probabilistic) look-up table? And a large enough ML model trained on human behaviour could in theory create a neural network functionally equivalent to said look-up table.What’s to say that a sufficiently large pile of linear algebra, seeded with a sufficiently large amount of data, and executed on a sufficiently fast computer, could not build an accurate world model, recursively rewrite more efficient versions of itself, reverse engineer human psychology, hide it’s intentions from us, create nanotech in secret, etc etc, on the way to turning the future lightcone into computronium in pursuit of the original goal programmed into it at its instantiation (making paperclips, making a better language model, making money on the stock market, or whatever), all without a single conscious subjective internal experience?
This especially considering that an all-things-considered (and IMO conservative) estimate for the advent of AGI is 10% chance in (now) 14 years! This is a huge amount of short-term risk! It should not be considered as (exclusively) part of the longtermist cause area.
X-risk as a focus for neartermismI think it’s unfortunate how x-risks are usually lumped in with longtermism , and longtermism is talked about a lot more as a top-level EA cause area these days, and x-risk less so. This considering that, arguably, x-risk is very important from a short-term (or at least medium-term) perspective too.
As OP says in #2, according to our best estimates, many (most?) people’s chances of dying in a global catastrophe over the next 10-25 years are higher than many (any?) other regular causes of death (car accidents, infectious disease, heart disease and cancer for those of median age or younger, etc). As Carl Shulman outlines in his common sense case for x-risk work on the 80k podcast:
If you believe that the risk of human extinction over the next century is something like one in six (as Toby Ord suggests is a reasonable figure in his book The Precipice), then it would be worth the US government spending up to $2.2 trillion to reduce that risk by just 1%, in terms of [present] American lives saved alone.
See also Martin Trouilloud’s comment arguing that “a great thing you can do for the short term is to make the long term go well”, and this post arguing that “the ‘far future’ is not just the far future” [due to near term transformative technology and x-risk].I think X-risk should be reinstated as a top-level EA cause area in it’s own right, distinct from Longtermism. I worry that having it as a sub-level concern, under the broad heading of longtermism, will lead to people seeing it as less urgent than it is; “longtermism” giving the impression that we have plenty of time to figure things out (when we really don’t, in expectation).
The way I sometimes phrase it to people is that I now think it’s more urgent than Climate Change (and people understand that Climate Change is getting quite urgent, and is something that will have a big impact within their lifetimes).
Median is ~3-4 decades away. I’d call that “a few”, rather than “several” (sorry to nitpick, but I think this is important: several implies “no need to worry about it, probably not going to happen in my lifetime”, whereas a few implies (for the majority of people) “this is within my lifetime; I should sit up and pay attention.”)
Yep. Watched Don’t Look Up last night; can imagine that.
What makes you confident that “Transformative AI is several decades away”? Holden estimates “more than a 10% chance we’ll see transformative AI within 15 years (by 2036)”, based on a variety of reports taking different approaches (that are IMO conservative). Given the magnitude of what is meant by “transformative”, governments (and people in general) should really be quite a bit more concerned. As the analogy goes—if you were told that there was a >10% chance of aliens landing on Earth in the next 15 years, then you should really be doing all you can to prepare, as soon as possible!
I think AGI would easily be capable of FOOM-ing 100x+ across the board. And as for AGI being developed, it seems like we are getting ever closer with each new breakthrough in ML (and there doesn’t seem to be anything fundamentally required that can be said to be “decades away” with high conviction).
Maybe your view is closer to Eric Drexler’s CAIS? That would be a good outcome, but it doesn’t seem very likely to be a stable state to me, given that the narrow AIs could be used to speed AGI development. I don’t think the world will coordinate around the idea of narrow AIs / CAIS being enough, without a lot of effort around getting people to recognise the dangers of AGI.
“Neural Descriptor Fields” is promising—their robot learns to grasp from only ten examples
Thanks for these links. Incredible (and scary) progress!
cheaply supply human-brain-scale AI to the nefarious individual
I think we’re coming at this from different worldviews. I’m coming from much more of a Yudkowsky/Bostrom perspective, where the thing I worry about is misaligned superintelligent AGI; an existential risk by default. For a ban on AGI to be effective against this, it has to stop every single project reaching AGI. There won’t be a stage that lasts any appreciable length of time (say, more than a few weeks) where there are AGIs that can be destroyed/stopped before reaching a point of no return.
then there must have been an earlier date when the amount of compute was only “100 trillion synapses-seconds per second”, enough for a real-time brain, only.
Yes, but my point above was that the very first prototype isn’t going to use all the compute available. Available compute is a function of money spent. So there will very likely be room to significantly speed up the first prototype AGI as soon as it’s deployed. We may very well be at a point now where if all the best algorithms were combined, and $10T spent on compute, we could have something approximating an AGI. But that’s unlikely to happen as there are only maybe 2 entities that can spend that amount of money (the US government and the Chinese government), and they aren’t close to doing so. However, if it gets to only needing $100M in compute, then that would be within reach of many players that could quickly ramp that up to $1B or $10B.
Each new improvement is a harder-fought victory, for a diminishing return.
Do you think this is true even in the limit of AGI designing AGI? Do you think human level is close to the maximum possible level of intelligence? When I mentioned “FOOM” I meant it in the classic Yudkowskian fast takeoff to superintelligence sense.
Thanks for the heads up about Hinton’s GLOM, Numenta’s Sparse Representations and Google’s Pathways. The latter in particular seems especially worrying, given Google’s resources.I don’t think your arguments regarding Sharpness and Hardness are particularly reassuring though. If an AGI can be made that runs at “real time”, what’s to stop someone throwing 10, 100, 1000x more compute at it to make it run correspondingly faster? Will they really have spent all the money they have at their disposal on the first prototype? And even if they did, others with more money could quickly up the ante. (And then obviously once the AGI is running much faster than a human, it can be applied to making itself smarter/faster still, etc → FOOM)And as for banning AGI—if only this were as easily done as said. How exactly would we go about banning AGI? Especially in such a way that Narrow AI was allowed to continue (so e.g. banning large GPU/TPU clusters wouldn’t be an option)?
Some related news: Peter Singer has released a (very limited) NFT series! They’re up for auction on OpenSea, with proceeds going to TLYCS.
Not exactly what you are looking for, but here is an actual [metaphorical] map (although it could do with updating; it’s from Feb 2020):
I don’t think mathematics should be a crux. As I say below, it could be generalised to being offered to anyone a panel of top people in AGI Safety would have on their dream team (who otherwise would be unlikely to work on the problem). Or perhaps “Fields Medalists, Nobel Prize winners in Physics, other equivalent prize recipients in Computer Science, or Philosophy[?], or Economics[?]”. And we could include additional criteria, such as being able to intuit what is being alluded to here. Basically, the idea is to headhunt the very best people for the job, using extreme financial incentives. We don’t need to artificially narrow our search to one domain, but maths ability is a good heuristic as a starting point.
Gift Cards are live now at https://www.tisbest.org/redefinegifting
[Half-baked global health idea based on a conversation with my doctor: earlier cholesterol checks and prescription of statins]I’ve recently found out that I’ve got high (bad) cholesterol, and have been prescribed statins. What surprised me was that my doctor said that they normally wait until the patient has a 10% chance of heart attack or stroke in the next 10 years before they do anything(!) This seems crazy in light of the amount of resources put into preventing things with a similar (or lower) risk profiles, such as Covid, or road traffic accidents. Would reducing that to, say 5%* across the board (i.e. worldwide), be a low hanging fruit? Say by adjusting things set at a high level. Or have I just got this totally wrong? (I’ve done ~zero research, apart from searching givewell.org for “statins”, from which I didn’t find anything relevant).*my risk is currently at 5%, and I was pro-active about getting my blood tested.