Planning to simply defy human nature doesn’t usually go very well.
trevor1
Karma: 233
Crosspost to Lesswrong?
Two explanations come to my mind:
Past Sam Altman didn’t trust his future self, and wanted to use the OpenAI governance structure to constrain himself.
His status game / reward gradient changed (at least subjectively from his perspective). At the time it was higher status to give EA more power / appear more safety-conscious, and now it’s higher status to take it back / race faster for AGI. (I note there was internal OpenAI discussion about wanting to disassociate with EA after the FTX debacle.)
I think it’s reasonable to think that “Constraining Sam in the future” was obviously a highly pareto-efficient deal. EA had every reason to want Sam constrained in the future. Sam had every reason to make that trade, gaining needed power in the short-term, in exchange for more accountability and oversight in the future.
This is clearly a sensible trade that actual good guys would make; not “Sam didn’t trust his future self” but rather “Sam had every reason to agree to sell off his future autonomy in exchange for cooperation and trust in the near term”.I think “the world changing around Sam and EA, rather than Sam or EA changing” is worth more nuance. I think that, over the last 5 years, the world changed to make groups of humans vastly more vulnerable than before, due to new AI capabilities facilitating general-purpose human manipulation and the world’s power players investing in those capabilities.
This dramatically increased the risk of outsider third parties creating or exploiting divisions in the AI safety community, to turn people against each other and use the chaos as a ladder. Given that this risk was escalating, then centralizing power was clearly the correct move in response.
I’ve been warning about this during the months before the OpenAI conflict started, in the preceeding weeks (including the concept of an annual discount rate for each person, based on the risk of that person becoming cognitively compromised and weaponized against the AI safety community), and I even described the risk of one of the big tech companies hijacking Anthropic 5 days before Sam Altman was dismissed. I think it’s possible that Sam or people in EA also noticed the world rapidly becoming less safe for AI safety orgs, discovering the threat from a different angle than I did.
Upvoted. I’m really glad that people like you are thinking about this.
Something that people often miss with bioattacks is the economic dimension. After the 2008 financial crisis, economic failure/collapse became perhaps the #1 goalpost of the US-China conflict.
It’s even debatable whether the 2008 financial crisis was the cause of the entire US China conflict (e.g. lots of people in DC and Beijing would put the odds at >60% that >50% of the current US-China conflict was caused by the 2008 recession alone, in contrast to other variables like the emergence of unpredictable changes in cybersecurity).
Unlike conventional war e.g. over Taiwan and cyberattacks, economic downturns have massive and clear effects on the balance of power between the US and China, with very little risk of a pyrrhic victory (I don’t currently know how this compares to things like cognitive warfare which also yield high-stakes victories and defeats that are hard to distinguish from natural causes).
Notably, the imperative to cause massive economic damage, rather than destroy the country itself, allows attackers to ratchet down the lethality as far as they want, so long as it’s enough to cause lockdowns which cause economic damage (maybe mass IQ reduction or other brain effects could achieve this instead).
GOF research is filled with people who spent >5 years deeply immersed in a medical perspective e.g. virology, so it seems fairly likely to me that GOF researchers will think about the wider variety of capabilities of bioattacks, rather than inflexibly sticking to the bodycount-maximizing mindset of the Cold War.
I think that due to disorganization and compartmentalization within intelligence agencies, as well as unclear patterns of emergence and decay of competent groups of competent people, it’s actually more likely that easier-access biological attacks would first be caused by radicals with privileged access within state agencies or state-adjacent organizations (like Booz Allen Hamilton, or the Internet Research Agency which was accused of interfering with the 2016 election on behalf of the Russian government).
These radicals might incorrectly (or even correctly) predict that their country is a sinking ship and that they only way out is to personally change the balance of power; theoretically, they could even correctly predict that they are the only ones left competent enough to do this before it’s too late.
If I was a Bay Area VC, and I had $5m to invest annually and $100k to donate to people researching the long-term future (e.g. because it’s interesting and I like the idea of being the one to drive the research), it would be foolish to spend some of the $5m investing in people researching nanofactories.
But it would also be foolish to donate some of the $100k to the kinds of people who say “nanorobotics is an obvious scam, they can just make up whatever they want”.
And people don’t realize that short-term investment and long-term predictions are separate domains that are both valuable in their own way, because there are so few people outside of the near-term focused private sector who are thinking seriously about the future.
They just assume that thinking about the long-term future is just a twisted, failed perversion of the private sector, because of how deeply immersed they are in the private sector’s perspective exclusively.
As a result, they never have a chance to notice that the long-term future is something that they and their families might end up living in.
As of January 2023, the institutional markets were not predicting AGI within 30 years.
Do you think you could linkpost your article to Lesswrong too?
I know this article mainly focuses on EA values, but it also overlaps with a bunch of stuff that LW users like to research and think about (e.g. in order to better understand the current socio-political and and geopolitical situation with AI safety).
There’s a lot of people on LW who mainly spend their days deep into quantitative technical alignment research, but are surprisingly insightful and helpful when given a fair chance to weigh in on the sociological and geopolitical environment that EA and AI safety take place in, e.g. johnswentworth’s participation in this dialogue.
Normally the barriers to entry are quite high, which discourages involvement from AI safety’s most insightful and quantitative thinkers. Non-experts typically start out, by default, with really bad takes on US politics or China (e.g. believing that the US military just hands over the entire nuclear arsenal to a new president every 4-8 years), and people have to call them out on that in order to preserve community epistemics.
But it also keeps alignment researchers and other quant people separated from the people thinking about the global and societal environment that EA and AI safety take place in, which currently needs as many people as possible understanding the problems and thinking through viable solutions.
Upvoted, I’m grateful for the sober analysis.
The previous point notwithstanding, people’s attention spans are extremely short, and the median outcome of a news story is ~nothing. I’ve commented before that FTX’s collapse had little effect on the average person’s perception of EA, and we might expect a similar thing to happen here.
I think this is an oversimplification. This effect is largely caused by competing messages; the modern internet optimizes information for memetic fitness e.g. by maximizing emotional intensity or persuasive effect, and people have so much routine exposure to stuff that leads their minds around in various directions that they get wary (or see having strong reactions to anything at all as immature, since a large portion of outcries on the internet are disproportionately from teenagers). This is the main reason why people take things with a grain of salt.
However, overton windows can still undergo big and lasting shifts (this process could also be engineered deliberately long before generative AI emerged, e.g. via clown attacks which exploit social status instincts to consistently hijack any person’s impressions of any targeted concept). The 80,000 hours podcast with Cass Sunstein covered how Overton windows are dominated by vague impressions of what ideas are acceptable or unacceptable to talk about (note: this podcast was from 2019). This dynamic could plausibly strangle EA’s access to fresh talent, and AI safety’s access to mission-critical policy influence, for several years (which would be far too long).
It can be frustrating to feel that a group you are part of is being judged by the actions of a couple people you’ve never met nor have any strong feelings about.
On the flip side, johnswentworth actually had a pretty good take on this; that the human brain is instinctively predisposed to over-focus on the risk of their in-group becoming unpopular among everyone else:
First, [AI safety being condemned by the public] sure does sound like the sort of thing which the human brain presents to us as a far larger, more important fact than it actually is. Ingroup losing status? Few things are more prone to distorted perception than that.
Strong upvoted. I’m a huge fan of optimized combinations of words as a communication strategy, and have been for almost two years now.
I think that converting key x-risk ideas into poetry has a ton of potential to produce communication value from the creation of galaxy-brained combinations of words (including solving many problems fundamental to the human mind and human groups, such as the one mentioned in Raemon’s You Get About 5 Words).
I recommend pioneering this idea and seeing how far you can run with it; I think the expected value makes it worth trying, even if there’s a risk that it won’t work out, or that you won’t be the one credited for getting it going.
(As a side note, I also think it’s valuable to say at the beginning whether LLM generation was used and to what extent. It might seem obvious to you, and it probably actually is obvious that this is human-written to people with tons of experience with both poetry and LLMs, but LLM capabilities are always changing, and modern readers might need to feel reassured, especially for people new to poetry. Skill building for Cyborg poetry might be high EV too, and it might be important to be an early adopter so that EA will be the first to ride the wave when things get serious).
If this is true, then I think the board has made a huge mess of things. They’ve taken a shot without any ammunition, and not realised that the other parties can shoot back. Now there are mass resignations, Microsoft is furious, seemingly all of silicon valley has turned against EA, and it’s even looking likely that Altman comes back.
How much of this is “according to anonymous sources”?
The Board was deeply aware of intricate details of other parties’s will and ability to shoot back. Probably nobody was aware of all of the details, since webs of allies are formed behind closed doors and rearrange during major conflicts, and since investors have a wide variety of retaliatory capabilities that they might not have been open about during the investment process.
No, Sam Altman and the members of the OpenAI board all don’t work at MIRI/SIAI or FHI, so it doesn’t seem to have anything to do with AI safety.
Agreed, I only used the word “dominance games” because it seemed helpful for understandability and the wordcount. But it was inaccurate enough to be worth effort to find a better combination of words.
Because humans are primates, we have a strong drive to gain social status and play dominance games. The problem is that humans tend to take important concepts and turn them into dominance games.
As a result, people anticipate some sort of dominance or status game whenever they hear about an important concept. For many people, this anticipation has become so strong that they stopped believing that important concepts can exist.
Henrik Karlsson’s post Childhoods of Exceptional People did research indicating that there are intensely positive effects from young children spending lots of time talking and interacting with smart, interested adults; so much so that we could even reconsider the paradigm of kids mostly spending time with other kids their age.
It’s probably really important to go to lots of events and meet and talk to a bunch of different people and get a wider variety of perspectives; there’s only so much that a couple can do from inside one’s own two heads.
Western culture is highly individualistic, especially when it comes to major life decisions. However, aligning oneself with empirical reality is typically best done by gathering information from lots of people.
5 Reasons Why Governments/Militaries Already Want AI for Information Warfare
The difficulty of pitching AI safety to someone has been going down by ~50% every ~18 months. This thanksgiving might be a great time to introduce it to family; run Murphyjitsu and be goal-oriented! 🦃
People in the 90%-98% range are still about as instinctively driven to pursue social status as people in the 98%-100% range, and also predisposed to misevaluating themselves as in the 98-100% range (since completely accurate self-evaluations are a specific state, and departures/mistakes in either direction are reasonably likely).
Even people in the 98%-99% range are enough of a risk, since they grew up in an environment that got them used to being the smartest person in the room most of the time. Also, smarter people often got treated like crap growing up, due to intelligence causing them to do something or other differently and therefore stand out and fail at conformity.
It causes all kinds of weird personality issues, some of which suddenly manifest at unexpected times. As a result, fewer people mean fewer liabilities.
Theoretical moloch-preventing civilizations like dath ilan have evaluated this problem and implemented solutions, but our real world has barely grappled with it.
That area is controlled by militaries, who might retaliate against people who find clever ways to smuggle aid into the conflict zone. So trying to help people there instead of elsewhere is the wrong move.
EA was probably wrong to prioritize bednets over a malaria vaccine, even though lots of children would have died horribly if a malaria vaccine was invented 5 years later instead of them getting bednets now. It might seem harsh, but saving fewer lives instead of more is even more harsh for the lives of the people themselves, even if it’s accidental.
What is “reference class skepticism”? This is the first time I’ve heard that phrase, I googled it and didn’t find anything.