Monitoring Nanotechnologies and APM
Nanotechnologies, and a catastrophic scenario linked to it called “Grey goo”, have received very little attention recently (more information here ), whereas nanotechnologies keep moving forward and that some think that it’s one of the most plausible ways of getting extinct.
We’d be excited that a person or an organization closely monitors the evolution of the field and produces content on how dangerous it is. Knowing whether there are actionable steps that could be taken now or not would be very valuable for both funders and researchers of the longtermist community.
simeon_c
Karma: 807
AGI Timelines in Governance: Different Strategies for Different Timeframes
AGI x Animal Welfare: A High-EV Outreach Opportunity?
New GPT3 Impressive Capabilities—InstructGPT3 [1/2]
Longtermists Should Work on AI—There is No “AI Neutral” Scenario
Responsible Scaling Policies Are Risk Management Done Wrong
New Cause area: The Meta-Cause [Cause Exploration Prize]
AI Takeover Scenario with Scaled LLMs
Decomposing Biological Risks: Harm, Potential, and Strategies
Is GPT3 a Good Rationalist? - InstructGPT3 [2/2]
[Question] What’s the likelihood of irrecoverable civilizational collapse if 90% of the population dies?
The Cruel Trade-Off Between AI Misuse and AI X-risk Concerns
Future Design: How To Include Future Generations in Today’s Decision-Making?
Navigating AI Risks (NAIR) #1: Slowing Down AI
I agree with the general underlying point.
I also think that another important issue is that reasoning on counterfactuals makes people more prone to do things that are unusual AND is more prone to errors (e.g. by not taking into account some other effects).
Both combined make counterfactual reasoning without empirical data pretty perilous on average IMO.In the case of Ali in your example above for instance, Ali could neglect that the performance he’ll have will determine the opportunities & impact he has 5y down the line and so that being excited/liking the job is a major variable. Without counterfactual reasoning, Ali would have intuitively relied much more on excitement to pick the job but by doing counterfactual reasoning which seemed convincing, he neglected this important variable and made a bad choice.
I think that counterfactual reasoning makes people very prone to ignoring Chesterton’s fence.
Note that saying “this isn’t my intention” doesn’t prevent net negative effects of a theory of change from applying. Otherwise, doing good would be a lot easier.
I also highly recommend clarifying what exactly you’re criticizing, i.e. the philosophy, the movement norms or some institutions that are core to the movement.
Finally, I usually find the criticism of people a) at the core of the movement and b) highly truth-seeking most relevant to improve the movement so I would expect that if you’re trying to improve the movement, you may want to focus on these people. There exists relevant criticisms external to the movement but usually they will lack of context and thus fail to address some key trade-offs that the movement cares about.
Here’s a small list of people I would be excited to hear on EA flaws and their recommandations for change:
Rob Bensinger
Eli Lifland
Ozzie Gooen
Nuno Sempere
Oliver Habryka
Some more ideas that are related to what you mentioned :
Exploring / Exploiting interventions on growth in developing countries. So for instance, what if we took an entire country and spent about 100$ or more per households (for a small country, that could be feasible) ? We could do direct transfer as GiveDirectly but I’d expect some public goods funding to be worth trying aswell.
Making AI safety prestigious setting up an institute that would hire top researchers for safety aligned research. I’m not a 100% sure but I feel like top AI people often go to Google in big part because they offer great working conditions. If an institute offered these working conditions and that we could hire top junior researchers quite massively to work on prosaic AGI alignment, that could help making AI Safety more prestigious. Maybe such an institute could run seminars, give some rewards in safety or even on the long run host a conference.
Very glad to see that happening, regranting solves a bunch of unsolved problems with centralized grantmaking.
Yes, scenarios are a good way to put a lower bound but if you’re not able to create one single scenario that’s a bad sign in my opinion.
For AGI there are many plausible scenarios where I can reach ~1-10% likelihood of dying. With biorisks it’s impossible with my current belief on the MVP (minimum viable population)
Thanks for your comment!
First, you have to have in mind that when people are talking about “AI” in industry and policymaking, they usually have mostly non-deep learning or vision deep learning techniques in mind simply because they mostly don’t know the ML academic field but they have heard that “AI” was becoming important in industry. So this sentence is little evidence that Russia (or any other country) is trying to build AGI, and I’m at ~60% Putin wasn’t thinking about AGI when he said that.
I think that you’re deeply wrong about this. Policymakers and people in industry, at least till ChatGPT had no idea what was going on (e.g at the AI World Summit, 2 months ago very few people even knew about GPT-3). SOTA large language models are not really properly deployed, so nobody cared about them or even knew about them (till ChatGPT at least). The level of investment right now in top training runs probably doesn’t go beyond $200M. The GDP of the US is 20 trillion. Likewise for China. Even a country like France could unilaterally put $50 billion in AGI development and accelerate timelines quite a lot within a couple of years.
Even post ChatGPT, people are very bad at projecting what it means for next years and still have a prior on the fact that human intelligence is very specific and can’t be beaten which prevents them from realizing all the power of this technology.
I really strongly encourage you to go talk to actual people from industry and policy to get a sense of their knowledge on the topic. And I would strongly recommend not publishing your book as long as you haven’t done that. I also hope that a lot of people who have thought about these issues have proofread your book because it’s the kind of thing that could really increase P(doom) substantially.
I think that to make your point, it would be easier to defend the line that “even if more governments got involved, that wouldn’t change much”. I don’t think that’s right because if you gave $10B more to some labs, it’s likely they’d move way faster. But I think that it’s less clear.
I agree that it would be something good to have. But the question is: is it even possible to have such a thing?
I think that within the scientific community, it’s roughly possible (but then your book/outreach medium must be highly targeted towards that community). Within the general public, I think that it’s ~impossible. Climate change, which is a problem which is much easier to understand and explain is already way too complex for the general public to have a good idea of what are the risks and what are the promising solutions to these risks (e.g. a lot people’s top priorities is to eat organic food, recycle and decrease plastic consumption).
I agree that communicating with the scientific community is good, which is why I said that you should avoid publicizing only among “the general public”. If you really want to publish a book, I’d recommend targeting the scientific community, which is not at all the same public as the general public.
“On the other hand, if most people think that strong AI poses a significant risk to their future and that of their kids, this might change how AI capabilities researchers are seen, and how they see themselves”
I agree with this theory of change and I think that it points a lot more towards “communicate in the ML community” than “communicate towards the general public”. Publishing great AI capabilities is mostly cool for other AI researchers and not that much for the general public. People in San Francisco (where most of the AGI labs are) also don’t care much about the general public and whatever it thinks ; the subculture there and what is considered to be “cool” is really different from what the general public thinks is cool. As a consequence, I think they mostly care about what their peers are thinking about them. So if you want to change the incentives, I’d recommend focusing your efforts on the scientific & the tech community.