Monitoring Nanotechnologies and APM
Nanotechnologies, and a catastrophic scenario linked to it called “Grey goo”, have received very little attention recently (more information here ), whereas nanotechnologies keep moving forward and that some think that it’s one of the most plausible ways of getting extinct.
We’d be excited that a person or an organization closely monitors the evolution of the field and produces content on how dangerous it is. Knowing whether there are actionable steps that could be taken now or not would be very valuable for both funders and researchers of the longtermist community.
simeon_c
I agree with the general underlying point.
I also think that another important issue is that reasoning on counterfactuals makes people more prone to do things that are unusual AND is more prone to errors (e.g. by not taking into account some other effects).
Both combined make counterfactual reasoning without empirical data pretty perilous on average IMO.In the case of Ali in your example above for instance, Ali could neglect that the performance he’ll have will determine the opportunities & impact he has 5y down the line and so that being excited/liking the job is a major variable. Without counterfactual reasoning, Ali would have intuitively relied much more on excitement to pick the job but by doing counterfactual reasoning which seemed convincing, he neglected this important variable and made a bad choice.
I think that counterfactual reasoning makes people very prone to ignoring Chesterton’s fence.
Note that saying “this isn’t my intention” doesn’t prevent net negative effects of a theory of change from applying. Otherwise, doing good would be a lot easier.
I also highly recommend clarifying what exactly you’re criticizing, i.e. the philosophy, the movement norms or some institutions that are core to the movement.
Finally, I usually find the criticism of people a) at the core of the movement and b) highly truth-seeking most relevant to improve the movement so I would expect that if you’re trying to improve the movement, you may want to focus on these people. There exists relevant criticisms external to the movement but usually they will lack of context and thus fail to address some key trade-offs that the movement cares about.
Here’s a small list of people I would be excited to hear on EA flaws and their recommandations for change:
Rob Bensinger
Eli Lifland
Ozzie Gooen
Nuno Sempere
Oliver Habryka
Some more ideas that are related to what you mentioned :
Exploring / Exploiting interventions on growth in developing countries. So for instance, what if we took an entire country and spent about 100$ or more per households (for a small country, that could be feasible) ? We could do direct transfer as GiveDirectly but I’d expect some public goods funding to be worth trying aswell.
Making AI safety prestigious setting up an institute that would hire top researchers for safety aligned research. I’m not a 100% sure but I feel like top AI people often go to Google in big part because they offer great working conditions. If an institute offered these working conditions and that we could hire top junior researchers quite massively to work on prosaic AGI alignment, that could help making AI Safety more prestigious. Maybe such an institute could run seminars, give some rewards in safety or even on the long run host a conference.
Very glad to see that happening, regranting solves a bunch of unsolved problems with centralized grantmaking.
Yes, scenarios are a good way to put a lower bound but if you’re not able to create one single scenario that’s a bad sign in my opinion.
For AGI there are many plausible scenarios where I can reach ~1-10% likelihood of dying. With biorisks it’s impossible with my current belief on the MVP (minimum viable population)
Ah ah you probably don’t realize it but “you” is actually 4 persons: Amber Dawn for the first draft of the post, me (Simeon) for the ideas, the table and the structure of the post, and me, Nicole Nohemi & Felicity Riddel for the partial rewriting of the draft to make it clearer.
So the credits are highly distributed! And thanks a lot, it’s great to hear that!
Thanks for your comment!
A couple of remarks:
Regulations that cut X-risks are strong regulations: My sense is that regulations that really cut X-risks at least a bit are pretty “strong”, i.e. in the reference class of “Constrain labs to airgap and box their SOTA models while they train them” or “Any model which is trained must be trained following these rules/applying these tests”. So what matters in terms of regulation is “will governments take such actions?” and my best guess is no, at least not without the public opinion caring a lot about that. Do you have in mind an example of regulation which is a) useful and b) softer than that?
Additional state funding would be bad by default: I think that the default of “more funding goes towards AGI” is that it accelerates capabilities more (e.g backing some labs so that they move faster than China, things like that). Which is why I’m not super excited about increasing the amount of funding governments put into AGI. But to the extent that there WILL be some funding, then it’s nice to steer it towards safety research.
And finally, I like the example you gave on cyber. The point I was making was something like “Your theories of change for pre-2030 timelines shouldn’t rely too much on national government policy” and my understanding of what you’re saying is something like “that may be right, but national governments are still likely to have a lot of (bad by default) influence, so we should care about them”.
I basically had in mind this kind of scenario where states don’t do the research themselves but are backing some private labs to accelerate their own capabilities, and it makes me more worried about encouraging states to think about AGI. But I don’t put that much weight on these scenarios yet.
How confident are you that governments will get involved in meaningful private-public collaboration around AGI by 2030? A way of operationalizing that could be “A national government spends more than a billion $ in a single year on a collaboration with a lab with the goal to accelerate research on AGI”.
If you believe that it’s >50%, that would definitely update me towards “we should still invest a significant share of our resources in national policy, at least in the UK and the US so that they don’t do really bad moves”.
Thanks David, that’s great !
“The first reason not to pursue the one-country approach from a policy perspective is that non-existential catastrophes seem likely, and investments in disease detection and prevention are a good investment from a immediate policy perspective. Given that, it seems ideal to invest everywhere and have existential threat detection be a benefit that is provided as a consequence of more general safety from biological threats. There are also returns to scale for investments, and capitalizing on them may require a global approach.”
I feel like there are two competing views here that are very well underlined thanks to your comment :
From a global perspective, the optimal policy is probably to put metagenomic sequencing in the key nodes of the travel network so that we’re aware of any pathogen as soon as possible. I feel like it’s roughly what you meant
From a marginalist perspective, given the governance by country it’s probably much easier to cover one country who cares about national security with metagenomic sequencing (e.g the US) than to apply the first strategy.
I expect the limiting factor not to be our own resource allocation but the opportunities to push for the relevant policies at the right moment. If we’re able to do the first strategy, i.e if there’s an opportunity for us to push in favor of a global metagenomic plan that has some chances to work, that’s great ! But if we’re not, we shouldn’t disregard the second strategy (i.e pushing in one single country to have a strong metagenomic sequencing policy being implemented) as a great way to greatly mitigate at least X-risks from GCBRs.
“Second, a key question for whether the proposed “one country” approach is more effective than other approaches is whether we think early detection is more important than post-detection response, and what they dynamics of the spread are. As we saw with COVID-19, once a disease is spreading widely, stopping it is very, very difficult. The earlier the response starts, the more likely it is that a disease can be stopped before spreading nearly universally. The post-detection response, however, can vary significantly between countries, and those most able to detect the thread weren’t the same as those best able to suppress cases—and for this and related reasons, putting our eggs all in one basket, so to speak, seems like a very dangerous approach.”
Yes, I agree with this for GCBRs in general but not for existential ones ! My point is just that conditionally on a very very bad virus and on awareness about this virus, I expect some agents who are aware quite early about it (hence the idea to put metagenomic sequencing in every entry points of a country) to find ways to survive it, either due to governments or due to personal preparation (personal bunkers or this kind of stuff).
I hope I answered your points correctly !
Thanks for the comment !
I mean, I agree that it has nuance but it’s still trained on a set of values that are pretty much current western people values, so it will probably put more or less emphasis on various values according to the weight western people give to each of those.
I may try to write something on that in the future. I’m personally more worried about accidents and think that solving accidents causes one to solve misuse pre-AGI. Post aligned AGI, misuse rebecomes a major worry.
I think that our disagreement comes from what we mean by “regulating and directing it.”
My rough model of what usually happens in national governments (and not the EU, which is a lot more independent from its citizen than the typical national government) is that there are two scenarios:
Scenario 1 in which national governments regulate or do things on something nobody is caring about (in particular, not the media). That gives birth to a lot of degrees of freedom and the possibility of doing fairly ambitious things (cf Secret Congress)
Scenario 2 in which national governments regulate things that many people care about and brings attention and then nothing gets done, most measures are fairly weak etc. In this scenario my rough model is that national governments do the smallest thing that satisfy their electorate + key stakeholders.
I feel like we’re extremely likely to be in scenario 2 regarding AI. And thus that no significant measure will be taken, which is why I put the emphasis of “no strong [positive] effect” on AI safety. So basically I feel like the best you can probably do in national policy is something like “avoid that they do bad things” (which is really good if it’s a big risk) or “do mildly good things”. But to me, it’s quite unlikely that we go from a world where we die to a world where we don’t die thanks to a theory of change which is focused on national policy.
The EU AI Act is a bit different in that as I said above, the EU is much less tied to the daily worries of citizen and thus is operating under less constraints. Thus I think that it’s indeed plausible that the EU does something ambitious on GPAIS but I think that unfortunately it’s unlikely that the US will replicate something locally and that the EU compliance mechanisms are not super likely to cut the worst risks for the UK and US companies.
Regulating the training of these models is different and harder, but even that seems plausible to me at some point
I think that it’s plausible but not likely, and given that it would be the intervention that would cut the most risks, I tend to prefer corporate governance which seems significantly more tractable and neglected to me.
Out of curiosity, could you refer to a specific event you’d expect to see “if we get closer to substantial leaps in capabilities”? I think that it’s a useful exercise to disagree fruitfully on timelines and I’d be happy to bet on some events if we disagree on one.
Thanks for your comment!
That’s an important point that you’re bringing up.My sense is that at the movement level, the consideration you bring up is super important. Indeed, even though I have fairly short timelines, I would like funders to hedge for long timelines (e.g. fund stuff for China AI Safety). Thus I think that big actors should have in mind their full distribution to optimize their resource allocation.
That said, despite that, I have two disagreements:
I feel like at the individual level (i.e. people working in governance for instance, or even organizations), it’s too expensive to optimize over a distribution and thus you should probably optimize with a strategy of “I want to have solved my part of the problem by 20XX”. And for that purpose, identifying the main characteristics of the strategic landscape at that point (which this post is trying to do) is useful.
“the EV gained in the worlds where things move quickly is not worth the expected cost in worlds where they don’t.” I disagree with this statement, even at the movement level. For instance I think that the trade-off of “should we fund this project which is not the ideal one but still quite good?” is one that funders often encounter and I would expect that funders have more risk adverseness than necessary because when you’re not highly time-constrained, it’s probably the best strategy (i.e. in every fields except in AI safety, it’s probably a way better strategy to trade-off a couple of years against better founders).
Finally, I agree that “the best strategies will have more variance” is not a good advice for everyone. The reason I decided to write it rather than not is because I think that the AI governance community tends to have a too high degree of risk adverseness (which is a good feature in their daily job) which penalizes mechanically a decent amount of actions that are way more useful under shorter timelines.
Yep, good point! I just wanted to make clear that IMO a good first-order approximation of your impact on the long-term future is: “What’s the causal impact of your work on AI?”
And even though UX designer for 80k / Community building are not focused on AI, they are instrumentally very useful towards AI, in particular if the person who does it has this theory of change in mind.
If there were no preferences, at least 95% and probably more around 99%. I think that this should update according to our timelines.
And just to clarify, that includes community building etc. as I mentioned.
I think that if you take these infohazards seriously enough, you probably even shouldn’t do that. Because if everyone has a 95% likelihood to keep it secret, with 10 persons in the know is 60%.
Thanks!
Do you think that biorisks/nuclear war could plausibly cause us never to recover our values? What’s the weight you give to such a scenario?(I want to know if the weight you put on “worse values” is due to stable totalitarianism due to new technologies or due to collapse → bad people win).
I know it’s not trivial to do that but if you included your AGI timelines into consideration for this type of forecast, you’d come up with very different estimates. For that reason, I’d be willing to bet on most estimates
Hi Lauren! Thanks for the post! Did you come across some literature on civil wars and life satisfaction ? Because I expect the effect of civil wars on the latter to be significant so I’d be curious to know if there were some estimates.
Thanks for your comment!
First, you have to have in mind that when people are talking about “AI” in industry and policymaking, they usually have mostly non-deep learning or vision deep learning techniques in mind simply because they mostly don’t know the ML academic field but they have heard that “AI” was becoming important in industry. So this sentence is little evidence that Russia (or any other country) is trying to build AGI, and I’m at ~60% Putin wasn’t thinking about AGI when he said that.
I think that you’re deeply wrong about this. Policymakers and people in industry, at least till ChatGPT had no idea what was going on (e.g at the AI World Summit, 2 months ago very few people even knew about GPT-3). SOTA large language models are not really properly deployed, so nobody cared about them or even knew about them (till ChatGPT at least). The level of investment right now in top training runs probably doesn’t go beyond $200M. The GDP of the US is 20 trillion. Likewise for China. Even a country like France could unilaterally put $50 billion in AGI development and accelerate timelines quite a lot within a couple of years.
Even post ChatGPT, people are very bad at projecting what it means for next years and still have a prior on the fact that human intelligence is very specific and can’t be beaten which prevents them from realizing all the power of this technology.
I really strongly encourage you to go talk to actual people from industry and policy to get a sense of their knowledge on the topic. And I would strongly recommend not publishing your book as long as you haven’t done that. I also hope that a lot of people who have thought about these issues have proofread your book because it’s the kind of thing that could really increase P(doom) substantially.
I think that to make your point, it would be easier to defend the line that “even if more governments got involved, that wouldn’t change much”. I don’t think that’s right because if you gave $10B more to some labs, it’s likely they’d move way faster. But I think that it’s less clear.
I agree that it would be something good to have. But the question is: is it even possible to have such a thing?
I think that within the scientific community, it’s roughly possible (but then your book/outreach medium must be highly targeted towards that community). Within the general public, I think that it’s ~impossible. Climate change, which is a problem which is much easier to understand and explain is already way too complex for the general public to have a good idea of what are the risks and what are the promising solutions to these risks (e.g. a lot people’s top priorities is to eat organic food, recycle and decrease plastic consumption).
I agree that communicating with the scientific community is good, which is why I said that you should avoid publicizing only among “the general public”. If you really want to publish a book, I’d recommend targeting the scientific community, which is not at all the same public as the general public.
“On the other hand, if most people think that strong AI poses a significant risk to their future and that of their kids, this might change how AI capabilities researchers are seen, and how they see themselves”
I agree with this theory of change and I think that it points a lot more towards “communicate in the ML community” than “communicate towards the general public”. Publishing great AI capabilities is mostly cool for other AI researchers and not that much for the general public. People in San Francisco (where most of the AGI labs are) also don’t care much about the general public and whatever it thinks ; the subculture there and what is considered to be “cool” is really different from what the general public thinks is cool. As a consequence, I think they mostly care about what their peers are thinking about them. So if you want to change the incentives, I’d recommend focusing your efforts on the scientific & the tech community.