I’ve been thinking a bit around secret efforts in AI safety research.
My current thoughts are around: if it is or does occur what non secret efforts might be needed? E.g. if it develops safe AI media that shows postive outcomes from AI might be needed so that people aren’t overly scared.
Oh and AI policy might be needed too, perhaps limiting certain types of AI (agentic stuff).
How should important ideas around topics like AI and biorisk be shared? Is there a best practice, or government departments that specialise in handling that?
Hi, I’m thinking about a possibly new approach to AI safety. Call it AI monitoring and safe shutdown.
Safe shutdown, riffs on the idea of the big red button, but adapts it for use in simpler systems. If there was a big red button, who gets to press it and how? This involves talking to law enforcement, legal and policy. Big red buttons might be useful for non learning systems, large autonomous drones and self-driving cars are two system that might suffer from software failings and need to be shutdown safely if possible (or precipitously if the risks from hard shutdown are less than it’s continued operation).
The monitoring side of thing asks what kind of registration and monitoring we should have for AIs and autonomous systems. Building on work on aircraft monitoring, what would the needs around autonomous system be?
Is this a neglected/valuable cause area? If so, I’m at an early stage and could use other people to help out.
I found this report on adaptation, which suggest adaptation with some forethought will be better than waiting for problems to get worse. Talks about things other than crops too. The headlines
Without adaptation, climate change may depress growth in global agriculture yields up to 30 percent by 2050. The 500 million small farms around the world will be most affected.
The number of people who may lack sufficient water, at least one month per year, will soar from 3.6 billion today to more than 5 billion by 2050.
Rising seas and greater storm surges could force hundreds of millions of people in coastal cities from their homes, with a total cost to coastal urban areas of more than $1 trillion each year by 2050.
Climate change could push more than 100 million people within developing countries below the poverty line by 2030. The costs of climate change on people and the economy are clear. The toll on human life is irrefutable. The question is how will the world respond: Will we delay and pay more or plan ahead and prosper?
I’ve been thinking for a while civilisational collapse scenarios impact some of the common assumptions about the expected value of movement building or saving for effective altruism. This has knock on implications to when things are most hingeist.
That said, I personally would be quite surprised if worldwide crop yields actually ended up decreasing by 10-30%. (Not an informed opinion, just vague intuitions about econ).
I hope they won’t too, if we manage to develop the changes we need to make before we need them. Economics isn’t magic
But I wanted to point out that there will probably be costs associated with stopping deaths associated with food shortages with adaptation. Are they bigger or smaller than mitigation by reducing CO2 output or geoengineering?
This case hasn’t been made either way to my knowledge and could help allocate resources effectively.
Are there any states that have committed to doing geoengineering, or even experimenting with geoengineering, if mitigation fails?
Having some publicly stated sufficient strategy would convince me that this was not a neglected area.
I’m expecting the richer nations to adapt more easily, So I’m expecting a swing away from food production in the less rich nations as poorer farmers would have a harder time adapting as there farms get less productive (and they have less food to sell). Also farmers with now unproductive land would struggle to buy food on the open market
I’d be happy to be pointed to the people thinking about this and planning on having funding for solving this problem. Who are the people that will be funding the teaching of subsistence rice farmers (of all nationalities) how to farm different crops they are not used to etc? Providing tools and processing equipment for the new crop. Most people interested in climate change I have met are still in the hopeful mitigation phase and if they are thinking about adaptation it is about their own localities.
This might not be a pressing problem now[1], but it could be worth having charities learning in the space about how to do it well (or how to help with migration if land becomes uninhabitable).
[1] https://blogs.ei.columbia.edu/2018/07/25/climate-change-food-agriculture/ suggests that some rice producing regions might have problems soon
On 1) not being able to read the full text of the impactlab report, but it seem they just model the link between heat and mortality, but not the impact of heat on crop production causing knock on health problems. E.g. http://dels.nas.edu/resources/static-assets/materials-based-on-reports/booklets/warming_world_final.pdf suggests that each degree of warming would reduce the current crop yields by 5-15%. So for 4 degrees warming (baseline according to https://climateactiontracker.org/global/temperatures/ ), this would be 20-60% of world food supply reduction.
If governments stick to their policies (which they have been notoriously bad at so far) then the reduction would only be 10-30%. I’d expect even a 10% decrease to have massive knock on effects to the nutrition and mortality of the world. I expect that is not included in the impact lab report because it is very hard to have papers that encompass the entire scope of the climate crisis.
Of course there could be a lot of changes to how and where we grow crops to avoid these problems, but making sure that we manage this transition well, so that people in the global south can adopt the appropriate crops for whatever their climate becomes seems like something that could use some detailed analysis. It seems neglected as far as I can tell, there may be simple things we can do to help. It is not mainstream climate change mitigation though, so might fit your bill?
As currently defined, long termists have two possible choices.
Direct work to reduce X-risk
Investing for the future (by saving or movement building) to then spend on reduction of x-risk at a later date
There are however other actions that may be more beneficial.
Let us look again at the definition of influential again
a time ti is more influential (from a longtermist perspective) than a time tj iff you would prefer to give an additional unit of resources,[1] that has to be spent doing direct work (rather than investment), to a longtermist altruist living at ti rather than to a longtermist altruist living at tj.
While direct work is not formally defined it can be seen here to be mainly referred to as near-term existential risk mitigation.
The most obvious implication, however, is regarding what proportion of resources longtermist EAs should be spending on near-term existential risk mitigation versus what I call ‘buck-passing’ strategies like saving or movement-building.
What happens if the answer is neither option? What are the other levers we have on the future. One is that we might be able to take actions that change the expected rate of return. Perhaps the expected rate of return on investments is very bad, but there are actions you can take to increase it to more normal levels? Or there is low-hanging fruit for vastly increasing the expected rate of return on investments in the long term.
So let us introduce another type of influential (and another version of hingeness). Influential-i, that is the time that has the most influential action being ones that attempt to impact interest rates.
a time ti is more influential-i (from a longtermist perspective) than a time tj iff you would prefer to give an additional unit of resources,[1] that has to be spent doing direct work to alter investment rates (rather than normal investment itself or X-risk reduction), to a longtermist altruist living at ti rather than to a longtermist altruist living at tj.
So what would increase the rate of return on investment? New energy sources with high energy returned on energy invested could increase the rate of return on investment. For example, if you manage to help invent nuclear fusion you would increase the amount of cleaner energy available to civilisation giving more resources to future altruists to use to solve problems.
Avoiding vast decreases in the rate of return would be actions that manage to stave of civilization collapse. Civilization collapse should be a lot higher on the radar for long term altruists, as it is.
More likely than existential risk (considering civilisations have collapsed in the past and there are inside views for it happening in the future)
Likely to cause a collapse in the effective altruism movement as well as people focus lower on maslow’s hierarchy of needs.
Likely to cause hyperinflation (at the minimum), wiping out savings.
Lowering/destroying existing existential risk reduction mitigation efforts (meteorite monitoring programs).
The long-termist community would do well to look at these options when thinking about the time frame of hundreds of years.
Let’s say they only mail you as much protein as one full human genome.
This doesn’t make sense. Do you mean proteome? There is not a 1-1 mapping between genome and proteome. There are at least 20,000 different proteins in the human proteome, it might be quite noticeable (and tie up the expensive protein producing machines), if there were 20,000 orders in a day. I don’t know the size of the market, so I may be off about that.
I will be impressed if the AI manages to make a biological nanotech that is not immediately eaten up or accidentally sabotaged by the soup of hostile nanotech that we swim in all the time.
There is a lot of uranium in the sea, only because there is a lot of sea. From the pages I have found, there is only 3 micrograms of U per liter, and 0.72 percent is U235. To get the uranium 235 (80% enriched 50Kg bomb) required for a single bomb you would need to process roughly 18 km3 of sea water or 1.8 * 10^13 liters.
This would be pretty noticeable if done in a short time scale (you might also have trouble with diluting the sea locally if you couldn’t wait for diffusion to even out the concentrations globally).
To build 1 million nukes you would need more sea water than the Mediterranean (3.75 million km3)
There might be a further consideration, people might not start or fund impactful startups if there wasn’t a good chance of getting investment. The initial investors (if not impact oriented), might still be counting on impact oriented people to buy the investment. So while each individual impact investor is not doing much in isolation, collectively they are creating a market for things that might not get funded otherwise. How you account for that I’m not sure.
It might be worth looking at the domains where it might be less worthwhile (formal chaotic systems, or systems with many sign flipping crucial considerations). If you can show that trying to make cost-effectiveness based decisions in such environments is not worth it, that might strengthen your case.
Hi Gregory,
A couple of musings generated by your comment.
2: I don’t think there’s a neat distinction between ‘technical dangerous information’ and ‘broader ideas about possible risks’, with the latter being generally safe to publicise and discuss.
I have this idea of independent infrastructure, trying to make infrastructure (electricity/water/food/computing) that is on a smaller scale than current infrastructure. This is for a number of reasons, one of which includes mitigating risks, How should I build broad-scale support for my ideas without talking about the risks I am mitigating?
4.1: In addition to the considerations around the unilateralist’s curse offered by Brian Wang (I have written a bit about this in the context of biotechnology here) there is also an asymmetry in the sense that it is much easier to disclose previously-secret information than make previously-disclosed information secret. The irreversibility of disclosure warrants further caution in cases of uncertainty like this.
Although in some scenarios non-disclosure is irreversible as well, as conditions change. Consider if someone had the idea of hacking a computer and had managed to convince the designers of C to create a more secure list indexing and also everyone not to use other insecure languages. Now we would not be fighting the network effect of all the bad C code when trying to get people to code computers securely.
This irreversibility of non-disclosure seems to only occur if if something is not a huge threat right now, but may become more so as technology develops and gets more widely used and locked in. Not really relevant to the biotech arena. that I can think of immediately at least. But an interesting scenario nonetheless.
For people outside of EA, I think those who are in possession of info hazard-y content are much more likely to be embedded in some sort of larger institution (e.g., a research scientist or a journal editor looking to publish something), where perhaps the best leverage is setting up certain policies, rather than trying to teach everyone the unilateralist’s curse.
There is a growing movement of maker’s and citizen scientists that are working on new technologies. It might be worth targeting them somewhat (although again probably without the math). I think the approaches for ea/non-ea seem sensible.
You’re right, strict consensus is the wrong prescription. A vote is probably better. I wonder if there’s mathematical modeling that you could do that would determine what fraction of votes is optimal, in order to minimize the harms of the standard unilateralist’s curse and the curse in reverse? Is it a majority vote? A 2/3s vote? l suspect this will depend on what the “true sign” of releasing the potentially dangerous info is likely to be; the more likely it is to be negative, the higher bar you should be expected to clear before releasing.
I also like to weigh the downside of the lack of releasing the information as well. If you don’t release information you are making everyone make marginally worse decisions (if you think someone will release it anyway later). For example in the nuclear fusion example, you think that everyone currently building new nuclear fission stations are wasting their time, that people training on how to manage coal plants should be training on something else etc, etc.
I also have another consideration which is possibly more controversial. I think we need some bias to action, because it seems like we can’t go on as we are for too much longer (another 1000 years might be pushing it). The level of resources and coordination towards global problems fielded by the status quo seems insufficient. So it is a default bad outcome.
With this consideration, going back to the fusion pioneers, they might try and find people to tell so that they could increase the bus factor (the number of people that would have to die to lose the knowledge). They wouldn’t want the knowledge to get lost (as it would be needed in the long term) and they would want to make sure that whoever they told understood the import and potential downsides of the technology.
Edit: Knowing the sign of an intervention is hard, even after the fact. Consider the invention and spread of the knowledge about nuclear chain reactions. Without it we would probably be burning a lot more fossil fuels, however with it we have the existential risk associated with it. If that risk never pays out, then it may have been a spur towards greater coordination and peace.
I’ll try and formalise these thoughts at some point, but I am bit work impaired for a while.
Ah right. I suppose the unilateralist’s curse is only a problem insofar as there are a number of other actors also capable of releasing the information; if you are a single actor then the curse doesn’t really apply. Although one wrinkle might be considering the unilateralist’s curse with regards to different actors through time (i.e., erring on the side of caution with the expectation that other actors in the future will gain access to and might release the information), but coordination in this case might be more challenging.
Interesting idea. This may be worth trying to develop more fully?
Probably it’s best to discuss privately with a number of other trusted individuals first, who also understand the unilateralist’s curse,
I’m still coming at this from a lens of “actionable advice for people not in ea”. It might be that the person doesn’t know many other trusted individuals, what should be the advice then? It would probably also be worth giving advice on how to have the conversation as well. The original article gives some advice on what happens if consensus can’t be reached (voting/such like).
As I understand it you shouldn’t wait for consensus else you have the unilateralist’s curse in reverse. Someone pessimistic about an intervention can block the deployment of an intervention needed to avoid disaster (this seems very possible if you consider crucial considerations flipping signs, rather than just random noise in beliefs in desirability).
Would you suggest discussion and vote (assuming no other courses of action can be agreed upon)? Do you see the need to correct for status quo bias in any way?
This seems very important to get right. I’ll think about this some more.
My understanding is that it applies regardless of whether or not you expect others to have the same information. All it requires is a number of actors making independent decisions, with randomly distributed error, with a unilaterally made decision having potentially negative consequences for all.
Information determines the decisions that can be made. For example you can’t spread the knowledge of how to create effective nuclear fusion without the information on how to make it.
If there is a single person with the knowledge of how to create safe efficient nuclear fusion they cannot expect other people to release it on their behalf. They may expect it to be net positive but they also expect some downsides and are unsure of whether it will be net good or not. To give a potential downside of nuclear fusion, let us say they are worried about creating excess heat over what the earth can dissipate due to widescale deployment in the world (even if it fixes global warming due to trapping solar energy, it might cause another heat related problem). I forget the technical term for this unfortunately.
The fusion expert(s) cannot expect other people to release this information for them, for as far as they know they are the only people making that exact decision.
I’m also not sure how the strategy of “preemptively release, but mitigate” would work in practice. Does this mean release potentially dangerous information, but with the most dangerous parts redacted? Release with lots of safety caveats inserted? How does this preclude the further release of the unmitigated info?
What the researcher can do is try and build consensus/lobby for a collective decision making body on the internal climate heating (ICH) problem. Planning to release the information when they are satisfied that there is going to be a solution in time for fixing the problem when it occurs.
If they find a greater than expected number of people lobbying for solutions to the ICH problem, then they can expect they are in a unilateralist’s curse scenario. And they may want to hold off on releasing information even when they are satisfied with the way things are going (in case there is some other issue they have not thought of).
They can look to see what the other people are doing that have been helping with ICH and see if there other initiatives they are starting, that may or may not be to do with the advent of nuclear fusion.
I think I am also objecting to the expected payoff being thought of as a fixed quantity. You can either learn more about the world to alter your knowledge of the payoff or try and introduce things/insituttions into the world to alter the expected payoff. Building useful institutions may rely on releasing some knowledge, that is where things become more hairy.
I’ve not had the best luck reaching out to talk to people about my ideas. I expect that the majority of new ideas will come from people not heavily inside the group and thus less influenced by group think. So you might want to think of solutions that take that into consideration.
I’m not sure I’m fully understanding you here. If you’re saying that the majority of potentially dangerous ideas will originate in those who don’t know what the unilateralist’s curse is, then I agree –– but I think this is just all the more reason to try to –– but I think this is just all the more reason to try to spread norms of consensus.
I was suggesting that more norm spreading should be done outwards, keeping it simple and avoiding too much jargon. Is there a presentation of the unilateralist’s curse aimed at micro biologists for example?
Also as the the unilaterlist’s curse suggests discussing with other people such that they can undertake the information release, sometimes increases the expectation of a bad out come. How should consensus be reached in those situations?
Increasing the number of agents capable of undertaking the initiative also exacerbates the problem: as N grows, the likelihood of someone proceeding incorrectly increases monotonically towards 1.7 The magnitude of this effect can be quite large even for relatively small number of agents. For example, with the same error assumptions as above, if the true value of the initiative V* = −1 (the initiative is undesirable), then the probability of erroneously undertaking the initiative grows rapidly with N, passing 50% for just 4 agents.
And if no one is working on it, is there an organisation that would be interested in starting working on it?