Have there been any models of other activities that might reduce existential risk.
E.g. convincing prospective AGI researchers that it is dangerous and should be handled carefully? It would seem that that might increase the pool of potential researchers and also give more time for a safe approach to be developed?
I think I broadly agree that take off is likely to be slow and that there is not a slam dunk argument for trying to make safe super intelligent agents.
However I think there is room for all sorts of work. Anything that can reduce the uncertainty of where AGI is going.
I think AI, as it is, is on slightly the wrong track. If we get on the right track we will get somewhere a lot quicker than the decades referenced above.
Computers as they stand are designed with the idea of having a human that looks after them and understands their inner workings, at least somewhat. Animals from the lowly nematode to humans do not have that assumption. Current deep learning assumes a human will create the input and output spaces and assign resources to that learning process.
If we can off load the administration of a computer to the computer itself, this would allow cheaper administration of computers and also the computer systems to become more complex. Computer systems are limited in complexity by the thing that debugs them.
I have an idea of what this might look like and if my current paradigm plays out, I think humanity will get the choice of creating separate agents or creating external lobes of our brains. Most likely humanity will pick the creating external lobes. The external lobes may act in a more economic fashion, but I think they still might have the capability of going bad. Minimising the probability of this is very important.
I think there is also probably a network effect, if we could get altruistically minded people to be the first to have the external brains then we might influence the future by preferentially helping other altruists to get external brains. This could create a social norms among people with external brains.
So I think technical work towards understanding administratively autonomous computers (no matter how intelligent they are) can reduce uncertainty and allow us to understand what choices face us.
My criticism of the HRAD research project is that it has no empirical feedback mechanisms and that the ignored physical aspect of computation can have a large impact on the type of systems you think about and design.
I think people thinking highly formally about AI systems might be useful as long as the real world can be used to constrain their thinking
My own 2 cents. It depends a bit what form of general intelligence is made first. There are at least two possible models.
Super intelligent agent with a specified goal
External brain lobe
With the first you need to be able to specify a human preferences in the form of a goal. Which enables it to pick the right actions.
The external brain lobe would start not very powerful and not come with any explicit goals but would be hooked into the human motivational system and develop goals shaped by human preferences.
HRAD is explicitly about the first. I would like both to be explored.
On the meta side of things:
I found ai impacts recently recently. There is a group I am loosely affiliated that is trying to make a MOOC about ai safety.
If you care about doing something about immense suffering risks (s-risks) you might like the foundational research institute.
There is an overview of other charities but it is more favourable of HRAD style papers.
I would like to set up an organisation that studies autonomy and our response to making more autonomous things (especially with regards to adminstrative autonomy). I have a book slowly brewing. So if you are interested in that get in contact.
Fixed, thanks.
I agree that HRAD might be useful. I read some of the stuff. I think we need a mix of theory and practice and only when we have community where they can feed into each other will we actually get somewhere. When an AI safety theory paper says, “Here is an experiment we can do to disprove this theory,” then I will pay more attention than I do.
The “ignored physical aspect of computation” is less about a direction to follow, but more an argument about the type of systems that are likely to be effective and so an argument about which ones we should study. There is no point studying how to make ineffective systems safe if the lessons don’t carry over to effective ones.
You don’t want a system that puts in the same computational resources trying to decide what brand of oil is best for its bearings as it does to deciding the question of what is a human or not. If you decide how much computational resources you want to put into each class of decision, you start to get into meta-decision territory. You also need to decide how much of your pool you want to put into making that meta-decision as making it will take away from making your other decisions.
I am thinking about a possible system which can allocate resources among decision making systems and this can be used to align the programs (at least somewhat). It cannot align a super intelligent malign program, work needs to done on the initial population of programs in the system, so that we can make sure they do not appear. Or we need a different way of allocating resources entirely.
I don’t pick this path because it is an easy path to safety, but because I think it is the only path that leads anywhere interesting/dangerous and so we need to think about how to make it safe.
Hmm, maybe
Global EA vs local EA
Total EA vs focused EA
But it’s equally plausible that the larger system will be enforcing morally correct standards and a minority of individuals will want to do something wrong (like slavery or pollution).
Both of these would impinge on the vital sets of others though (slavery directly, pollution by disrupting the natural environment people rely on). So it would still be a bad outcome from the autonomy viewpoint if these things happened.
The autonomy viewpoint is only arguing that lots of actions should physically possible for people, all the actions physically possible aren’t necessarily morally allowed.
Which of these three scenarios is best?:
No one has guns so no-one gets shot
Everyone has guns and people get shot because they have accidents
Everyone has guns but no-one gets shot because they are well trained and smart.
The autonomy viewpoint argues that the third is the best possible outcome and tries to work towards it. There are legitimate uses for guns.
I don’t go into how these things should be regulated, as this is a very complicated subject. I’ll just point out that to get the robust free society that I want you would need to not regulate the ability to do these things, but make sure the incentive structures and education is correct.
I don’t see a reason that we ought to intrinsically value autonomy. The reasons you gave only support autonomy instrumentally through other values.
I don’t think I can argue for intrinsically valuing anything. I agree with not being able to argue ought from is. So the best I can do is either claim it as a value, which I do, and refer to other value systems that people might share. I suppose I could talk about other people who value autonomy in itself. Would you find that convincing?
But by the original criteria, a single AI would (probably) be robust to catastrophe due to being extremely intelligent and having no local competitors.
Unless it is omniscient I don’t see how it will see all threats to itself. It may lose a gamble on the logical induction lottery and make an ill-advised change to itself.
Also what happens if a decisive strategic advantage is not possible and this hypothetical single AI does not come into existence. What is the strategy for that chunk of probability space?
If it is a good friendly AI, then it will treat people as they deserve, not on the basis of thin economic need, and likewise it will always be morally correct. It won’t be racist or oppressive.
I’m personally highly skeptical that this will happen.
I bet no one will want to leave its society, but if we think that that right is important then we can design an AI which allows for that right.
How would that be allowed if those people might create a competitor AI?
you reach some inappropriate conclusions because future scenarios are often different from present ones in ways that remove the importance of our present social contexts.
Like I said in the article, if I was convinced of decisive strategic advantage my views of the future would be very different. However as I am not, I have to think that the future will remain similar to the present in many ways.
Thanks. I know I need to do more reading around this. This looks like a good place to start.
I suspect it might be because I’ve not couched the website enough in terms of EA? I like EA and would love to work with people from the community and perhaps get the website more EA friendly. I’ve not been massively encouraged by the EA yet.
There is lots to say about autonomy, so I look forward to any forthcoming comments.
Thanks fixed. I should put some money towards a copy editor at some point or time to figuring out an automated solution.
What are the most important considerations for assessing charities doing uncertain-return stuff?
I think an important one is: How likely is the project to reduce the uncertainty of the return?
E.g. will it decide a crucial consideration
Edit to give more detail:
Resolving a crucial consideration increases the value of all your future research massively. Take for example the question of whether that will be a hard or slow take off. Hard take off favours AI safety now, whereas soft take off favours building political and social institutions that encourage cooperation and avoid wars. As they both have humanity’s future on the line they are both equally massively important, conditioned on them being the scenario that might happen.
Resolving the question (or at least driving down the uncertainty) would allow the whole community to focus on the right scenario and get a lot better bang for their buck. Even if it doesn’t directly address the problem.
The is-ought problem doesn’t say that you can’t intrinsically value anything
I never said it did, I said it means I can’t argue that you should intrinsically value anything. What arguments could I give to a paper clipper to stop its paper clipping ways.
That said I do think I can argue for a plurality of intrinsic values.
1) They allow you to break ties. If there are two situations with equal well being then having a second intrinsic value would give you a way of picking between the two.
2) It might be computationally complicated or informationally complicated to calculate your intrinsic value. Having another that is not at odds with it and generally correlates with it allows you to optimise for that. For example you could optimise for political freedom which would probabilistically lead to more eudaemonia, even if it is not the case that more political freedom always leads to more eudaemonia under all cases. As you can’t measure the eudaemonia of everyone.
It will see most threats to itself in virtue of being very intelligent and having a lot of data, and will have a much easier time by not being in direct competition. Basically all known x-risks can be eliminated if you have zero coordination and competition problems.
I am thinking about unknown internal threats. One possibility that I alluded to is if it modifies itself to improve itself, but it does that on shakey premise and destroys itself. Another possibility is that parts of may degrade and/or get damaged and it gets the equivalent of cancers.
I’m personally highly skeptical that this will happen.
Okay, but the question was “is a single AI a good thing,” not “will a single AI happen”
I was assuming a single non-morally perfect AI as that seems like the most likely outcome of the drive to a single AI to me.
It will be allowed by allowing them to exist without allowing them to create a competitor AI. What specific part of this do you think would be difficult? Do you think that everyone who is allowed to exist must have access to supercomputers free of surveillance?
If they are not free of surveillance, then they have not left the society. I think it a preferable world if we can allow everyone to have supercomputers because they are smart and wise enough to use them well.
I think we want different things from our moral systems. I think my morality/value is complicated and best represented by different heuristics that guide how I think or what I aim for. It would take more time than I am willing to invest at the moment to try and explain my views fully.
Why should we care about someone’s desire to have a supercomputer which doesn’t get checked for the presence of dangerous AGI...?
Why should we care about someone’s desire to have their thoughts not checked for the presence of malicious genius? They may use their thinking to create something equally dangerous that we have not yet thought of.
Why care about freedom at all?
If I upload and then want to take a spaceship somewhere hard to monitor, will I be allowed to take a super computer, if I need it to perform science?
What is in my pocket was once considered a dangerous super computer. The majority of the world is now trusted with it, or at least the benefits of having them out weigh the potential costs.
Drawdown a book on possible climate change solutions seems EA relevant. It is interesting that it only allows peer reviewed data/models in it and systematically surveys all the solutions they could find.
I’ve been reading Superforecasting and my take away is that to have good predictions at the world you need to have a multiplicity of view points and quantify and breakdown the estimates fermi-style.
So my question is, has there been any collective attempts at model building for prediction purposes? Try and get all the hedgehogs together with their big ideas and synthesize them to form a collective fox-y model?
I know there are prediction markets, but you don’t know what information that a price has synthesized so it is hard to bet on them, if you only have a small bit of information and do not think you know better than the market as a whole.
It would seem that if we could share a pool of predictive power between us we could make better decisions about how to intervene in the world.
One last one.
I’m writing more on my blog about my approach to intelligence augmentation
I’ll be coding and thinking about how to judge it’s impact this week (a lot of it depends on things like hard vs soft takeoff, possibilities of singletons and other crucial considerations). I’m also up for spending a few hours helping people with IA or autonomy based EA work, if anyone needs it?
This post covers what I want to get out of my autonomy measure. I think I might have pitched it wrong to start with. It is more along the lines of trying to increase the world economy than a base value. It also covers some of my initial forrays into the research into whether freedom is psychologically useful (I need to find out how reliable these are).
I’d not thought to look at it, I assumed it was/stayed an IARPA thing and so focused on world affairs. Thanks!
It looks like it has become a for-profit endeavour now with an open component.
From the looks of it there are no ways to submit questions and you can’t see the models of the world used to make the predictions, so I’m not sure if charities (or people investing in charities) can gain much value from it.
We would want questions of the form: if intervention Y occurs what is the expected magnitude of outcome Z.
I’m not sure how best to tackle this.
There has recently been an effort started to make the pipeline better for getting people up to speed with AGI safety. I’m trying to champion a broad view of AGI safety including psychology.
Would anyone be interested in providing digested content? It would also be good to have an exit for the pipeline for psychology people interested in AGI. Would that be FHI? Who else would be good to talk to about what is required.