Thanks for this! I think it would be helpful to plot the median changes in extinction probabilities against the number of words in the article/video. I’m noticing a correlation as I click through the links and would be curious how strong it is (so this effect can be disentangled from the style of the source).
Joshc
Karma: 549
Yep, I have some ideas. Please DM me and give some info about yourself if you are interested in hearing them :)
Thanks for the referral. I agree that the distinction between serial time and parallel time is important and that serial time is more valuable. I’m not sure if it is astronomically more valuable though. There are two points we could have differing views on:
- the amount of expected serial time a successful (let’s say $10 billion dollar) AI startup is likely to counterfactually burn. In the post I claimed that this seems unlikely to be more than a few weeks. Would you agree with this?
- the relative value of serial time to money (which is exchangeable with parallel time). If you agree with the first statement, would you trade $10 billion dollars for 3 weeks of serial time at the current margin?
If you would not trade $10 billion for 3 weeks that could be because:
- I’m more optimistic about empirical research / think the time iterating later when we have the systems is significantly more important than the time now when we can only try to reason about them.
- you think money will be much less useful than I expect it to be
I would be interested in pinning down where I disagree with you (and others who probably disagree with the post for similar reasons).
If your claim is that ‘applying AI models for economically valuable tasks seems dangerous, i.e. the AIs themselves could be dangerous’ then I agree. A scrappy applications company might be more likely to end the world than OpenAI/DeepMind… it seems like it would be good, then, if more of these companies were run by safety conscious people.
A separate claim is the one about capabilities externalities. I basically agree that AI startups will have capabilities externalities, even if I don’t expect them to be very large. The question, then, is how much expected money we would be trading for expected time and what is the relative value between these two currencies.
It’s unclear to me that having EA people starting an AI startup is more tractable than convincing other people that the work is worth funding
Yeah, this is unclear to me to. But you can encourage lots of people to pursue earn-to-give paths (maybe a few will succeed). Not many are in a position to persuade people, and more people having this as an explicit goal seems dangerous.
Also, as an undergraduate student with short timelines, the startup path seems like a better fit.I don’t see how the flexibility of money makes any difference? Isn’t it frustratingly difficult to predict which uses of money will actually be useful for AI safety? In that case, you still have the same problem.
I have to make important career decisions right now. It’s hard to know what will be useful in the future, but it seems likely that money will be. I could have made that point clearer.
That’s a good point. Here’s another possibility:
Require that students go through a ‘research training program’ before they can participate in the research program. It would have to actually help prepare them for technical research though. Relabeling AGISF as a research training program would be misleading, so you would want to add a lot more technical content (reading papers, coding assignments, etc.) It would probably be pretty easy to gauge how much the training program participants care about X-risk / safety and factor that in when deciding whether to accept them into the research program.
The social atmosphere can also probably go a long way in influencing people’s attitudes towards safety. Making AI risk an explicit focus of the club, talking about it a lot at socials, inviting AI safety researchers to dinners, etc might do most of the work tbh.
Oo exciting. Yeah, the research program looks like it is closer to what I’m pitching.
Though I’d also be excited about putting research projects right at the start of the pipeline (if they aren’t already). It looks like AGISF is still at the top of your funnel and I’m not sure if discussion groups like these will be as good for attracting talent.
Late to the party here, but I was wondering why these organizations need aligned engineering talent. Anthropic seems like the kind of org that talented, non-aligned people would be interested in...
These are reasonable concerns, thanks for voicing them. As a result of unforeseen events, we became responsible for running this iteration only a couple of weeks ago. We thought that getting the program started quickly — and potentially running it at a smaller scale as a result — would be better than running no program at all or significantly cutting it down.
The materials (lectures, readings, homework assignments) are essentially ready to go and have already been used for MLSS last summer. Course notes are supplementary and are an ongoing project.
We are putting a lot of hours into making sure this program gets started without a hitch and runs smoothly. We are sorry the deadlines are so aggressive and agree that it would have been better to launch earlier. If you have trouble getting your application in on time, please don’t hesitate to contact us about getting an extension. We also plan to run another iteration in the Spring and announce the program further in advance.
Yes! Thanks for asking.
Fixed, thanks!
Yeah, I would be in favor of interaction in simulated environments—other’s might disagree, but I don’t think this influences the general argument very much as I don’t think leaving some matter for computers will reduce the number of brains by more than an order of magnitude or so.
Having a superintelligence aligned to normal human values seems like a big win to me!
Not super sure what this means but the ‘normal human values’ outcome as I’ve defined it hardly contributes to EV calculations at all compared to the utopia outcome. If you disagree with this, please look at the math and let me know if I made a mistake.
Yep, I didn’t initially understand you. That’s a great point!
This means the framework I presented in this post is wrong. I agree now with your statement:the EV of partly utilitarian AI is higher than that of fully utilitarian AI.
I think the framework in this post can be modified to incorporate this and the conclusions are similar. The quantity that dominates the utility calculation is now the expected representation of utilitarianism in the AGI’s values.
The two handles become:
(1) The probability of misalignment.
(2) The expected representation of utilitarianism in the moral parliament conditional on alignment.
The conclusion of the post, then, should be something like “interventions that increase (2) might be underrated” instead of “interventions that increase the probability of fully utilitarian AGI are underrated.”
Yep, thanks for pointing that out! Fixed it.
...I haven’t seen much discussion about the downsides of delaying
I’m not sure how your first point relates to what I was saying in this post; but, I’ll take a guess. I said something about how investing in capabilities at anthropic could be good. An upside to this would be increasing the probability that EAs end up controlling the super-intelligent AGI in the future. The downside is that it could shorten timelines, but hopefully this can be mitigated by keeping all of the research under wraps (which is what they are doing). This is a controversial issue though. I haven’t thought very much about whether the upsides outweigh the downsides, but the argument in this post caused me to believe the upsides were larger than I thought before.
Also I’m not sure about outcome 1 having zero utility...
It doesn’t matter what outcome you assign zero value to as long as the relative values are the same since if a utility function is an affine function of another utility function then they produce equivalent decisions.
I agree with Zach Stein-Perlman. I did some BOTECs to justify this (see ‘evaluating outcome 3’). If a reasonable candidate for a ‘partially-utilitarian AI’ leads to an outcome where there are 10 billion happy humans on average per star, then an AI that is using every last Joule of energy to produce positive experiences would produce at least ~ 10^15 times more utility.
These are great! I’ll add that you should be careful not to overbook yourself. I would leave an hour and a half in the middle of the day open in case you want to take a nap.
This could be helpful. Maybe posting questions on the EA forum and allowing the debate to happen in the comments could be a good format for this.
Got it! I edited the point about in-person reading so that it provides a more accurate portrayal of what you all are doing.
The competition was cancelled. I think the funding for it was cut, though @Oliver Z can say more. I was not involved in this decision.