Vojta Kovarik. AI alignment and game theory researcher.
Pivocajs
Karma: 83
Of course, my views on this issue are by no means set in stone and still evolving. I’m happy to elaborate on my reasons for preferring this more modest usage if you are interested.
I think the more modest usage is reasonable choice.
Maybe you had a different country in mind. [regarding top-secret security clearance]
I am Czech. We do have the institute, and use it. But, as far as I know, our president doesn’t have it, and a bunch of other people don’t have it. (I.e., it seems that people who need secret information on a daily basis have it. But you don’t need it for many other positions from which you could put pressure on people who have the clearance.)
I would like to highlight an aspect you mention in the “other caveats”: How much should you discount for Goodharting vs doing things for the right reasons? Or, relatedly, if you work on some relevant topic (say, Embedded Agency) without knowing that AI X-risk could be a thing, how much less useful will your work be? I am very uncertain about the size of this effect—maybe it is merely a 10% decrease in impact, but I wouldn’t be too surprised if it decreased the amount of useful work by 98% either.
Personally, I view this as the main potential argument against the usefulness of academia. However, even if the effect is large, the implication is not that we should ignore academia. Rather, it would suggest that we can get huge gains by increasing the degree to which academics do the research because of the right reasons.
(Standard disclaimers apply: This can be done in various ways. Viktoria Krakovna’s list of specification gaming examples is a good one. Screaming about how everybody is going to die tomorrow isn’t :P.)
Vojta
Some thoughts:
1) Most importantly: In your planning, I would explicitly include the variable of how happy you are. In particular, if the AI Safety option would result in a break-up of a long-term & happy relationship, or cause you to be otherwise miserable, it is totally legitimate to not do the AI Safety option. Even if it was higher “direct” impact. (If you need an impact-motivated excuse—which might even be true—then think about the indirect impact of avoiding signalling “we only want people who are so hardcore that they will be miserable just to do this job”.)2) My guess: Given that you think your QC work is unlikely to be relevant to AI Safety, I personally believe that (ignoring the effect on you), the AI Safety job is higher impact.
3) Why is it hard to hire world experts to work on this? (Some thoughts, possibly overlapping with what other people wrote.)
“world experts in AI/ML” are—kinda tautologically—experts in AI/ML, not in AI Safety. (EG, “even” you and me have more “AI Safety” expertise than most AI/ML experts.)
Most problems around AI Safety seem vague, and thus hard to delegate to people who don’t have their own models of the topic. Such models take time to develop. So these people might not be productive for a year (or two? or more? I am not sure) even if they are genuine about AI Safety work.
Top people might be more motivated by prestige than money. (And being “bought off” seems bad from this point of view, I guess.)
Top people might be more motivated by personal beliefs than money. (So the bottleneck is convincing them, not money.)
4) I am tempted to say that all the people who could be effectively bought with money are already being bought with money, so you donating doesn’t help here. But I think a more careful phrasing is “recruiting existing experts is bottlenecked on other things than money (including people coming up with good recruiting strategies)”.
5) Phrased differently: In our quest for developing the AI Safety field, there is basically no tradeoff between “hiring ‘more junior’ people (like you)” and “recruiting senior people”, even if those more junior people would go earning to give otherwise.
This gave me an idea for an experiment/argument. Posting here, in case somebody wants come up with a more thought-out version of it and do it.
[On describing what would change his mind:] You couldn’t find weird behaviors [in the AI], no matter how hard you tried.
People like to take an AI, poke it, and then argue “it is doing [all these silly mistakes], therefore [not AGI/not something to worry about/...]”. Now, the conclusion might be right, but the argument is wrong—even dangerous things can be stupid is some settings. Nevertheless, the argument seems convincing.
My prediction is that people make a lot of mistakes[1] that would seem equally laughable if it was AI that made them. Except that we are so used to them that we don’t appreciate it. So if one buys the argument above, they should conclude that humans are also [not general intelligence/something to worry about/...]. So perhaps if we presented the human mistakes right, it could become a memorable counterargument to “AI makes silly mistakes, hence no need to worry about it”.
Some example formats:
“Look at these silly AI mistakes! Surprise, that’s normal people.” or
“Quizz: AI mistake or human mistake?”
(uhm, or “Quizz: AI or Trump?”; wouldn’t mention this, except bots on that guy already exist).
Obligatory disclaimer: It might turn out that humans really don’t make [any sorts of] [silly mistakes current AI makes], or make [so few that it doesn’t matter]. If you could operationalize this, that would also be valuable.
- ^
What is “these mistakes”? I don’t know . Exercise for the reader.
Yes, the expected utility is larger. The claim is that there is nothing incoherent about not maximising expected utility in this case.
To try rephrasing: Principle 1: if you have to choose a X% chance of getting some outcome A, and a >=X% chance of a strictly better outcome B, you should take B. Principle 2: if you will be facing a long series of comparably significant choices, you should decide each of them based on expected utility maximisation. Principle 3: you should do expected utility maximisation for every single choice. Even if that is the last/most important choice you will ever make.
The claim is that: P1 is solid. P2 follows from P1 (via Central Limit Theorem, or whatever math). But P3 does not follow from P1/P2, and there will be cases where it might be justified to not obey P3. (Like the case with 51% chance of doubling the world’s goodness, 49% chance of destroying it.)
Note that I am not claiming it’s wrong to do expected utility maximisation in all scenarios. Just saying that it both doing and not doing it is OK. And therefore it is (very?) non-strategic to associate your philosophical movement with it. (Given that most people’s intuitions seem to be against it.)
Does this explanation make sense? Maybe I should change the title to something with expected utility?
Agree with Acylhalide’s point—you only need to be non-Dutchbookable by bets that you could actually be exposed to.
To address a potential misunderstanding: I agree with both Sharmake’s examples. But they don’t imply you have to maximise expected utility always. Just when the assumptions apply.
More generally: expected utility maximisation is an instrumental principle. But it is justified by some assumptions, which don’t always hold.
Yes, sure, probabilities are only in the map. But I don’t think that matters for this. Or I just don’t see what argument you are making here. (CLT is in the map, expectations are taken in the map, and decisions are made in the map (then somehow translated into the territory via actions). I don’t see how that says anything about what EV reasoning relies on.)
Uhm, yes, that seems right. That’s why this matters.
“National greatness, and staying ahead of China for world influence requires that we have the biggest economy. To do that, we need more people.” -Matt Yglesias, One Billion Americans.
Yeah, the guy who has chosen to have one child is going to inspire me to make the sacrifices involved in having four. It might be good for America, but the ‘ask’ here looks like it is that I sacrifice my utility for Matt’s one kid, and thus is not cooperate-cooperate. I’ll jump when you jump.Two pushbacks here:
(1) The counterargument seems rather weak here, right? Even if Matt Yglesias had no kids, that doesn’t mean his argument isn’t valid. EG, if a single non-vegan person claims that more people should be vegan, will you view that as evidence that people should not be vegan? ;-) (Not that I disagree with your claim. Just with your argument.)
(2) Did you actually read One Billion Americans, or are you just taking the citation and interpretting it as Matt Yglesias making an argument for people having more children? I didn’t read the book, so I am not sure. But I listened to a podcast with Matt Yglesias, about the book, and my impression was that he was primarily arguing for changing the immigration policy, and (if memory serves) not really making any strong claims about how many kids people should have.
I saw the line “found no good evidence that anything failed any criterion”, but just to check explicitly: What do the confidence levels mean? In particular, should I read “low confidence” as “weak evidence that X feels pain-as-operationalized-by-Criterion Y”? Or as “strong evidence that X does not feel pain-as-operationalized-by-Criterion Y”?
In other words:
Suppose you did the same evaluation for the order Rock-optera (uhm, I mean literal rocks). (And suppose there was literature on that :-).) How would the corresponding row look like? All white, or would you need to add a new colour for that?
Suppose you found 1000 high-quality papers on order X and Criterion Y, and all of them suggested that X is precisely borderline between satisfying Y vs not satisfying it. How would this show up in the tables?
Nitpicky feedback on the presentation:
If I am understanding it correctly, the current format of the tables makes them fundamentally incapable of expressing evidence for insects being unable to feel pain. (The colour coding goes from green=evidence for to red=no evidence, and how would you express ??=evidence against?) I would be more comfortable with a format without this issue, in particularly since it seems justified to expect the authors to be biased towards wanting to find evidence for. [Just to be clear, I am not pushing against the results, or against for caring about insects. Just against the particular presentation :-).]
After thinking about it more, I would interpret (parts of) the post as follows:
To the extent that we found research on these orders O and criteria C, each of the orders satisfies each of the criteria.
We are not saying anything about the degree to which a particular O satisfies a particular C. [Uhm, I am not sure why. Are the criteria extremely binary, even if you measure them statistically? Or were you looking at the degrees, and every O satisfied every C to a high enough degree that you just decided not to talk about it in the post?]
To recap: you don’t talk about the degrees-of-satisfying-criteria, and any research that existed pointed towards sufficient-degree-of-C, for any O and C. Given this, the tables in this post essentially just depict “How much quality-adjusted research we found on this.”
In particular, the tables do not depict anything like “Do we think these insects can feel pain, according to this measure?”. Actually, you believe that probably once there is enough high-quality research, the research will conclude that all insects will satisfy all of the criteria. (Or all orders of insects sufficiently similar to the ones you studied.)
[Here, I mean “believe” in the Bayesian sense where if you had to bet, this is what you would bet on. Not in the sense of you being confident that all the research will come up this way. In particular, no offense meant by this :-) .]
Is this interpretation correct? If so, then I register the complaint that the post is a bit confusing—not particularly sure why, just noticing that it made me confused. Perhaps it’s the thing where I first understood the tables/conclusions as “how much pain do these types of insects feel?”. (And I expect others might get similarly confused.)
I suggest editing the post by adding a tl;dr section to the top of the post. Or maybe change the title to something like Why “just make an agent which cares only about binary rewards” doesn’t work.
Reasoning: To me, the considerations in the post mostly read as rehashing standard arguments, which one should be familiar if they thought about the problem themselves, or went through AGI Safety Fundamentals, etc. It might be interesting to some people, but it would be good to have the clear indication that this isn’t novel.Also: When I read the start of the post, I went “obviously this doesn’t work”. Then I spent several minutes reading the post to see where the flaw in your argument is, and point it out. Only to find that your conclusion is “yeah, this doesn’t help”. If you edit the post, you might save other people from wasting their time in a similar manner :-).
Just to highlight a particular example: suppose you have a prediction market on “How much will be inflation of USD over the next 2 years?”, that is priced in USD.
Re “Middle management is toxic, we should avoid it.”:
I want to flag that: your counterargument here does not properly address the points from Middle Manager Hell / the Immoral Mazes sequences. (Less constructively, “Middle management being toxic” seems like a quite weak version of the arguments against large orgs. Which suggests that your counterargument might not work against the stronger version. More constructively, one difference between current EA structure and large orgs is that small EA orgs are not married to a single funder. This imo reduces the “toxicity” you might otherwise get by the invectives structure in large companies. There might be other important differences; I just haven’t thought about this enough.)
All that said, perhaps we can get the best of the both worlds by using larger orgs for some things but not all? And inventing some tools that make it easier to get the benefits you want without all of the costs? (Example: something that allows people to temporarily/tentatively switch jobs without having to deal with all the paperwork.)
In terms of feedback/reaction: I work on AI alignment, game theory, and cooperative AI, so Moloch is basically my key concern. And from that position, I highly approve of the overall talk, and of all of the content in particular—except for one point, where I felt a bit so-so. And that is the part about what the company leaders can do to help the situation.
The key thing is 9:58-10:09 (“We need leaders who are willing to flip the Moloch’s playbook. …”) , but I think this part then changes how people interpret 10:59-10:11 (“Perhaps companies can start competing over who … ”). I don’t mean to say that I strongly disagree here—rather, I mean that this part seems objectively speculative, which was in contrast with everything else in the talk (which seemed super solid).
More specifically, the talk’s formulation suggested to me that the key thing is whether the leaders would be willing to not play the Moloch game. In contrast, it seems quite possible that this by itself wouldn’t help at all, for example because they would just get fired if they tried. My personal guess is that “the key thing” is affordance the leaders have for not playing the Moloch game / the costs they incur for doing so. Or perhaps the combination of this and the willingness to not play the Moloch game. And this is also how I would frame the 10:59-10:11 part—that we should try to make it such that the companies can compete on those other things that turn this into a race to the top. (As opposed to “the companies should compete on those other things”.)
My personal reason for not digging into this is that my naive model of how good the AI future is: quality_of_future * amount_of_the_stuff. And there is distinction I haven’t seen you acknowledged: while high “quality” doesn’t require humans to be around, I ultimately judge quality by my values. (Thing being conscious is an example. But this also includes things like not copy-pasting the same thing all over, not wiping out aliens, and presumably many other things I am not aware of. IIRC Yudkowsky talks about cosmopolitanism being a human value.) Because of this, my impression is that if we hand over the future to a random AI, the “quality” will be very low. And so we can currently have a much larger impact by focusing on increasing the quality. Which we can do by delaying “handing over the future to AI” and picking a good AI to hand over to. IE, alignment.
(Still, I agree it would be nice if there was a better analysis of this, which exposed the assumptions.)
In my opinion, the main relevant alternative to this view is to be partial to the human species, as opposed to being partial to either one’s current generation, or oneself. And I think the human species is kind of a weird category to be partial to, relative to those other things. Do you disagree?
I agree with this.
the best way to advance your own values is generally to actually “be there” when AI happens.
I (strongly) disagree with this. Me being alive is a relatively small part of my values. And since I am not the director of the world, me personally being around to influence things is unlikely to have a decisive impact on things I value.
In more detail: Sure, all else being equal, me being there when AI happens is mildly helpful. But the outcome of building AI seems to be a function of, among other things, (i) values of the people building it + (ii) how much reflection they can do on those values + (iii) the environment dynamics these people are subject to (e.g., the current race dynamics between AI companies). And over time, I expect the potential decrease in (i) to be far outweighed by gains in (ii) and (iii).
The first issue is about (i), that it is not actually me building the AGI, either now or in the future. But I am willing to grant that (all else being equal) current generation is more likely to have values closer to my values.
However, I expect that the factors are (ii) and (iii) are just as influential. Regarding (ii), it seems we keep making progress at philosophy, ethics, etc, and to me, this currently far outweighs the value drift in (i).
Regarding (iii), my impression is that the current situation is so bad that it can’t get much worse, and we might as well wait. This of course depends on how likely you think we are likely to get a bad outcome if we either (a) get superintelligence without additional progress on alignment or (b) get widespread human-level AI with no progress on alignment, institution design, etc.
Some thoughts that occured to me while reading:
1) Research suggestion: From afar, malevolence-detection techniques seem like a better version of the already-existing tool of top-secret security clearance (or tests similar to it). I am not confident about this, but it already seems that if top-secret security clearance was a requirement for holding important posts, a lot of grief would be avoided (at least where I am from). Yet we generally do not use this tool. Why is this? I suspect that whatever the answer is, it will apply to malevolence-detection techniques as well.
2) Potential bottleneck: Suppose you succeed and develop 100% accurate malevolence-detection technique. I think that, by default, you would have trouble convincing people to use it. (“I mean, what if I score high on it? You know, I am keeping my dark side in check and I don’t plan to become too influential either, so my malevolence doesn’t really hurt anybody. But the other people don’t know that! If I get branded as malevolent, nobody will talk to me ever, or hire me, or anything!”) I conjecture that the impact of this agenda will be bottlenecked on figuring out how to leave the malevolent people a line of retreat; making sure that if you score high on this, the implications aren’t that bad. I see three reasons for this:
a) non-malevolent people might not know they are non-malevolent, and hence be afraid of this,
b) malevolent-and-know-it people might have enough power to hinder this,
c) reasonable general concerns about any test like this getting out of hand.
3) Relatedly to (2), would it make sense to consider some alternative branding that more accurately suggests what you intend to do with the concept (and doesn’t suggest other things)? Unwieldly suggestion, to illustrate what I mean: Being publicly known as “potentially too risky to be in a position of great power” indicates that you shouldn’t be a president, but you might still have friends, a spouse, and a prestigeous job. Being publicly known as “malevolent”, however, … . (Also, it seems plausible that there are people who are malevolent, but do not endorse being so, similarly to how, I think, there are paedophiles who wish they weren’t so.)
(Also, it might not be obvious from my nitpicking, but I really like the post, thanks for it :-).)