Vojta Kovarik. AI alignment and game theory researcher.
Pivocajs
Karma: 86
Apply now to Human-aligned AI Summer School 2025
What (if any) is the overlap of cooperative AI, AI ethics, and AI safety? Perhaps preventing catastrophic harm that is somehow tied to failures of fairness or inclusion?
I imagine that failures as Moloch / runaway capitalism / you get what you can measure would qualify. (Or more precisely, harms caused by these would include things that AI Ethics is concerned about, in a way that Cooperative AI / AI Safety also tries to prevent.)
I think the summary at the start of this post is too easy to misinterpret as “if you think of yourself as a smart and moral person, it’s ok to go for these companies”.
(None of the things the summary says seem false. But the overall impression seems too vulnerable to rationalisation along the lines of “surely I would not fall prey to these bad incentives”. When reality is probably that most people fall prety to them. So at the minimum, it might be more fair to change the recommendation to something like “it’s complicated, but err on the side of not joining” or “it’s complicated, but we wouldn’t recommend this for 95% of people who can get a job at these companies”[1].
- ^
Or whatever qualifier you think is fair. The main point is to make it clear that the warnings apply to the reader as well, not just to “all the other people”.
- ^
In my opinion, the main relevant alternative to this view is to be partial to the human species, as opposed to being partial to either one’s current generation, or oneself. And I think the human species is kind of a weird category to be partial to, relative to those other things. Do you disagree?
I agree with this.
the best way to advance your own values is generally to actually “be there” when AI happens.
I (strongly) disagree with this. Me being alive is a relatively small part of my values. And since I am not the director of the world, me personally being around to influence things is unlikely to have a decisive impact on things I value.
In more detail: Sure, all else being equal, me being there when AI happens is mildly helpful. But the outcome of building AI seems to be a function of, among other things, (i) values of the people building it + (ii) how much reflection they can do on those values + (iii) the environment dynamics these people are subject to (e.g., the current race dynamics between AI companies). And over time, I expect the potential decrease in (i) to be far outweighed by gains in (ii) and (iii).
The first issue is about (i), that it is not actually me building the AGI, either now or in the future. But I am willing to grant that (all else being equal) current generation is more likely to have values closer to my values.
However, I expect that the factors are (ii) and (iii) are just as influential. Regarding (ii), it seems we keep making progress at philosophy, ethics, etc, and to me, this currently far outweighs the value drift in (i).
Regarding (iii), my impression is that the current situation is so bad that it can’t get much worse, and we might as well wait. This of course depends on how likely you think we are likely to get a bad outcome if we either (a) get superintelligence without additional progress on alignment or (b) get widespread human-level AI with no progress on alignment, institution design, etc.
My personal reason for not digging into this is that my naive model of how good the AI future is: quality_of_future * amount_of_the_stuff. And there is distinction I haven’t seen you acknowledged: while high “quality” doesn’t require humans to be around, I ultimately judge quality by my values. (Thing being conscious is an example. But this also includes things like not copy-pasting the same thing all over, not wiping out aliens, and presumably many other things I am not aware of. IIRC Yudkowsky talks about cosmopolitanism being a human value.) Because of this, my impression is that if we hand over the future to a random AI, the “quality” will be very low. And so we can currently have a much larger impact by focusing on increasing the quality. Which we can do by delaying “handing over the future to AI” and picking a good AI to hand over to. IE, alignment.
(Still, I agree it would be nice if there was a better analysis of this, which exposed the assumptions.)
In terms of feedback/reaction: I work on AI alignment, game theory, and cooperative AI, so Moloch is basically my key concern. And from that position, I highly approve of the overall talk, and of all of the content in particular—except for one point, where I felt a bit so-so. And that is the part about what the company leaders can do to help the situation.
The key thing is 9:58-10:09 (“We need leaders who are willing to flip the Moloch’s playbook. …”) , but I think this part then changes how people interpret 10:59-10:11 (“Perhaps companies can start competing over who … ”). I don’t mean to say that I strongly disagree here—rather, I mean that this part seems objectively speculative, which was in contrast with everything else in the talk (which seemed super solid).
More specifically, the talk’s formulation suggested to me that the key thing is whether the leaders would be willing to not play the Moloch game. In contrast, it seems quite possible that this by itself wouldn’t help at all, for example because they would just get fired if they tried. My personal guess is that “the key thing” is affordance the leaders have for not playing the Moloch game / the costs they incur for doing so. Or perhaps the combination of this and the willingness to not play the Moloch game. And this is also how I would frame the 10:59-10:11 part—that we should try to make it such that the companies can compete on those other things that turn this into a race to the top. (As opposed to “the companies should compete on those other things”.)
Re “Middle management is toxic, we should avoid it.”:
I want to flag that: your counterargument here does not properly address the points from Middle Manager Hell / the Immoral Mazes sequences. (Less constructively, “Middle management being toxic” seems like a quite weak version of the arguments against large orgs. Which suggests that your counterargument might not work against the stronger version. More constructively, one difference between current EA structure and large orgs is that small EA orgs are not married to a single funder. This imo reduces the “toxicity” you might otherwise get by the invectives structure in large companies. There might be other important differences; I just haven’t thought about this enough.)
All that said, perhaps we can get the best of the both worlds by using larger orgs for some things but not all? And inventing some tools that make it easier to get the benefits you want without all of the costs? (Example: something that allows people to temporarily/tentatively switch jobs without having to deal with all the paperwork.)
Just to highlight a particular example: suppose you have a prediction market on “How much will be inflation of USD over the next 2 years?”, that is priced in USD.
I suggest editing the post by adding a tl;dr section to the top of the post. Or maybe change the title to something like Why “just make an agent which cares only about binary rewards” doesn’t work.
Reasoning: To me, the considerations in the post mostly read as rehashing standard arguments, which one should be familiar if they thought about the problem themselves, or went through AGI Safety Fundamentals, etc. It might be interesting to some people, but it would be good to have the clear indication that this isn’t novel.Also: When I read the start of the post, I went “obviously this doesn’t work”. Then I spent several minutes reading the post to see where the flaw in your argument is, and point it out. Only to find that your conclusion is “yeah, this doesn’t help”. If you edit the post, you might save other people from wasting their time in a similar manner :-).
I am at high P(doom|AGI pre-2035), but not at near-certainty. Say, 75% but not 99.9%.
The reason for that is that I find both “fast takeoff takeover” and “continous multipolar takeoff” scenarios plausible (with no decisive evidence for one or the other). In “continuous multipolar takeoff”, you still get superintelligences running around. However, they would be “superintelligent with respect civilization-2023″ but not necessarily wrt civilization-then. And for the standard somewhat-well-thought-out AI takeover arguments to aply, you need to be superintelligent wrt civilization-then.
Two disclaimers: (1) Just because you don’t get discontinuity in influence around human level does not mean you can’t get it later. In my book, world can look “Christiano-like”, until suddenly it looks “Yudkowsky-like”. (2) Even if we never get AI singleton, things can still go horribly wrong (ie, Christiano’s what failure looks like). But imo those scenarios are much harder to reason about, and we have haven’t thought them out in enough detail to justify high certainty of either outcome.
My intuitive aggregation of this gives, say, 80% P(doom this century|AGI pre-2035). On top of that, I add some 5-10% on “I am so wrong about some of this that even the high-level reasoning doesn’t apply”. (Which includes being wrong about where the burden of proofs, and priors, lie for P(doom|AGI).) And that puts me at the (ass-) number 75%.
Nitpicky feedback on the presentation:
If I am understanding it correctly, the current format of the tables makes them fundamentally incapable of expressing evidence for insects being unable to feel pain. (The colour coding goes from green=evidence for to red=no evidence, and how would you express ??=evidence against?) I would be more comfortable with a format without this issue, in particularly since it seems justified to expect the authors to be biased towards wanting to find evidence for. [Just to be clear, I am not pushing against the results, or against for caring about insects. Just against the particular presentation :-).]
After thinking about it more, I would interpret (parts of) the post as follows:
To the extent that we found research on these orders O and criteria C, each of the orders satisfies each of the criteria.
We are not saying anything about the degree to which a particular O satisfies a particular C. [Uhm, I am not sure why. Are the criteria extremely binary, even if you measure them statistically? Or were you looking at the degrees, and every O satisfied every C to a high enough degree that you just decided not to talk about it in the post?]
To recap: you don’t talk about the degrees-of-satisfying-criteria, and any research that existed pointed towards sufficient-degree-of-C, for any O and C. Given this, the tables in this post essentially just depict “How much quality-adjusted research we found on this.”
In particular, the tables do not depict anything like “Do we think these insects can feel pain, according to this measure?”. Actually, you believe that probably once there is enough high-quality research, the research will conclude that all insects will satisfy all of the criteria. (Or all orders of insects sufficiently similar to the ones you studied.)
[Here, I mean “believe” in the Bayesian sense where if you had to bet, this is what you would bet on. Not in the sense of you being confident that all the research will come up this way. In particular, no offense meant by this :-) .]
Is this interpretation correct? If so, then I register the complaint that the post is a bit confusing—not particularly sure why, just noticing that it made me confused. Perhaps it’s the thing where I first understood the tables/conclusions as “how much pain do these types of insects feel?”. (And I expect others might get similarly confused.)
I saw the line “found no good evidence that anything failed any criterion”, but just to check explicitly: What do the confidence levels mean? In particular, should I read “low confidence” as “weak evidence that X feels pain-as-operationalized-by-Criterion Y”? Or as “strong evidence that X does not feel pain-as-operationalized-by-Criterion Y”?
In other words:
Suppose you did the same evaluation for the order Rock-optera (uhm, I mean literal rocks). (And suppose there was literature on that :-).) How would the corresponding row look like? All white, or would you need to add a new colour for that?
Suppose you found 1000 high-quality papers on order X and Criterion Y, and all of them suggested that X is precisely borderline between satisfying Y vs not satisfying it. How would this show up in the tables?
“National greatness, and staying ahead of China for world influence requires that we have the biggest economy. To do that, we need more people.” -Matt Yglesias, One Billion Americans.
Yeah, the guy who has chosen to have one child is going to inspire me to make the sacrifices involved in having four. It might be good for America, but the ‘ask’ here looks like it is that I sacrifice my utility for Matt’s one kid, and thus is not cooperate-cooperate. I’ll jump when you jump.Two pushbacks here:
(1) The counterargument seems rather weak here, right? Even if Matt Yglesias had no kids, that doesn’t mean his argument isn’t valid. EG, if a single non-vegan person claims that more people should be vegan, will you view that as evidence that people should not be vegan? ;-) (Not that I disagree with your claim. Just with your argument.)
(2) Did you actually read One Billion Americans, or are you just taking the citation and interpretting it as Matt Yglesias making an argument for people having more children? I didn’t read the book, so I am not sure. But I listened to a podcast with Matt Yglesias, about the book, and my impression was that he was primarily arguing for changing the immigration policy, and (if memory serves) not really making any strong claims about how many kids people should have.
Uhm, yes, that seems right. That’s why this matters.
Yes, sure, probabilities are only in the map. But I don’t think that matters for this. Or I just don’t see what argument you are making here. (CLT is in the map, expectations are taken in the map, and decisions are made in the map (then somehow translated into the territory via actions). I don’t see how that says anything about what EV reasoning relies on.)
Agree with Acylhalide’s point—you only need to be non-Dutchbookable by bets that you could actually be exposed to.
To address a potential misunderstanding: I agree with both Sharmake’s examples. But they don’t imply you have to maximise expected utility always. Just when the assumptions apply.
More generally: expected utility maximisation is an instrumental principle. But it is justified by some assumptions, which don’t always hold.
Yes, the expected utility is larger. The claim is that there is nothing incoherent about not maximising expected utility in this case.
To try rephrasing: Principle 1: if you have to choose a X% chance of getting some outcome A, and a >=X% chance of a strictly better outcome B, you should take B. Principle 2: if you will be facing a long series of comparably significant choices, you should decide each of them based on expected utility maximisation. Principle 3: you should do expected utility maximisation for every single choice. Even if that is the last/most important choice you will ever make.
The claim is that: P1 is solid. P2 follows from P1 (via Central Limit Theorem, or whatever math). But P3 does not follow from P1/P2, and there will be cases where it might be justified to not obey P3. (Like the case with 51% chance of doubling the world’s goodness, 49% chance of destroying it.)
Note that I am not claiming it’s wrong to do expected utility maximisation in all scenarios. Just saying that it both doing and not doing it is OK. And therefore it is (very?) non-strategic to associate your philosophical movement with it. (Given that most people’s intuitions seem to be against it.)
Does this explanation make sense? Maybe I should change the title to something with expected utility?
In extremely high-stakes scenarios, it’s ok not to maximise expected utility
This gave me an idea for an experiment/argument. Posting here, in case somebody wants come up with a more thought-out version of it and do it.
[On describing what would change his mind:] You couldn’t find weird behaviors [in the AI], no matter how hard you tried.
People like to take an AI, poke it, and then argue “it is doing [all these silly mistakes], therefore [not AGI/not something to worry about/...]”. Now, the conclusion might be right, but the argument is wrong—even dangerous things can be stupid is some settings. Nevertheless, the argument seems convincing.
My prediction is that people make a lot of mistakes[1] that would seem equally laughable if it was AI that made them. Except that we are so used to them that we don’t appreciate it. So if one buys the argument above, they should conclude that humans are also [not general intelligence/something to worry about/...]. So perhaps if we presented the human mistakes right, it could become a memorable counterargument to “AI makes silly mistakes, hence no need to worry about it”.
Some example formats:
“Look at these silly AI mistakes! Surprise, that’s normal people.” or
“Quizz: AI mistake or human mistake?”
(uhm, or “Quizz: AI or Trump?”; wouldn’t mention this, except bots on that guy already exist).
Obligatory disclaimer: It might turn out that humans really don’t make [any sorts of] [silly mistakes current AI makes], or make [so few that it doesn’t matter]. If you could operationalize this, that would also be valuable.
- ^
What is “these mistakes”? I don’t know . Exercise for the reader.
I want to flag that even with short timelines and selfish goals, the terms of the bet seem like a bad deal.
Reason: Many people with short timelines also tend to put high probability on superintelligent AI being bad news (eg, me). From that point if view, an over-simplified interpretation of the terms is:
Either we get SAI by 2028, in which case I am dead (and get 10k).
Or we don’t, in which case I have to pay 10k.
If you wanted to account for this, the bet should be modified somehow. EG, you give me 10k now, and if [the question didn’t resolve] / [I am alive] by January 2029, I send you your 10k back and pay you 10k * x—where your proposal corresponds to x=1. (FWIW, I personally wouldn’t take the bet for x=1. But I would start thinking about it for x=0.5 or so.)