Most of my stuff (even the stuff of interest to EAs) can be found on LessWrong: https://www.lesswrong.com/users/daniel-kokotajlo
kokotajlod
Karma: 1,998
(Typo: You say the US when you meant the USSR.)
The philosophy literature has stuff on this. If I recall correctly I linked some of it in the bibliography of this post. It’s been a while since I thought about this I’m afraid so I don’t have references in memory. Probably you should search the Stanford Encyclopedia of Philosophy for the “Pasadena Game” and “st petersburg paradox”
Welcome to the fantastic world of philosophy, friend! :) If you are like me you will enjoy thinking and learning more about this stuff. Your mind will be blown many times over.
I do in fact think that utilitarianism as normally conceived is just wrong, and one reason why it is wrong is that it says every action is equally choiceworthy because they all have undefined expected utility.
But maybe there is a way to reconceive utilitarianism that avoids this problem. Maybe.
Personally I think you might be interested in thinking about metaethics next. What do we even mean when we say something matters, or something is good? I currently think that it’s something like “what I would choose, if I was idealized in various ways, e.g. if I had more time to think and reflect, if I knew more relevant facts, etc.”
I’m a fan of lengthy asynchronous intellectual exchanges like this one, so no need to apologize for the delay. I hope you don’t mind my delay either? As usual, no need to reply to this message.
If we condition on not having extreme capabilities for persuasion or research/engineering, I’m quite skeptical that something in the “business/military/political strategy” category is a great candidate to have transformative impact on its own.
I think I agree with this.
Re: quantification: I agree; currently I don’t have good metrics to forecast on, much less good forecasts, for persuasion stuff and AI-PONR stuff. I am working on fixing that problem. :)
Re persuasion: For the past two years I have agreed with the claims made in “The misinformation problem seems like misinformation.”(!!!) The problem isn’t lack of access to information; information is more available than it ever was before. Nor is the problem “fake news” or other falsehoods. (Most propaganda is true.) Being politically polarized and extremist correlates positively with being well-informed, not negatively! (Anecdotally, my grad school friends with the craziest/most-extreme/most-dangerous/least-epistemically-virtuous political beliefs were generally the people best informed about politics. Analogous to how 9/11 truthers will probably know a lot more about 9/11 than you or me.) This is indeed an epistemic golden age… for people who are able to resist the temptations of various filter bubbles and the propaganda of various ideologies. (And everyone thinks themself one such person, so everyone thinks this is an epistemic golden age for them.)I do disagree with your claim that this is currently an epistemic golden age. I think it’s important to distinguish between ways in which it is and isn’t. I mentioned above a way that it is.
If we made a chart of some number capturing “how easy it is to convince key parts of society to recognize and navigate a tricky novel problem” … since the dawn of civilization, what would that chart look like? My guess is that it would be pretty chaotic; that it would sometimes go quite low and sometimes go quite high
Agreed. I argued this, in fact.
and that it would be very hard to predict the impact of a given technology or other development on epistemic responsiveness.
Disagree. I mean, I don’t know, maybe this is true. But I feel like we shouldn’t just throw our hands up in the air here, we haven’t even tried! I’ve sketched an argument for why we should expect epistemic responsiveness to decrease in the near future (propaganda and censorship are bad for epistemic responsiveness & they are getting a lot cheaper and more effective & no pro-epistemic-responsiveness-force seems to be rising to counter it)
Maybe there have been one-off points in history when epistemic responsiveness was very high; maybe it is much lower today compared to peak, such that someone could already claim we have passed the “point of no return”; maybe “persuasion AI” will drive it lower or higher, depending partly on who you think will have access to the biggest and best persuasion AIs and how they will use them.
Agreed. I argued this, in fact. (Note: “point of no return” is a relative notion; it may be that relative to us in 2010 the point of no return was e.g. the founding of OpenAI, and nevertheless relative to us now the point of no return is still years in the future.)
So I think even if we grant a lot of your views about how much AI could change the “memetic environment,” it’s not clear how this relates to the “point of no return.”
The conclusion I built was “We should direct more research effort at understanding and forecasting this stuff because it seems important.” I think that conclusion is supported by the above claims about the possible effects of persuasion tools.
What has/had higher ex ante probability of leading to a dramatic change in the memetic environment: further development of AI language models that could be used to write more propaganda, or the recent (last 20 years) explosion in communication channels and data, or many other changes over the last few hundred years such as the advent of radio and television, or the change in business models for media that we’re living through now? This comparison is intended to be an argument both that “your kind of reasoning would’ve led us to expect many previous persuasion-related PONRs without needing special AI advances” and that “if we condition on persuasion-related PONRs being the big thing to think about, we shouldn’t necessarily be all that focused on AI.”
Good argument. To hazard a guess:
1. Explosion in communication channels and data (i.e. the Internet + Big Data)
2. AI language models useful for propaganda and censorship
3. Advent of radio and television
4. Change in business models for media
However I’m pretty uncertain about this, I could easily see the order being different. Note that from what I’ve heard the advent of radio and television DID have a big effect on public epistemology; e.g. it partly enabled totalitarianism. Prior to that, the printing press is argued to have also had disruptive effects.
This is why I emphasized elsewhere that I’m not arguing for anything unprecedented. Public epistemology / epistemic responsiveness has waxed and waned over time and has occasionally gotten extremely bad (e.g. in totalitarian regimes and the freer societies that went totalitarian) and so we shouldn’t be surprised if it happens again and if someone has an argument that it might be about to happen again it should be taken seriously and investigated. (I’m not saying you yourself need to investigate this, you probably have better things to do.) Also I totally agree that we shouldn’t just be focused on AI; in fact I’d go further and say that most of the improvements in propaganda+censorship will come from non-AI stuff like Big Data. But AI will help too; it seems to make censorship a lot cheaper for example.
I’d be interested in seeing literature on how big an effect size you can get out of things like focus groups and A/B testing. My guess is that going from completely incompetent at persuasion (e.g., basically modeling your audience as yourself, which is where most people start) to “empirically understanding and incorporating your audience’s different-from-you characteristics” causes a big jump from a very low level of effectiveness, but that things flatten out quickly after that, and that pouring more effort into focus groups and testing leads to only moderate effects, such that “doubling effectiveness” on the margin shouldn’t be a very impressive/scary idea.I think most media is optimizing for engagement rather than persuasion, and that it’s natural for things to continue this way as AI advances. Engagement is dramatically easier to measure than persuasion, so data-hungry AI should help more with engagement than persuasion; targeting engagement is in some sense “self-reinforcing” and “self-funding” in a way that targeting persuasion isn’t (so persuasion targeters need some sort of subsidy to compete with engagement targeters); and there are norms against targeting persuasion as well. I do expect some people and institutions to invest a lot in persuasion targeting (as they do today), but my modal expectation does not involve it becoming pervasive on nearly all websites, the way yours seems to.
I feel like a lot of today’s “persuasion” is either (a) extremely immersive (someone is raised in a social setting that is very committed to some set of views or practices); or (b) involves persuading previously-close-to-indifferent people to believe things that call for low-cost actions (in many cases this means voting and social media posting; in some cases it can mean more consequential, but still ultimately not-super-high-personal-cost, actions). (b) can lead over time to shifting coalitions and identities, but the transition from (b) to (a) seems long.
I particularly don’t feel that today’s “persuaders” have much ability to accomplish the things that you’re pointing to with “chatbots,” “coaches,” “Imperius curses” and “drugs.” (Are there cases of drugs being used to systematically cause people to make durable, sustained, action-relevant changes to their views, especially when not accompanied by broader social immersion?)
These are all good points. This is exactly the sort of thing I wish there was more research into, and that I’m considering doing more research on myself.
Re: pervasiveness on almost all websites: Currently propaganda and censorship both seem pretty widespread and also seem to be on a trend of becoming more so. (The list of things that get censored is growing, not shrinking, for example.) This is despite the fact that censorship is costly and so theoretically platforms that do it should be outcompeted by platforms that just maximize engagement. Also, IIRC facebook uses large language models to do the censoring more efficiently and cheaply, and I assume the other companies do too. As far as I know they aren’t measuring user opinions and directly using that as a feedback signal, thank goodness, but… is it that much of a stretch to think that they might? It’s only been two years since GPT-3.
Oh nice, thanks, this is the sort of thing I was looking for!
Fair enough! I’m grateful for all the work you’ve already done and don’t think it’s your job to do more research in the areas that would be more convincing to me.
Agreed. Though that was a century ago & with different governments, as you pointed out elsewhere. Also no nukes; presumably nukes make escalation less likely than it was then (I didn’t realize this until now when I just read the wiki—apparently the Allies almost declared war on the soviet union due to the invasion of finland!).
OK, thanks!
Ah, that’s a good argument, thanks! Updating downwards.
That makes sense. 2 OOMs is clearly too high now that you mention it. But I stand by my 1 OOM claim though, until people convince me that really this is much more like an ordinary business-as-usual month than I currently think it is. Which could totally happen! I am not by any means an expert on this stuff, this is just my hot take!
That’s why it was my upper bound. I too think it’s pretty implausible. How would you feel about a bet on the +1 OOM odds?
To be clear I haven’t thought about this nearly as much as you. I just think that this is clearly an unusually risky month and so the number should be substantially higher than for an average month.
Seems less mild than the invasion of Georgia and the Armenia and Azerbaijan war. But more mild than the cuban missile crisis for sure. Anyhow, roughly how many examples do you think you’d want to pick for the reference class? If it’s a dozen or so, then I think you’d get a substantially higher base rate.
FWIW I’d wildly guess this analysis underestimates by 1-2 orders of magnitude, see this comment thread. ETA: People (e.g. Misha) have convinced me that 2 is way too high, but I still think 1 is reasonable. ETA: This forecaster has a much more in-depth analysis that says it’s 1 OOM.
I don’t think this is a situation where the per-month base rate applies. The history of nuclear war is not a flat rate of risk every month; instead, the risk is lower most months and then much higher on some months, and it averages out to whatever flat rate you think is appropriate. (Looking at the list of nuclear close calls it seems hard to believe the overall chance of nuclear war was <50% for the last 70 years. Individual incidents like the cuban missile crisis seem to contribute at least 20%.) So there should be some analysis of how this month compares to past months—is it more like one of the high-risk months, or more like a regular month?
ETA: Analogy: Just a few hours ago I was wondering how likely it was that my flight would be cancelled or delayed. The vast majority of flights are not cancelled or delayed. However, I didn’t bother to calculate this base rate, because I have the relevant information that there is a winter storm happening right now around the airport. So instead I tried to use a narrower reference class—how often do flights get cancelled in this region during winter storms? Similarly, I think the reference class to use is something like “Major flare-up in tensions / crisis between NATO and Russia” not “average month in average year.” We should be looking at the cuban missile crisis, the fall of the soviet union, and maybe a half-dozen other examples, and using those as our reference class. Unless we think the current situation is milder than that.
I take the point no 3 above that it’s relatively late in the game, the edges seem mostly agreed-upon, etc. This is the sort of analysis that I think should be done. I would like to see a brainstorm of scenarios that could lead to war & an attempt to estimate their probability. Like, if Putin uses nukes in Ukraine, might that escalate? What if he uses chemical WMDs? What if NATO decides to ship in more warplanes and drones? What if they start going after each other’s satellites, since satellites are so militarily important? What if there is an attempted assassination or coup?
(Re point 1: Sure, yeah, but probably NATO vs. Russia risk is within an order of magnitude of general nuclear war risk, and plausibly more than 50% as high. Most of the past nuclear close calls were NATO vs. Russia, the current crisis is a NATO vs. Russia crisis, and if there is a nuclear war right now it seems especially likely to escalate to London being hit compared to e.g. pakistan vs. India in a typical year etc.)
Thanks for doing this. Why is the next-month chance of nuclear war so low? (0.034%) That seems like basically what the base rate should be, i.e. the chance of an average month in an average year resulting in nuclear war. But this is clearly not an average month, the situation is clearly much more risky. How much more, I dunno. But probably at least an order of magnitude more, maybe two? Did you model this, or do we just disagree that this is a significantly riskier month than average?
OK, cool, thanks. I flat-out disagree with Braumoeller here. The Korean war was way more intense than the Vietnam war. It just was over quickly, whereas the vietnam war dragged on for a long time and thus was worse/bloodier/moredestructive overall.
If the word you used was bloodiest I’d agree. But you used intensity.
The Middle East was rocked by a series of deadly clashes: measured by intensity (deaths divided by the population of the countries involved), the Iran-Iraq War of the 1980s was the third bloodiest war of all time.[1]
Wait, really? I thought there were plenty of wars in the past where a significant fraction of an entire region was killed. Like, 75%.
Also, shouldn’t intensity be deaths per year divided by population of the countries involved?
Think about how hard you would try to avoid getting the next wave of COVID if it turned out it had a 1% chance of killing you. Not even 1% conditional on you getting it; 1% unconditional. (So for concreteness, imagine that your doctor at the next checkup tells you that based on your blood type and DNA you actually have a 10% chance of dying from COVID if you were to get it, and based on your current default behavior and prevalence in the population it seems like you have a 10% chance of getting it before a better vaccine for your specific blood type is developed.)
Well, I claim, you personally are more than 1% likely to die of x-risk. (Because we all are.)
Don’t you mean 1-that?