Thinking, writing, and tweeting from Berkeley California. Previously, I ran programs at the Institute for Law & AI, worked on the one-on-one advising team at 80,000 Hours in London and as a patent litigator at Sidley Austin in Chicago.
Mjreard
Karma: 2,234
Slides are quite good. Maybe this is somewhat played-out, but liberal influencers should take up a tack of commenting on how right-wing influencers claims just don’t square with people’s everyday lives. Like “have you really seen cartels murdering people, in your neighborhood? Or is that just something people on the internet are talking about?”
I agree. Basically anyone not in a politically sensitive role (this category is broader than it might intuitively seem) should be looking to make large donations in this area now and others should be reaching out to EAs focused on US politics if they feel well equipped to run or contribute to a high leverage project.
Unfortunately there is no AMF/GiveDirectly for politics and most things you can donate too are very poorly leveraged. Likewise it is hard to both scope a leveraged project and execute well on it. I know of one general exception at the moment which I’m happy to recommend privately.
I’m also happy to speak to anyone who intends to devote considerable money or work resources to this and pass them along to the people doing the best work here if that makes sense.
Over the last decade, we should have invested more in community growth at the expense of research.
Being very confident on this question because would be questioning a pretty marked success, but it does seems like 1) we’re short of the absolute talent/power threshold big problems demand and 2) like energy/talent/resources have been sucked out of good growth engines multiple times in the past decade.
I agree this is quite bad practice in general, though see my other comment for why I think these are not especially bad cases.
A central error in these cases is assuming audiences will draw the wrong inferences from your true view and do bad things because of that. As far as I can tell, no one has full command of the epistemic dynamics here to be able to say that with confidence and then act on it. If you aren’t explicit and transparent about your reasoning, people can make any number of assumptions, others can poke holes in your less-than-fully-endorsed claim and undermine the claim or undermine your credibility and people can use that to justify all kinds of things.
You need to trust that your audience will understand your true view or that you can communicate it properly. Any alternative assumption is speculation whose consequences you should feel more, not less, responsible for since you decided to mislead people for the sake of the consequences rather than simply being transparent and letting the audience take responsibility for how they react to what you say.
I think people who do the bad version of this often have this ~thought experiment in mind: “my audience would rather I tell them the thing that makes their lives better than the literal content of my thoughts.” As a member of your audience, I agree. I don’t, however, agree with the subtly altered, but more realistic version of the thought experiment: “my audience would rather I tell them the thing that I think makes their lives better than the literal content of my thoughts.”
I agree that people should be doing a better job here. As you say, you can just explain what you’re doing and articulate your confidence in specific claims.
The thing you want to track is confidence*importance. MacAskill and Ball do worse than Piper here. Both of them were making fundamental claims about their primary projects/areas of expertise, and all claims in those two areas are somewhat low confidence and people adjust their expectations that.
MacAskill and Ball both have defenses too. In MackAskill’s case, he’s got a big body of other work that makes it fairly clear DGB was not a comprehensive account of his all-things-considered views. It’d be nice to clear up the confusion by stating how he resolves the tension between different works of his, but the audience can also read them and resolve the tension for themselves. The specific content of William MacAskill’s brain is just not the thing that matters and its fine for him to act that way as long as he’s not being systematically misleading.
Ball looks worse, but I wouldn’t be surprised if he alluded to his true view somewhere public and he merely chose not to emphasize it so as to better navigate an insane political environment. If not, that’s bad, but again there’s a valid move of saying “here are some rationales for doing X” that doesn’t obligate you to disclose the ones you care most about, though this is risky business and a mild negative update on your trustworthiness.
Contrarian marketing like this seems like it would only work well if the thing being opposed was extremely well known, which I don’t think Veganuary is.
Many creators act as though Youtube’s algorithm disfavors content that refers to graphic acts of sex and violence, i.e., bleeping words like ‘kill’ or ‘suicide’ or referring to these in very circuitous ways. I would guess these are incomplete methods of avoidance and that YT tries to keep up by detecting these workarounds. Seems like a potential issue for the MechaHitler video.
@Andy Masley and @Richard Y Chappell🔸 are my top recommendations here
Don’t do illegal things (or things that excessively pollute the commons), but I think there’s value in having attractive signage with QR codes to central resources in places where that’s accepted, especially e.g. university message boards.
Bloggers for Shrimp!
Good characterization; I should have watched the video. Seems like she may be unwilling to consider that the weird Silicon Valley stuff is correct, but explicitly says she’s just raising the question of motivated reasoning.
The “writing scifi with your smart friends” is quite an unfair characterization, but fundamentally on us to counter. I think it will all turn on whether people find AI risk compelling.
For that, there’s always going to be a large constituency scoffing. There’s a level at which we should just tolerate that, but we’re still at a place where communicating the nature of AI risk work more broadly and more clearly is important on the margin.
I agree there should be a counter narrative. It is also important to realize that people who create, like, and comment on mean-spirited TikToks who are self-absorbed in their own misguided ideology are far enough from the target market that you really shouldn’t worry about changing their behavior.
A 10% chance of transformative AI this decade justifies current EA efforts to make AI go well. That includes the opportunity costs of that money not going to other things in the 90% worlds. Spending money on e.g. nuclear disarmament instead of AI also implies harm in the 10% of worlds where TAI was coming. Just calculating the expected vale of each accounts for both of these costs.
It’s also important to understand that Hendrycks and Yudkowsky were simply describing/predicting the geopolitical equilibrium that follows from their strategies, not independently advocating for the airstrikes or sabotage. Leopold is a more ambiguous case, but even he says that the race is already the reality, not something he prefers independently. I also think very few “EA” dollars are going to any of these groups/individuals.
My list is very similar to yours. I believe items 1, 2, 3, 4, and 5 have already been achieved to substantial degrees and we continue to see progress in the relevant areas on a quarterly basis. I don’t know about the status of 6.
For clarity on item 1, AI company revenues in 2025 are on track to cover 2024 costs, so on a product basis, AI models are profitable; it’s the cost of new models that pull annual figures into the red. I think this will stop being true soon, but that’s my speculation, not evidence, so I remain open that scaling will continue to make progress towards AGI, potentially soon.
Your picture of EA work on AGI preparation is inaccurate to the extent I don’t think you made a serious effort to understand the space you’re criticizing. Most of the work looks like METR benchmarking, model card/RSP policy (companies should test new models for dangerous capabilities a propose mitigations/make safety cases), mech interp, compute monitoring/export controls research, and trying to test for undesirable behavior in current models.
Other people do make forecasts that rely on philosophical priors, but those forecasts are extrapolating and responding to the evidence being generated. You’re welcome to argue that their priors are wrong or that they’re overconfident, but comparing this to preparing for an alien invasion based on Oumuamua is bad faith. We understand the physics of space travel well enough to confidently put a very low prior on alien invasion. One thing basically everyone in the AI debate agrees on is that we do not understand where the limits of progress are as data reflecting continued progress continues to flow.
I agree there’s logical space for something less than less than AGI making the investments rational, but I think the gap between that and full AGI is pretty small. Peculiarity of my own world model though, so not something to bank on.
My interpretation of the survey responses is selecting “unlikely” when there are also “not sure” and “very unlikely” options suggests substantial probability (i.e. > 10%) on the part of the respondents who say “unlikely,” or “don’t know.” Reasonable uncertainty is all you need to justify work on something so important if-true and the cited survey seems to provide that.
I directionally agree that EAs are overestimating the imminence of AGI and will incur some credibility costs, but the bits of circumstantial evidence you present here don’t warrant the confidence you express. 76% of experts saying it’s “unlikely” the current paradigm will lead to AGI leaves ample room for a majority thinking there’s a 10%+ chance it will, which is more than enough to justify EA efforts here.
And most of what EAs are working on is determining whether we’re in that world and what practical steps you can take to safeguard value given what we know. It’s premature to declare case closed when the markets and the field are still mostly against you (at the 10% threshold).
I wish EA were a bigger and broader movement such that we could do more hedging, but given that you only have a few hundred people and a few $100m/yr, it’s reasonable to stake that on something this potentially important that no one else is doing effective work on.
I would like to bring back more of the pre-ChatGPT disposition where people were more comfortable emphasizing their uncertainty, but standing by the expected value of AI safety work. I’m also open to the idea that that modesty too heavily burdens our ability to have impact in the 10%+ of worlds where it really matters.
Yes, but this shows your claim here is actually just empirical skepticism about how general and how capable AI systems will be.
It is true that loose talk of AIs being “[merely] better than” all humans at all tasks does not imply doom, but the “merely” part is not what doomers believe.
If AIs are a perfect substitute for humans with lower absolute costs of production – where “costs” mean the physical resources needed to keep a flesh-and-blood human alive and productive – humans will have a comparative advantage only in theory. In practice, it would make more sense to get rid of the humans and use the inputs that would have sustained them to produce more AI labor.
I’ll need to reread Scott’s post to see how reductive it is,[1] but negotiation and motivated cognition here do feel like a slightly lower level of abstraction in the sense that they are composed or different kinds of (and proportions of) conflicts and mistakes. The dynamics you discuss here follow pretty intuitively from the basic conflict/mistake paradigm.
This is still great analysis and a useful addendum to Scott’s post.
actually pretty reductive on a skim, but he does have a savings clause at the end: “But obviously both can be true in parts and reality can be way more complicated than either.”