I think there’s something epistemically off about allowing users to filter only bad AI news. The first tag doesn’t have that problem, but I’d still worry about missing important info. I prefer the approach of just requesting users be vigilant against the phenomenon I described.
tcelferact
Karma: 323
I don’t object to folks vocalizing their outrage. I’d be skeptical of ‘outrage-only’ posts, but I think people expressing their outrage while describing what they are doing and wish the reader to do would be in line with what I’m requesting here.
Your post more than meets my requested criteria, thank you!
I agree with this. Where there is a tradeoff, err on the side of truthfulness.
This seems aimed at regulators; I’d be more interested in a version for orgs like the CIA or NSA.
Both those orgs seem to have a lot more flexibility than regulators to more or less do what they want when national security is an issue, and AI could plausibly become just that kind of issue.
So ‘policy ideas for the NSA/CIA’ could be at once both more ambitious and more actionable.
I did write the survey assuming AI researchers have at least been exposed to these ideas, even if they were completely unconvinced by them, as that’s my personal experience of AI researchers who don’t care about alignment. But if my experiences don’t generalize, I agree that more explanation is necessary.
I definitely think “that’s just one final safety to rely on” applies to this suggestion. I hope we do a lot more than this!
The idea here is to prepare for an emergency stop if we are lucky enough to notice things going spectacularly wrong before it’s too late. I don’t think there’s any hamstringing of well-intentioned people implied by that!
I agree that private docs and group chats are totally fine and normal. The bit that concerns me is ‘discuss how to position themselves and how to hide their more controversial views or make them seem palatable’, which seems a problematic thing for leaders to be doing in private. (Just to reiterate I have zero evidence for or against this happening though.)
Thanks Arden! I should probably have said it explicitly in the post, but I have benefited a huge amount from the work you folks do, and although I obviously have criticisms, I think 80K’s impact is highly net-positive.
I think you’re correct that they aren’t being dishonest, but I disagree that the discrepancy is because ‘they’re answering two different questions’.
If 80K’s opinion is that a Philosophy PhD is probably a bad idea for most people, I would still expect that to show up in the Global Priorities information. For example, I don’t see any reason they couldn’t write something like this:
In general, for foundational global priorities research the best graduate subject is an economics PhD. The next most useful subject is philosophy … but the academic job market for philosophy is extremely challenging, and the career capital you acquire working toward a career in philosophy isn’t particularly transferable. For these reasons, we strongly recommend approaching GPR via economics instead of philosophy unless you are a particularly gifted philosopher and comfortable with a high risk of failure...
Maybe I’m nitpicking, as you say it is mentioned on the ‘philosophy academia’ page. I was trying to draw attention to a general discomfort I have with the site that it seems to underemphasise risk of failure, but perhaps I need to find a better example!
Upvoted. I think these are all fair points.
I agree that ‘utilitarian-flavoured’ isn’t an inherently bad answer from Ben. My internal reaction at the time, perhaps due to how the night had been marketed, was something like ‘ah he doesn’t want to scare me off if I’m a Kantian or something’, and this probably wasn’t a charitable interpretation.
On the Elon stuff, I agree that talking to Elon is not something that should require reporting. I think the shock for me was that I saw Will’s tweet in August, which as wock agreed implied to me they didn’t know each other, so when I saw the signal conversation I felt misled and started wondering how close they actually were. That said I had no idea Elon was an EAG keynote speaker, which is obviously public knowledge and makes the whole thing a lot less suspicious. I would also remove the word ‘controversial’ if I was to write this again, and that I think Elon’s done harm re: AI, as I agree it’s not relevant to the point I’m trying to make.
This reads (at least to me) as taking a softer line than the original piece, so there’s not as much I disagree with, and quite a lot that’s closer to my own thinking too. I might add more later, but this was already a useful exchange for me, so thanks again for writing and for the reply! I have upvoted (I upvoted the original also), and I hope you find your interactions on here constructive.
Edit: One thing that seems worth acknowledging: I agree there is a distinctive form of ‘meta-’ reflection that is required if you want be meaningfully inclusive, and my reply didn’t capture that with ‘listen to diverse viewpoints, use that to update very hard...‘. I think your ‘challenge the knowledge architecture’ phrase is fuzzy but is getting at something useful, as the process definitely involves updating your heuristics around what sorts of contributions are valuable (versus e.g. just listening to people from different backgrounds for contributions that you consider valuable). I am inclined to credit social movements and not critical theory with figuring out how to do this though, and participating in social movements with being the best way to get better at it yourself!
Thanks for taking the time to write this up. I have a few reactions to reading it:
EA as a consequence of capitalism
I just want to call out that this in itself isn’t a valid criticism of EA, any more than it would be a valid criticism of the social movements that you favour. But I suspect you agree with this, so let’s move on.
EA as a form of capitalism
Simultaneously, EA is also a form of capitalism because it is founded on a need to maximize what a unit of resources like time, money, and labour can achieve
I think you’ve made a category error here. I hear your comment that ‘critical theorists have long viewed capital in extra-monetary terms’, but whatever resource we’re talking about, the kind of capitalist system you’re describing is people trying to grab as much of that resource as possible for themselves. That’s what the ‘maximization’ is all about.
EA is not trying to use time, money, and labour to maximally hoard resources, it’s using them to try and maximally improve the long-term future/alleviate suffering/avoid extinction risks/etc.
I would expect any social movement you care about to be doing something similar with regards to its own goals. I do hear that you have concerns about focusing too hard on efficiency/optimization, but I don’t agree that this is the property of capitalism that causes harm, rather its lack of a means to incentivise optimizing for public (vs private) goods.
EA as a facilitator of capitalism
I would really like concrete examples if you’re going to make this argument. My impression is that people tend to make this case without providing any, and as a result I’m highly sceptical of the claim.
Can you show me a couple of case studies where an EA-backed aid program plausibly thwarted meaningful political change that would otherwise have occurred in some area of the world? Without that, I don’t think we can have a productive conversation on this point.
I accept that aid is sometimes used to disingenuously manipulate public opinion, but I do not think the correct response to this is to stop trying to help people! (I think this would be true even if most aid was given in bad faith).
I also think the idea that EA funds help bad actors to better disingenuously manipulate public opinion doesn’t make sense. I think most of the public would consider EA funds a pretty weird place to put money, and even if some bad actor could claim they saved 100x as many lives and EA had helped them do it, our cognitive biases around large numbers mean this probably wouldn’t play significantly better in terms of PR. The extra 99x lives saved, however, would remain saved.
Finally, I am strongly against any line of thinking that implies we should deliberately be more neglectful so people in need get angry and revolt, ultimately making things better in the long run. You don’t go this far in your piece, but I think you get pretty close.
There are many reasons I think this kind of idea is wrongheaded, but for a start I think it’s disrespectful to those in suffering to act like they somehow need to be ‘prodded’ into realizing things could work better than they do, and doubly so to try and do this by deliberately abstaining from helping them when it’s within your ability to do so. I hope we can agree on that!
Foucault, Critical Theory, etc.
one of the most important contributions of critical theory has been to dismantle the idea that objective evidence exists
I actually spent some time at university in the post-Kantian philosophy space. There was a point when I really liked it, but now I find it problematically navel-gazey.
For example, claiming a worldview that values ‘joy, community, and non-human life’ would somehow ‘de-reify’ scarcity as something that actually exists in the world seems completely unhelpful to me. Scarcity pretty straightforwardly predates capitalism and feudalism (see starvation), and I think having a joy- and community-based value system comes nowhere close to building the structures that will let us avoid it.
That said, ‘EA-admissible data therefore only captures a small fraction of total ideas’ is correct, and I tend to agree with you that EA should act a lot more like it (post here). I just encourage you not to push this to the point where you’re making statements like ‘objective evidence doesn’t exist’. Even if they are in some sense true, they are totally impractical, and so are (rightfully, I think) offputting to most people.
I think you mostly bring critical theory up in the context of deciding what evidence to act on. For all of its flaws, including, as you say, that it inevitably falls short of being fully inclusive, focused and quantitative approaches have yielded some pretty amazing results.
What pushed me over into the EA consensus here is Philip Tetlock’s Superforecasting. There’s something about being able to consistently predict the future better than everyone else that I find pretty convincing, and Tetlock’s background is also in the humanities. I really recommend reading it.
I agree it is very important that people get to work on projects in areas without pre-researched interventions or randomized control trials they can use to argue their ideas will work, because I think your observations relating to diversity and bias in who gets to have those things funded are correct. I just don’t think there’s anything wrong with an ecosystem that decides to focus on areas where the RCTs do already exist.
Finally, I care about a method of change’s track record, and I’m not particularly convinced by critical theory in this area (you might want to look into Martin Heidegger’s politics). I want to take their insights around how atypical and underleveraged it is to listen to diverse viewpoints, use that to update very hard on the views of people with different experiences and backgrounds to my own, and then get to work.
In the volunteering that I have done, no other part of the critical theory you cite throughout your piece has proven particularly useful. It’s worth noting that unions and social movements predate critical theory!
General comment on the idea that EA is opposed to social change
EA is not opposed to social change movements. I donate to Sunrise Movement on the recommendation of https://www.givinggreen.earth/ , there is also https://www.socialchangelab.org/ .
I regularly hear criticism along the lines of ‘EA by its very structure cannot question the dynamics of power, it can only work within the existing political system’, and I think this is straightforwardly false.
Political system change certainly isn’t a focus of EA from what I’ve seen, but that is mostly because EA folks tend to like numbers and statistics, which can’t be leveraged in quite such interesting ways when working with grassroots organizations. The typical elite background of EAs probably also makes grassroots organization unappealing on some aesthetic level too to be fair, which seems more problematic.
That said, this says something about the personal preferences of the EA community, but it does not render EA opposed to other communities doing grassroots work. In specific cases where EA gets in the way of another community, of course they should communicate and try to resolve the issue, but generally I think the best solution is pretty clearly to live and let live.
Some people like doing good with statistics, some people like doing good with organizing, those preferences lend themselves to different cause areas, and I am very grateful to both groups of people.
Tropical rainforest example
I want to hear more about this, as based on what you’ve written it sounds like a great cause to prioritize. (I acknowledge that you’re worried EA cause-prioritizing the amazon will lead to commodifying the amazon, and hopefully I’ve explained why I disagree with that above).
I worry about our implicit social structures sending the message “all the cool people hang around the centrally EA spaces”
I agree that I don’t hear EAs explicitly stating this, but it might be a position that a lot of people are indirectly commited to. e.g. Perhaps a lot of the community have a high degree of confidence in existing cause prioritization and interventions and so don’t see much reason to look elsewhere.
I like your proposed suggestions! I would just add a footnote that if we run into resistance trying to implement them, it could be useful to get curious about the community attitudes that are causing that resistance, to try to identify them, and to explicitly challenge them if they appear problematic.
Though I do recognize this response reads like me moving the goal posts....
Yep, I think this is my difficulty with your viewpoint. You argue that there’s no way to predict future human discoveries, and if I give you counterexamples your response seems to be ‘that’s not what I mean by discovery’. I’m not convinced the ‘discovery-like’ concept you’re trying to identify and make claims about is coherent.
Maybe a better example here would be the theory of relativity and the subsequent invention of nuclear weapons. I’m not a physicist, but I would guess the scientific breakthrough that led to nuclear weapons would have been almost impossible to predict unless you were Einstein or Einstein-adjacent.
I agree we should be very scared of these sorts of breakthroughs, and the good news is many EAs agree with you! See Nick Bostrom’s Vulnerable World Hypothesis for example. You don’t need to argue against our ability to predict if/when all future discoveries will occur to make this case.
This is as it must be with all human events
I think there are some straightforward counterexamples here:
Superforecasters. You mentioned Tetlock; the best forecasters in his study are consistently better than average at predicting geopolitical events. Those events are influenced by people.
Speech and language development in infants. There’s a lot of discovery going on here, and we’ve spent enough time observing it that we can be pretty confident about how long it will take (link).
Periodic table. Maybe you are most interested in new discoveries. Mendeleev successfully used the periodic table to predict the discovery of new elements ahead of time (link).
If predictions of anything with human involvement are nonsensical, superforecasters shouldn’t exist, humans should learn to talk at wildly differing times, and Mendeleev wasn’t doing meaningful inference, he just got really lucky. So I think your claim is far too strong.
All that said, I think that predicting if, and especially when, events that have never previously happened will occur is usually very difficult and merits a lot of humility and uncertainty.
I also think we agree on a weaker version of your claim, which is that EA underestimates the value of data it hasn’t yet seen (post here).
I had not noticed that those aren’t the same, thank you for correcting me! And I agree that applying to it makes a lot more sense than applying to the incubation program.
On this particular point
message testing from Rethink suggests that longtermism and existential risk have similarly-good reactions from the educated general public
I can’t find info on Rethink’s site, is there anything you can link to?
Of the three best-performing messages you’ve linked, I think the first two emphasise risk much more heavily than longtermism. The third does sound more longtermist, but I still suspect the risk-ish phrase ‘ensure a good future’ is a large part of what resonates.
All that said, more info on the tests they ran would obviously update my position.
So people actually pretty like messages that are about unspecified, and not necessarily high-probability threats, to the (albeit nearer-term) future.
This seems correct to me, and I would be excited to see more of them. However, I wouldn’t interpret this as meaning ‘longtermism and existential risk have similarly-good reactions from the educated general public’, I would read this as risk messaging performing better.
Also, messages ‘about unspecified, and not necessarily high-probability threats’ is not how I would characterize most of the EA-related press I’ve seen recently (NYTimes, BBC, Time, Vox).
(More generally, I mostly see journalists trying to convince their readers that an issue is important using negative emphasis. Questioning existing practices is important: they might be ineffective; they might be unsuitable to EA aims (e.g. manipulative, insufficiently truth-seeking, geared to persuade as many people as possible which isn’t EA’s objective, etc.). But I think the amount of buy-in this strategy has in high-stakes, high-interest situations (e.g. US presidential elections) is enough that it would be valuable to be clear on when EA deviates from it and why).
tl;dr: I suspect risk-ish messaging works better. Journalists seem to have a strong preference for it. Most of the EA messaging I’ve seen recently departs from this. I think it would be great to be very clear on why. I’m aware I’m missing a lot of data. It would be great to see the data from rethink that you referenced. Thanks!
“EA-aligned” was probably a poor choice of words, maybe “EA-influenced” would be better. I agree that e.g. the EA forum’s attitude to OpenAI is strongly negative.