I can highly recommend following Sentinel’s weekly minutes, a weekly update from superforecasters on the likelihood of any events which plausibly could cause worldwide catastrophe.
Perhaps the weekly newsletter I look the most forward to at this point. Read previous issues here:
I’ve been reviewing some old Forum posts for an upcoming post I’m writing, and incidentally came across this by Howie Lempel for noticing in what spirit you’re engaging with someone’s ideas:
“Did I ask this question because I think they will have a good answer or because I think they will not have a good answer?”
I felt pretty called out :P
To be fair, I think the latter is sometimes a reasonable persuasive tactic, and it’s fine to put yourself in a teaching role rather than a learning role if that’s your endorsed intention and the other party is on board. But the value of this quote to me is that it successfully highlights how easily we can tell ourselves we’re being intellectually curious, when we’re actually doing something else.
Quick[1] thoughts on the Silicon Valley ‘Vibe-Shift’
I wanted to get this idea out of my head and into a quick-take. I think there’s something here, but a lot more to say, and I’ve really haven’t done the in-depth research for it. There was a longer post idea I had for this, but honestly diving more than I have here into it is not a good use of my life I think.
The political outlook in Silicon Valley has changed.
Since the attempted assassination attempt on President Trump, the mood in Silicon Valley has changed. There have been open endorsements, e/acc has claimed political victory, and lots of people have noticed the ‘vibe shift’.[2] I think that, rather than this being a change in opinions, it’s more an event allowing for the beginning of a preference cascade, but at least in Silicon Valley (if not yet reflected in national polling) it has happened.
So it seems that a large section of Silicon Valley is now openly and confidently supporting Trump, and to a greater or lesser extent aligned with the a16z/e-acc worldview,[3] we know it’s already reached the ears of VP candidate JD Vance.
How did we get here
You could probably write a book on this, so this is a highly opinionated take. But I think this is somewhat, though not exclusively, an own goal of the AI Safety movement.
As ChatGPT starts to bring AI, and AI Safety, into the mainstream discourse, the e/acc countermovement begins. It positions itself as opposite effective altruism, especially in the wake of SBF.
Guillaume Verdon, under the alias “Beff Jezos”, realises the memetic weakness of the AI Safety movement and launches a full memetic war against it. Regardless of his rightness or wrongness, you do to some extent got to hand it to him. He’s like right-wing Émile Torres, ambitious and relentless and driven by ideological zeal against a hated foe.
Memetic war is total war. This means nuance dies to get it to spread. I don’t know if, for example, Marc Andreessen actually thinks antimalarial bednets are a ‘triple threat’ of badness, but it’s a war and you don’t take prisoners. Does Beff think that people running a uni-group session on Animal Welfare are ‘basically terrorists’, I don’t know. But EA is the enemy, and the enemy must be defeated, and the war is total.
The OpenAI board fiasco is, I think, a critical moment here. It doesn’t matter what the reasoning we’ve come out with at the end of the day was, I think it was perceived as ‘a doomer coup’ and it did radicalize the valley. In his recent post Richard Ngo called on the AI Safety movement to show more legitimacy and competence. The board fiasco torpedoed my trust in the legitimacy and competence of many senior AI safety people, so god knows how strong the update was for Silicon Valley as a whole.
This new movement became increasingly right-wing coded. Partly as a response to the culture wars in America and the increasing vitriol thrown by the left against ‘tech bros’, partly as a response to the California Ideology being threatened by any sense of AI oversight or regulation, and partly because EA is the enemy and EA was being increasingly seen by this group as left-wing, woke, or part of the Democratic Party due to the funding patterns of SBF and Moskovitz. I think this has led, fairly predictably, to the right-ward shift in SV and direct political affiliation with a (prospective) second Trump presidency
Across all of this my impression is that, just like with Torres, there was little to no direct pushback. I can understand not wanting to be dragged into a memetic war, or to be involved in the darker parts of Twitter discourse. But the e-acc/technooptimist/RW-Silicon-Valley movement was being driven by something, and I don’t think AI Safety ever really argued against it convincingly, and definitely not in a convincing enough way to ‘win’ the memetic war. Like, the a16z cluster literally lied to Congress and to Parliament, but nothing much come of that fact.
I think this is very much linked to playing a strong ‘inside game’ to access the halls of power and no ‘outside game’ to gain legitimacy for that use of power. It’s also I think due to EA not wanting to use social media to make its case, whereas the e-acc cluster was born and lives on social media.
Where are we now?
I’m not a part of the Bay Area scene and culture,[4] but it seems to me that the AI Safety movement has lost the ‘mandate of heaven’ to whatever extent it did have it. SB-1047 is a push to change policy that has resulted in backlash, and may result in further polarisation and counter-attempts to fight back in a zero-sum political game. I don’t know if it’s constitutional for a Trump/Vance administration to use the Supremacy Clause to void SB-1047 but I don’t doubt that they might try. Biden’s executive order seems certain for the chopping block. I expect a Trump administration to be a lot less sympathetic to the Bay Area/DC AI Safety movements, and the right-wing part of Silicon Valley will be at the very least energised to fight back harder.
One concerning thing for both Silicon Valley and the AI Safety movement is what happens as a result of the ideological consequences of SV accepting this trend. Already a strong fault-line is the extreme social conservatism and incipient nationalism brought about by this. In the recent a16z podcast, Ben Horowitz literally accuses the Biden administration of breaking the rule of law, and says nothing about Trump literally refusing to concede the 2020 election and declaring that there was electoral fraud. Mike Solana seems to think that all risks of democratic backsliding under a Trump administration were/are overblown (or at least that people in the Bay agreeing was preference falsification). On the Moments-of-Zen Podcast (which has also hosted Curtis Yarvin twice), Balaji Srinivasan accused the ‘Blue Tribe’ of ethnically cleansing him out of SF[5]and called on the grey tribe to push all the blues out of SF.e-acc sympathetic people are noting that anti-trans ideas bubbling up in the new movement. You cannot seriously engage with ideas and shape them without those ideas changing you.[6] This right-wing shift will have further consequences, especially under a second Trump presidency.
What next for the AI Safety field?
I think this is a bad sign for the field of AI Safety. Political polarisation has escaped AI for a while. Current polls may lean in support , but polls and political support are fickle, especially in the age of hyper-polarisation.[7] I feel like my fears around the perception of Open Philanthropy are re-occuring here but for the AI Safety movement at large.
I think the consistent defeats to the e-acc school and the fact that the tech sector as a whole seems very much unconvinced by the arguments for AI Safety should at some point lead to a reflection from the movement. Where you stand on this very much depends on your object-level beliefs. While this is a lot of e-acc discourse around transhumanism, replacing humanity, and the AI eschaton, I don’t really buy it. I think that they don’t think ASI is possible soon, and thus all arguments for AI Safety are bunk. Now, while the tech sector as a whole might not be as hostile, they don’t seem at all convinced of the ‘ASI-soon’ idea.
A key point I want to emphasise is that one cannot expect to wield power successfully without also having legitimacy.[8]And to the extent that the AI Safety movement’s strategy is trying to thread this needle it will fail.
Anyway, long ramble over, and given this was basically a one-shot ramble it will have many inaccuracies and flaws. Nevertheless I hope that it can be directionally useful and lead to productive discussion.
See here, here, and here. These examples are from Twitter because, for better or for worse, it seems much of SV/tech opinions are formed by Twitter discourse.
A good example in fiction is in Warhammer40K, where Horus originally accepts the power of Chaos to fight against Imperial Tyranny, but ends up turning into their slave.
Due to polarisation, views can dramatically shift on even major topics such as the economy and national security (i know these are messy examples!). Current poll leads for AI regulation should not, in any way, be considered secure
I guess you could also have overwhelming might and force, but even that requires legitimacy. Caesar needed to be seen as legitimate by Marc Anthony, Alexander didn’t have the legitimacy to get his army to cross the Hyphasis etc.
I think this case it’s ok (but happy to change my mind) - afaict he owns the connection now and the two names are a bit like separate personas. He’s gone on podcasts under his true name, for instance.
@thoth hermes (or https://x.com/thoth_iv if someone can get it to them if you’re Twitter friends then pls go ahead.[1] I’m responding to this thread here—I am not saying “that EA is losing the memetic war because of its high epistemic standards”, in fact quite the opposite r.e. AI Safety, and maybe because of misunderstanding of how politics work/not caring about the social perception of the movement. My reply to Iyngkarran below fleshes it out a bit more, but if there’s a way for you to get in touch directly, I’d love to clarify what I think, and also hear your thoughts more. But I think I was trying to come from a similar place that Richard Ngo is, and many of his comments on the LessWrong thread here very much chime with my own point-of-view. What I am trying to push for is the AI Safety movement reflecting on losing ground memetically and then asking ‘why is that? what are we getting wrong?’ rather than doubling down into lowest-common denominator communication. I think we actually agree here? Maybe I didn’t make that clear enough in my OP though.
@Iyngkarran Kumar—Thanks for sharing your thoughts, but I must say that I disagree with it. I don’t think that the epistemic standards are working against us by being too polite, quite the opposite. I think the epistemic standards in AI Safety have been too low relative to the attempts to wield power. If you are potentialy going to criminalise existing Open-Source models,[2] you better bring the epistemic goods. And for many people in the AI Safety field, the goods have not been brought (which is why I see people like Jeremy Howard, Sara Hooker, Rohit Krishnan etc get increasingly frustrated by the AI Safety field). This is on the field of AI Safety imo for not being more persuasive. If the AI Safety field was right, the arguments would have been more convincing. I think, while it’s good for Eliezer to say what he thinks accurately, the ‘bomb the datacenters’[3] piece has probably been harmful for AI Safety’s cause, and things like it a very liable to turn people away from supporting AI Safety. I also don’t think it’s good to say that it’s a claim of ‘what we believe’, as I don’t really agree with Eliezer on much.
(r.e. inside vs outside game, see this post from Holly Elmore)
@anormative/ @David Mathers—Yeah it’s difficult to manage the exact hypothesis here, especially for falsified preferences. I’m pretty sure SV is ‘liberal’ overall, but I wouldn’t be surprised if Trump % is greater than 16 and 20, and it definitely seems to be a lot more open this time, e.g. a16z and Musk openly endorsing Trump, Sequoia Capital partners claiming that Biden dropping out was worse than the Jan 6th riot. Things seem very different this time around, different enough to be paid attention to.
- - - - - - - - - - - -
Once again, if you disagree, I’d love to actually here why. Up/down voting is a crude feedback to, and discussion of ideas leads to much quicker sharing of knowledge. If you want to respond but don’t want to publicly, then by all means please send a DM :)
This piece from @1a3orn is excellent and to absence of evidence of good arguments against it is evidence of the absence of said arguments. (tl;dr—AI Safety people, engage with 1a3orn more!)
I disagree FWIW. I think that the political activation of Silicon Valley is the sort of thing which could reshape american politics, and that twitter is a leading indicator.
I think that the political activation of Silicon Valley is the sort of thing which could reshape american politics, and that twitter is a leading indicator.
I don’t disagree with this statement, but also think the original comment is reading into twitter way too much.
absence of evidence of good arguments against it is evidence of the absence of said arguments. (tl;dr—AI Safety people, engage with 1a3orn more!)
There are many (edit: 2) comments responding and offering to talk. 1a3orn doesn’t appear to have replied to any of these comments. (To be clear, I’m not saying they’re under any obligation here, just that there isn’t a absence of attempted engagement and thus you shouldn’t update in the direction you seem to be updating here.)
a) r.e. Twitter, almost tautologically true I’m sure. I think it is a bit of signal though, just very noisy. And one of the few ways for non-Bay people such as myself to try to get a sense of the pulse of the Bay, though obviously very prone to error, and perhaps not worth doing at all.
Across all of this my impression is that, just like with Torres, there was little to no direct pushback
Strongly agree. I think the TESCREAL/e-acc movements badly mischaracterise the EA community with extremely poor, unsubstantiated arguments, but there doesn’t seem to be much response to this from the EA side.
I think this is very much linked to playing a strong ‘inside game’ to access the halls of power and no ‘outside game’ to gain legitimacy for that use of power
What does this refer to? I’m not familiar.
Other thoughts on this:
Publicly, the quietness from the EA side in response to TESCREAL/e-acc/etc. allegations is harming the community’s image and what it stands for. But ‘winning’ the memetic war is important. If not, then the world outside EA—which has many smart, influential people—ends up seeing the community as a doomer cult (in the case of AI safety) or assigns some equally damaging label that lets them quickly dismiss many of the arguments being made.
I think this is a case where the the epistemic standards of the EA community work against it. Rigorous analysis, expressing second/third-order considerations, etc. are seen as the norm for most writing on the forum. However, in places such as Twitter, these sorts of analyses aren’t ‘memetically fit’ [1].
So, I think we’re in need of more pieces like the Time essay on Pausing AI—a no-punches-pulled sort of piece that gets across the seriousness of what we’re claiming. I’d like to see more Twitter threads and op-ed’s that dismantle claims like “advancements in AI have solved it’s black-box nature”, ones that don’t let clearly false claims like this see the light of day in serious public discourse.
Don’t get me wrong—epistemically rigorous work is great. But when responding to TESCREAL/e-acc ‘critiques’ that continuously hit below the belt, other tactics may be better.
I’ve often found it hard to tell whether an ideology/movement/view has just found a few advocates among a group, or whether it has totally permeated that group. For example, I’m not sure that Srinivasan’s politics have really changed recently or that it would be fair to generalize from his beliefs to all of the valley. How much of this is actually Silicon Valley’s political center shifting to e/acc and the right, as opposed to people just having the usual distribution of political beliefs (in addition to a valley-unspecific decline of the EA brand)?
The EA Infrastructure Fund will fund and support projects that build and empower the community of people trying to identify actions that do the greatest good from a scope-sensitive and impartial welfarist view.
And a rationale (there’s more detail in the post):
[...] EA is doing something special.
[...] fighting for EA right now could make it meaningfully more likely to thrive long term.
[...] we could make EA much better than it currently is—particularly on the “beacon for thoughtful, sincere, and selfless” front. [...]
Here I’m spending some time thinking about this, in particular:
What does it mean for EA to thrive?
What projects could push EA in the direction of thriving?
(I work at EAIF. These are my personal views/ thoughts. I’m not speaking on behalf of EAIF here)
For many people, having an EA job is pretty important.
It’s pretty competitive and many people who want EA jobs will not in fact get them.
There’s been some discussion related to this on the EA Forum, focusing in particular on jobseekers. I’m also interested in exploring this dynamic with people who are working in EA jobs.
I expect EA job scarcity not only have an impact on EA jobseekers, but also people who are working in EA jobs.
Given 1 and 2, it seems like for people working in EA jobs it will be pretty important for them to keep their jobs. If the job market is competitive it may not be obvious that they can get another one. (For people who have got one EA job, it will presumably be easier to get another, but maybe not guaranteed).
For someone who’s in a position of scarcity about their EA job, I can imagine this meaning they focus primarily on performing well/ being seen to perform well.
This becomes a problem if what counts as performing well and what is actually good to do comes into conflict. Eg. this might involve things like:
Agreeing with the organisational strategy or one’s manager more than one endorses
Focusing on ensuring that they have achieved certain outputs independent of whether that output seems good
In general I expect that under conditions of scarcity people will be less able to do valuable work (and I mean valuable here as ‘actually good’ as opposed to ’work that is perceived to be valuable).
(If I’m right about this, then one potential answer to ‘what is it for EA to thrive’, is: EAs aren’t in a position of scarcity).
Things I’d be interested to ask people who are working at EA jobs to understand whether this is in fact a thing:
How concerned are you about your perceived performance?
If your employer/ manager/ funder/ relevant people said something like: ‘We have full confidence in you, your job is guaranteed and we want you to focus on whatever you think is best’ - would that change what you focus on? How much?
If your employer/ manager/ funder/ relevant people said something like: ‘We have full confidence in you, your job is guaranteed and we want you to focus on whatever you think is best’ - would that change what you focus on? How much?
My personal impression is that significant increases in unrestricted funding (even if it were a 1-1 replacement for restricted funding) would dramatically change orgs and individual prioritisations in many cases.
To the extent that one thinks that researchers are better placed to identify high value research questions (which, to be clear, one may not in many cases), this seems bad.
Are these EA phenomena? Also, are they psychological phenomena?
These things (I guess excluding EA disillusionment), don’t just exist within EA they exist within society in general, so it’s plausibly unfair to call them EA phenomena. Though it also seems to me that for each of these things, there’s somewhat strong fit with EA, and EA culture.
Taking impostor syndrome as an example: EA often particularly values ambitious and talented people. Also, it seems to me there’s something of a culture of assessing and prioritising people on this basis. Insofar as it’s important for people to be successful within EA, it’s also important for people to be seen in a certain way by others (talented, ambitious etc.). In general, the stronger the pressure there is for people to be perceived in a certain way, the more prominent I expect impostor syndrome to be.
(I’m a bit wary of ‘just so’ stories here, but my best guess is that this in fact explanatory).
I think impostor syndrome and other things in this ballpark are often discussed as an individual/ psychological phenomena. I think such framings are pretty useful. And there’s another framing which is seeing it instead as a ~sociological phenomena—these are things which happen in a social context, as a result of different social pressures and incentives within the environment.
I don’t know quite what to conclude here, in a large part because I don’t know how common these things are within EA, and how this compares to other places (or even what the relevant comparison class is). Though tentatively, if I’m asking ‘What does it look like for EA to thrive?’, then part of my answer is ‘being an environment where impostor syndrome, burnout, impact obsession and EA disillusionment are less common’.
There’s a thing where lots of people will say that they are EA Adjacent rather than EA (funny post related to this). In particular, it seems to me that the closer to the core people are, the less inclined they are to identify themselves with EA. What’s going on here? I don’t know, but it’s an interesting trailhead to me.
Plausibly there are some aspects of EA, the culture, norms, worldview, individuals, organisations etc. that people disagree with or don’t endorse, and so prefer to not identify as EAs.
I’m unsure how much to treat this as reflective of a substantive issue vs. a quirk, or reflective of things being actually fine. At least in terms of EA being a ‘beacon for thoughtful, sincere, and selfless’, it seems a little bit worrying to me that some of the core members of the community aren’t willing to describe themselves as EA.
Perhaps a way of getting to the heart of this is asking people something like: Imagine you’re talking to someone who is thoughtful, sincere and selfless. Would you recommend EA to them? Which parts? How strongly? Would you express any reservations?
Looping back to the question of ‘What is it for EA to thrive?’, one answer is: It’s the kind of community that EA’s would strongly recommend to a thoughtful, sincere and selfless friend.
(Maybe this is too strong—people will probably reasonably have disagreements about what aspects of EA are good and aren’t, and if everyone is very positive on EA in this way, this plausibly means that there’s not enough disagreement in the community. )
In particular, it seems to me that the closer to the core people are, the less inclined they are to identify themselves with EA. What’s going on here? I don’t know, but it’s an interesting trailhead to me.
I think this implies that there is a substantive non-quirky effect. That said, I imagine some of this may be explained by new EAs simply being particularly enthusiastic in ways which explain stronger identification with EA and higher satisfaction.[1]
One dynamic which I expect explains this is the narcissism of small differences, as people become closer to EA, differences and disagreements become more salient, and so people may become more inclined to want to distance themselves from EA as a whole.
If being thoughtful, sincere and selfless is a core value, it seems like it would be more of a problem if influential people in the community felt they had to embrace the label even if they didn’t think it was valuable or accurate
I suspect a lot of the ‘EA adjacent’ description comes from question marks about particular characteristics EA stances or image rather than doubting some of their friends could benefit from participating in the community, and that part of that is less a rejection of EA altogether and more an acknowledgement they often find themselves at least as closely aligned with people doing great work outside the community.
(Fwiw I technically fit into the “adjacent” bracket from the other side: never been significantly active in the community, like some of its ideas and values—many of which I believed in before ‘EA’ was a thing—and don’t identify with or disagree with other ideas commonly associated with EA, so it wouldn’t really make much sense to call myself an EA)
Here’s a story you could tell about academia. Academia, is in some sense supposed to be about generating knowledge. But it ends up being ineffective at doing this because of something something incentives. Eg.
Academic jobs are highly competitive
In order to get an academic job, it’s more important to have done things like original research than things like replications.
Things like replications are undersupplied, and the replication crisis happens.
What are the incentives within EA? How does this affect how well EA ends up ‘doing the most good?’. I don’t have a full theory here, though I also suspect that there are ways in which incentives in EA can push against doing the most good. Professional EA group funding is one example:
Professional EA group organisers are often in a bit of a precarious position. Their job depends on their ability to get funding from organisations like CEA or EAIF.
One of the main ways that EA group organisers are assessed is on the basis of things like how well they produce highly engaged EAs, or career plan changes or other such things (I think this is broadly true, though I don’t have a great insight into how CEA assesses groups).
Professional EA group organisers are incentivised to produce these kinds of things. Some potential problems here: It’s hard to assess what counts as a good eg. career, which pushes in the direction of non-standard career options being discounted, often it may make sense for someone to focus on building career capital over working at an EA organisation, but these kinds of things are less obviously/ legibly impactful…
It’s hard to assess what counts as a good eg. career, which pushes in the direction of non-standard career options being discounted, often it may make sense for someone to focus on building career capital over working at an EA organisation, but these kinds of things are less obviously/ legibly impactful…
I agree with the general gist, but my impression is that organisations that focus on career changes and grantmakers have high epistemic humility. When looking at meta organisations focussing on career change, most seem not to break down the changes into types in their quantitative analysis. This leads to a greater focus on case studies where different aspects like prior achievements and unusual career paths can be explained. I assume there is some signalling going on between grantmakers and group organisers where a low-fidelity version might point to standard options, whereas thoughtful grantmakers showcasing a wider variety of pathways as potentially impactful can make a difference.
There’s a comment by Saulius on an old EA Forum post: ‘[...] I see EA as something that is mostly useful when you are deciding how you want to do good. After you figured it out, there is little reason to continue engaging with it [...]’.
I found this pretty interesting, and it inspired a bunch of thoughts. Here’s a couple of oversimplified models of what it is to pursue the EA project:
Model 1:
EA involves figuring out how to do the most good and doing it.
Figuring out how to do the most good involves working out what cause area is most important and what career path to pursue.
Doing it involves getting such a job and executing well on it.
Model 2:
EA involves figuring out how to do the most good and doing it.
This is less of a two step process, and more of an ongoing cultivation of virtues and attitudes like scout mindset, effectiveness and so on. It involves constant vigilance, or constant attunement. It’s an ongoing process of development, personal, epistemic, moral etc.
I see most EA community building efforts as mostly framed by model 1. For example EA Groups, the EA Forum (perhaps EAG/ EAGx as well, though this is less clear). It seems to me a common pattern for people to engage in these things heavily when getting involved in EA, and then when people are ‘in’ they stop engaging with them, and focus on executing at their job.
Insofar as engaging with these things (EA groups, EA Forum etc.) is a key component of what it is to engage with EA, I’m inclined to agree with the above comment—once you’ve figured out what to do there’s little reason to continue engaging with EA.
I’d like to see more community building efforts, or EA Infrastructure that’s framed around model 2 - things that give EA’s who are already ‘in’ a reason to continue engaging with EA, things that provide them with value in pursuing this project.
I don’t think model 1 and 2 necessarily have to come into conflict. Or at least, I think it’s fine and good for there to be people that see EA as mostly being relevant to a career decision process. And, for people that want to treat the EA project as more like model 2 (an ongoing process of cultivating virtues like scout mindset, effectiveness and so on, I’d be excited to see more community building, or infrastructure which is designed to support them in these aims.
I like this framing and agree that most CB effort seems to go into model 1, which I also spend most of my time working on. Model 2 efforts could help people choose career paths where they upskill or do direct work in organisations that are not EA-aligned. This could reduce the frustration connected with job searches of early career individuals.
With open-source models being released and on ramps to downstream innovation lowering, the safety challenges may not be a single threshold but rather an ongoing, iterative cat-and-mouse game.
Just underscores the importance of people in the policy/safety field thinking far ahead
Reading and engaging with the forum as good for a meta reason, engaging and encouraging other people to keep making posts because engagement seems to exist and they’re incentivized to post. Or even more junior people to try and contribute, idk what the ea forum felt like ~10 years ago, but probably lower standards for engagement.
Hey everyone, in collaboration with Apart Research, I’m helping organize a hackathon this weekend to build tools for accelerating alignment research. This hackathon is very much related to my effort in building an “Alignment Research Assistant.”
Here’s the announcement post:
2 days until we revolutionize AI alignment research at the Research Augmentation Hackathon!
As AI safety researchers, we pour countless hours into crucial work. It’s time we built tools to accelerate our efforts! Join us in creating AI assistants that could supercharge the very research we’re passionate about.
Date: July 26th to 28th, online and in-person Prizes:$2,000 in prizes
Why join?
* Build tools that matter for the future of AI * Learn from top minds in AI alignment * Boost your skills and portfolio
We’ve got a Hackbook with an exciting project to work on waiting for you! No advanced AI knowledge required—just bring your creativity!
Register now: Sign up on the website here, and don’t miss this chance to shape the future of AI research!
These benchmarks are reporting the second of the two.
LLMs (at least without scaffolding) still do badly on ARC, and I’d wager Llama 405B still doesn’t do well on the ARC-AGI challenge, and it’s telling that all the big labs release the 95%+ number they get on AI2-ARC, and not whatever default result they get with ARC-AGI...
(Or in general, reporting benchmarks where they can go OMG SOTA!!!! and not helpfully advance the general understanding of what models can do and how far they generalise. Basically, traditional benchmark cards should be seen as the AI equivalent of “IN MICE”)
I want to say thanks to people involved in the EA endeavour. I know things can be tough at times, but you didn’t have to care about this stuff, but you do. Thank you, it means a lot to me. Let’s make the world better!
A lot of people have said sharing these notes were helpful, so sharing it here on the EAF! Here are notes on NTI | bio’s recent event with Dr. Lu Borio on H5N1 Bird Flu, in case anyone here would find it helpful!
Inspired in part by the EA Forum’s recent debate week, Metaculus is running a “focus week” this week, aimed at trying to make intellectual progress on the issue of “What will the world look like five years after AGI (assuming that humans are not extinct)[1]?”
Leaders of AGI companies, while vocal about some things they anticipate in a post-AGI world (for example, bullishness in AGI making scientific advances), seem deliberately vague about other aspects. For example, power (will AGI companies have a lot of it? all of it?), whether some of the scientific advances might backfire (e.g., a vulnerable world scenario or a race-to-the-bottom digital minds takeoff), and how exactly AGI will be used for “the benefit of all.”
Those interested: head over here. You can participate by:
Forecasting
Commenting
Comments are especially valuable on long-term questions, because the forecasting community has less of a track record at these time scales.[2][3]
Writing questions
There may well be some gaps in the admin-created question set.[4] We welcome question contributions from users.
The focus week will likely be followed by an essay contest, since a large part of the value in this initiative, we believe, lies in generating concrete stories for how the future might play out (and for what the inflection points might be). More details to come.
This is not to say that we firmly believe extinction won’t happen. I personally put p(doom) at around 60%. At the same time, however, as I have previouslywritten, I believe that more important trajectory changes lie ahead if humanity does manage to avoid extinction, and that it is worth planning for these things now.
With short-term questions on things like geopolitics, I think one should just basically defer to the Community Prediction. Conversely, with certain long-term questions I believe it’s important to interrogate how forecasters are reasoning about the issue at hand before assigning their predictions too much weight. Forecasters can help themselves by writing comments that explain their reasoning.
In addition, stakeholders we work with, who look at our questions with a view to informing their grantmaking, policymaking, etc., frequently say that they would find more comments valuable in helping bring context to the Community Prediction.
I am very concerned about the future of US democracy and rule of law and its intersection with US dominance in AI. On my Manifold question, forecasters (n=100) estimate a 37% that the US will no longer be a liberal democracy by the start of 2029 [edit: as defined by V-DEM political scientists].
Project 2025 is an authoritarian playbook, including steps like 50,000 political appointees (there are ~4,000 appointable positions, of which ~1,000 change in a normal presidency). Trump’s chances of winning are significantly above 50%, and even if he loses, Republicans get another chance every 4 years.
The presidential immunity ruling by SCOTUS is very far-reaching and further empowers an autocratic president. (From a Western European standpoint it’s already insane that judges are generally appointed by a single party.)
I do not know what this means for effective altruism. With respect to global affairs, Republicans are very divided on foreign policy, with Trump being far more isolationist that the majority of Republican legislators, so anything could happen there.
However, a Republican government being in charge when AGI is close to being achieved (I don’t know if we’re close) seems very dangerous, and I’m surprised that Ashenbrenner pushed for nationalisation given this risk, and that no one (?) made this criticism. If the alignment problem got solved, MAGA controlling AGI would very likely be very bad, while I think Democrat leadership is more likely (but by no means guaranteed) to actually attempt to do the right thing on impartial grounds.
I don’t know if it’s tractable to work on this. In the medium term, it might be worthwhile to introduce/support introducing ballot initiatives to ban gerrymandering. This worked well in Michigan in 2018. However, I have no idea about the details of tractability (how many people are working on this, how many states have this possibility).
On my Manifold question, forecasters (n=100) estimate a 37% that the US will no longer be a liberal democracy by the start of 2029.
Your question is about V-Dem’s ratings for the US, but these have a lot of problems, so I think don’t shine nearly as much light on the underlying reality as you suggest. Your Manifold question isn’t really about the US being a liberal democracy, it’s about V-Dem’s evaluation. The methodology is not particularly scientific—it’s basically just what some random academics say. In particular, this leaves it quite vulnerable to the biases of academic political scientists.
The election of Trump is a huge outlier, giving a major reduction to ‘Liberal Democracy’.
This impact occurs immediately despite the fact that he didn’t actually pass that many laws or make that many changes in 2017; I think this is more about perception than reality.
The impact appears to be larger than the abolition of slavery, the passage of the civil rights act, the second world war, conscription or female suffrage. This seems very implausible to me.
Freedom of domestic movement increased in 2020, despite the introduction of essentially unprecedented covid-related restrictions. (Other countries with even more draconian rules, like the UK, also do not see major declines here, even though the entire population was essentially under house arrest for much of the year)
The subnational civil liberties unevenness index does not seem to reflect the fact that covid restrictions were very different in rural and urban areas.
On the whole I think these ratings often tell us more about the political views of the authors (pro-lockdown, anti-trump) than they do about the actual levels of liberty or democracy in a country.
V-Dem indicators seem to take into account statements made by powerful politicians, not only their policies or other deeds. For example, I found this in one of their annual reports:
The V-Dem indicator of government attacks on the judiciary, which reveals government rhetoric calling into question the integrity of the judiciary, dropped precipitously in 2010, likely reflecting President Barack Obama’s State of the Union address in which he criticized the Supreme Court’s decision in Citizens United vs. Federal Election Commission. President Donald Trump has sharply increased the pointedness of verbal attacks on the judiciary, referring to one of the judges who blocked his first executive order on immigration as a “so-called judge.” Public criticism of the judiciary can be a healthy part of maintaining the balance between judicial independence and judicial accountability. Yet it can also be part of an unraveling of core checks on power. Coupled with the politicization of the judicial nominations process and the dismantling of super-majoritarian rules of appointing all Article Ill judges, supporters of democracy would be wise to pay close attention to executive -judicial relations in the United States.
My guess is that statements made by Trump were extreme outliers in how they betrayed little respect to democratic institutions, compared to statements made by earlier US presidents, and that affected their model.
I think that’s reasonable. It might not be fully reflective of lived reality for US citizens at the moment the statements are made, but it sure captures the beliefs and motives of powerful people, which is predictive of their future actions.
Indeed, one way to see the drop in 2017 is that it was able to predict a major blow to American democracy (Trump refusing to concede an election) 4 years in advance.
I’m not really sure this contradicts what I said very much. I agree the V-Dem evaluators were reacting to Trump’s comments, and this made them reduce their rating for America. I think they will react to Trump’s comments again in the future, and this will again make them likely reduce their rating for America. This will happen regardless of whether policy changes, and be poorly calibrated for actual importance—contra V-Dem, Trump getting elected was less important than the abolition of slavery. Since I think Siebe was interested in policy changes rather than commentary, this means V-Dem is a bad metric for him to look at.
Thank you, these are some good points. When I made the question, I believed V-DEM had a more rigorous methodology, and I can’t change it now.
I don’t think the specific probability is necessary for my argument (and it depends on how one defines ‘liberal democracy’): a Trump presidency with an enabling Supreme Court would be very harmful to US liberal democracy and the rule of law, and a nationalized AGI project under such a government would be very risky.
P.P.S. I am also concerned about silencing/chilling effects: if you want to get anything political done in the next few years, it’s probably strategic to refrain from criticizing Trump & his allies anywhere publicly, including the EA Forum.
P.S. I don’t think the Forum norm of non-partisanship should apply in its strong form in the case of the US. The Republican party has clearly become an anti-democratic, anti-rule of law, and anti-facts party. This has been concluded by many political scientists and legal scholars.
I wanted to figure out where EA community building has been successful. Therefore, I asked Claude to use EAG London 2024 data to assess the relative strength of EA communities across different countries. This quick take is the result.
The report presents an analysis of factors influencing the strength of effective altruism communities across different countries. Using attendance data from EA Global London 2024 as a proxy for community engagement, we employed multiple regression analysis to identify key predictors of EA participation. The model incorporates geographic, economic, linguistic, and social factors, explaining 52.3% of the variance in per capita attendance.
Methodology
Data Source
Per capita attendance at EAG London 2024 for 62 countries
Variables
Geographic proximity (distance from London, UK)
Economic development (GDP per capita)
English language proficiency
Income inequality (Gini coefficient)
Youth population percentage
Social media penetration
Religiosity
Trust in institutions
Philanthropic culture
Analytical Approach
Multiple regression analysis with per capita EAG London 2024 attendance as the dependent variable
Key Findings
Model Performance:
R-squared value: 0.523
The model explains 52.3% of the variance in per capita attendance
Significant Predictors:
GDP per capita (positive relationship)
Distance from London (negative relationship)
Philanthropic culture (positive relationship)
Social media penetration (positive relationship)
English proficiency (positive relationship)
Non-significant Factors:
Gini coefficient
Youth population percentage
Religiosity
Trust in institutions
Top 10 Countries with Strongest EA Communities (based on positive residuals):
Switzerland
Norway
Netherlands
Estonia
Sweden
Singapore
Finland
Czech Republic
Australia
Ireland
Interpretation of Results
Economic Factors: GDP per capita emerges as the strongest predictor of EA engagement, suggesting that economic development plays a crucial role in fostering EA communities.
Geographic Considerations: Distance from London negatively correlates with engagement, which is expected given that the event was held in London. This finding should be interpreted cautiously: a) It may reflect the ease of attending for nearby countries rather than overall EA engagement. b) The strong performance of some distant countries (e.g., Australia, Singapore) suggests that highly engaged EA communities can overcome geographic barriers.
Cultural Factors: A country’s philanthropic culture strongly predicts EA engagement, highlighting the importance of pre-existing charitable inclinations.
Technological Infrastructure: Social media penetration positively correlates with EA engagement, underscoring the role of online connectivity in community building.
Language: English proficiency remains a significant factor, though its impact is less pronounced than economic and cultural variables.
Limitations
Proxy Measure: EAG attendance may not fully capture all forms of EA engagement.
Single Time Point: The analysis is based on one conference in one year, potentially missing temporal variations.
Unmeasured Variables: Other factors not included in the model may influence EA community strength.
Linear Model Assumptions: The analysis assumes linear relationships, potentially missing complex interactions or non-linear effects.
Host Country Bias: The UK’s attendance is likely inflated due to hosting the event, potentially skewing overall results.
I’m curious if you fed Claude the variables or if it fetched them itself? In the latter case, there’s a risk of having the wrong values, isn’t there?
Otherwise, really interesting project. Curious of the insights to take out of this, esp. for me the fact that Switzerland comes up first. Also surprising that Germany’s not on the list, maybe?
I’m surprised that the “top 10” doesn’t include Denmark, Austria, Belgium, and Germany, since they all have more population-adjusted participants than Ireland, are not English-speaking, are more distant from London, and have lower GDP per capita[1]
Are we using different data?
In general, I’m a bit sceptical of these analyses, compared to looking at the countries/cities with the most participants in absolute terms. I also expect Claude to make lots of random mistakes.
But absolute terms isn’t very useful if we’re trying to spot success stories, right? Or am I misunderstanding something?
But yeah, something seems off about Ireland. The rest of the list feels quite good though. David Moss said they have some per capita estimates in the pipeline, so I’m excited to see what they produce!
This could be a long slog but I think it could be valuable to identify the top ~100 OS libraries and identify their level of resourcing to avoid future attacks like the XZ attack. In general, I think work on hardening systems is an underrated aspect of defending against future highly capable autonomous AI agents.
To further comment, this seems like it might be an intractable task, as the term “dependency hell” kind of implies. You’d have to scrap likely all of GitHub and calculate what libraries are used most frequently in all projects to get an accurate assessment. Then it’s not clear to me how you’d identify their level of resourcing. Number of contributors? Frequency of commits?
Also, with your example of the XZ attack, it’s not even clear who made the attack. If you suspect it was, say, the NSA, would you want to thwart them if their purpose was to protect American interests? (I’m assuming you’re pro-American) Things like zero-days are frequently used by various state actors, and it’s a morally grey question whether or not those uses are justified.
I also, as a comp sci and programmer, have doubts you’d ever be able to 100% prevent the risk of zero-days or something like the XZ attack from happening in open source code. Given how common zero-days seem to be, I suspect there are many in existing open source work that still haven’t been discovered, and that XZ was just a rare exception where someone was caught.
Yes, hardening these systems might somewhat mitigate the risk, but I wouldn’t know how to evaluate how effective such an intervention would be, or even, how you’d harden them exactly. Even if you identify the at-risk projects, you’d need to do something about them. Would you hire software engineers to shore up the weaker projects? Given the cost of competent SWEs these days, that seems potentially expensive, and could compete for funding with actual AI safety work.
I’d be interested in exploring funding this and the broader question of ensuring funding stability and security robustness for critical OS infrastructure. @Peter Wildeford is this something you guys are considering looking at?
not sure if such a study would naturally also be helpful to potential attackers, perhaps even more helpful to attackers than defenders, so might need to be careful about whether / how you disseminate the information
My sense is that 100 is an underestimate for the number of OS libraries as important as that one. But I’m not sure if the correct number is 1k, 10k or 100k.
I’m extremely excited that EAGxIndia 2024 is confirmed for October 19–20 in Bengaluru! The team will post a full forum post with more details in the coming days, but I wanted a quick note to get out immediately so people can begin considering travel plans. You can sign up to be notified about admissions opening, or to express interest in presenting, via the forms linked on the event page:
Do you like SB 1047, the California AI bill? Do you live outside the state of California? If you answered “yes” to both these questions, you can e-mail your state legislators and urge them to adopt a similar bill for your state. I’ve done this and am currently awaiting a response; it really wasn’t that difficult. All it takes is a few links to good news articles or opinions about the bill and a paragraph or two summarizing what it does and why you care about it. You don’t have to be an expert on every provision of the bill, nor do you have to have a group of people backing you. It’s not nothing, but at least for me it was a lot easier than it sounded like it would be. I’ll keep y’all updated on if I get a response.
Both my state senator and my state representative have responded to say that they’ll take a look at it. It’s non-commital, but it still shows how easy it is to contact these people.
I can highly recommend following Sentinel’s weekly minutes, a weekly update from superforecasters on the likelihood of any events which plausibly could cause worldwide catastrophe.
Perhaps the weekly newsletter I look the most forward to at this point. Read previous issues here:
https://sentinel-team.org/blog/
They seem really good! I feel like an idiot for asking this, but where on their website can I subscribe to the newsletter?
IDK if this actually works since I only just signed up, but, the “Join us” button in top right leads to, “https://sentinel-team.org/contact/″
Seems you can add yourself to mailing list from there.
Good question, not sure how I get it into my email actually, I can’t find it on the website either
edit: I think it’s through the forecasting newsletter
Anthropic issues questionable letter on SB 1047 (Axios). I can’t find a copy of the original letter online.
The full letter is available here — was recently posted online as part of this tweet thread.
I’ve been reviewing some old Forum posts for an upcoming post I’m writing, and incidentally came across this by Howie Lempel for noticing in what spirit you’re engaging with someone’s ideas:
I felt pretty called out :P
To be fair, I think the latter is sometimes a reasonable persuasive tactic, and it’s fine to put yourself in a teaching role rather than a learning role if that’s your endorsed intention and the other party is on board. But the value of this quote to me is that it successfully highlights how easily we can tell ourselves we’re being intellectually curious, when we’re actually doing something else.
Quick[1] thoughts on the Silicon Valley ‘Vibe-Shift’
I wanted to get this idea out of my head and into a quick-take. I think there’s something here, but a lot more to say, and I’ve really haven’t done the in-depth research for it. There was a longer post idea I had for this, but honestly diving more than I have here into it is not a good use of my life I think.
The political outlook in Silicon Valley has changed.
Since the attempted assassination attempt on President Trump, the mood in Silicon Valley has changed. There have been open endorsements, e/acc has claimed political victory, and lots of people have noticed the ‘vibe shift’.[2] I think that, rather than this being a change in opinions, it’s more an event allowing for the beginning of a preference cascade, but at least in Silicon Valley (if not yet reflected in national polling) it has happened.
So it seems that a large section of Silicon Valley is now openly and confidently supporting Trump, and to a greater or lesser extent aligned with the a16z/e-acc worldview,[3] we know it’s already reached the ears of VP candidate JD Vance.
How did we get here
You could probably write a book on this, so this is a highly opinionated take. But I think this is somewhat, though not exclusively, an own goal of the AI Safety movement.
As ChatGPT starts to bring AI, and AI Safety, into the mainstream discourse, the e/acc countermovement begins. It positions itself as opposite effective altruism, especially in the wake of SBF.
Guillaume Verdon, under the alias “Beff Jezos”, realises the memetic weakness of the AI Safety movement and launches a full memetic war against it. Regardless of his rightness or wrongness, you do to some extent got to hand it to him. He’s like right-wing Émile Torres, ambitious and relentless and driven by ideological zeal against a hated foe.
Memetic war is total war. This means nuance dies to get it to spread. I don’t know if, for example, Marc Andreessen actually thinks antimalarial bednets are a ‘triple threat’ of badness, but it’s a war and you don’t take prisoners. Does Beff think that people running a uni-group session on Animal Welfare are ‘basically terrorists’, I don’t know. But EA is the enemy, and the enemy must be defeated, and the war is total.
The OpenAI board fiasco is, I think, a critical moment here. It doesn’t matter what the reasoning we’ve come out with at the end of the day was, I think it was perceived as ‘a doomer coup’ and it did radicalize the valley. In his recent post Richard Ngo called on the AI Safety movement to show more legitimacy and competence. The board fiasco torpedoed my trust in the legitimacy and competence of many senior AI safety people, so god knows how strong the update was for Silicon Valley as a whole.
As some evidence this is known in EA circles, I think this is exactly what Dwarkesh is alluding to when asked ‘what happened to the EA brand’. For many people in Silicon Valley, I think the answer is that it got thrown in the dustbin of history.
This new movement became increasingly right-wing coded. Partly as a response to the culture wars in America and the increasing vitriol thrown by the left against ‘tech bros’, partly as a response to the California Ideology being threatened by any sense of AI oversight or regulation, and partly because EA is the enemy and EA was being increasingly seen by this group as left-wing, woke, or part of the Democratic Party due to the funding patterns of SBF and Moskovitz. I think this has led, fairly predictably, to the right-ward shift in SV and direct political affiliation with a (prospective) second Trump presidency
Across all of this my impression is that, just like with Torres, there was little to no direct pushback. I can understand not wanting to be dragged into a memetic war, or to be involved in the darker parts of Twitter discourse. But the e-acc/technooptimist/RW-Silicon-Valley movement was being driven by something, and I don’t think AI Safety ever really argued against it convincingly, and definitely not in a convincing enough way to ‘win’ the memetic war. Like, the a16z cluster literally lied to Congress and to Parliament, but nothing much come of that fact.
I think this is very much linked to playing a strong ‘inside game’ to access the halls of power and no ‘outside game’ to gain legitimacy for that use of power. It’s also I think due to EA not wanting to use social media to make its case, whereas the e-acc cluster was born and lives on social media.
Where are we now?
I’m not a part of the Bay Area scene and culture,[4] but it seems to me that the AI Safety movement has lost the ‘mandate of heaven’ to whatever extent it did have it. SB-1047 is a push to change policy that has resulted in backlash, and may result in further polarisation and counter-attempts to fight back in a zero-sum political game. I don’t know if it’s constitutional for a Trump/Vance administration to use the Supremacy Clause to void SB-1047 but I don’t doubt that they might try. Biden’s executive order seems certain for the chopping block. I expect a Trump administration to be a lot less sympathetic to the Bay Area/DC AI Safety movements, and the right-wing part of Silicon Valley will be at the very least energised to fight back harder.
One concerning thing for both Silicon Valley and the AI Safety movement is what happens as a result of the ideological consequences of SV accepting this trend. Already a strong fault-line is the extreme social conservatism and incipient nationalism brought about by this. In the recent a16z podcast, Ben Horowitz literally accuses the Biden administration of breaking the rule of law, and says nothing about Trump literally refusing to concede the 2020 election and declaring that there was electoral fraud. Mike Solana seems to think that all risks of democratic backsliding under a Trump administration were/are overblown (or at least that people in the Bay agreeing was preference falsification). On the Moments-of-Zen Podcast (which has also hosted Curtis Yarvin twice), Balaji Srinivasan accused the ‘Blue Tribe’ of ethnically cleansing him out of SF[5] and called on the grey tribe to push all the blues out of SF. e-acc sympathetic people are noting that anti-trans ideas bubbling up in the new movement. You cannot seriously engage with ideas and shape them without those ideas changing you.[6] This right-wing shift will have further consequences, especially under a second Trump presidency.
What next for the AI Safety field?
I think this is a bad sign for the field of AI Safety. Political polarisation has escaped AI for a while. Current polls may lean in support , but polls and political support are fickle, especially in the age of hyper-polarisation.[7] I feel like my fears around the perception of Open Philanthropy are re-occuring here but for the AI Safety movement at large.
I think the consistent defeats to the e-acc school and the fact that the tech sector as a whole seems very much unconvinced by the arguments for AI Safety should at some point lead to a reflection from the movement. Where you stand on this very much depends on your object-level beliefs. While this is a lot of e-acc discourse around transhumanism, replacing humanity, and the AI eschaton, I don’t really buy it. I think that they don’t think ASI is possible soon, and thus all arguments for AI Safety are bunk. Now, while the tech sector as a whole might not be as hostile, they don’t seem at all convinced of the ‘ASI-soon’ idea.
A key point I want to emphasise is that one cannot expect to wield power successfully without also having legitimacy.[8] And to the extent that the AI Safety movement’s strategy is trying to thread this needle it will fail.
Anyway, long ramble over, and given this was basically a one-shot ramble it will have many inaccuracies and flaws. Nevertheless I hope that it can be directionally useful and lead to productive discussion.
lol, lmao
See here, here, and here. These examples are from Twitter because, for better or for worse, it seems much of SV/tech opinions are formed by Twitter discourse.
Would be very interested to hear the thoughts of people in the Bay on this
And if invited to be I would almost certainly decline,
He literally used the phrase ‘ethnically cleanse’. This is extraordinarily dangerous language in a political context.
A good example in fiction is in Warhammer40K, where Horus originally accepts the power of Chaos to fight against Imperial Tyranny, but ends up turning into their slave.
Due to polarisation, views can dramatically shift on even major topics such as the economy and national security (i know these are messy examples!). Current poll leads for AI regulation should not, in any way, be considered secure
I guess you could also have overwhelming might and force, but even that requires legitimacy. Caesar needed to be seen as legitimate by Marc Anthony, Alexander didn’t have the legitimacy to get his army to cross the Hyphasis etc.
Nit: Beff Jezos was doxxed and repeating him name seems uncool, even if you don’t like him.
I think this case it’s ok (but happy to change my mind) - afaict he owns the connection now and the two names are a bit like separate personas. He’s gone on podcasts under his true name, for instance.
Ok thanks, I didn’t know that.
Folding in Responses here
@thoth hermes (or https://x.com/thoth_iv if someone can get it to them if you’re Twitter friends then pls go ahead.[1] I’m responding to this thread here—I am not saying “that EA is losing the memetic war because of its high epistemic standards”, in fact quite the opposite r.e. AI Safety, and maybe because of misunderstanding of how politics work/not caring about the social perception of the movement. My reply to Iyngkarran below fleshes it out a bit more, but if there’s a way for you to get in touch directly, I’d love to clarify what I think, and also hear your thoughts more. But I think I was trying to come from a similar place that Richard Ngo is, and many of his comments on the LessWrong thread here very much chime with my own point-of-view. What I am trying to push for is the AI Safety movement reflecting on losing ground memetically and then asking ‘why is that? what are we getting wrong?’ rather than doubling down into lowest-common denominator communication. I think we actually agree here? Maybe I didn’t make that clear enough in my OP though.
@Iyngkarran Kumar—Thanks for sharing your thoughts, but I must say that I disagree with it. I don’t think that the epistemic standards are working against us by being too polite, quite the opposite. I think the epistemic standards in AI Safety have been too low relative to the attempts to wield power. If you are potentialy going to criminalise existing Open-Source models,[2] you better bring the epistemic goods. And for many people in the AI Safety field, the goods have not been brought (which is why I see people like Jeremy Howard, Sara Hooker, Rohit Krishnan etc get increasingly frustrated by the AI Safety field). This is on the field of AI Safety imo for not being more persuasive. If the AI Safety field was right, the arguments would have been more convincing. I think, while it’s good for Eliezer to say what he thinks accurately, the ‘bomb the datacenters’[3] piece has probably been harmful for AI Safety’s cause, and things like it a very liable to turn people away from supporting AI Safety. I also don’t think it’s good to say that it’s a claim of ‘what we believe’, as I don’t really agree with Eliezer on much.
(r.e. inside vs outside game, see this post from Holly Elmore)
@anormative/ @David Mathers—Yeah it’s difficult to manage the exact hypothesis here, especially for falsified preferences. I’m pretty sure SV is ‘liberal’ overall, but I wouldn’t be surprised if Trump % is greater than 16 and 20, and it definitely seems to be a lot more open this time, e.g. a16z and Musk openly endorsing Trump, Sequoia Capital partners claiming that Biden dropping out was worse than the Jan 6th riot. Things seem very different this time around, different enough to be paid attention to.
- - - - - - - - - - - -
Once again, if you disagree, I’d love to actually here why. Up/down voting is a crude feedback to, and discussion of ideas leads to much quicker sharing of knowledge. If you want to respond but don’t want to publicly, then by all means please send a DM :)
I don’t have Twitter and think it’d be harmful for my epistemic & mental health if I did get an account and become immersed in ‘The Discourse’
This piece from @1a3orn is excellent and to absence of evidence of good arguments against it is evidence of the absence of said arguments. (tl;dr—AI Safety people, engage with 1a3orn more!)
I know that’s not what it literally says but it’s what people know it as
I think you’re reading into twitter way too much.
I disagree FWIW. I think that the political activation of Silicon Valley is the sort of thing which could reshape american politics, and that twitter is a leading indicator.
I don’t disagree with this statement, but also think the original comment is reading into twitter way too much.
There are many (edit: 2) comments responding and offering to talk. 1a3orn doesn’t appear to have replied to any of these comments. (To be clear, I’m not saying they’re under any obligation here, just that there isn’t a absence of attempted engagement and thus you shouldn’t update in the direction you seem to be updating here.)
a) r.e. Twitter, almost tautologically true I’m sure. I think it is a bit of signal though, just very noisy. And one of the few ways for non-Bay people such as myself to try to get a sense of the pulse of the Bay, though obviously very prone to error, and perhaps not worth doing at all.
b) I haven’t seen those comments,[1] could you point me to them or where they happened? I know there was a bunch of discussion around their concerns about the Biorisk paper, but I’m particularly concerned with the “Many AI Safety Orgs Have Tried to Criminalize Currently-Existing Open-Source AI” article—which I haven’t seen good pushback to. Again, welcome to being wrong on this.
Ok, I’ve seen Ladish and Kokotajlo offer to talk which is good, would have like 1a3orn to take them up on that offer for sure.
Scroll down to see comments.
Strongly agree. I think the TESCREAL/e-acc movements badly mischaracterise the EA community with extremely poor, unsubstantiated arguments, but there doesn’t seem to be much response to this from the EA side.
What does this refer to? I’m not familiar.
Other thoughts on this:
Publicly, the quietness from the EA side in response to TESCREAL/e-acc/etc. allegations is harming the community’s image and what it stands for. But ‘winning’ the memetic war is important. If not, then the world outside EA—which has many smart, influential people—ends up seeing the community as a doomer cult (in the case of AI safety) or assigns some equally damaging label that lets them quickly dismiss many of the arguments being made.
I think this is a case where the the epistemic standards of the EA community work against it. Rigorous analysis, expressing second/third-order considerations, etc. are seen as the norm for most writing on the forum. However, in places such as Twitter, these sorts of analyses aren’t ‘memetically fit’ [1].
So, I think we’re in need of more pieces like the Time essay on Pausing AI—a no-punches-pulled sort of piece that gets across the seriousness of what we’re claiming. I’d like to see more Twitter threads and op-ed’s that dismantle claims like “advancements in AI have solved it’s black-box nature”, ones that don’t let clearly false claims like this see the light of day in serious public discourse.
Don’t get me wrong—epistemically rigorous work is great. But when responding to TESCREAL/e-acc ‘critiques’ that continuously hit below the belt, other tactics may be better.
I’ve often found it hard to tell whether an ideology/movement/view has just found a few advocates among a group, or whether it has totally permeated that group. For example, I’m not sure that Srinivasan’s politics have really changed recently or that it would be fair to generalize from his beliefs to all of the valley. How much of this is actually Silicon Valley’s political center shifting to e/acc and the right, as opposed to people just having the usual distribution of political beliefs (in addition to a valley-unspecific decline of the EA brand)?
A NYT article I read a couple of days ago claimed Silicon Valley remains liberal overall.
What is it for EA to thrive?
EA Infrastructure Fund’s Plan to Focus on Principles-First EA includes a proposal:
And a rationale (there’s more detail in the post):
Here I’m spending some time thinking about this, in particular:
What does it mean for EA to thrive?
What projects could push EA in the direction of thriving?
(I work at EAIF. These are my personal views/ thoughts. I’m not speaking on behalf of EAIF here)
EA Jobs, Scarcity and Performance
It seems like:
For many people, having an EA job is pretty important.
It’s pretty competitive and many people who want EA jobs will not in fact get them.
There’s been some discussion related to this on the EA Forum, focusing in particular on jobseekers. I’m also interested in exploring this dynamic with people who are working in EA jobs.
I expect EA job scarcity not only have an impact on EA jobseekers, but also people who are working in EA jobs.
Given 1 and 2, it seems like for people working in EA jobs it will be pretty important for them to keep their jobs. If the job market is competitive it may not be obvious that they can get another one. (For people who have got one EA job, it will presumably be easier to get another, but maybe not guaranteed).
For someone who’s in a position of scarcity about their EA job, I can imagine this meaning they focus primarily on performing well/ being seen to perform well.
This becomes a problem if what counts as performing well and what is actually good to do comes into conflict. Eg. this might involve things like:
Agreeing with the organisational strategy or one’s manager more than one endorses
Focusing on ensuring that they have achieved certain outputs independent of whether that output seems good
In general I expect that under conditions of scarcity people will be less able to do valuable work (and I mean valuable here as ‘actually good’ as opposed to ’work that is perceived to be valuable).
(If I’m right about this, then one potential answer to ‘what is it for EA to thrive’, is: EAs aren’t in a position of scarcity).
Things I’d be interested to ask people who are working at EA jobs to understand whether this is in fact a thing:
How concerned are you about your perceived performance?
If your employer/ manager/ funder/ relevant people said something like: ‘We have full confidence in you, your job is guaranteed and we want you to focus on whatever you think is best’ - would that change what you focus on? How much?
My personal impression is that significant increases in unrestricted funding (even if it were a 1-1 replacement for restricted funding) would dramatically change orgs and individual prioritisations in many cases.
To the extent that one thinks that researchers are better placed to identify high value research questions (which, to be clear, one may not in many cases), this seems bad.
Some EA psychological phenomena
Some things that people report in EA:
Impostor Syndrome
Impact obsession
Burnout
EA Disillusionment
Are these EA phenomena? Also, are they psychological phenomena?
These things (I guess excluding EA disillusionment), don’t just exist within EA they exist within society in general, so it’s plausibly unfair to call them EA phenomena. Though it also seems to me that for each of these things, there’s somewhat strong fit with EA, and EA culture.
Taking impostor syndrome as an example: EA often particularly values ambitious and talented people. Also, it seems to me there’s something of a culture of assessing and prioritising people on this basis. Insofar as it’s important for people to be successful within EA, it’s also important for people to be seen in a certain way by others (talented, ambitious etc.). In general, the stronger the pressure there is for people to be perceived in a certain way, the more prominent I expect impostor syndrome to be.
(I’m a bit wary of ‘just so’ stories here, but my best guess is that this in fact explanatory).
I think impostor syndrome and other things in this ballpark are often discussed as an individual/ psychological phenomena. I think such framings are pretty useful. And there’s another framing which is seeing it instead as a ~sociological phenomena—these are things which happen in a social context, as a result of different social pressures and incentives within the environment.
I don’t know quite what to conclude here, in a large part because I don’t know how common these things are within EA, and how this compares to other places (or even what the relevant comparison class is). Though tentatively, if I’m asking ‘What does it look like for EA to thrive?’, then part of my answer is ‘being an environment where impostor syndrome, burnout, impact obsession and EA disillusionment are less common’.
What’s going on with ‘EA Adjacents’?
There’s a thing where lots of people will say that they are EA Adjacent rather than EA (funny post related to this). In particular, it seems to me that the closer to the core people are, the less inclined they are to identify themselves with EA. What’s going on here? I don’t know, but it’s an interesting trailhead to me.
Plausibly there are some aspects of EA, the culture, norms, worldview, individuals, organisations etc. that people disagree with or don’t endorse, and so prefer to not identify as EAs.
I’m unsure how much to treat this as reflective of a substantive issue vs. a quirk, or reflective of things being actually fine. At least in terms of EA being a ‘beacon for thoughtful, sincere, and selfless’, it seems a little bit worrying to me that some of the core members of the community aren’t willing to describe themselves as EA.
Perhaps a way of getting to the heart of this is asking people something like: Imagine you’re talking to someone who is thoughtful, sincere and selfless. Would you recommend EA to them? Which parts? How strongly? Would you express any reservations?
Looping back to the question of ‘What is it for EA to thrive?’, one answer is: It’s the kind of community that EA’s would strongly recommend to a thoughtful, sincere and selfless friend.
(Maybe this is too strong—people will probably reasonably have disagreements about what aspects of EA are good and aren’t, and if everyone is very positive on EA in this way, this plausibly means that there’s not enough disagreement in the community. )
I share this impression. Also, we see that satisfaction is lower among people who have been in EA longer compared to newer EAs (though this is not true for self-reported engagement), which seems potentially related. Note that we would expect to see pressure in the opposite direction due to less satisfied people dropping out over time.
I think this implies that there is a substantive non-quirky effect. That said, I imagine some of this may be explained by new EAs simply being particularly enthusiastic in ways which explain stronger identification with EA and higher satisfaction.[1]
One dynamic which I expect explains this is the narcissism of small differences, as people become closer to EA, differences and disagreements become more salient, and so people may become more inclined to want to distance themselves from EA as a whole.
I’m not suggesting any particular causal theory about the relationship between satisfaction and identification.
If being thoughtful, sincere and selfless is a core value, it seems like it would be more of a problem if influential people in the community felt they had to embrace the label even if they didn’t think it was valuable or accurate
I suspect a lot of the ‘EA adjacent’ description comes from question marks about particular characteristics EA stances or image rather than doubting some of their friends could benefit from participating in the community, and that part of that is less a rejection of EA altogether and more an acknowledgement they often find themselves at least as closely aligned with people doing great work outside the community.
(Fwiw I technically fit into the “adjacent” bracket from the other side: never been significantly active in the community, like some of its ideas and values—many of which I believed in before ‘EA’ was a thing—and don’t identify with or disagree with other ideas commonly associated with EA, so it wouldn’t really make much sense to call myself an EA)
Incentives within EA
Here’s a story you could tell about academia. Academia, is in some sense supposed to be about generating knowledge. But it ends up being ineffective at doing this because of something something incentives. Eg.
Academic jobs are highly competitive
In order to get an academic job, it’s more important to have done things like original research than things like replications.
Things like replications are undersupplied, and the replication crisis happens.
What are the incentives within EA? How does this affect how well EA ends up ‘doing the most good?’. I don’t have a full theory here, though I also suspect that there are ways in which incentives in EA can push against doing the most good. Professional EA group funding is one example:
Professional EA group organisers are often in a bit of a precarious position. Their job depends on their ability to get funding from organisations like CEA or EAIF.
One of the main ways that EA group organisers are assessed is on the basis of things like how well they produce highly engaged EAs, or career plan changes or other such things (I think this is broadly true, though I don’t have a great insight into how CEA assesses groups).
Professional EA group organisers are incentivised to produce these kinds of things. Some potential problems here: It’s hard to assess what counts as a good eg. career, which pushes in the direction of non-standard career options being discounted, often it may make sense for someone to focus on building career capital over working at an EA organisation, but these kinds of things are less obviously/ legibly impactful…
I agree with the general gist, but my impression is that organisations that focus on career changes and grantmakers have high epistemic humility. When looking at meta organisations focussing on career change, most seem not to break down the changes into types in their quantitative analysis. This leads to a greater focus on case studies where different aspects like prior achievements and unusual career paths can be explained. I assume there is some signalling going on between grantmakers and group organisers where a low-fidelity version might point to standard options, whereas thoughtful grantmakers showcasing a wider variety of pathways as potentially impactful can make a difference.
What is the EA project? Also, who is it for?
There’s a comment by Saulius on an old EA Forum post: ‘[...] I see EA as something that is mostly useful when you are deciding how you want to do good. After you figured it out, there is little reason to continue engaging with it [...]’.
I found this pretty interesting, and it inspired a bunch of thoughts. Here’s a couple of oversimplified models of what it is to pursue the EA project:
Model 1:
EA involves figuring out how to do the most good and doing it.
Figuring out how to do the most good involves working out what cause area is most important and what career path to pursue.
Doing it involves getting such a job and executing well on it.
Model 2:
EA involves figuring out how to do the most good and doing it.
This is less of a two step process, and more of an ongoing cultivation of virtues and attitudes like scout mindset, effectiveness and so on. It involves constant vigilance, or constant attunement. It’s an ongoing process of development, personal, epistemic, moral etc.
I see most EA community building efforts as mostly framed by model 1. For example EA Groups, the EA Forum (perhaps EAG/ EAGx as well, though this is less clear). It seems to me a common pattern for people to engage in these things heavily when getting involved in EA, and then when people are ‘in’ they stop engaging with them, and focus on executing at their job.
Insofar as engaging with these things (EA groups, EA Forum etc.) is a key component of what it is to engage with EA, I’m inclined to agree with the above comment—once you’ve figured out what to do there’s little reason to continue engaging with EA.
I’d like to see more community building efforts, or EA Infrastructure that’s framed around model 2 - things that give EA’s who are already ‘in’ a reason to continue engaging with EA, things that provide them with value in pursuing this project.
I don’t think model 1 and 2 necessarily have to come into conflict. Or at least, I think it’s fine and good for there to be people that see EA as mostly being relevant to a career decision process. And, for people that want to treat the EA project as more like model 2 (an ongoing process of cultivating virtues like scout mindset, effectiveness and so on, I’d be excited to see more community building, or infrastructure which is designed to support them in these aims.
I like this framing and agree that most CB effort seems to go into model 1, which I also spend most of my time working on. Model 2 efforts could help people choose career paths where they upskill or do direct work in organisations that are not EA-aligned. This could reduce the frustration connected with job searches of early career individuals.
With open-source models being released and on ramps to downstream innovation lowering, the safety challenges may not be a single threshold but rather an ongoing, iterative cat-and-mouse game.
Just underscores the importance of people in the policy/safety field thinking far ahead
Reading and engaging with the forum as good for a meta reason, engaging and encouraging other people to keep making posts because engagement seems to exist and they’re incentivized to post. Or even more junior people to try and contribute, idk what the ea forum felt like ~10 years ago, but probably lower standards for engagement.
Hey everyone, in collaboration with Apart Research, I’m helping organize a hackathon this weekend to build tools for accelerating alignment research. This hackathon is very much related to my effort in building an “Alignment Research Assistant.”
Here’s the announcement post:
2 days until we revolutionize AI alignment research at the Research Augmentation Hackathon!
As AI safety researchers, we pour countless hours into crucial work. It’s time we built tools to accelerate our efforts! Join us in creating AI assistants that could supercharge the very research we’re passionate about.
Date: July 26th to 28th, online and in-person
Prizes: $2,000 in prizes
Why join?
* Build tools that matter for the future of AI
* Learn from top minds in AI alignment
* Boost your skills and portfolio
We’ve got a Hackbook with an exciting project to work on waiting for you! No advanced AI knowledge required—just bring your creativity!
Register now: Sign up on the website here, and don’t miss this chance to shape the future of AI research!
Mental health org in India that follows the paraprofessional model
https://reasonstobecheerful.world/maanasi-mental-health-care-women/
#mental-health-cause-area
Meta has just released Llama 3.1 405B. It’s open-source and in many benchmarks it beats GPT-4o and Claude 3.5 Sonnet:
Zuck’s letter “Open Source AI Is the Path Forward”.
Wait, all the LLMs get 90+ on ARC? I thought LLMs were supposed to do badly on ARC.
It’s an unfortunate naming clash, there are different ARC Challenges:
ARC-AGI (Chollet et al) - https://github.com/fchollet/ARC-AGI
ARC (AI2 Reasoning Challenge) - https://allenai.org/data/arc
These benchmarks are reporting the second of the two.
LLMs (at least without scaffolding) still do badly on ARC, and I’d wager Llama 405B still doesn’t do well on the ARC-AGI challenge, and it’s telling that all the big labs release the 95%+ number they get on AI2-ARC, and not whatever default result they get with ARC-AGI...
(Or in general, reporting benchmarks where they can go OMG SOTA!!!! and not helpfully advance the general understanding of what models can do and how far they generalise. Basically, traditional benchmark cards should be seen as the AI equivalent of “IN MICE”)
Thanks!
I want to say thanks to people involved in the EA endeavour. I know things can be tough at times, but you didn’t have to care about this stuff, but you do. Thank you, it means a lot to me. Let’s make the world better!
A lot of people have said sharing these notes were helpful, so sharing it here on the EAF! Here are notes on NTI | bio’s recent event with Dr. Lu Borio on H5N1 Bird Flu, in case anyone here would find it helpful!
‘Five Years After AGI’ Focus Week happening over at Metaculus.
Inspired in part by the EA Forum’s recent debate week, Metaculus is running a “focus week” this week, aimed at trying to make intellectual progress on the issue of “What will the world look like five years after AGI (assuming that humans are not extinct)[1]?”
Leaders of AGI companies, while vocal about some things they anticipate in a post-AGI world (for example, bullishness in AGI making scientific advances), seem deliberately vague about other aspects. For example, power (will AGI companies have a lot of it? all of it?), whether some of the scientific advances might backfire (e.g., a vulnerable world scenario or a race-to-the-bottom digital minds takeoff), and how exactly AGI will be used for “the benefit of all.”
Forecasting questions for the week range from “Percentage living in poverty?” to “Nuclear deterrence undermined?” to “‘Long reflection’ underway?”
Those interested: head over here. You can participate by:
Forecasting
Commenting
Comments are especially valuable on long-term questions, because the forecasting community has less of a track record at these time scales.[2][3]
Writing questions
There may well be some gaps in the admin-created question set.[4] We welcome question contributions from users.
The focus week will likely be followed by an essay contest, since a large part of the value in this initiative, we believe, lies in generating concrete stories for how the future might play out (and for what the inflection points might be). More details to come.
This is not to say that we firmly believe extinction won’t happen. I personally put p(doom) at around 60%. At the same time, however, as I have previouslywritten, I believe that more important trajectory changes lie ahead if humanity does manage to avoid extinction, and that it is worth planning for these things now.
Moreover, I personally take Nuño Sempere’s “Hurdles of using forecasting as a tool for making sense of AI progress” piece seriously, especially the “Excellent forecasters and Superforecasters™ have an imperfect fit for long-term questions” part.
With short-term questions on things like geopolitics, I think one should just basically defer to the Community Prediction. Conversely, with certain long-term questions I believe it’s important to interrogate how forecasters are reasoning about the issue at hand before assigning their predictions too much weight. Forecasters can help themselves by writing comments that explain their reasoning.
In addition, stakeholders we work with, who look at our questions with a view to informing their grantmaking, policymaking, etc., frequently say that they would find more comments valuable in helping bring context to the Community Prediction.
All blame on me, if so.
I just preordered Yuval Noah Harari’s new book
I am very concerned about the future of US democracy and rule of law and its intersection with US dominance in AI. On my Manifold question, forecasters (n=100) estimate a 37% that the US will no longer be a liberal democracy by the start of 2029 [edit: as defined by V-DEM political scientists].
Project 2025 is an authoritarian playbook, including steps like 50,000 political appointees (there are ~4,000 appointable positions, of which ~1,000 change in a normal presidency). Trump’s chances of winning are significantly above 50%, and even if he loses, Republicans get another chance every 4 years.
The presidential immunity ruling by SCOTUS is very far-reaching and further empowers an autocratic president. (From a Western European standpoint it’s already insane that judges are generally appointed by a single party.)
I do not know what this means for effective altruism. With respect to global affairs, Republicans are very divided on foreign policy, with Trump being far more isolationist that the majority of Republican legislators, so anything could happen there.
However, a Republican government being in charge when AGI is close to being achieved (I don’t know if we’re close) seems very dangerous, and I’m surprised that Ashenbrenner pushed for nationalisation given this risk, and that no one (?) made this criticism. If the alignment problem got solved, MAGA controlling AGI would very likely be very bad, while I think Democrat leadership is more likely (but by no means guaranteed) to actually attempt to do the right thing on impartial grounds.
I don’t know if it’s tractable to work on this. In the medium term, it might be worthwhile to introduce/support introducing ballot initiatives to ban gerrymandering. This worked well in Michigan in 2018. However, I have no idea about the details of tractability (how many people are working on this, how many states have this possibility).
I don’t really understand why so many people are downvoting this. If anyone would like to explain, that’d be nice!
Your question is about V-Dem’s ratings for the US, but these have a lot of problems, so I think don’t shine nearly as much light on the underlying reality as you suggest. Your Manifold question isn’t really about the US being a liberal democracy, it’s about V-Dem’s evaluation. The methodology is not particularly scientific—it’s basically just what some random academics say. In particular, this leaves it quite vulnerable to the biases of academic political scientists.
For example, if we look at the US:
A couple of things jump out at me here:
The election of Trump is a huge outlier, giving a major reduction to ‘Liberal Democracy’.
This impact occurs immediately despite the fact that he didn’t actually pass that many laws or make that many changes in 2017; I think this is more about perception than reality.
The impact appears to be larger than the abolition of slavery, the passage of the civil rights act, the second world war, conscription or female suffrage. This seems very implausible to me.
Freedom of domestic movement increased in 2020, despite the introduction of essentially unprecedented covid-related restrictions. (Other countries with even more draconian rules, like the UK, also do not see major declines here, even though the entire population was essentially under house arrest for much of the year)
The subnational civil liberties unevenness index does not seem to reflect the fact that covid restrictions were very different in rural and urban areas.
On the whole I think these ratings often tell us more about the political views of the authors (pro-lockdown, anti-trump) than they do about the actual levels of liberty or democracy in a country.
V-Dem indicators seem to take into account statements made by powerful politicians, not only their policies or other deeds. For example, I found this in one of their annual reports:
My guess is that statements made by Trump were extreme outliers in how they betrayed little respect to democratic institutions, compared to statements made by earlier US presidents, and that affected their model.
I think that’s reasonable. It might not be fully reflective of lived reality for US citizens at the moment the statements are made, but it sure captures the beliefs and motives of powerful people, which is predictive of their future actions.
Indeed, one way to see the drop in 2017 is that it was able to predict a major blow to American democracy (Trump refusing to concede an election) 4 years in advance.
I’m not really sure this contradicts what I said very much. I agree the V-Dem evaluators were reacting to Trump’s comments, and this made them reduce their rating for America. I think they will react to Trump’s comments again in the future, and this will again make them likely reduce their rating for America. This will happen regardless of whether policy changes, and be poorly calibrated for actual importance—contra V-Dem, Trump getting elected was less important than the abolition of slavery. Since I think Siebe was interested in policy changes rather than commentary, this means V-Dem is a bad metric for him to look at.
Thank you, these are some good points. When I made the question, I believed V-DEM had a more rigorous methodology, and I can’t change it now.
I don’t think the specific probability is necessary for my argument (and it depends on how one defines ‘liberal democracy’): a Trump presidency with an enabling Supreme Court would be very harmful to US liberal democracy and the rule of law, and a nationalized AGI project under such a government would be very risky.
P.P.S. I am also concerned about silencing/chilling effects: if you want to get anything political done in the next few years, it’s probably strategic to refrain from criticizing Trump & his allies anywhere publicly, including the EA Forum.
P.S. I don’t think the Forum norm of non-partisanship should apply in its strong form in the case of the US. The Republican party has clearly become an anti-democratic, anti-rule of law, and anti-facts party. This has been concluded by many political scientists and legal scholars.
I wanted to figure out where EA community building has been successful. Therefore, I asked Claude to use EAG London 2024 data to assess the relative strength of EA communities across different countries. This quick take is the result.
The report presents an analysis of factors influencing the strength of effective altruism communities across different countries. Using attendance data from EA Global London 2024 as a proxy for community engagement, we employed multiple regression analysis to identify key predictors of EA participation. The model incorporates geographic, economic, linguistic, and social factors, explaining 52.3% of the variance in per capita attendance.
Methodology
Data Source
Per capita attendance at EAG London 2024 for 62 countries
Variables
Geographic proximity (distance from London, UK)
Economic development (GDP per capita)
English language proficiency
Income inequality (Gini coefficient)
Youth population percentage
Social media penetration
Religiosity
Trust in institutions
Philanthropic culture
Analytical Approach
Multiple regression analysis with per capita EAG London 2024 attendance as the dependent variable
Key Findings
Model Performance:
R-squared value: 0.523
The model explains 52.3% of the variance in per capita attendance
Significant Predictors:
GDP per capita (positive relationship)
Distance from London (negative relationship)
Philanthropic culture (positive relationship)
Social media penetration (positive relationship)
English proficiency (positive relationship)
Non-significant Factors:
Gini coefficient
Youth population percentage
Religiosity
Trust in institutions
Top 10 Countries with Strongest EA Communities (based on positive residuals):
Switzerland
Norway
Netherlands
Estonia
Sweden
Singapore
Finland
Czech Republic
Australia
Ireland
Interpretation of Results
Economic Factors: GDP per capita emerges as the strongest predictor of EA engagement, suggesting that economic development plays a crucial role in fostering EA communities.
Geographic Considerations: Distance from London negatively correlates with engagement, which is expected given that the event was held in London. This finding should be interpreted cautiously: a) It may reflect the ease of attending for nearby countries rather than overall EA engagement. b) The strong performance of some distant countries (e.g., Australia, Singapore) suggests that highly engaged EA communities can overcome geographic barriers.
Cultural Factors: A country’s philanthropic culture strongly predicts EA engagement, highlighting the importance of pre-existing charitable inclinations.
Technological Infrastructure: Social media penetration positively correlates with EA engagement, underscoring the role of online connectivity in community building.
Language: English proficiency remains a significant factor, though its impact is less pronounced than economic and cultural variables.
Limitations
Proxy Measure: EAG attendance may not fully capture all forms of EA engagement.
Single Time Point: The analysis is based on one conference in one year, potentially missing temporal variations.
Unmeasured Variables: Other factors not included in the model may influence EA community strength.
Linear Model Assumptions: The analysis assumes linear relationships, potentially missing complex interactions or non-linear effects.
Host Country Bias: The UK’s attendance is likely inflated due to hosting the event, potentially skewing overall results.
I’m curious if you fed Claude the variables or if it fetched them itself? In the latter case, there’s a risk of having the wrong values, isn’t there?
Otherwise, really interesting project. Curious of the insights to take out of this, esp. for me the fact that Switzerland comes up first. Also surprising that Germany’s not on the list, maybe?
Thanks!
I’m surprised that the “top 10” doesn’t include Denmark, Austria, Belgium, and Germany, since they all have more population-adjusted participants than Ireland, are not English-speaking, are more distant from London, and have lower GDP per capita[1]
Are we using different data?
In general, I’m a bit sceptical of these analyses, compared to looking at the countries/cities with the most participants in absolute terms. I also expect Claude to make lots of random mistakes.
But of course, Ireland’s GDP is very artificial
But absolute terms isn’t very useful if we’re trying to spot success stories, right? Or am I misunderstanding something?
But yeah, something seems off about Ireland. The rest of the list feels quite good though. David Moss said they have some per capita estimates in the pipeline, so I’m excited to see what they produce!
Looking for people (probably from US/UK) to do donation swaps with. My local EA group currently allows tax-deductible donations to:
GiveWell—Top Charities Fund
Animal Charity Evaluators—Top Charities Fund
Against Malaria Foundation
Good Food Institute
<One other org that I don’t want to include here>
However, I would like to donate to the following:
GiveWell—All Grants Fund (~$1230)
GiveDirectly (~$820)
The Humane League (~$580)
If anyone is willing to donate these sums and have me donate an equal sum to one of the funds mentioned above—please contact me.
This could be a long slog but I think it could be valuable to identify the top ~100 OS libraries and identify their level of resourcing to avoid future attacks like the XZ attack. In general, I think work on hardening systems is an underrated aspect of defending against future highly capable autonomous AI agents.
Relevant XKCD comic.
To further comment, this seems like it might be an intractable task, as the term “dependency hell” kind of implies. You’d have to scrap likely all of GitHub and calculate what libraries are used most frequently in all projects to get an accurate assessment. Then it’s not clear to me how you’d identify their level of resourcing. Number of contributors? Frequency of commits?
Also, with your example of the XZ attack, it’s not even clear who made the attack. If you suspect it was, say, the NSA, would you want to thwart them if their purpose was to protect American interests? (I’m assuming you’re pro-American) Things like zero-days are frequently used by various state actors, and it’s a morally grey question whether or not those uses are justified.
I also, as a comp sci and programmer, have doubts you’d ever be able to 100% prevent the risk of zero-days or something like the XZ attack from happening in open source code. Given how common zero-days seem to be, I suspect there are many in existing open source work that still haven’t been discovered, and that XZ was just a rare exception where someone was caught.
Yes, hardening these systems might somewhat mitigate the risk, but I wouldn’t know how to evaluate how effective such an intervention would be, or even, how you’d harden them exactly. Even if you identify the at-risk projects, you’d need to do something about them. Would you hire software engineers to shore up the weaker projects? Given the cost of competent SWEs these days, that seems potentially expensive, and could compete for funding with actual AI safety work.
I’d be interested in exploring funding this and the broader question of ensuring funding stability and security robustness for critical OS infrastructure. @Peter Wildeford is this something you guys are considering looking at?
@Peter Wildeford @Matt_Lerner interested in similar. This in-depth analysis’ was a bit strict in my opinion looking at file-level criteria:
https://www.metabase.com/blog/bus-factor
These massive projects were mostly maintained by 1 person last I checked a year ago:
https://github.com/curl/curl/graphs/contributors
https://github.com/vuejs/vue/graphs/contributors
https://github.com/twbs/bootstrap/graphs/contributors
https://github.com/laravel/laravel/graphs/contributors
https://github.com/pallets/flask/graphs/contributors
https://github.com/expressjs/express/graphs/contributors
https://github.com/redis/redis/graphs/contributors
https://github.com/tiangolo/fastapi/graphs/contributors
https://github.com/lodash/lodash/graphs/contributors
https://github.com/psf/requests/graphs/contributors
https://github.com/babel/babel/graphs/contributors
https://github.com/mastodon/mastodon/graphs/contributors (seemingly improved since)
https://github.com/BurntSushi/ripgrep/graphs/contributors
https://github.com/FFmpeg/FFmpeg/graphs/contributors
https://github.com/gorhill/uBlock/graphs/contributors
https://github.com/evanw/esbuild/graphs/contributors
I’d love to be able to maintain more polished current data
not sure if such a study would naturally also be helpful to potential attackers, perhaps even more helpful to attackers than defenders, so might need to be careful about whether / how you disseminate the information
My sense is that 100 is an underestimate for the number of OS libraries as important as that one. But I’m not sure if the correct number is 1k, 10k or 100k.
That said, this is a nice project, if you have a budget it shouldn’t be hard to find one or a few OS enthusiasts to delegate this to.
I’m extremely excited that EAGxIndia 2024 is confirmed for October 19–20 in Bengaluru! The team will post a full forum post with more details in the coming days, but I wanted a quick note to get out immediately so people can begin considering travel plans. You can sign up to be notified about admissions opening, or to express interest in presenting, via the forms linked on the event page:
https://www.effectivealtruism.org/ea-global/events/eagxindia-2024
Hope to see many of you there!!
Do you like SB 1047, the California AI bill? Do you live outside the state of California? If you answered “yes” to both these questions, you can e-mail your state legislators and urge them to adopt a similar bill for your state. I’ve done this and am currently awaiting a response; it really wasn’t that difficult. All it takes is a few links to good news articles or opinions about the bill and a paragraph or two summarizing what it does and why you care about it. You don’t have to be an expert on every provision of the bill, nor do you have to have a group of people backing you. It’s not nothing, but at least for me it was a lot easier than it sounded like it would be. I’ll keep y’all updated on if I get a response.
Both my state senator and my state representative have responded to say that they’ll take a look at it. It’s non-commital, but it still shows how easy it is to contact these people.