I currently lead EA funds.
Before that, I worked on improving epistemics in the EA community at CEA (as a contractor), as a research assistant at the Global Priorities Institute, on community building, and Global Health Policy.
Unless explicitly stated otherwise, opinions are my own, not my employer’s.
You can give me positive and negative feedback here.
My impression is that ARC-AGI (1) is close to being solved, which is why they brought our ARC-AGI-2 a few weeks ago.
Benchmarks are often adversarially selected so they take longer to be saturated, so I don’t think little progress on ARC-AGI-2 a few weeks after release (and iirc after any major model release) tells us much at all.
-Most AI experts are skeptical that scaling up LLMs could lead to AGI.
I don’t think this is true. Do you have a source? My guess is that I wouldn’t consider many of the people “experts”.
-It seems like there are deep, fundamental scientific discoveries and breakthroughs that would need to be made for building AGI to become possible. There is no evidence we’re on the cusp of those happening and it seems like they could easily take many decades.
I think this is a pretty strange take. It seems like basically all progress on AI has involved approximately 0 “deep, fundamental scientific discoveries”, so I think you need some argument for why the trend will change. Alternatively, if you think we have made lots of discoveries and that explains AI progress so far, then you need an argument for why these discoveries will stop. Or, if you think we have made little AI progress since ~2010 then I think most readers would strongly disagree with you.
-If you look at easy benchmarks like ARC-AGI and ARC-AGI-2 that are easy for humans to solve and intentionally designed to be a low bar for AI to clear, the weaknesses of frontier AI models are starkly revealed.
I don’t think they are designed to be a low bar to clear. They seem very adversarially selected, though I agree that LMs do poorly on them relative to subjectively more difficult tasks like coding. It seems pretty hard to make a timelines update from ARC-AGI unless you are very confident in the importance of abstract shape rotation problems for much more concrete problems, or you care about some notion of “intelligence” much more than automating intellectual labour.
I feel quite confused about the case where someone earns much less than their earning potential in another altruistically motivated but less impactful career doing work that uses a similar skillset (e.g. joining a think tank after working on policy at an AI company). This seems somewhere between A and B.
We don’t listen to … Tobacco CEOs on the safety of tobacco
Tbf we might have done if Tobacco CEOs had said that tobacco products were very harmful.
Pragmatically, I think many of the old folks around EA are either doing very well, or are kind of lost/exploring other avenues. Other areas allow people to have more reputable positions, but these are typically not very EA/effective areas. Often E2G isn’t very high-status in these clusters, so I think a lot of these people just stop doing much effective work.
I haven’t really noticed this happening very much empirically, but I do think the effect you are talking about is quite intuitive. Have you seen many cases of this that you’re confident are correct (e.g. they aren’t lost for other reasons like working on non-public projects or being burnt out)? No need to mention specific names.
In theory, EAs are people who try to maximize their expected impact. In practice, EA is a light ideology that typically has a limited impact on people. I think that the EA scene has demonstrated success at getting people to adjust careers (in circumstances where it’s fairly cheap and/or favorable to do so)
This seems incorrect to me, in absolute terms. By the standards of ~any social movement, EAs are very sacrificial and focused on increasing their impact. I suspect you somewhat underrate how rare it is outside of EA to be highly committed to ~any non-self-serving principles seriously enough to sacrifice significant income and change careers, particularly in new institutions/movements.
Yeah all seems plausible. I suspect that lack of a great “seed” for community projects is more predictive—it just happens to be the case that few people have done high effort projects that got product market fit. Maybe this is the rich-get-richer thing you mentioned.
I think this is a great initiative. SF is one of the most important (possibly the most important) places for EA/AIS work, but there aren’t many high-effort community/field-building projects there. There are lots in Berkeley, but travelling from one place to the other happens less than you might naively expect.
Austen and his team are some of the best executors I have met in EA/AIS. I’m really excited to see where this goes!
I don’t know if there is an official answer, but I would be very surprised if the 10% pledge required including your spouse’s income as well.
I think the GWWC team generally (IMO correctly) cares more about people fulfilling the “spirit” of the pledge than splitting hairs over who has and hasn’t fulfilled it in some technical sense. Including your spouse’s income may make sense in some cases, but it probably depends on specifics that you should just make a call on.
I’m really excited about this change in direction. My impression is that 80k staff increasingly have wanted to double down on making AI go well for a while, and I think it’s important that the outward brand/image is aligned with what people in the organisation are most excited about.
My impression is that many commenters who haven’t run or worked at cause-neutral organisations will underestimate the challenges of having an org vision and mission that doesn’t feel coherent and consistent to its employees. One way I expect this to improve 80k as an organisation is that 80k may have an easier time hiring people who care a lot about AI and are deeply knowledgeable on the topic, even if (hypothetically) the case for working at 80k for people who mostly care about AI risk was about as strong as before the official switch.
I really appreciate that 80k leadership are bold enough to focus on what they think is most useful. Pivoting 80k seems much better to me than having most senior people leave 80k to work on a different project and then 80k being a very different org people-wise with the same brand.
(I haven’t been through the many comments on this post—apologies if this is wrong in meaningful ways that have been clarified in other comments)
This has been on the stack to look into for a few weeks. I think we might just take the graphs down until we’re confident they are all accurate. They were broken (and then fixed) after we moved accounting systems, but it looks like they might have broken again.
I agree that a lot of EAs seem to make this mistake but I don’t think the issue is with the neglectedness measure, ime people often incorrectly scope the area they are analysing and fail to notice that that specific area can be highly neglected whilst also being tractable and important even if the wider area it’s part of is not very neglected.
For example, working on information security in USG is imo not very neglected but working on standards for datacentres that train frontier LMs is.
Fwiw I think the “deepfakes will be a huge deal” stuff has been pretty overhyped and the main reason we haven’t seen huge negative impacts is that society already has reasonable defences against fake images that prevent many people from getting misled by them.
I don’t think this applies to many other mouse style risks that the AI X-risk community cares about.
For example the main differences in my view between AI-enabled deepfakes and AI-enabled biorisks are:
* marginal people getting access to bioweapons is just a much bigger deal than marginal people being able to make deepfakes
* there is much less room for the price of deepfakes to decrease than the cost of developing a bioweapon (photoshop has existed for a long time and expertise is relatively cheap).
My point was that I don’t think it was marketing or a historical accident, and it’s actually quite different to the other companies that you named which were all just straightforward revenue generating companies from ~day 1.
People call these companies labs due to some combination of marketing and historical accident. To my knowledge no one ever called Facebook, Amazon, Apple, or Netflix “labs”, despite each of them employing many researchers and pushing a lot of genuine innovation in many fields of technology.
I agree overall but fwiw I think that for the first few years of Open AI and Deepmind’s existence, they were mostly pursuing blue sky research with few obvious nearby commercial applications (e.g. training NNs to play video games). I think a lab was a pretty reasonable term—or at least similarly reasonable to calling say, bell labs a lab.
How is that better than individuals just donating to wherever they think makes sense on the margin? If it’s about reducing the influence of large donors, what is the incentive for large donors to participate?
Fwiw I think that donor lotteries are great and I’m glad they have a place in the effective giving ecosystem. I’m not sure I follow most of your points analysis but I’d push back on
Donor lotteries assume there’s demand for the model....
My understanding is that donor lotteries don’t take up much time or attention from people who aren’t participating in them. They are pretty low-cost to run relative to managed funds and people entering them have a good sense of their chance of winning and the pool they’ll be able to direct if they do win.
Not sure right now, but probably sometime next quarter.
Yes it’s only for the EAIF as of the time of writing this comment.
My impression is that the EAIF is currently getting back to people in much less than 8 weeks, but I haven’t checked in the last few weeks. In any case, that’s a different non-round process to the ltff.
In my view, there are many good reasons to work at an AI company, including:
* productively steering an AI lab during crunch time
* doing well-resourced AI safety research
* increasing the ability for safety-conscious people to blow the whistle to governments
* learning about the AI frontier from the best people in the field
* giving to effective charities
* influencing the views of other employees
* influencing how powerful AI systems are deployed and what they are used for during deployment
I don’t think these necessarily outweigh the costs of working at an AI company, but the altruistic benefits are sometimes large, and it seems good for people to consider the option thoughtfully.