Currently doing local AI safety Movement Building in Australia and NZ.
Chris Leong
Karma: 6,974
I guess the issue for arguing for AI tutoring interventions to increase earnings is that it would have to compete against AI tutoring interventions to assist folk working directly on high-priority issues and that comparison is unlikely to come out favourably (though the former has the advantage of being more sellable to traditional funders).
a) The link to your post on defining alignment research is broken
b) “Governing with AI opens up this whole new expanse of different possibilities”—Agreed. This is part of the reason why my current research focus is on wise AI advisors.
c) Regarding malaria vaccines, I suspect it’s because the people who focused on high-quality evidence preferred bet nets and folk who were interested in more speculative interventions were drawn more towards long-termism.
In retrospect, it seems that LLM’s were initially successful because they allowed engineers to produce certain capabilities in a way that almost maximally leaned on crystallized knowledge and minimally leaned on fluid intelligence.
It appears that LLM’s have continued to be successful because we’ve gradually been able to get them to rely more on fluid intelligence.
The AI Safety Fundamentals course has done a good job of building up the AI safety community and you might want to consider running something similar for moral alignment.
One advantage of developing on a broader moral alignment field is that you might be able to produce a course that would still appeal to folks who are skeptical of either the animal rights or AI sentience strands.
I can share a few comments on my thoughts here if this is something you’d consider pursuing.
(I also see possible intersections with my Wise AI advisor research direction).
Sorry to hear it didn’t work out and thank you for your service.
For what it’s worth, often it’s valuable to take a step back rather than to just keep hitting your head against a wall. This can provide space to develop a better sense of perspective and why things went the way they did, whether you might have had a shot if you approached things in a different way or whether something else might be a better fit for you.
One thing I’d be much more excited about seeing rather than “quantifying post-training variables and their effects” (but which I’m not planning to pursue) would be to take an old model and then to try to map post-training enhancements discovered over time and see how the maximum elicitable capabilities change.
I’m worried that quantifying post-training variables directly has significant capabilities externalities and that there’s no obvious limit to how far post-training can be pushed.
Comments:
I wouldn’t really frame it as us wanting more folk to take AI safety courses as I think quality is more important than quantity. Instead, I would personally focus more on a) local knowledge about people’s potential b) building up viable local scenes for people to grow/develop in.
Regarding marketing, I’d be much more interested in niche marketing targeting people who are top of the distribution in some way than just increasing numbers due to decreasing marginal returns.
Your model here seemed to be that you should only run a fellowship program if you already have a strong community, but running a fellowship program is a way to build up a community. It has a strong track record with university groups. Admittedly, it is slightly less suited to the country context, but it can still work either remotely or by running groups in the most major cities.
As an AI safety person who believes short timelines are very possible, I’m extremely glad to see this shift.
For those who are disappointed, I think it’s worth mentioning that I just took a look at the Probably Good website and it seems much better than the last time I looked. I had previously been a bit reluctant to recommend it, but it now seems like a pretty good resource and I’m sure they’ll be able to make it even better with more support.
Given that The 80,000 Hours Podcast is increasing its focus on AI, it’s worth highlighting Asterisk Magazine as a good resource for exploring a broader set of EA-adjacent ideas.
Change in strategic direction?
Feels like Anthropic has been putting out a lot of good papers recently that help build the case for various AI threats. Given this, “no meaningful path to impact” seem a bit strong.
I’ve only just CTL-F’d the report so I could have missed something, but I guess the key question for me is what does a multilateral project mean in terms of security/diffusion of the technology?
My intuition is that preventing diffusion of the tech in a multilateral project would be hard, if not impossible and I see this as consideration as something that could kill the desirability of such a project by itself, even if there are several other strong arguments in favour.
I know you mention this in the potential future work section, but I do think it is worthwhile editing in a paragraph or two on why you think we might want to consider this model anyway (it’s impossible to address everyone’s pet objection, but my guess is that this will prove to be one of the major objections that people make).
I expect that some of the older EA’s are more senior and therefore have more responsibilities competing against attending EA Global.
I have neither upvoted nor downvoted this post.
I suspect that the downvoting is because the post assumes this is a good donation target rather than making the argument for it (even a paragraph or two would likely make a difference). Some folks may feel that it’s bad for the community for posts like this to be at +100, even if they agree with the concrete message, as it undermines the norm of EA forum posts containing high-quality reasoning, as opposed to other appeals.
I think it’s worth bringing in the idea of an “endgame” here, defined as “a state in which existential risk from AI is negligible either indefinitely or for long enough that humanity can carefully plan its future”.
Some waypoints are endgames, some aren’t and some may be treated as an endgame by one strategy, but not by another.
It’s quite unclear that attempts to “boost scientific and technological progress” are net-positive at this point in time. I’d much rather see an effort to shift science towards differential technological development.
It’s very hard to say since it wasn’t tried.
I think incremental progress in this direction still would be better than the comparative.
The section “Most discussions about AGI fall into one of three categories” is rather weak, so I wouldn’t place too much confidence in what the AI says yet.
I agree that the role that capitalism plays in pushing us towards doom is an under-discussed angle.
I personally believe that a wisdom explosion would have made more sense for our society to pursue rather than an intelligence explosion given the constraints of capitalism.
Well, there’s also direct work on AI safety and governance.
These proposals seem pretty good. One area I’m a bit less certain about though is the focus on growth.
I hadn’t really thought very much about the morale implications of growing EA before. These could be strong reasons to aim for growth.
At the same time, I do think it’s worth noting that there’s a certain tension between a principles-first approach and emphasising growth. Firstly, if we’re aiming to find people who strongly align with EA principles, rather than just resonating with one of the cause areas, that significantly narrows the pool. Secondly, it’s easier to built a movement where people have a deep understanding of that movement’s principles when the movement isn’t growing too fast. Thirdly, when a community has a strong commitment to principles, it can often access strategies that are less dependent on the size of the community, than when the community’s commitment to principles are weaker, leading to less value in growth.
I’m not saying that a growth strategy would be a mistake, just noting a deep tension here.
I’ll also note one argument on the growth side: to the extent that EA talent is being pulled into focusing more narrowly on AI safety, EA needs to increase the rate at which it brings in new talent in order to keep the movement healthy/viable. I don’t know how strong this consideration is as I don’t have a deep understanding of how EA is doing outside of Australia (within Australia more growth would be beneficial b/c so much of our talent gets pulled overseas).