Thank you!
Worth noting that our input was also very unevenly distributed – our original idea list includes ~40% AI-related ideas, ~15% bio, ~25% movement building / community infrastructure, and only ~20% other. (this was mainly due to us having better access to AI-related project ideas via our networks). If you’re interested in pursuing biosecurity- or movement building-related projects, feel free to get in touch and I can share some of our additional ideas – for the other areas I think we don’t necessarily have great ideas.
Buhl
Karma: 571
Project proposal: Scenario analysis group for AI safety strategy
Thanks, appreciate your comment and the compliment!
On your questions:2. The research process does consider cost-effectiveness as a key factor – e.g., the weighted factor model we used included both an “impact potential” and a “cost” item, so projects were favoured if they had high estimated impact potential and/or a low estimated cost. “Impact potential” here means “impact with really successful (~90th percentile) execution” – we’re focusing on the extreme rather than the average case because we expect most of our expected impact to come from tail outcomes (but have a separate item in the model to account for downside risk). The “cost” score was usually based on a rough proxy, but the “impact potential” score was basically just a guess – so it’s quite different from how CE (presumably) uses cost-effectiveness, in that we don’t make an explicit cost-effectiveness estimate and in that we don’t consult evidence from empirical studies (which typically don’t exist for the kinds of projects we consider).
Re: “For each of the ideas do you feel like you have a sense of why this thing has not happened already?” – we didn’t consider this explicitly in the process (though it somewhat indirectly featured as part of considering tractability and impact potential). I feel like I have a rough sense for each of the projects listed – and we wouldn’t include projects where we didn’t think it was plausible that the project would be feasible, that there’d be a good founder out there etc. – but I could easily be missing important reasons. Definitely an important question – would be curious to hear how CE takes it into account.
3. Appreciate the input! The idea here wouldn’t be to just shove people into government jobs, but also making sure that they have the right context, knowledge, skills and opportunities to have a positive impact once there. I agree that policy is an ecosystem and that people are needed in many kinds of roles. I think it could make sense for an individual project to focus just/primarily on one or a few types of role (analogously to how the Horizon Institute focuses primarily on technocratic staffer and executive branch roles + think tank roles), but am generally in favour of high-quality projects in multiple policy-related areas (including advocacy/lobbying and developing think tank pipelines).
The quick explanation is that I don’t want people to over-anchor on it, given that the inputs are extremely uncertain, and that I think that a ranked list produced by a relatively well-respected research organisation is the kind of thing people could very easily over-anchor on, even if you caveat it heavily
20 concrete projects for reducing existential risk
[Question] What longtermist projects would you like to see implemented?
(I’m in a similar position to Amber: Limited background (technical or otherwise) in AI safety and just trying to make sense of things by discussing them.)
Re: “I think you need to say more about what the system is being trained for (and how we train it for that). Just saying “facts about humans are in the data” doesn’t provide a causal mechanism by which the AI acts in human-like ways, any more than “facts about clouds are in the data” provides a mechanism by which the AI role-plays being a cloud.”
The (main) training process for LLMs is exactly to predict human text, which seems like it could reasonably be described as being trained to impersonate humans. If so, it seems natural to me to think that LLMs will by default acquire goals that are similar to human goals. (So it’s not just that “facts about humans are in the data”, but rather that state-of-the-art models are (in some sense) being trained to act like humans.)
I can see some ways this could go wrong – e.g., maybe “predicting what a human would do” is importantly different from “acting like a human would” in terms of the goals internalised; maybe fine-tuning changes the picture; or maybe we’ll soon move to a different training paradigm where this doesn’t apply. And of course, even if there’s some chance this doesn’t happen (even if it isn’t the default), it warrants concern. But, naively, this argument still feels pretty compelling to me.
Speedrun: Demonstrate the ability to rapidly scale food production in the case of nuclear winter
Speedrun: Develop an affordable super PPE
Scalable longtermist projects: Speedrun series – Introduction
Thank you for the important post!
“we might question how well neuron counts predict overall information-processing capacity”
My naive prediction would be that many other factors predicting information-processing capacity (e.g., number of connections, conduction velocity, and refractory period) are positively correlated with neuron count, such that neuron count is pretty strongly correlated with information processing even if it only plays a minor part in causing more information processing to happen.
You cite one paper (Chitka 2009) that provides some evidence against my prediction (based on skimming the abstract, this seemed to be roughly by arguing that insect brains are not necessarily worse at information processing than vertebrate brains). Curious if you think this is the general trend of the literature on this topic?
Curious what you’re referring to here and if there’s any publicly available information about it? Couldn’t find anything in ALLFEDs 2020 and 2021 updates. (I’m trying to estimate the cost-effectiveness of this kind of project as part of my work at Rethink Priorities)
Another failure mode I couldn’t easily fit into the taxonomy that might warrant a new category:
Competency failures—EAs are just ineffective at achieving things in the world due to lack of skills (eg comms, politics, org running) or bad judgement. Maybe this could be classed as a resource failure (for failing to attract people with certain skills) or a rigor failure (for failing to develop them/learn from others). Will try to think of a title beginning with R…
Minor points:
I was also considering something like value failures (EAs have the wrong moral theories/values), but that could probably be classified as a failure of rigor.
+1 to separating internal strife and reputation risks.
Curious what people think of the argument that, given that people in the EA community have different rankings of the top causes, a close-to-optimal community outcome could be reached if individuals argmax using their own ranking?
(At least assuming that the number of people who rank a certain cause as the top one is proportional to how likely it is to be the top one.)
[Shortform version of this comment here.]
Update: I helped Linch collect data on the undergrad degrees of exceptionally successful people (using some of the ex post metrics Linch mentioned).
Of the 32 Turing Award winners in the last 20 years, 6 attended a top 10 US university, 16 attended another US university, 3 attended Oxbridge, and 7 attended other non-US universities. (full data)
Of the 97 Decacorn company founders I could find education data for, 19 attended a top 10 US university, 32 attended another US university, and 46 attended non-US universities (no Oxbridge). (full data)
So it seems like people who are successful on these metrics are pretty spread out across both US/elsewhere and elite/non-elite unis, but concentrated enough that having considerable focus on top US universities makes sense (assuming a key aim is to target people with the potential to be extremely successful).
The concentration gets a bit higher for PhDs for the Turing Award winners (28% at top 10 US universities). It’s also higher for younger Decacorn company founders (e.g., 50% of under-35s in the US at MIT or Stanford) – so that gives some (relatively weak) evidence that concentration at top US universities has increased in the last few decades.
There’s a doc with more details here for anyone interested.
Tl;dr: Most Turing Award winners and Decacorn company founders (i.e., exceptionally successful people) don’t attend US top universities, but there’s a fair amount of concentration.
In response to the post Most Ivy-smart students aren’t at Ivy-tier schools and as a follow-up to Linch’s comment tallying the educational background of Field Medalists, I collected some data on the undergrad degrees of exceptionally successful people (using some of the (imperfect) ex post metrics suggested by Linch).
Of the 32 Turing Award winners in the last 20 years, 6 attended a top 10 US university, 16 attended another US university, 3 attended Oxbridge, and 7 attended other non-US universities. (full data)
Of the 97 Decacorn company founders I could find education data for, 19 attended a top 10 US university, 32 attended another US university, and 46 attended non-US universities (no Oxbridge). (full data)
So it seems like people who are successful on these metrics are pretty spread out across both US/elsewhere and elite/non-elite unis, but concentrated enough that having considerable focus on top US universities makes sense (assuming a key aim is to target people with the potential to be extremely successful).
The concentration gets a bit higher for PhDs for the Turing Award winners (28% at top 10 US universities). It’s also higher for younger Decacorn company founders (e.g., 50% of under-35s in the US at MIT or Stanford) – so that gives some (relatively weak) evidence that concentration at top US universities has increased in the last few decades.
There’s a doc with more details here for anyone interested.
[Also for full disclosure: I collected this data as part of my job, not just as a fun after hours project.]
Buhl’s Quick takes
Thought-provoking post, thanks a lot for writing it!
I broadly agree that it’s good for community builders to spend significant time on learning/direct work, especially if their long-term plan is not to do community building, but I think I disagree with some of your specific reasons.
I think the post sometimes conflates two senses of marketing. One is “pure” marketing, the other is marketing as you define it (i.e., marketing and ops), which includes things like organising content-heavy events and programs like fellowships. My instinct is that:
A. Most of the negative effects of “too much marketing” that you identify are negative effects of “pure” marketing, rather than marketing-and-operations. I think this is especially true of claim 2 and 4: It doesn’t seem to me like organising a talk or fellowship creates bad epistemics or makes EA comes across as pushy or single-minded. It’s maybe not always the best thing organisers could be doing (e.g., because of claim 1 and 3), but doesn’t seem harmful otherwise.
B. It’s not true that 60% of community builders spend 70-80% of their time on “pure” marketing.
I’m curious if you disagree with either of these claims. (But even if not, I think the central argument could still be true, though for slightly different reasons, e.g., that organisers spend too much time on “pure” marketing, or that spending significant time on learning/direct work makes you a better community builder.)
No worries!
I don’t have strong opinions on a 4-week fellowship, no! I think my quick take would be that (a) it’s harder to teach the core EA ideas well in 4x1.5h sessions, (b) it’s harder to create a social community/have people become friends in 4 weeks, and (c) the group of people who’d commit to a 4-week program but not an 8-week program is relatively small, at least in a university group context. But I’m not too sure about this. It also seems plausible to me that 4 weeks could be better in contexts like professional or city groups.
I’d be excited to see a group running both and comparing the outcomes (e.g., in terms of retention, later engagement, number of friends made, whether participants say they’d like a shorter/longer program).
My outsider impression is that CG is so bottlenecked on grantmakers because of having a relatively high epistemic bar for giving out grants (ie CG grantmakers spend a fair amount of time developing detailed inside views about grants). What’s the case for hiring more grantmakers rather than lowering the rigorousness bar?
I’m guessing the answer might be something like “active grantmaking produces much better grants at the current margin”. My own heuristic (from doing a bit of incubator work) is that it’s usually more effective to found one thing yourself than to get others to found a couple of things, because it’s hard to convey a vision and also hard to find someone talented enough who isn’t already doing something equally good or better. Interested in your takes on that!