I’m curious about what’s the original source of the funding you’re giving out here. According to this Nonlinear received $250k from Future Fund and $600k from Survival and Flourishing Fund. Is the funding being distributed here coming solely from the SFF grant? Does Nonlinear have other funding sources besides Future Fund and SFF?
(I didn’t do any deeper dive than looking at Nonlinear’s website, where I couldn’t find anything about funding sources.)
Aman Patel
Karma: 168
Thanks for writing this—even though I’ve been familiar with AI x-risk for a while, it didn’t really hit me on an emotional level that dying from misaligned AI would happen to me too, and not just “humanity” in the abstract. This post changed that.
Might eventually be useful to have one of these that accounts for biorisk too, although biorisk “timelines” aren’t as straightforward as trying to estimate the date that humanity builds the first AGI.
Thanks for posting your attempt! Yeah, it does seem like you ran into some of those issues in your attempt, and it’s useful information to know that this task is very hard. I guess one lesson here is that we probably won’t be able to build perfect institutions on the first try, even in safety-critical cases like AGI governance.
Just stumbled upon this post—I like the general vein in which you’re thinking. Not sure if you’re aware of it already, but this post by Paul Christiano addresses the “inevitable dangerous technology” argument as it relates to AI alignment.
- “First-principles design is intractable and misses important situation-specific details”—This could easily be true, I don’t have a strong opinion on it, just intutions.
I think this objection is pretty compelling. The specific tools that an institution can use to ensure that a technology is deployed safely will ultimately depend on the nature of that technology itself, its accessibility/difficulty of replication, the political/economic systems it’s integrated into, and the incentives surrounding its deployment. (Not an exhaustive list.)
Usually, any type of regulation or “responsible power-wielding” comes with tradeoffs (to freedom, efficiency, equitability, etc.), and it’ll be hard to assess whether these accepting these tradeoffs is prudent without a specific technology in mind.
That said, I think it can still be a worthwhile exercise to think about how we can build governance practices that are robust to worst-case scenarios for all of the above. I can imagine some useful insights coming out of that kind of exercise!
Thanks, great points (and counterpoints)!
If you are a community builder (especially one with a lot of social status), be loudly transparent with what you are building your corner of the movement into and what tradeoffs you are/aren’t willing to make.
I like this suggestion—what do you imagine this transparency looks like? Do you think, e.g., EA groups should have pages outlining their community-building philosophies on their websites? Should university groups should write public Forum posts about their plans and reasoning before every semester/quarter or academic year? Would you advocate for more community-building roundtables at EAGs? (These are just a few possible example modalities of transparency that just came to my head, very interested in hearing more.)
Yeah, I’ve had several (non-exchange) students ask me what altruism means—my go-to answer is “selflessly helping others,” which I hope makes it clear that it describes a practice rather than a dogma.
Thanks for the comment! I agree with your points—there are definitely elements of EA, whether they’re core to EA or just cultural norms within the community, that bear stronger resemblances to cult characteristics.
My main point in this post was to explore why someone who hasn’t interacted with EA before (and might not be aware of most of the things you mentioned) might still get a cult impression. I didn’t mean to claim that the Google search results for “altruism” are the most common reason why people come away with a cult impression. Rather, I think that they might explain a few perplexing cases of cult impressions that occur before people become more familiar with EA. I should have made this distinction clearer, thanks for pointing it out :)
Hey Jordan! Great to see another USC person here. The best writing advice I’ve gotten (that I have yet to implement) is to identify a theory of change for each potential piece—something to keep in mind!
6 sounds interesting, if you can make a strong case for it. Aligning humans isn’t an easy task (as most parents, employers, governments, and activists know very well), so I’m curious to hear if you have tractable proposals.
7 sounds important given that a decent number of EAs are vegan, and I’m quite surprised I haven’t heard of this before. 15 IQ points is a whole standard deviation, so I’d love to see the evidence for that.
8 might be interesting. I suspect most people are already aware of groupthink, but it could be good to be aware of other relevant phenomena that might not be as widely-known (if there are any).
From what I can tell, 11 proposes a somewhat major reconsideration of how we should approach improving the long-term future. If you have a good argument, I’m always in favor of more people challenging the EA community’s current approach. I’m interested in 21 for the same reason.
(In my experience, the answer to 19 is no, probably because there isn’t a clear, easy-to-calculate metric to use for longtermist projects in the way that GiveWell uses cost-effectiveness estimates.)
Out of all of these, I think you could whip up a draft post for 7 pretty quickly, and I’d be interested to read it!
A visualization of some orgs in the AI Safety Pipeline
Thanks Linch! This list is really helpful. One clarifying question on this point:
Relatedly, what does the learning/exploration value of this project look like?
To the researcher/entrepreneur?
To the institution? (if they’re working in an EA-institutional context)
To the EA or longtermist ecosystem as a whole?
For 1) and 2), I assume you’re referring to the skills gained by the person/institution completing the project, which they could then apply to future projects.
For 3), are you referring to the possibility of “ruling out intervention X as a feasible way to tackle x-risks”? That’s what I’m assuming, but I’m just asking to make sure I understand properly.Thanks again!
[Question] What are some heuristics for longtermist project evaluation?
This thinking has come up in a few separate intro fellowship cohorts I’ve facilitated. Usually, somebody tries to flesh it out by asking whether it’s “more effective” to save one doctor (who could then be expected to save five more lives) or two mechanics (who wouldn’t save any other lives) in trolley-problem scenarios. This discussion often gets muddled, and many people have the impression that “EAs” would think it’s better to save the doctor, even though I doubt that’s a consensus opinion among EAs. I’ve found this to be a surprisingly large snag point that isn’t discussed much in community-building circles.
I think it would be worth it to clarify the difference between intrinsic and instrumental value in career advice/intro fellowships/other first interactions with the EA community, because there are some people who might agree with other EA ideas but find that this argument undermines our basic principles (as well as the claim that you don’t need to be utilitarian to be an EA). Maybe we could extend current messaging about ideological diversity within EA.
That said, I read Objection 4 differently. Many people (especially in cultures that glorify work) tie their sense of self-worth to their jobs. I don’t know how universal this is, but at least in my middle-class American upbringing, there was a strong sense that your career choice and achievement is a large part of your value as a person.
As a result, some people feel personally judged when their intended careers aren’t branded as “effective”. If you equate your career value with your personal value, you won’t feel very good if someone tells you that your career isn’t very valuable, and so you’ll resist that judgment.
I don’t think that this feeling precludes people from being EAs. It takes time to separate yourself from your current or intended career, and Objection 4 strikes me as a knee-jerk defensive reaction. Students planning to work in shipping logistics won’t immediately like the idea that the job they’ve been working hard to prepare for is “ineffective,” but they might come around to it after some deeper reflection.
I could be misreading Objection 4, though. It could also mean something like “shipping logistics is valuable because the world would grind to a halt if nobody worked in shipping logistics,” but then that’s just a variant of Objection 5.
I’m very curious to know more about the sense in which these students gave Objection 4.
Changed “guilt” to “responsibility,” but I’m not sure if that’s much better.
Thanks for the feedback! I think this is probably a failure of the story more than a failure of your understanding—after all, a story that’s hard to understand isn’t fulfilling its purpose very well. Jackson Wagner’s comment below is a good summary of the main points I was intending to get across.
Next time I write, I’ll try to be more clear about the points I’m trying to convey.
“As tagged, this story strikes me as a fable intended to explain one of the mechanisms behind so-called “S-risks”, hellish scenarios that might be a fate worse than the “death” represented by X-risks.”
That’s what I was going for, although I’m aware that I didn’t make this as clear as I should have.
”Of course it’s a little confusing to have the twist with the sentient birds—I think rather than a literal “farmed animal welfare” thing, this is intended to showcase a situation where two different civilizations have very different values.”S
ame thing here. This is what I was trying to get at, but couldn’t think of many other scenarios involving suffering agents where one group of people cares and another doesn’t.”
I don’t really understand why the story is a frame story, or why the main purpose of the ritual is for all the Kunus to feel “collective guilt”… EA is usually trying to steer away from giving the impression that we want everyone to feel guilty all the time.”T
his is really helpful feedback—I didn’t realize that “collective guilt” came across as the point of the story, and I definitely agree that making people feel guilty is counterproductive. I can’t remember why I threw in that phrase (probably because I couldn’t think of anything else), but I’ll change it now.
Totally unrelated point, but I thought the economics of this story were a little wacky.
Yup, definitely more than a “little” wacky :) Maybe using another resource like food or water or land would be better—but then it would have been harder to make the point that each country thought were doing the right thing
.This is a good part of the parable—if S-risks ever occur, the civilizations that commit those galactic war crimes will probably be convinced of their righteousness, and indeed probably won’t even recognize that they are committing a wrong.
This is the central point that I wanted to get across. Whether we’re considering a civilization or an advanced AI, s-risks need not result from intentional malevolence. I’m glad it didn’t get too distorted, but it seems like there are better ways to build a story around this point.
Another side-note: a lot of the ideas behind this story are discussed in the Center on Long-Term Risk’s research agenda. I don’t know whether they would agree with my presentation or conceptualization of those ideas.
Thank you so much for the feedback!
Thanks!
Thanks! I’m glad you enjoyed it. The main reason I wrote this was to practice creative writing—and the Forum contest seemed to be a good place to do that. This is the first time I tried writing short stories—the only other creative writing piece I’ve published anywhere is this one, which I also wrote for the Forum contest: https://forum.effectivealtruism.org/posts/sGTHctACf73gunnk7/creative-writing-contest-the-legend-of-the-goldseeker
I hope that helps!
I recently learned about Training for Good, a Charity Entrepreneurship-incubated project, which seems to address some of these problems. They might be worth checking out.
I think this is a great exercise to think about, especially in light of somewhat-recent discussion on how competitive jobs at EA orgs are. There seems to be plenty of room for more people working on EA projects, and I agree that it’s probably good to fill that opportunity. Some loose thoughts:There seem to be two basic ways of getting skilled people working on EA cause areas:
1. Selectively recruiting people who already have skills.
2. Recruiting promising people who might not yet have needed skills and train them.Individual organizations can choose both options, depending on their level of resources. But if most organizations choose option 1, the EA community might be underutilizing its potential pool of human resources. So we might want the community in general to use option 2, so that everyone who wants to be involved with EA can have a role—even if individual EA organizations still choose option 1. For this to happen, the EA community would probably need a program whereby motivated people can choose a skillset to learn, are taught that skillset, and are matched with a job at the end of the process.
Currently, motivated people who don’t yet possess skills are placed into a jumble of 1-on-1 conversations, 80k advising calls, and fellowship and internship listings. Having those calls and filling out internship and fellowship applications takes a ton of time and mental energy, and might leave people more confused than they were initially. A well-run training program could eliminate many of these inefficiencies and reduce the risk that interested people won’t be able to find a job in EA.
We can roughly rank skill-building methods by the number of people they reach (“scale”), and the depth of training that they provide. In the list below, “high depth” skill development could lead to being hired for that skill (when one would not have been hired for that skill otherwise), “medium depth” as warranting a promotion or increase in seniority level, and “low depth” as an enhancement of knowledge that can help someone perform their job better, but probably won’t lead to new positions or higher status.
Internal development within organizations, like Aaron Gertler mentioned (small scale, medium depth)
Internship/fellowship programs (medium scale, medium depth)
One-off workshops and lectures (small scale, low depth)
Cause area-specific fellowships, like EA Cambridge’s AGI Safety Fellowship (large scale, low depth)
A training program like the one I described above (large scale, high depth)
An EA university, as proposed here (large scale, high depth)
If we choose option 2, we probably want large scale, high depth ways to train people. I’m interested in hearing people’s thoughts on whether this is a good way to evaluate skill-building methods.
One caveat: there’s a lot more interest in working for the military than there is in working for EA orgs. Since this interest already exists, the military just needs to capitalize on it (although they still spend lots of money on recruitment ads and programs like ROTC). The EA community doesn’t even have great name recognition, so it’s probably premature to assume that we’d have waves of people signing up for such a training program—but it’s possible that we could get to that point with time.
Thanks for this post! Reading through these lessons has been really informative. I have a few more questions that I’d love to hear your thinking on:
1) Why did you choose to run the fellowship as a part-time rather than full-time program?2) Are there any particular reasons why fellowship participants tended to pursue non-venture projects?
3) Throughout your efforts, were you optimizing for project success or project volume, or were you instead focused on gathering data on the incubator space?
4) Do you consider the longtermist incubation space to be distinct from the x-risk reduction incubation space?
5) Was there a reason you didn’t have a public online presence, or was it just not a priority?
The hygiene hypothesis (especially the autoimmune disease variant, brief 2-paragraph summary here if you Ctrl+F “Before we go”) could be another example.
On a somewhat related note, Section V of this SlateStarCodex post goes through some similar examples where humans departing from long-lived tradition has negative effects that don’t become visible for a long time.