Newer EAs are too junior to have good takes yet. It’s just that the growth rate has increased so there’s a higher proportion of them.
People who have better thoughts get hired at EA orgs and are too busy to post. There is anticorrelation between the amount of time people have to post on EA Forum and the quality of person.
Although we want more object-level discussion, everyone can weigh in on meta/community stuff, whereas they only know about their own cause areas. Therefore community content, especially shallow criticism, gets upvoted more. There could be a similar effect for posts by well-known EA figures.
Contests like the criticism contest decrease average quality, because the type of person who would enter a contest to win money on average has worse takes than the type of person who has genuine deep criticism. There were 232 posts for the criticism contest, and 158 for the Cause Exploration Prizes, which combined is more top-level posts than the entire forum in any month except August 2022.
EA Forum is turning into a place primarily optimized for people to feel welcome and talk about EA, rather than impact.
All of this is exacerbated as the most careful and rational thinkers flee somewhere else, expecting that they won’t get good quality engagement on EA Forum
So napkin math suggests that the per-post cost of a contest post is something like 1% of the per-post cost of a RP publication. A typical RP publication is probably much higher quality. But maybe sometimes getting a lot of shallow explorations quickly is what’s desired. (Disclaimer: I haven’t been reading the forum much, didn’t read many contest posts, and don’t have an opinion about their quality. But I did notice the organizers of the ELK contest were “surprised by the number and quality of submissions”.)
A related point re: quality is that smaller prize pools presumably select for people with lower opportunity costs. If I’m a talented professional who commands a high hourly rate, I might do the expected value math on e.g. the criticism prize and decide it’s not worthwhile to enter.
It’s also not clear if the large number of entries will persist in the longer term. Not winning can be pretty demoralizing. Supposing a talented professional goes against their better judgement and puts a lot of time into their entry, then loses and has no idea why. Will they enter the next contest they see? Probably not. They’re liable to interpret lack of a prize as “the contest organizers didn’t think it was worth my time to make a submission”.
Hey just want to weigh in here that you can’t divide our FTE by our total publication count, since that doesn’t include a large amount of work we’ve produced that is not able to be made public or is not yet public but will be. Right now I think a majority of our output is not public right now for one reason or another, though we’re working on finding routes to make more of it public.
I do think your general point though that the per-post cost of a contest post is less / much less than an RP post is accurate though.
BTW, I hope it doesn’t seem like it was picking on you—it just occurred to me that I could do math for Rethink Priorities because your salaries are public. I have no reason to believe a cost-per-public-report estimate would be different for any other randomly chosen EA research organization in either direction. And of course most EA organizations correctly focus on making a positive impact rather than maximizing publication count.
We also seem to get a fair number of posts that make basically the same point as an earlier article, but the author presumably either didn’t read the earlier one or wanted to re-iterate it.
I think there are many people who have very high bars for how good something should be to post on the forum. Thus the forum becomes dominated by a few people (often people who aren’t aware of or care about forum norms) who have much lower bars to posting.
This is a plausible mechanism for explaining why content is of lower quality than one would otherwise expect, but it doesn’t explain differences in quality over time (and specifically quality decline), unless you add extra assumptions such that the proportion of people with low bars to posting has increased recently. (Cf. Ryan’s comment)
often people who aren’t aware of or care about forum norms
EA has grown a lot recently, so I think there are more people recently who aren’t aware of or care about the “high bar” norm. This is in part due to others explicitly saying the bar should be lower, which (as others here have noted) has a stronger effect on some than on others.
Edit: I don’t have time to do this right now, but I would be interested to plot the proportion of posts on the EA forum from people who have been on the forum for less than a year over time. I suspect that it would be trending upwards (but could be wrong). This would be a way to empirically verify part of my claim.
I’m interested in learning how plausible people find each of these mechanisms, so I created a short (anonymous) survey. I’ll release the results in a few days [ETA: see below]. Estimated completion time is ~90 seconds.
Re 3, ‘There is anticorrelation between the amount of time people have to post on EA Forum and the quality of person.’ - this makes me wince. A language point is that I think talking about how ‘good quality’ people are overall is unkind and leads to people feeling bad about themselves for not having such-and-such an attribute. An object level point is I don’t think there is an anticorrelation—I think being a busy EA org person does make it more likely that they’ll have valuable takes, but not being a busy-EA-org-person doesn’t make it less likely—there aren’t that many busy-EA-org-person jobs, and some people aren’t a good fit for busy jobs (eg because of their health or family commitments) but they still have interesting ideas.
Re 7: I’m literally working on a post with someone about how lots of people feel too intimidated to post on the Forum because of its perceived high standards! So I think though the Forum team are trying to make people feel welcome, it’s not true that it’s (yet) optimized for this, imo.
There’s a kind of general problem whereby any messaging or mechanism that’s designed to dissuade people from posting low-quality things will (a) just not work on some people—some people just have a lot of confidence in their not-very-good opinions, shrug, and (b) dissuade people who would post high-quality things, but who have impostor syndrome or are perfectionist or over self-critical. I think the number of people that the mechanism works as intended on—ie people who would have posted a low quality post but are now dissuaded from it—is probably pretty low. Since there are lots of people in EA with impostor syndrome/perfectionism/over-scrupulosity, I’m pretty in favour of the Forum having a ‘welcoming’ vibe over a We Are Very Serious and Important vibe.… because I’d rather have more good takes and more bad takes, than get rid of the bad takes and also get rid of good takes from impostors.
I think it’s fairly clear which of these are the main factors, and which are not. Explanations (3-5) and (7) do not account for the recent decline, because they have always been true. Also, (6) is a weak explanation, because the quality wasn’t substantially worse than an average post.
On the other hand, (1-2) +/- (8) fit perfectly with the fact that volume has increased over the last 18 months, over the same period as community-building has happened on a large scale. And I can’t think of any major contributors outside of (1-8), so I think the main causes are simply community dilution + a flood of newbies.
Though the other factors could still partially explain why the level (as opposed to the trend) isn’t better, and arguably the level is what we’re ultimately interested in.
I wouldn’t be quick to dismiss (3-5) and (7) as factors we should pay attention to. These sorts of memetic pressures are present in many communities, and yet communities vary dramatically in quality. This is because things like (3-5) and (7) can be modulated by other facts about the community:
How intrinsically susceptible are people to clickbait?
Have they been taught things like politics is the mind-killer and the dangers of platforms where controversial ideas outcompete broadly good ones?
What is the variance in how busy people are?
To what degree do people feel like they can weigh in on meta? To what degree can they weigh in on cause areas that are not their own?
Are the people on EA Forum mostly trying for impact, or to feel like they’re part of a community (including instrumentally towards impact)?
So even if they cannot be solely reponsible for changes, they could have been necessary to produce any declines in quality we’ve observed, and be important for the future.
I agree that (4) could be modulated by the character of the community. The same is true for (3,5), except that, the direction is wrong. Old-timers are more likely to be professional EAs, and know more about the community, so their decreased prevalence should reduce problems from (3,5). And (7) seems more like an effect of the changing nature of the forum, rather than a cause of it.
Eternal September is a slightly different hypothesis that those listed. It’s that if new people come into the community then there is an erosion of norms that make the community distinctive.
So as I see it the main phenomenon is that there’s just much more being posted on the forum. I think there’s two factors behind that 1) community growth and 2) strong encouragement to post on the Forum. Eg there’s lots of encouragement to post on the forum from: the undergraduate introductory/onboarding fellowships, the AGI/etc ‘Fundamentals’ courses, the SERI/CERI/etc Summer Fellowships, or this or this (h/t John below).
The main phenomenon is that there is a lot more posted on the forum, mostly from newer/more junior people. It could well be the case that the average quality of posts has gone down. However, I’m not so sure that the quality of the best posts has gone down, and I’m not so sure that there are fewer of the best posts every month. Nevertheless, spotting the signal from the noise has become harder.
But then the forum serves several purposes. To take two of them: One (which is the one commenters here are most focussed on) is “signal”—producing really high-quality content—and its certainly got harder to find that. But another purpose is more instrumental—its for more junior people to demonstrate their writing/reasoning ability to potential employees. Or its to act as an incentive/endgoal for them to do some research—where the benefit is more that they see whether its a fit for them or not, but they wouldn’t actually do the work if it wasn’t structured towards writing something public.
So the main thing that those of us who are looking for “signal” need to do is find better/new ways to do so. The curated posts are a postive step in this direction, as are the weekly summaries and the monthly summaries.
I’d reframe this slightly, though I agree with all your key points. EA forum is finding a new comparative advantage. There are other platforms for deep, impact-focused research. Some of the best research has crystallized into founding efforts.
There will always be the need for an onboarding site and watering hole, and EA forum is filling that niche.
There are other platforms for deep, impact-focused research.
Could you name them? I’m not sure which ones are out there, other than LW and Alignment Forum for AI alignment research.
E.g. I’m not sure where else is a better place to post research on forecasting, research on EA community building, research on animal welfare, or new project proposals. There are private groups and slacks, but sometimes what you want is public or community engagement.
I was thinking about our biggest institutions, OpenPhil, 80k, that sort of thing—the work produced by their on-staff researchers. It sounds like you’re wanting a space that’s like the EA forum, but has a higher concentration of impact-focused research especially by independent researchers? Or maybe that you’d like to see the new work other orgs are doing get aggregated in one place?
I’m worried about EA values being wrong because EAs are unrepresentative of humanity and reasoning from first principles is likely to go wrong somewhere. But naively deferring to “conventional” human values seems worse, for a variety of reasons:
There is no single “conventional morality”, and it seems very difficult to compile a list of what every human culture thinks of as good, and not obvious how one would form a “weighted average” between these.
most people don’t think about morality much, so their beliefs are likely to contradict known empirical facts (e.g. cost of saving lives in the developing world) or be absurd (placing higher moral weight on beings that are physically closer to you).
Human cultures have gone through millennia of cultural evolution, such that values of existing people are skewed to be adaptive, leading to greed, tribalism, etc.; Ian Morris says “each age gets the thought it needs”.
However, these problems all seem surmountable with a lot of effort. The idea is a team of EA anthropologists who would look at existing knowledge about what different cultures value (possibly doing additional research) and work with philosophers to cross-reference between these while fixing inconsistencies and removing values that seem to have an “unfair” competitive edge in the battle between ideas (whatever that means!).
The potential payoff seems huge, as it would expand the basis of EA moral reasoning from the intuitions of a tiny fraction of humanity to that of thousands of human cultures, and allow us to be more confident about our actions. Is there a reason this isn’t being done? Is it just too expensive?
I also agree that research into how laypeople actually think about morality is probably a very important input into our moral thinking. I mentioned some reasons for this in this post for example. This project on descriptive population ethics also outlines the case for this kind of descriptive research. If we take moral uncertainty and epistemic modesty/outside-view thinking seriously, and if on the normative level we think respecting people’s moral beliefs is valuable either intrinsicaially or instrumentally, then this sort of research seems entirely vital.
I also agree that incorporating this data into our considered moral judgements requires a stage of theoretical normative reflection, not merely “naively deferring” to whatever people in aggregate actually believe and that we should probably go back and forth between these stages to bring our judgements into reflective equillibrium (or some such).
That said, it seems like what you are proposing is less a project and more an enormous research agenda spanning several fields of research, a lot of which is ongoing across multiple disciplines, though much of it is in its early stages. For example, there is much work in moral psychology, which tries to understand what people believe, and why, at different levels, (influential paradigms include Haidt’s Moral Foundations Theory, and Oliver Scott Curry’s (Morality as Cooperation / Moral Molecules theory), a whole new field of sociology of morality (see also here) , anthropology of morality is a whole long-standing field, and experimental philosophy has just started to seek to empirically examine how people think about morality too.
Unfortunately, I think our understanding of folk morality remains exceptionally unclear and in its very early stages. For example, despite a much touted “new synthesis” between different disciplines and approaches, there remains much distance between different approaches, to the extent that people in psychology, sociology and anthropology are barely investigating the same questions >90% of the time. Similarly, experimental philosophy of morality seems utterly crippled by validity issues (see my recent paper with Lance Bush here) . There is also, I have argued, a necessity to also gather qualitative data, in part due to the limitations with survey methodology for understanding people’s moral views, which experimental philosophy and most psychology have essentially not started to do at all.
I would also note that there already cross-cultural moral research on various questions, but this is usually limited to fairly narrow paradigms: for example, aside from those I mentioned above, the World Values Survey’s focus on Traditional/Secular-Rational and Survival/Self-expressive values; research on the trolley problem (which also dominates the rest of moral psychology), or the Schwartz Values Survey. So these lines of research doesn’t really give us insight into people’s moral thinking in different cultures as a whole.
I think the complexity and ambition involved in measuring folk morality becomes even clearer when we consider what is involved in studying specific moral issues. For example, see Jason Schukraft’s discussion of how we might investigate how much moral weight the folk ascribe to the experiences of animals of different species.
There are lots of other possible complications with cross-cultural moral research. For example, there is some anthropological evidence that the western concept of morality is idiosyncratic and does not overlap particularly neatly with other cultures, see here.
So I think, given this, the problem is not simply that it’s “too expensive”, as we might say of a really large survey, but that it would be a huge endeavour where we’re not even really clear about much of the relevant theory and categories. Also training a significant number of EA anthropologists, who are competent in ethnography and the relevant moral philosophy would be quite a logistical challenge.
---
That said I think there are plenty of more tractable research projects that one could do roughly within this area. For example, more large scale representative surveys examining people’s views and their predictors across a wider variety of issues relevant to effective altruism/prioritisation would be relatively easy to do with a budget of <$10,000, by existing EA researchers. This would also potentially contribute to understanding influences on the prioritisation of EAs, rather than just what non-EA things, which would also plausibly be valuable.
I have recently been thinking about the exact same thing, down to getting anthropologists to look into it! My thoughts on this were that interviewing anthropologists who have done fieldwork in different places is probably the more functional version of the idea. I have tried reading fairly random ethnographies to built better intuitions in this area, but did not find it as helpful as I was hoping, since they rarely discuss moral worldviews in as much detail as needed.
My current moral views seem to be something close to “reflected” preference utilitarianism, but now that I think this is my view, I find it quite hard to figure out what this actually means in practice.
My impression is that most EAs don’t have a very preference utilitarian view and prefer to advocate for their own moral views. You may want to look at my most recent post on my shortform on this topic.
If you would like to set up a call sometime to discuss further, please PM!
Is there a reason this isn’t being done? Is it just too expensive?
From where I’m sitting, there are a whole bunch of potentially highly useful things that aren’t being done. After several years around the EA community, I’ve gotten a better model of why that is:
1) There’s a very limited set of EAs who are entrepreneurial, trusted by funders, and have the necessary specific skills and interests to do many specific things. (Which respected EAs want to take a 5 to 20 year bet on field anthropology?) 2) It often takes a fair amount of funder buy-in to do new projects. This can take several years to develop, especially for an research area that’s new. 3) Outside of OpenPhil, funding is quite limited. It’s pretty scary and risky to start something new and go for it. You might get funding from EA Funds this year, but who’s to say if you’ll have to fire your staff in 3 years.
On doing anthropology, I personally think there might be lower hanging fruit first engaging with other written moral systems we haven’t engaged with. I’d be curious to get an EA interpretation of parts of Continental Philosophy, Conservative Philosophy, and the philosophies and writings of many of the great international traditions. That said, doing more traditional anthropology could also be pretty interesting.
I agree—I’m especially worried that focusing too much on longtermism will make us seem out of touch with the rest of humanity, relative to other schools of EA thought. I would support conducting a public opinion poll to learn about people’s moral beliefs, particularly how important and practical they believe focusing on the long-term future would be. I hypothesize that people who support ideas such as sustainability will be more sympathetic to longtermism.
I noticed this a while ago. I don’t see large numbers of low-quality low-karma posts as a big problem though (except that it has some reputation cost for people finding the Forum for the first time). What really worries me is the fraction of high-karma posts that neither original, rigorous, or useful. I suggested some server-side fixes for this.
PS: #3 has always been true, unless you’re claiming that more of their output is private these days.
Should the EA Forum team stop optimizing for engagement? I heard that the EA forum team tries to optimize the forum for engagement (tests features to see if they improve engagement). There are positives to this, but on net it worries me. Taken to the extreme, this is a destructive practice, as it would
normalize and encourage clickbait;
cause thoughtful comments to be replaced by louder and more abundant voices (for a constant time spent thinking, you can post either 1 thoughtful comment or several hasty comments. Measuring session length fixes this but adds more problems);
cause people with important jobs to spend more time on EA Forum than is optimal;
avoid community members and “EA” itself from keeping their identities small, as politics is an endless source of engagement;
distract from other possible directions of improvement, like giving topics proportionate attention, adding epistemic technology like polls and prediction market integration, improving moderation, and generally increasing quality of discussion.
I’m not confident that EA Forum is getting worse, or that tracking engagement is currently net negative, but we should at least avoid failing this exercise in Goodhart’s Law.
Thanks for this shortform! I’d like to quickly clarify a bit about our strategy. TL;DR: I don’t think the Forum team optimizes for engagement.
We do track engagement, and engagement is important to us, since we think a lot of the ways in which the Forum has an impact are diffuse or hard to measure, and they’d roughly grow or diminish with engagement.
But we definitely don’t optimize for it, and we’re very aware of worries about Goodharting.
Besides engagement, we try to track estimates for a number of other things we care about (like how good the discussions have been, how many people have gotten jobs as a result of the Forum, etc), and we’re actively working on doing that more carefully.
And for what it’s worth, I think that none of our major projects in the near future (like developing subforums) are aimed at increasing engagement, and neither have been our recent projects (like promoting impactful jobs).
And for what it’s worth, I think that none of our major projects in the near future (like developing subforums) are aimed at increasing engagement, and neither have been our recent projects (like promoting impactful jobs).
I wasn’t counting that as a major project, but Draft Amnesty Day also wasn’t aimed at optimizing engagement (and I’d be surprised[1] if it helped or hurt engagement in a significant way). That was motivated by a desire to get people to publish drafts (which could have cool ideas) that they’ve been sitting on for a while. :)
Terminology proposal: a class-n (or tier-n) megaproject reduces x-risk by between 10^-n and 10^-(n+1). This is intended as a short way to talk about the scale of longtermist megaprojects, inspired by 80k’s scale-scale but a bit cleaner because people can actually remember how to use it.
Class-0 project: reduces x-risk by >10%, e.g. creating 1,000 new AI safety researchers as good as Paul Christiano
Class-1 project: reduces x-risk by 1-10%, e.g. reducing pandemic risk to zero
Class-2 project: reduces x-risk by 0.1-1%, e.g. the Anthropic interpretability team
Class-3 project: reduces x-risk by 0.01-0.1%, e.g. most of these, though some make it into class 2
The classes could also be non-integer for extra precision, so if I thought creating 1,000 Paul Christianos reduced x-risk by 20%, I could call it a -log10(20%) = class-0.70 megaproject.
I’m still not sure about some details, so leave a comment if you have opinions:
“class” vs “tier”
I originally thought of having the percentages be absolute, but perhaps one could also make the case for relative percentages.
should class-n be between 10^-n and 10^-(n+1), or between 10^-(n-1) and 10^-n?
Are we evaluating outcomes or projects? What should the tier of a project be with a 10% chance to produce some outcome that reduces x-risk by 2%? I think it’s a class-1 (2%) project and the likelihood to succeed falls under tractability or neglectedness.
I might want to become a billionaire for roughly the reasons in this post [1] (tl;dr EV is tens of millions per year and might be the highest EV thing I can do), and crypto seems like one particularly promising way. I have other possible career paths, but my current plan is to
- accumulate a list of ~25 problems in crypto that could be worth $1B if solved
- hire a research assistant to look over the list for ~100 hours and compile basic stats like “how crowded is this” and estimating market size
- talk with experts about the most promising ones
- if one is particularly promising, do the standard startup things (hire smart contract and front-end devs, get funding somehow) and potentially drop out of school
Does this sound reasonable? Can you think of improvements to this plan? Are there people I should talk to?
How much equity does SBF actually have in FTX? Posts like this imply he has 90%, but the first article I found said that he actually had 90% equity in Alameda (which is owned by FTX or something?) and nothing I can find gives a percentage equity in FTX. Also, FTX keepsraising money, so even if he had 90% at once point, surely much of that has been sold.
It’s common for people to make tradeoffs between their selfish and altruistic goals with a rule of thumb or pledge like “I want to donate X% of my income to EA causes” or “I spend X% of my time doing EA direct work” where X is whatever they’re comfortable with. But among more dedicated EAs where X>>50, maybe a more useful mantra is “I want to produce at least Y% of the expected altruistic impact that I would if I totally optimized my life for impact”. Some reasons why this might be good:
Impact is ultimately what we care about, not sacrifice. The new framing shifts people out of a mindset of zero-sum tradeoffs between a selfish and altruistic part.
In particular, this promotes ambition. Thoughts similar to this have helped me realize that by being more ambitious, I can double my impact without sacrificing much personal well-being. This is a much better thing to do than working 70 hours a week at an “EA job” because I think my commitment level is X=85% or something.
It also helps me not stress about small things. My current diet is to avoid chicken, eat fewer eggs, and offset any eggs I eat. Some people around me are vegan, and some people think offsetting is antithetical to EA. Even though neither group was pushy I would have spent more energy being stressed/sad if I didn’t have the Y mindset, even though I personally believe that diet is a pretty small part of my impact.
Some reasons it might be bad (not a complete list, since I’m biased in favor):
Creating unhealthy dynamics between people trying to prove they have the highest value of Y. This is also a problem with X, but Y is more subjective.
Even Y=20% is really hard to achieve. This is motivating to me but might discourage some people.
I think I like the thinking that’s in this general direction, but just to list some additional counter-considerations:
almost all of the predictable difference in your realized impact from your theoretical maximum would be due to contigent factors outside of your control.
You can try to solve this problem somewhat by saying Y% of your ex ante expected value
But it’s hard (but not impossible) to avoid problems with evidential updates here (like there’ll be situations where your policy prevents you from seeking evidential updates)
the toy example that comes to mind is that unless you’re careful, this policy would in theory prevent you from learning about much more ambitious things you could’ve done, because that’d be evidence that your theoretical maximum is much higher than you’ve previously thought!
The subjectivity of Y is problematic not just for interpersonal dynamics, but from a motivational perspective.Because it’s so hard to know both the numerator and especially denominator, the figures may be too noisy to optimize for/have a clean target to aim at.
In practice I think this hasn’t been too much of a problem for me, and I can easily switch from honest evaluation mode to execution mode. Curious if other people have different experiences.
I’m looking for AI safety projects with people with some amount of experience. I have 3⁄4 of a CS degree from Caltech, one year at MIRI, and have finished the WMLB and ARENA bootcamps. I’m most excited about activation engineering, but willing to do anything that builds research and engineering skill.
If you’ve published 2 papers in top ML conferences or have a PhD in something CS related, and are interested in working with me, send me a DM.
With all the scandals in the last year or two, has anyone looked at which recruitment sources are least likely to produce someone extremely net negative in direct impact or to the community (i.e. a justified scandal)? Maybe this should inform outreach efforts.
Women in longtermism and EA are consistently better in respects of character, responsibility and diligence (there are outliers in animal welfare, who have been power-seeking for ideological and been destructive, implicated in ACE’s fate, but that’s probably because of the demographics).
Women do not engage in as much power-seeking as much or interact as poorly with the social fictions/status/funding dynamics that produce bad outcomes in EA (they tend to do more real things).
As we will see, even Caroline did the “least crime”. In the non-linear case, my guess is that Kat Woods was more self-involved and highly unqualified as a manager, with less tones of systemic malice that Emerson gives off.
Is there any evidence for this claim? One can speculate about how average personality gender differences would affect p(scandal), but you’ve just cited two cases where women caused huge harms, which seems to argue neutrally or against you.
But with no evidence, just your guesses. IMO we should wait until things shake out and even then the evidence will require lots of careful interpretation. Also EA is 2⁄3 male, which means that even minor contributions of women to scandals could mean they cause proportionate harms.
I’m fine with downvotes on stuff, I’m not OK with not being seen.
Zooming out, I think there’s a danger some people will see this as a game to be won. If that’s true, you’re going to see different classes of people “play the game” to win the online dynamics. Some classes of people are not going to play.
I think there are other ways of doing this. Another distinct class of people are going to play a game that “they only need to win once”.
Is it possible to donate appreciated assets (e.g. stocks) to one of the EA Funds? The tax benefits would be substantially larger than donating cash.
I know that MIRI and GiveWell as well as some other EA-aligned nonprofits do support donating stocks. GiveWell even has a DAF with Vanguard Charitable. But I don’t see such an option for the EA Funds.
Probably the easiest way to do this is to give to a donor-advised fund, and then instruct the fund to give to the EA Fund. Even for charities that can accept stock, my experience has been that donating through a donor-advised fund is much easier (it requires less paperwork).
If I’m capable of running an AI safety reading group (at my school, and I learn that someone else is doing it, I might be jealous that my impact is “being taken”.
If I want to maximize total impact, I don’t endorse this feeling. But what feeling does make sense from an impact maximization perspective? Based on Shapley values, you should
update downwards on the impact they get (because they’re replaceable)
update downwards on the impact you get, if you thought this was your comparative advantage (because you’re replaceable).
want to find a new task/niche where you’re less replaceable.
I claim that something like this is the good form of impact jealousy. (Of course, you should also be happy the work is happening).
Ah… this is something I struggle with. Especially since I’ve had the same goals for years. It would be a hard transition, I’ve done it before. I like to think of it as the next thing I find will be better in ways I didn’t expect, as long as I’m putting effort in.
What’s the right way to interact with people whose time is extremely valuable, equivalent to $10,000-$1M per hour of OpenPhil’s last dollar? How afraid should we be of taking up their time? Some thoughts:
Sometimes people conflate time cost with status, and the resulting shyness/fear can prevent you from meeting someone—this seems unwarranted because introducing yourself only takes like 20 seconds.
The time cost should nevertheless not be ignored; how prepared you are for a 1-hour meeting might be the largest source of variance in the impact you produce/destroy in a week.
At a $24k/hour valuation, a one-hour meeting is $24k but you might only need 2 slack messages, which maybe take one minute = $400 to respond to.
Being net-negative in immediate impact by taking up more mentor/manager time than the impact you create is not necessarily bad, because you build skills towards being much more valuable.
I personally have a difficult time with this. Usually in these conversations they are looking for something specific, and it’s hard to know what I have to say that would be helpful for them. For me sometimes I see the big picture too much and it’s hard to find something meaningful out of that. Ex. I want to solve systemic issues that the person I’m talking to can’t change. I don’t know how to balance what’s realistic and helpful with what’s needed.
I think there are currently too few infosec people and people trying to become billionaires.
Infosec: this seems really helpful for AI safety and biosecurity in a lot of worlds, and I’m guessing it’s just much less sexy / popular than technical research. Maybe I’m wrong about the number of people here, but from attendance at an EAGxSF event it didn’t seem like we would be saturated.
Entrepreneurship: I think the basic argument for making tens of billions of dollars still holds. Just because many longtermist orgs are well-funded now doesn’t mean they will be in the future (crypto and other risk), and there might be ways to spend hundreds of billions of dollars on other things. My understanding is that even at the crypto peak there were <50 EAs trying to become billionaires, and there are even fewer now, which seems like a mistake.
I’ve thought about both of these paths myself, and I think they’re not quite as good as technical alignment research for me, but I can’t rule out that I’m just being a coward.
What percent of Solana is held by EAs? I’ve heard FTX holds some, but unknown how much. This is important because if I do a large crypto project on Solana, much of the value might come from increasing the value of the Solana ecosystem, and thus other EAs’ investments.
Are there GiveWell-style estimates of the cost-effectiveness of the world’s most popular charities (say UNICEF), preferably by independent sources and/or based on past results? I want to be able to talk to quantitatively-minded people and have more data than just saying some interventions are 1000x more effective.
Unfortunately most cost-effectiveness estimates are calculated by focusing on the specific intervention the charity implements, a method which is a poor fit for large diversified charities.
Hmm, that’s what I suspected. Maybe it’s possible to estimate anyway though—quick and dirty method would be to identify the most effective interventions a large charity has, estimate that the rest follow a power law, take the average and add error bars upwards for the possibility we underestimated an intervention’s effectiveness?
One argument against the effectiveness from mega charities who does a bunch of different, unrelated interventions is that from the Central Limit Theorem (https://en.m.wikipedia.org/wiki/Central_limit_theorem) the average effectiveness of a large sample of interventions is apriori more likely to be close to the population mean effectiveness—that is the mean effectiveness of all relevant interventions. In other words, it’s hard to be one of the very best if you are doing lots of different stuff. Even if some of the interventions you do are really effective, your average effectiveness will be dragged down by the other interventions.
A lot of EAs I know have had strong intuitions towards scope sensitivity, but I also remember having strong intuitions towards moral obligation, e.g. I remember being slightly angry at Michael Phelps’ first retirement, thinking I would never do this and that top athletes should have a duty to maximize their excellence over their career. Curious how common this is.
I agree with these points. I’ve worked in the space for a few years (most notably for IOHK working on Cardano) and am happy to offer some advice. Saying that, I would much rather work on something directly valuable (climate change or food security) than earning to give at the moment…
I want to skill up in pandas/numpy/data science over the next few months. Where can I find a data science project that is relevant to EA? Some rough requirements:
Takes between 1 and 3 months of full-time work
Helps me (a pretty strong CS undergrad) become fluent in pandas quickly, and maybe use some machine learning techniques I’ve studied in class
About as open-ended as a research internship
Feels meaningful
Should be important enough that I enjoy doing it, but it’s okay if it has e.g. 5% as much direct benefit as the highest-impact thing I could be doing
I’m interested in AI safety and other long-term cause areas
Bonus: working with time-series data, because I’m particularly confused about how it works.
I’ve already looked at the top datasets on kaggle and other places, and don’t feel inclined to work on them because they don’t seem relevant and have probably been analyzed to death. Also, I’ve only taken a few ML and no data science classes, so I might not be asking the right questions.
EA forum content might be declining in quality. Here are some possible mechanisms:
Newer EAs have worse takes on average, because the current processes of recruitment and outreach produce a worse distribution than the old ones
Newer EAs are too junior to have good takes yet. It’s just that the growth rate has increased so there’s a higher proportion of them.
People who have better thoughts get hired at EA orgs and are too busy to post. There is anticorrelation between the amount of time people have to post on EA Forum and the quality of person.
Controversial content, rather than good content, gets the most engagement.
Although we want more object-level discussion, everyone can weigh in on meta/community stuff, whereas they only know about their own cause areas. Therefore community content, especially shallow criticism, gets upvoted more. There could be a similar effect for posts by well-known EA figures.
Contests like the criticism contest decrease average quality, because the type of person who would enter a contest to win money on average has worse takes than the type of person who has genuine deep criticism. There were 232 posts for the criticism contest, and 158 for the Cause Exploration Prizes, which combined is more top-level posts than the entire forum in any month except August 2022.
EA Forum is turning into a place primarily optimized for people to feel welcome and talk about EA, rather than impact.
All of this is exacerbated as the most careful and rational thinkers flee somewhere else, expecting that they won’t get good quality engagement on EA Forum
Another possible mechanism is forum leadership encouraging people to be less intimidated and write more off-the-cuff posts—see e.g. this or this.
Side note: It seems like a small amount of prize money goes a long way.
E.g. Rethink Priorities makes their salaries public: they pay senior researchers $105,000 – $115,000 per year.
Their headcount near the end of 2021 was 24.75 full-time equivalents.
And their publications page lists 30 publications in 2021.
So napkin math suggests that the per-post cost of a contest post is something like 1% of the per-post cost of a RP publication. A typical RP publication is probably much higher quality. But maybe sometimes getting a lot of shallow explorations quickly is what’s desired. (Disclaimer: I haven’t been reading the forum much, didn’t read many contest posts, and don’t have an opinion about their quality. But I did notice the organizers of the ELK contest were “surprised by the number and quality of submissions”.)
A related point re: quality is that smaller prize pools presumably select for people with lower opportunity costs. If I’m a talented professional who commands a high hourly rate, I might do the expected value math on e.g. the criticism prize and decide it’s not worthwhile to enter.
It’s also not clear if the large number of entries will persist in the longer term. Not winning can be pretty demoralizing. Supposing a talented professional goes against their better judgement and puts a lot of time into their entry, then loses and has no idea why. Will they enter the next contest they see? Probably not. They’re liable to interpret lack of a prize as “the contest organizers didn’t think it was worth my time to make a submission”.
Hey just want to weigh in here that you can’t divide our FTE by our total publication count, since that doesn’t include a large amount of work we’ve produced that is not able to be made public or is not yet public but will be. Right now I think a majority of our output is not public right now for one reason or another, though we’re working on finding routes to make more of it public.
I do think your general point though that the per-post cost of a contest post is less / much less than an RP post is accurate though.
-Peter (Co-CEO of Rethink Priorities)
Thanks for the correction!
BTW, I hope it doesn’t seem like it was picking on you—it just occurred to me that I could do math for Rethink Priorities because your salaries are public. I have no reason to believe a cost-per-public-report estimate would be different for any other randomly chosen EA research organization in either direction. And of course most EA organizations correctly focus on making a positive impact rather than maximizing publication count.
We also seem to get a fair number of posts that make basically the same point as an earlier article, but the author presumably either didn’t read the earlier one or wanted to re-iterate it.
I’ll add another mechanism:
I think there are many people who have very high bars for how good something should be to post on the forum. Thus the forum becomes dominated by a few people (often people who aren’t aware of or care about forum norms) who have much lower bars to posting.
This is a plausible mechanism for explaining why content is of lower quality than one would otherwise expect, but it doesn’t explain differences in quality over time (and specifically quality decline), unless you add extra assumptions such that the proportion of people with low bars to posting has increased recently. (Cf. Ryan’s comment)
You’re quite right, it was left too implicit.
EA has grown a lot recently, so I think there are more people recently who aren’t aware of or care about the “high bar” norm. This is in part due to others explicitly saying the bar should be lower, which (as others here have noted) has a stronger effect on some than on others.
Edit: I don’t have time to do this right now, but I would be interested to plot the proportion of posts on the EA forum from people who have been on the forum for less than a year over time. I suspect that it would be trending upwards (but could be wrong). This would be a way to empirically verify part of my claim.
I’m interested in learning how plausible people find each of these mechanisms, so I created a short (anonymous) survey. I’ll release the results in a few days [ETA: see below]. Estimated completion time is ~90 seconds.
The results are below. The data is here.
I broadly agree with 5 and 6.
Re 3, ‘There is anticorrelation between the amount of time people have to post on EA Forum and the quality of person.’ - this makes me wince. A language point is that I think talking about how ‘good quality’ people are overall is unkind and leads to people feeling bad about themselves for not having such-and-such an attribute. An object level point is I don’t think there is an anticorrelation—I think being a busy EA org person does make it more likely that they’ll have valuable takes, but not being a busy-EA-org-person doesn’t make it less likely—there aren’t that many busy-EA-org-person jobs, and some people aren’t a good fit for busy jobs (eg because of their health or family commitments) but they still have interesting ideas.
Re 7: I’m literally working on a post with someone about how lots of people feel too intimidated to post on the Forum because of its perceived high standards! So I think though the Forum team are trying to make people feel welcome, it’s not true that it’s (yet) optimized for this, imo.
There’s a kind of general problem whereby any messaging or mechanism that’s designed to dissuade people from posting low-quality things will (a) just not work on some people—some people just have a lot of confidence in their not-very-good opinions, shrug, and (b) dissuade people who would post high-quality things, but who have impostor syndrome or are perfectionist or over self-critical. I think the number of people that the mechanism works as intended on—ie people who would have posted a low quality post but are now dissuaded from it—is probably pretty low. Since there are lots of people in EA with impostor syndrome/perfectionism/over-scrupulosity, I’m pretty in favour of the Forum having a ‘welcoming’ vibe over a We Are Very Serious and Important vibe.… because I’d rather have more good takes and more bad takes, than get rid of the bad takes and also get rid of good takes from impostors.
I think it’s fairly clear which of these are the main factors, and which are not. Explanations (3-5) and (7) do not account for the recent decline, because they have always been true. Also, (6) is a weak explanation, because the quality wasn’t substantially worse than an average post.
On the other hand, (1-2) +/- (8) fit perfectly with the fact that volume has increased over the last 18 months, over the same period as community-building has happened on a large scale. And I can’t think of any major contributors outside of (1-8), so I think the main causes are simply community dilution + a flood of newbies.
Though the other factors could still partially explain why the level (as opposed to the trend) isn’t better, and arguably the level is what we’re ultimately interested in.
I wouldn’t be quick to dismiss (3-5) and (7) as factors we should pay attention to. These sorts of memetic pressures are present in many communities, and yet communities vary dramatically in quality. This is because things like (3-5) and (7) can be modulated by other facts about the community:
How intrinsically susceptible are people to clickbait?
Have they been taught things like politics is the mind-killer and the dangers of platforms where controversial ideas outcompete broadly good ones?
What is the variance in how busy people are?
To what degree do people feel like they can weigh in on meta? To what degree can they weigh in on cause areas that are not their own?
Are the people on EA Forum mostly trying for impact, or to feel like they’re part of a community (including instrumentally towards impact)?
So even if they cannot be solely reponsible for changes, they could have been necessary to produce any declines in quality we’ve observed, and be important for the future.
I agree that (4) could be modulated by the character of the community. The same is true for (3,5), except that, the direction is wrong. Old-timers are more likely to be professional EAs, and know more about the community, so their decreased prevalence should reduce problems from (3,5). And (7) seems more like an effect of the changing nature of the forum, rather than a cause of it.
My comment got detached, woops
Eternal September is a slightly different hypothesis that those listed. It’s that if new people come into the community then there is an erosion of norms that make the community distinctive.
So as I see it the main phenomenon is that there’s just much more being posted on the forum. I think there’s two factors behind that 1) community growth and 2) strong encouragement to post on the Forum. Eg there’s lots of encouragement to post on the forum from: the undergraduate introductory/onboarding fellowships, the AGI/etc ‘Fundamentals’ courses, the SERI/CERI/etc Summer Fellowships, or this or this (h/t John below).
The main phenomenon is that there is a lot more posted on the forum, mostly from newer/more junior people. It could well be the case that the average quality of posts has gone down. However, I’m not so sure that the quality of the best posts has gone down, and I’m not so sure that there are fewer of the best posts every month. Nevertheless, spotting the signal from the noise has become harder.
But then the forum serves several purposes. To take two of them: One (which is the one commenters here are most focussed on) is “signal”—producing really high-quality content—and its certainly got harder to find that. But another purpose is more instrumental—its for more junior people to demonstrate their writing/reasoning ability to potential employees. Or its to act as an incentive/endgoal for them to do some research—where the benefit is more that they see whether its a fit for them or not, but they wouldn’t actually do the work if it wasn’t structured towards writing something public.
So the main thing that those of us who are looking for “signal” need to do is find better/new ways to do so. The curated posts are a postive step in this direction, as are the weekly summaries and the monthly summaries.
Are there examples of typical bad takes you’ve seen newer EAs post?
Small formatting thought: making these numbered instead of bulleted will make it easier to have conversations about them
Done
I’d reframe this slightly, though I agree with all your key points. EA forum is finding a new comparative advantage. There are other platforms for deep, impact-focused research. Some of the best research has crystallized into founding efforts.
There will always be the need for an onboarding site and watering hole, and EA forum is filling that niche.
Could you name them? I’m not sure which ones are out there, other than LW and Alignment Forum for AI alignment research.
E.g. I’m not sure where else is a better place to post research on forecasting, research on EA community building, research on animal welfare, or new project proposals. There are private groups and slacks, but sometimes what you want is public or community engagement.
I was thinking about our biggest institutions, OpenPhil, 80k, that sort of thing—the work produced by their on-staff researchers. It sounds like you’re wanting a space that’s like the EA forum, but has a higher concentration of impact-focused research especially by independent researchers? Or maybe that you’d like to see the new work other orgs are doing get aggregated in one place?
I’m worried about EA values being wrong because EAs are unrepresentative of humanity and reasoning from first principles is likely to go wrong somewhere. But naively deferring to “conventional” human values seems worse, for a variety of reasons:
There is no single “conventional morality”, and it seems very difficult to compile a list of what every human culture thinks of as good, and not obvious how one would form a “weighted average” between these.
most people don’t think about morality much, so their beliefs are likely to contradict known empirical facts (e.g. cost of saving lives in the developing world) or be absurd (placing higher moral weight on beings that are physically closer to you).
Human cultures have gone through millennia of cultural evolution, such that values of existing people are skewed to be adaptive, leading to greed, tribalism, etc.; Ian Morris says “each age gets the thought it needs”.
However, these problems all seem surmountable with a lot of effort. The idea is a team of EA anthropologists who would look at existing knowledge about what different cultures value (possibly doing additional research) and work with philosophers to cross-reference between these while fixing inconsistencies and removing values that seem to have an “unfair” competitive edge in the battle between ideas (whatever that means!).
The potential payoff seems huge, as it would expand the basis of EA moral reasoning from the intuitions of a tiny fraction of humanity to that of thousands of human cultures, and allow us to be more confident about our actions. Is there a reason this isn’t being done? Is it just too expensive?
Thanks for writing this.
I also agree that research into how laypeople actually think about morality is probably a very important input into our moral thinking. I mentioned some reasons for this in this post for example. This project on descriptive population ethics also outlines the case for this kind of descriptive research. If we take moral uncertainty and epistemic modesty/outside-view thinking seriously, and if on the normative level we think respecting people’s moral beliefs is valuable either intrinsicaially or instrumentally, then this sort of research seems entirely vital.
I also agree that incorporating this data into our considered moral judgements requires a stage of theoretical normative reflection, not merely “naively deferring” to whatever people in aggregate actually believe and that we should probably go back and forth between these stages to bring our judgements into reflective equillibrium (or some such).
That said, it seems like what you are proposing is less a project and more an enormous research agenda spanning several fields of research, a lot of which is ongoing across multiple disciplines, though much of it is in its early stages. For example, there is much work in moral psychology, which tries to understand what people believe, and why, at different levels, (influential paradigms include Haidt’s Moral Foundations Theory, and Oliver Scott Curry’s (Morality as Cooperation / Moral Molecules theory), a whole new field of sociology of morality (see also here) , anthropology of morality is a whole long-standing field, and experimental philosophy has just started to seek to empirically examine how people think about morality too.
Unfortunately, I think our understanding of folk morality remains exceptionally unclear and in its very early stages. For example, despite a much touted “new synthesis” between different disciplines and approaches, there remains much distance between different approaches, to the extent that people in psychology, sociology and anthropology are barely investigating the same questions >90% of the time. Similarly, experimental philosophy of morality seems utterly crippled by validity issues (see my recent paper with Lance Bush here) . There is also, I have argued, a necessity to also gather qualitative data, in part due to the limitations with survey methodology for understanding people’s moral views, which experimental philosophy and most psychology have essentially not started to do at all.
I would also note that there already cross-cultural moral research on various questions, but this is usually limited to fairly narrow paradigms: for example, aside from those I mentioned above, the World Values Survey’s focus on Traditional/Secular-Rational and Survival/Self-expressive values; research on the trolley problem (which also dominates the rest of moral psychology), or the Schwartz Values Survey. So these lines of research doesn’t really give us insight into people’s moral thinking in different cultures as a whole.
I think the complexity and ambition involved in measuring folk morality becomes even clearer when we consider what is involved in studying specific moral issues. For example, see Jason Schukraft’s discussion of how we might investigate how much moral weight the folk ascribe to the experiences of animals of different species.
There are lots of other possible complications with cross-cultural moral research. For example, there is some anthropological evidence that the western concept of morality is idiosyncratic and does not overlap particularly neatly with other cultures, see here.
So I think, given this, the problem is not simply that it’s “too expensive”, as we might say of a really large survey, but that it would be a huge endeavour where we’re not even really clear about much of the relevant theory and categories. Also training a significant number of EA anthropologists, who are competent in ethnography and the relevant moral philosophy would be quite a logistical challenge.
---
That said I think there are plenty of more tractable research projects that one could do roughly within this area. For example, more large scale representative surveys examining people’s views and their predictors across a wider variety of issues relevant to effective altruism/prioritisation would be relatively easy to do with a budget of <$10,000, by existing EA researchers. This would also potentially contribute to understanding influences on the prioritisation of EAs, rather than just what non-EA things, which would also plausibly be valuable.
Strong upvoted. Thank you so much for providing further resources, extremely helpful, downloading them all on my Kindle now!
I have recently been thinking about the exact same thing, down to getting anthropologists to look into it! My thoughts on this were that interviewing anthropologists who have done fieldwork in different places is probably the more functional version of the idea. I have tried reading fairly random ethnographies to built better intuitions in this area, but did not find it as helpful as I was hoping, since they rarely discuss moral worldviews in as much detail as needed.
My current moral views seem to be something close to “reflected” preference utilitarianism, but now that I think this is my view, I find it quite hard to figure out what this actually means in practice.
My impression is that most EAs don’t have a very preference utilitarian view and prefer to advocate for their own moral views. You may want to look at my most recent post on my shortform on this topic.
If you would like to set up a call sometime to discuss further, please PM!
First, neat idea, and thanks for suggesting it!
From where I’m sitting, there are a whole bunch of potentially highly useful things that aren’t being done. After several years around the EA community, I’ve gotten a better model of why that is:
1) There’s a very limited set of EAs who are entrepreneurial, trusted by funders, and have the necessary specific skills and interests to do many specific things. (Which respected EAs want to take a 5 to 20 year bet on field anthropology?)
2) It often takes a fair amount of funder buy-in to do new projects. This can take several years to develop, especially for an research area that’s new.
3) Outside of OpenPhil, funding is quite limited. It’s pretty scary and risky to start something new and go for it. You might get funding from EA Funds this year, but who’s to say if you’ll have to fire your staff in 3 years.
On doing anthropology, I personally think there might be lower hanging fruit first engaging with other written moral systems we haven’t engaged with. I’d be curious to get an EA interpretation of parts of Continental Philosophy, Conservative Philosophy, and the philosophies and writings of many of the great international traditions. That said, doing more traditional anthropology could also be pretty interesting.
I agree—I’m especially worried that focusing too much on longtermism will make us seem out of touch with the rest of humanity, relative to other schools of EA thought. I would support conducting a public opinion poll to learn about people’s moral beliefs, particularly how important and practical they believe focusing on the long-term future would be. I hypothesize that people who support ideas such as sustainability will be more sympathetic to longtermism.
I noticed this a while ago. I don’t see large numbers of low-quality low-karma posts as a big problem though (except that it has some reputation cost for people finding the Forum for the first time). What really worries me is the fraction of high-karma posts that neither original, rigorous, or useful. I suggested some server-side fixes for this.
PS: #3 has always been true, unless you’re claiming that more of their output is private these days.
#4 has also been true for many years
Should the EA Forum team stop optimizing for engagement?
I heard that the EA forum team tries to optimize the forum for engagement (tests features to see if they improve engagement). There are positives to this, but on net it worries me. Taken to the extreme, this is a destructive practice, as it would
normalize and encourage clickbait;
cause thoughtful comments to be replaced by louder and more abundant voices (for a constant time spent thinking, you can post either 1 thoughtful comment or several hasty comments. Measuring session length fixes this but adds more problems);
cause people with important jobs to spend more time on EA Forum than is optimal;
avoid community members and “EA” itself from keeping their identities small, as politics is an endless source of engagement;
distract from other possible directions of improvement, like giving topics proportionate attention, adding epistemic technology like polls and prediction market integration, improving moderation, and generally increasing quality of discussion.
I’m not confident that EA Forum is getting worse, or that tracking engagement is currently net negative, but we should at least avoid failing this exercise in Goodhart’s Law.
Thanks for this shortform! I’d like to quickly clarify a bit about our strategy. TL;DR: I don’t think the Forum team optimizes for engagement.
We do track engagement, and engagement is important to us, since we think a lot of the ways in which the Forum has an impact are diffuse or hard to measure, and they’d roughly grow or diminish with engagement.
But we definitely don’t optimize for it, and we’re very aware of worries about Goodharting.
Besides engagement, we try to track estimates for a number of other things we care about (like how good the discussions have been, how many people have gotten jobs as a result of the Forum, etc), and we’re actively working on doing that more carefully.
And for what it’s worth, I think that none of our major projects in the near future (like developing subforums) are aimed at increasing engagement, and neither have been our recent projects (like promoting impactful jobs).
What about Draft Amnesty?
I wasn’t counting that as a major project, but Draft Amnesty Day also wasn’t aimed at optimizing engagement (and I’d be surprised[1] if it helped or hurt engagement in a significant way). That was motivated by a desire to get people to publish drafts (which could have cool ideas) that they’ve been sitting on for a while. :)
Edit: can confirm that at a glance, engagement on Friday/this weekend looks normal
Terminology proposal: a class-n (or tier-n) megaproject reduces x-risk by between 10^-n and 10^-(n+1). This is intended as a short way to talk about the scale of longtermist megaprojects, inspired by 80k’s scale-scale but a bit cleaner because people can actually remember how to use it.
Class-0 project: reduces x-risk by >10%, e.g. creating 1,000 new AI safety researchers as good as Paul Christiano
Class-1 project: reduces x-risk by 1-10%, e.g. reducing pandemic risk to zero
Class-2 project: reduces x-risk by 0.1-1%, e.g. the Anthropic interpretability team
Class-3 project: reduces x-risk by 0.01-0.1%, e.g. most of these, though some make it into class 2
The classes could also be non-integer for extra precision, so if I thought creating 1,000 Paul Christianos reduced x-risk by 20%, I could call it a -log10(20%) = class-0.70 megaproject.
I’m still not sure about some details, so leave a comment if you have opinions:
“class” vs “tier”
I originally thought of having the percentages be absolute, but perhaps one could also make the case for relative percentages.
should class-n be between 10^-n and 10^-(n+1), or between 10^-(n-1) and 10^-n?
Are we evaluating outcomes or projects? What should the tier of a project be with a 10% chance to produce some outcome that reduces x-risk by 2%? I think it’s a class-1 (2%) project and the likelihood to succeed falls under tractability or neglectedness.
edit: changed my opinion on last bullet point
I might want to become a billionaire for roughly the reasons in this post [1] (tl;dr EV is tens of millions per year and might be the highest EV thing I can do), and crypto seems like one particularly promising way. I have other possible career paths, but my current plan is to
- accumulate a list of ~25 problems in crypto that could be worth $1B if solved
- hire a research assistant to look over the list for ~100 hours and compile basic stats like “how crowded is this” and estimating market size
- talk with experts about the most promising ones
- if one is particularly promising, do the standard startup things (hire smart contract and front-end devs, get funding somehow) and potentially drop out of school
Does this sound reasonable? Can you think of improvements to this plan? Are there people I should talk to?
[1]: https://forum.effectivealtruism.org/.../an-update-in...
How much equity does SBF actually have in FTX? Posts like this imply he has 90%, but the first article I found said that he actually had 90% equity in Alameda (which is owned by FTX or something?) and nothing I can find gives a percentage equity in FTX. Also, FTX keeps raising money, so even if he had 90% at once point, surely much of that has been sold.
It’s common for people to make tradeoffs between their selfish and altruistic goals with a rule of thumb or pledge like “I want to donate X% of my income to EA causes” or “I spend X% of my time doing EA direct work” where X is whatever they’re comfortable with. But among more dedicated EAs where X>>50, maybe a more useful mantra is “I want to produce at least Y% of the expected altruistic impact that I would if I totally optimized my life for impact”. Some reasons why this might be good:
Impact is ultimately what we care about, not sacrifice. The new framing shifts people out of a mindset of zero-sum tradeoffs between a selfish and altruistic part.
In particular, this promotes ambition. Thoughts similar to this have helped me realize that by being more ambitious, I can double my impact without sacrificing much personal well-being. This is a much better thing to do than working 70 hours a week at an “EA job” because I think my commitment level is X=85% or something.
It also helps me not stress about small things. My current diet is to avoid chicken, eat fewer eggs, and offset any eggs I eat. Some people around me are vegan, and some people think offsetting is antithetical to EA. Even though neither group was pushy I would have spent more energy being stressed/sad if I didn’t have the Y mindset, even though I personally believe that diet is a pretty small part of my impact.
Some reasons it might be bad (not a complete list, since I’m biased in favor):
Creating unhealthy dynamics between people trying to prove they have the highest value of Y. This is also a problem with X, but Y is more subjective.
Even Y=20% is really hard to achieve. This is motivating to me but might discourage some people.
I think I like the thinking that’s in this general direction, but just to list some additional counter-considerations:
almost all of the predictable difference in your realized impact from your theoretical maximum would be due to contigent factors outside of your control.
You can try to solve this problem somewhat by saying Y% of your ex ante expected value
But it’s hard (but not impossible) to avoid problems with evidential updates here (like there’ll be situations where your policy prevents you from seeking evidential updates)
the toy example that comes to mind is that unless you’re careful, this policy would in theory prevent you from learning about much more ambitious things you could’ve done, because that’d be evidence that your theoretical maximum is much higher than you’ve previously thought!
The subjectivity of Y is problematic not just for interpersonal dynamics, but from a motivational perspective.Because it’s so hard to know both the numerator and especially denominator, the figures may be too noisy to optimize for/have a clean target to aim at.
In practice I think this hasn’t been too much of a problem for me, and I can easily switch from honest evaluation mode to execution mode. Curious if other people have different experiences.
I’m looking for AI safety projects with people with some amount of experience. I have 3⁄4 of a CS degree from Caltech, one year at MIRI, and have finished the WMLB and ARENA bootcamps. I’m most excited about activation engineering, but willing to do anything that builds research and engineering skill.
If you’ve published 2 papers in top ML conferences or have a PhD in something CS related, and are interested in working with me, send me a DM.
Who tends to be clean?
With all the scandals in the last year or two, has anyone looked at which recruitment sources are least likely to produce someone extremely net negative in direct impact or to the community (i.e. a justified scandal)? Maybe this should inform outreach efforts.
Women in longtermism and EA are consistently better in respects of character, responsibility and diligence (there are outliers in animal welfare, who have been power-seeking for ideological and been destructive, implicated in ACE’s fate, but that’s probably because of the demographics).
Women do not engage in as much power-seeking as much or interact as poorly with the social fictions/status/funding dynamics that produce bad outcomes in EA (they tend to do more real things).
As we will see, even Caroline did the “least crime”. In the non-linear case, my guess is that Kat Woods was more self-involved and highly unqualified as a manager, with less tones of systemic malice that Emerson gives off.
Is there any evidence for this claim? One can speculate about how average personality gender differences would affect p(scandal), but you’ve just cited two cases where women caused huge harms, which seems to argue neutrally or against you.
In both cases, the examples of women have an explicit favorable comparison to their male counterparts.
But with no evidence, just your guesses. IMO we should wait until things shake out and even then the evidence will require lots of careful interpretation. Also EA is 2⁄3 male, which means that even minor contributions of women to scandals could mean they cause proportionate harms.
I’m fine with downvotes on stuff, I’m not OK with not being seen.
Zooming out, I think there’s a danger some people will see this as a game to be won. If that’s true, you’re going to see different classes of people “play the game” to win the online dynamics. Some classes of people are not going to play.
I think there are other ways of doing this. Another distinct class of people are going to play a game that “they only need to win once”.
Is it possible to donate appreciated assets (e.g. stocks) to one of the EA Funds? The tax benefits would be substantially larger than donating cash.
I know that MIRI and GiveWell as well as some other EA-aligned nonprofits do support donating stocks. GiveWell even has a DAF with Vanguard Charitable. But I don’t see such an option for the EA Funds.
edit: DAF = donor-advised fund
Probably the easiest way to do this is to give to a donor-advised fund, and then instruct the fund to give to the EA Fund. Even for charities that can accept stock, my experience has been that donating through a donor-advised fund is much easier (it requires less paperwork).
To clarify, you mean a donor-advised fund I have an account with (say Fidelity, Vanguard, etc.) which I manage myself?
Yes
Epistemic status: showerthought
If I’m capable of running an AI safety reading group (at my school, and I learn that someone else is doing it, I might be jealous that my impact is “being taken”.
If I want to maximize total impact, I don’t endorse this feeling. But what feeling does make sense from an impact maximization perspective? Based on Shapley values, you should
update downwards on the impact they get (because they’re replaceable)
update downwards on the impact you get, if you thought this was your comparative advantage (because you’re replaceable).
want to find a new task/niche where you’re less replaceable.
I claim that something like this is the good form of impact jealousy. (Of course, you should also be happy the work is happening).
Ah… this is something I struggle with. Especially since I’ve had the same goals for years. It would be a hard transition, I’ve done it before. I like to think of it as the next thing I find will be better in ways I didn’t expect, as long as I’m putting effort in.
What’s the right way to interact with people whose time is extremely valuable, equivalent to $10,000-$1M per hour of OpenPhil’s last dollar? How afraid should we be of taking up their time? Some thoughts:
Sometimes people conflate time cost with status, and the resulting shyness/fear can prevent you from meeting someone—this seems unwarranted because introducing yourself only takes like 20 seconds.
The time cost should nevertheless not be ignored; how prepared you are for a 1-hour meeting might be the largest source of variance in the impact you produce/destroy in a week.
At a $24k/hour valuation, a one-hour meeting is $24k but you might only need 2 slack messages, which maybe take one minute = $400 to respond to.
Being net-negative in immediate impact by taking up more mentor/manager time than the impact you create is not necessarily bad, because you build skills towards being much more valuable.
I personally have a difficult time with this. Usually in these conversations they are looking for something specific, and it’s hard to know what I have to say that would be helpful for them. For me sometimes I see the big picture too much and it’s hard to find something meaningful out of that. Ex. I want to solve systemic issues that the person I’m talking to can’t change. I don’t know how to balance what’s realistic and helpful with what’s needed.
I think there are currently too few infosec people and people trying to become billionaires.
Infosec: this seems really helpful for AI safety and biosecurity in a lot of worlds, and I’m guessing it’s just much less sexy / popular than technical research. Maybe I’m wrong about the number of people here, but from attendance at an EAGxSF event it didn’t seem like we would be saturated.
Entrepreneurship: I think the basic argument for making tens of billions of dollars still holds. Just because many longtermist orgs are well-funded now doesn’t mean they will be in the future (crypto and other risk), and there might be ways to spend hundreds of billions of dollars on other things. My understanding is that even at the crypto peak there were <50 EAs trying to become billionaires, and there are even fewer now, which seems like a mistake.
I’ve thought about both of these paths myself, and I think they’re not quite as good as technical alignment research for me, but I can’t rule out that I’m just being a coward.
What percent of Solana is held by EAs? I’ve heard FTX holds some, but unknown how much. This is important because if I do a large crypto project on Solana, much of the value might come from increasing the value of the Solana ecosystem, and thus other EAs’ investments.
Are there GiveWell-style estimates of the cost-effectiveness of the world’s most popular charities (say UNICEF), preferably by independent sources and/or based on past results? I want to be able to talk to quantitatively-minded people and have more data than just saying some interventions are 1000x more effective.
Unfortunately most cost-effectiveness estimates are calculated by focusing on the specific intervention the charity implements, a method which is a poor fit for large diversified charities.
Hmm, that’s what I suspected. Maybe it’s possible to estimate anyway though—quick and dirty method would be to identify the most effective interventions a large charity has, estimate that the rest follow a power law, take the average and add error bars upwards for the possibility we underestimated an intervention’s effectiveness?
One argument against the effectiveness from mega charities who does a bunch of different, unrelated interventions is that from the Central Limit Theorem (https://en.m.wikipedia.org/wiki/Central_limit_theorem) the average effectiveness of a large sample of interventions is apriori more likely to be close to the population mean effectiveness—that is the mean effectiveness of all relevant interventions. In other words, it’s hard to be one of the very best if you are doing lots of different stuff. Even if some of the interventions you do are really effective, your average effectiveness will be dragged down by the other interventions.
A lot of EAs I know have had strong intuitions towards scope sensitivity, but I also remember having strong intuitions towards moral obligation, e.g. I remember being slightly angry at Michael Phelps’ first retirement, thinking I would never do this and that top athletes should have a duty to maximize their excellence over their career. Curious how common this is.
I agree with these points. I’ve worked in the space for a few years (most notably for IOHK working on Cardano) and am happy to offer some advice. Saying that, I would much rather work on something directly valuable (climate change or food security) than earning to give at the moment…
I want to skill up in pandas/numpy/data science over the next few months. Where can I find a data science project that is relevant to EA? Some rough requirements:
Takes between 1 and 3 months of full-time work
Helps me (a pretty strong CS undergrad) become fluent in pandas quickly, and maybe use some machine learning techniques I’ve studied in class
About as open-ended as a research internship
Feels meaningful
Should be important enough that I enjoy doing it, but it’s okay if it has e.g. 5% as much direct benefit as the highest-impact thing I could be doing
I’m interested in AI safety and other long-term cause areas
Bonus: working with time-series data, because I’m particularly confused about how it works.
I’ve already looked at the top datasets on kaggle and other places, and don’t feel inclined to work on them because they don’t seem relevant and have probably been analyzed to death. Also, I’ve only taken a few ML and no data science classes, so I might not be asking the right questions.